Representation Norm Amplification for Out-of-Distribution Detection in Long-Tail Learning
Authors: Dong Geun Shin
Hye Won Chung
Venue: Accepted by TMLR ()

Abstract: Detecting out-of-distribution (OOD) samples is a critical task for reliable machine learning. However, it becomes particularly challenging when the models are trained on long-tailed datasets, as the models often struggle to distinguish tail-class in-distribution samples from OOD samples. We examine the main challenges in this problem by identifying the trade-offs between OOD detection and in-distribution (ID) classification, faced by existing methods. We then introduce our method, called *Representation Norm Amplification* (RNA), which solves this challenge by decoupling the two problems. The main idea is to use the norm of the representation as a new dimension for OOD detection, and to develop a training method that generates a noticeable discrepancy in the representation norm between ID and OOD data, while not perturbing the feature learning for ID classification. Our experiments show that RNA achieves superior performance in both OOD detection and classification compared to the state-of-the-art methods, by 1.70\% and 9.46\% in FPR95 and 2.43\% and 6.87\% in classification accuracy on CIFAR10-LT and ImageNet-LT, respectively. The code for this work is available at <https://github.com/dgshin21/RNA>.

Published in Transactions on Machine Learning Research (08/2024)
Representation Norm Amplification for Out-of-Distribttio Detection in Laig Learning
Dong Geun Shin
dgshin@kaist.oo..k
School of Electrical Engineering
Korea Advanced Institute of Science and Technology (KAIST)
Hye Won Chung
hucckuggGkaist.ccu oc.kr
School of Electrical Engineering
Korea Advanced Institute of Science and Technology (KAIST)
Reviewed on OpenReview: https: // openreview. net/ pett/formm id= 24b4Wfuoof
Abstract
Detecting out-of-distribution (OOD) samples is a critical task for reliable machinc learning. However, it becomes particularly challenging when the models are trained on long-tailed datasets, as the models often struggle to distinguish tail-class in-distribution samples from OOD samples. We examine the main challenges in this problem by identifying the trade-offs between OOD detection and in-distribution (ID) classification, faced by existing methods. We then introduce our method, called Representation Norm Amplification (RNA), which solves this challenge by decoupling the two problems. The main idea is to use the norm of the representation as a new dimension for OOD detection, and to develop a training method that generates a noticeable discrepancy in the representation norm between ID and OOD data, while not perturbing the feature learning for ID classification. Our experiments show that RNA achieves superior performance in both OOD detection and classification compared to the state-of-the-art methods, by 1.70% and 9.46% in FPR95 and 2.43% and 6.87% in classification accuracy CIFARIO-LT and ImageNeet-LT, respectively. The code for this work is available at https://github com com//gshin2////
1 Introduction
The issue of overconfidence machine learning has received significant allention due to its potential to produce unreliable or even harmful decisions. One common case when the overconfidence can be harmful is when the model is presented with inputs that are outside its training distribution, also known out-of-distribntion (OOD) samples. To address this problem, OOD detection, i.e., the task of identifying inputs outside the training distribution, has become an important area of research (Nguyen et al. 2015 Bendale & Boult 2016 Hendrycks & Gimpel, (017). Among the notable approaches is Outlier Exposure (OE) (Hendrycks ct al. 2019a), which uses an auxiliary dataset as an OOD training set and regulates the model's confidence on that dataset. by regularizing cross-entropy loss (Hendrycks et al. (019a),, free energy (Liu et al. 2020), or total variance loss (Papadopoulos et al., 2021). These methods have proven effective and bocome the state-of-the-art in OOD detection. However, recent. findings have highlighted the new challenges in OOD detection posed by tailed datasets, characterized by class imbalance, which are common in practice. Even OE and other existing techniques struggle to distinguish tail-class in-distribution samples from OOD samples in this scenario (Wang et al. 2022b Thus, improved methods are needed to tackle this new challenge.
To address OOD detection in loig-tail learning, we need to solve two challenging problems: (1) achieving high classification accuracy on balanced test sets, and (ii) distinguishing OOD samples from both general and tail-class in-distribution (ID) data, which is underrepresented in the training set. While previous works have explored each problem separately, simply combining the long-tailed recognition (LTR) methods (Kang et. al.
1

---

Published in Transactions on Machine Learning Research (08/2024)
RNA (ours)
78-
LA
LA+0E+CL (PASCL)
Accuracy (%)
76-
Trade-offs between
LA+OE
75
ODD- LTR-tallored methods
CE+OE
Baseline
OOD methed
73
CE
LTR method
+LTR methed
50
20
FPR95 (%)
Figure 1: The OOD detection performance (FPR95) and classification accuracy (Accuracy) of models trained by various methods. OF (Hendrycks et. al. 2019a), designed for OOD detection, and LA (Menon al., 2021), aimed at long-tailed recognition (LTR), exhibit trade-offs between OOD) detection and ID classification when combined (E). In contrast, method (RNA) excels in both FPR95 and Accuracy, effectively overcoming these trade-offs.
2020
Menon et al. 2021 with OOD methods does not effectively solve this challenge (Wang al. 2022b
This is because the cxisting LTR and OOD methods often pursue confficting goals in the logit space, creating trade-offs between OOD detection and classification, especially for tail classes (see Figure 1 and Table For example, OE (Hendrycks et al. 2019a) uses auxiliary OOD samples to train the model to output uniform logits for rare samples, while the LTR method such as Logit Adjustment (LA) encourages a relatively large margin between the logits tail versus head classes. Combining these approaches results in better OOD detection but inferior classification compared L-R-tailored methods, while achieving worse OOD detection but better classification than OOD-tailored methods. The crucial question is: how can we achieve both goals without compromising each other?
To address this challenge, provide a new OOD detection method in long-tail learning, called Representation Norm Amplification (RNA), which can effectively decouple the classification and OOD detection problems, by performing the classification in the logit space and the OOD detection in the embedding space. Our proposed training method creates a distinction between ID and OOD samples based on the norm their representations, which serves as our OOD scoring method. The main idea is train the classifier to minimize classification loss only for ID samples, regularized by loss enlarging the norm of ID representations. Meanwhile, we pass auxiliary OOD samples through the network to regularize the Batch Normalization (BN) (loffe & Szegedy 2015) layers using both ID and OOD data (Figure 3a). This simple yet effective training method generates discernible difference in the activation ratio at the last ReLU layer (before the classifier) and the representation norm between ID) vs. OOD data, which proves valuable for OOD detection. In addition, since the loss only involves ID data, only ID data is involved in the gradient for updating model parameters. This focus on learning representations for classification, rather than controlling network output for OOD data, is a key aspect in achieving high classification accuracy. Our contributions can be summarized as follows:
We introduce Representation Norm Amplification (RNA), novel training method that successfully disentangles TD) classification and OOD detection in long-tail learning. RNA achieves this by intentionally inducing noticeable difference in the representation norm between ID and OOD data. This approach enables elfective OOD delection without compromising the models' capability to learn representations for achieving high classification accuracy.
We evaluate RNA on diverse OOD detection benchmarks and show that it improves FPR95 by 1.70% and 9.46% and classification accuracy by 2.43% and 6.87% on CIFARIO-LT and ImageNet-LT, respectively, compared to the state-of-the-a methods without fine-tuning.
2

---

Published in Transactions on Machine Learning Research (08/2024)
2 Related works
OOD detection in long-tail learning While substantial research has been conducted in OQD detection and long-tail learning independently, the investigation of their combined task, referred to as LT-OOD, has emerged only recently. The LT-OOD task encompasses three main lines of research: training methodologies Choi et al.) 2023). post-hoe scoring techniques (Jiang al.) 2023). and abstention class
(Wang et al., 2022b
learning (Mia et al.
2023
Wei ct al. 2023 for LT-OOD tasks.
2022b) first addressed OOD detection in the context of long-tail learning, introducing a contrastive learning idea to improve the separability between ID tail classes and OOD data. BEL Choi
PASCT. Wang et al.
ict al. 2023) tackled the LT-OOD task, highlighting class imbalances in the distribution of auxiliary OOD set. They proposed a method to regularize auxiliary OOD samples from majority classes more heavily during energy-basod regularization. However, recent methods like PASCI, and BEL employ the two-branch technique, where each branch is tailored for OOD detection and classification, respectively. In contrast, our method does not use such two- branch technique. Instead, effectively address both challenges through a single model, leveraging the idea to disentangle ID classification and OOD detection, as will be observed in Section 6.2
Jiang et al. 2023) tackled the LT-OOD task using a post-hoc method, adapting traditional OOD scoring to the long-taill tailed setting. Despite its effectiveness, this approach relies on model bias induced by class imbalances in the training dataset. Thus, it is hard to be combined with training methodologies aimed at resolving class imbalances to improve the classification accuracy.
Recently, abstention class-based methods tailored for L.T-OOD tasks have emerged, including EAT (Wei et al., 2 and COCL (Miao et al. 2023). EAT integrats four techniques: multiple abstention classes, CutMix Yun et al. 2019) augmentation for tail-class ID data, the auxiliary branch fine-tuning, and ensemble models. COCL combines two techniques: debiased large margin learning and outlier-class-ware logit calibration. Debiased large margin learning entails training networks with two contrastive loss terms: one for tail-class ID prototypes and OOD samples, and the other for head-class ID samples and OOD samples. While these methods offer substantial improvements, they require a comprehensive approach with multiple techniques, and face challenges with OOD representations collapsing into the same class despite semantic differences.
Norm-based OOD detection Previous studies have investigated the utilization of norms for OOD detection scores. Dhamija et al. (2018) pioneered the use of representation norms for OOD detection, with a training method, Objectosphere, directly attemuating representation norms of auxiliary OOD data during training to increase separation between ID and OOD data in the feature space. However, training with RNA loss proves more effective than directly altenuating OOD representation norms to reduce activation ratios and representation norms (see Section 6.3.1). CSI (Tack et al. (020)) incorporated norms into their scoring method, obscrving that the gap in representation norms between ID and OOD data merges implicitly in contrastive learning. NAN (Park et al. 2023) and Block Selection (Yu et al. 2023 also utilized feature norms for OOD detection, considering deactivated feature coordinates and selecting the most suitable hidden layer for norm utilization, respectively. While these approaches leverage representation norms for OOD scoring, they lack a training method aimed at widening the gap between the norms of ID and OOD data. In contrast, our method not only ad vocaates for using representation norms for OOD detection in long-tail context. but also propose a corresponding traning method.
There also exist techniques to address overconfidence issue for ID data by controlling the norm of logits. ODIN Liang et al., 2018) uses temperature scaling for softmax scores at test time and LogitNorm (Wei ct al. 2022) normalizes logits of all samples during training. However, these methods do not consider the underconfidence issue of lail classes. In long-tail learning, networks often produce underconfident outputs for tail predictions. Thus, solving both the overconfidence problem of OOD data and the underconfideence problem of tail ID data is crucial. Our method tackles this challenge by amplifying the norms of only ID representations, while indirectly reducing the representation norms of OOD data by BN statistics updates using TD and OOD data. More reviews of related works are available in Appendix A
3

---

Published in Transactions on Machine Learning Research (08/2024)
3 Background
3.1 OOD detection
In this section, we formally define the task of OOD detection. Consider classification model that takes images as input and outputs predicted labels, consisting of a representation extractor fo with learnable parameters 0 and linear classifier W (sec FigureBa). The model is trained on a labeled datasct {(z,, yi) to optimize such that fa(x;) = y; for all i. In-distribution data is sampled from the training data distribution, while dustr--distiitution data comes from a different distribution. The main challenge of OOD detection task is that overparameterized deep neural networks tend to be overconfident in their predictions for OOD data. This overconfidence hinders the model's ability distinguish between ID and OOD samples.
OOD samples are detected using a decision function G, which employs an OOD scoring function S. The function S assigns scalar score to each sample, indicating how the sample is likely to be an OOD data. The decision function G is defined as:
ID if ((;;0, < GOOD
G(r
(1)
if S(; 2
where & is a threshold.
Outlier Exposure (OE) (Hendrycks et al. 2019a) is an effective training methodology for OOD detection, using
auxiliary OOD set. The OOD data in this set are from distribution disjoint with the distribution of both ID and the test OOD data. OE trains the model regulating the output softmax vectors of auxiliary OOD samples to have a uniform distribution over classes. The detailed loss function is explained in Section 5.1
3.2 Long-tail learning
In real-world scenarios, datasets often exhibit imbalanced class distributions, leading to models biased towards classes with larger number of samples. This issue becomes more pronounced the imbalance incrcases. Long-tailed dalasets is defined imbalanced datasets where a few classes (head) have many samples and many classes (tail) have few samples. Long-tailed datasets typically have extremely imbalanced label distributions, often following exponential distribution. Addressing this problem has led to extensive research in long-tail learning, as detailed in Appendix A
Logit Adjustment (LA) (Menon et al. 2021 is a prominent method for tackling the long-tail learning task. The rationale behind LA is to adjust the model to be Bayes-optimal for balanced test sets by subtracting the logarithm of the label probability from the logits. This is formulated as follows:
arg max Ipbal (y|z) = arg max P(zly) = arg max P(yl=)/P()) vE[C]
yE(C)
DE[C]
arg max S(WT So(x)) T
WE(C]
arg max W (z)y - log Ty.
vE[C]
where S(WT [o(x)) is the softmax output for label given the weight matrix W = [wi, w2,..., "c]] C' is the number of classes, [C] = {I,..., and x E 1] is the estimate of the label distribution over C' classes. In the formulation above, S(WTfo represents the output of the biased model trained on an imbalancod dataset. To achieve a Bayes-optimal classification model for a balanced distribution during training, the logits are adjusted by adding the logarithm of the label probability. The LA loss is defined as:
LLA Ei logl Σ exp (w: fo(x) T log n.)) (w,, fo(x) -log Fy))
(2)
cE(C]
where > 0 is hyperparameter, which we set to 1.
4

---

Published in Transactions on Machine Learning Research (08/2024)
Table 1: The existing OOD detection method (OE) and long-tail learning method (LA) aim to achieve better OOD detection and classification, respectively, by enforcing desired confidence gaps between ID vs. OOD or head vs. tail classes. Simply combining OE and LA (LA+OE) leads to trade-offs between OOD detection performance (FPR95) and classification accuracy, especially for the tail classes (Few). Our method (RNA) achieves superior OOD detection and classification performance by decoupling the two problems.
Softmax Confidence
FPR95 (.)
Accuracy (1)
Method
Avg.
Few
OOD
Avg.
Few
Avg.
Fow
CE
93.51
90.16
87.03
61.49
80.17
72.65
51.32
CE+OB
65.04
46.64
10.91
15.74
25.64
74.85
57.60
LA
92.75
89.54
81.65
52.64
66,04
77.30
64.18
LA+OE
64.01
46.63
10.94
17.92
28.45
75.32
57.93
PASCL
65.70
48.84
11.00
13.40
18.41
76.48
62.84
RNA (ours)
85.69
77.76
15.36
10.94
15.32
78.73
66.90
4 Motivation: trade-offs between OOD detection and long-tailed recognition
We first examine the OOD detection in long-tailed recognition to understand the main challenges. Specifically, we observe tradc-offs between the two problems, when the state-of-tho-art OOD detection method, Outlier Exposure (OE) (Hendrycks el. al.) 2019a), is simply combined with the widely used long-lail learning method, Logit Adjustment (LA) (Menon et al. 2021). OE exposes auxiliary OOD samples during training to discourage confident predictions on rare OOD samples by enforcing a uniform logit for OOD samples, while LA encourages confident predictions on rare tail-class ID samples by applying label frequency-dependent offset to cach logit during training.
In Table D we compare the performance of models trained with the cross-entropy (CE) loss, CE+OE loss (OE), LA, LA+OE, and PASCL (Wang et al., 2022b), which is recently developed method for OOD detection in long-tailed recognition by using the contrastive learning (CI.) idea, combined with LA+ +F.. We report the results of ResNet18 trained on CIFARIO-LT (Cui et al. 2019 when the performance is evaluated for an OOD test set, SVHN (Netzer et al., 2011). FPR95 is an OOD detection score, indicating the false positive rate (FPR) for ID data at a threshold achieving the true positive rate (TPR) of 95%. FPR95 can be computed for each class separately, and we report the FPR95 scores averaged over all classes (Avg.) and the bottom 33% of classes (Few) (in terms of the number of samples in the training set), and similarly for the classification accuracies.
Comparing the results for CE vs. CE+OE and for LA vs. LA+OE, we can observe that OE inereases the gap between the softmax confidence of ID and OOD data, which makes it easier to distinguish ID data from OOD data by the confidence score, resulting in the lower FPR95. However, compared to the model trained with LA, the model trained with LA+OE achieves a lower classification accuracy, reduced by 1.98% on average and 6. 6.25% for the 'Few' class group.
One can view LA+OF as method that trades off the OOD detection performance and TD classification accuracy, especially for tail classes, since it achieves worse FPR95 but better classification accuracy compared to the toD-tailored method, CE+OE, while achieves better FPR95 but worse accuracy than the LTR- tailored method, LA. PASCT,, which uses contrastive learning to push the representations of the tail-class ID samples and the OOD auxiliary samples, improves both the FPR95 and the classification accuracy for the 'Few' class group, each compared to those of CE+OE and LA, respectively. However, it still achieves a slightly lower average classification accuracy (-0.82%) than the model trained only for LTR by LA loss.
The main question is then whether such tradc-offs between OOD detection and long-tailed recognition are fundamental, or whether we can achieve the two different goals simultaneously, without particularly compromising each other. All the existing methods reviewed in Table [ aim to achieve better OOD detection, long-tailed recognition, or both by enforcing the desired logit distributions for ID vs. OOD data or head vs. tail classes, respectively. However, we observe the limitations of such approaches in achieving the two possibly conflicting goals simultaneously through the logit space, even with the help of contrastive learning.
5

---

Published in Transactions on Machine Learning Research (08/2024)
Based on this observation, we propose to utilize new dimension for OOD detection in the long-tail scenario, the norm of represenlation vectors, to decouple the OOD detection problem and the long-tailed recognition problem, and to achieve the two different goals without perturbing each other. As shown in Table D our method, Representation Norm Amplification (RNA), achieves the best FPR95 and classification accuracy for both the average case and the 'Few' class group, compared lo other methods, including even those lailored to only one of the two goals, such as CE+QE or LA. In particular, our method achieves a desirable gap between the confidence scores of ID (Avg.) vs. OOD data (85.69 vs. 15.36%), while maintaining relatively high confidence score of 77.76%, for the 'Few' ID classes.
5
Representation Norm Amplification (RNA)
In Section 5.1 we discuss the limitations of previous approaches in the way of exposing OOD samples during training by analyzing the loss gradients, and in Section 5.2
we present our method.
5.1 Previous ways of exposing auxiliary OOD data during training
Let Din and Doon denote an ID training dataset and an auxiliary OOD datasel, respectively. The stale-of- the-art OOD detection methods expose the auxiliary OOD data during training to minimize the combination of classification loss LID for ID data and an extra loss LOOD for OOD data, i..,, f LID + XLOOD for some tunable parameter À 0. As an example, one can choose the regular softmax cross-entropy loss as the ID loss, LID (( log Emetey exp(w. fo(x))) - (we,] fa(=)))
(o)))) where fa(x) E RD is the representation
vector of the input data and We E IR is the classifier weight (i.e., the last-layer weight) for the label e [C]. For long-tailed recognition, the LA method (Menon et al. 2021 uses the logil-adjjsted loss to improve the classification accuracy of ID data, i.e., LID = LLA in Equation 2 On the other hand, the loss for OOD training data is often designed to control the logit distribution or the softmax confidence of the OOD samples. For example, the OE method minimizes the cross entropy wetween the uniform distribution over classes and the softmax of the OOD sample, i..., LOOD Eg'nDoon
log [Eeejo exp(w. fo(z))) -ccejoj è (w, fo(2"))
OE perturbs tail classification: gradient analysis We explain why training with OOD data to minimize the loss of the form L = LID + ACOOD can lead to a degradation in classification accuracy, particularly for tail classes. Let T = {x; M1), x = be a training batch sampled from the sets of ID training samples {(x;, v:)} and unlabeled OOD samples {a'h: respectively. For simpleness of analysis, we assume a fixed feature extractor fo()) and examine how ID/OOn samples in the batch affect the updates of the classifier weights {w=)C_1- Note that the gradient of the loss L = LID + ALoOD on the batch T with respect to the classifier weight w. is
Oc Dw.
1 B
B E
(o(a)))((WT)) fo(s)) - -i) +
E i=1
B
fo(a)) (o(()))())))///// -
(3)
B
where of 1 if y4 = c and 0 otherwise. For LID LLA, S(WT ( fz:)) softmax output for the logits adjusted by the label-dependent offsets as in Equation 2
in the first term is replaced by the
From Equation 3 the gradient is a weighted sum of the representations of ID data (f(cx)) and OOD data {fo(=))) where the weights depend on the gap between the softmax output S(W fo(z;))e and the desired label, yf for ID data and 1/C for OOD data, respectively. Thus, the gradient balances the effects of ID samples, tuned to minimize the classification error, and those of OOD samples, tuned to control their confidence levels. For a tail class t, however, the proportion of ID samples with the label y, t is relatively small. As result, during training, the gradient for the classifier weight we of the tail class is dominated by the representations of OOD samples rather than those of the tail-class ID samples, particularly at the early stage of training when the label distribution for OOD samples deviates significantly from a uniform distribution. In Figure 2a, we plot the log-ratios (Iog(I) V...lll////... between ID and OOD samples with respect to the classifier weights of bottom 3 (tail) classes vs. top 3 (hoad) classes. Classifier weights for tail classes are more heavily influenced by OOD samples than those for head classes, especially in the early stages of training, resulting in greater degradation of classification accuracy,
of the gradient norms
6

---

Published in Transactions on Machine Learning Research (08/2024)
0.8
Gradient norm ratio
10
0.7
Activation ratio
0.5
0.4
0.3
Head class
0.2
RNA-ID
RNA-0OD
Tail class
LA-ID
LV-000
OE-ID
OE-00D
-10
0.0
25
50
75
100 Epoch
125
150
175 200
25
50
75
100
125
150
175
200
Epoch
(a) Log-ratio of gradient
of head/Lail class dala
(b) Activation ratio at the last ReLU layer
Figure 2: (a) The gradient ratio of ID classification loss to OOD detection loss with respect to the classifier weight of LA+OE model trained on CIFARIO-LT, i.c. V.. LiDll//// w..l///V..ooooo fOOD|li) In particular, LLA and Loe are used as LID and LOOD, respectively. Note that the log-ratio for tail classes is less than zero at the early stage of training, indicating that the gradient update is dominated by OOD data rather than ID data. (b) The activation ratio of ID) and OOD representations the last ReLU layer in the models trained by RNA, LA, and OE. CIFAR10 and SVHN are usod ID and OOD sots, respectively.
as observed in Table 1 The question is then how to effectively expose OOD data during training learn the representations distinguish ID and OOD data, while not degrading the classification.
5.2 Proposed OOD scoring and training method with auxiliary OOD data
Representation Norm (RN) score We present simple yet effective approach to expose OOD data during training to create a discrepancy in the representation between ID and OOD data. The main strategy is to utilize the norm of representations (i.e., feature vectors). Note that the representation vector fa(x) can decomposed into ils norm [])(()]]2 and direction (x) = fe(=)//llf((>))] Simply scaling the norm [[S(()) l2 of the representation without changing its direction does not change the class prediction, since for any scaling factor 1, we have arg max.e[[] S(WT "ff(())) arg max S(WT)a(x)) We thus leverage the norm of representations to generate discrepancy between the representations of ID vs. OOD data, while not enforcing the desired logit distribution for OOD data. This approach, termed Representation Norm (RN) score, is formulated
(RN(F) -111o
(4)
for test sample æ. Following the convention for OOD detection, samples with larger scores are dotoctod OOD data, while those with smaller scores considered ID data.
Representation Norm Amplification (RNA) We propose training method obtain desired repre- sentations for TD vs. OOD data. Our goal in the representation learning twofold: (i) to make the norms of the reprosentations for ID data relatively larger than those of OOD data, and (ii) to learn the representations that achieve high classification accuracy even when trained on long-tailed datasets. To achieve the first goal, we propose Representation Norm Amplification (RNA) loss CRNA:
CRNA -EzDDi log(1 [fo((f())))l))]
where hop is a 2-layer MLP projection function. This loss designed increase the norm of the representation for ID data, employing a logarithmic transformation on the L2-norm of the representation. The gradient of the RNA loss with respect to the projected representation is expressed as:
δCRNA Öh; (1 [1elll2) 1h:112'
h
(6)
7

---

Published in Transactions on Machine Learning Research (08/2024)
Pre-Normalizztion value
activated deactivated fo (Xt))
Classifier
LA
W
XiD
Representation extractor fe
-2
Tra hing BN running mean ming
Xoon
(XOO))
Projection
h
RRA
25
125 150 175 200 Epochs
Overview of the proposed method, Representation Norm Amplification (RNA) (b) Values before the last BN layer
Figure 3: (a) During training, RNA uses both IT) and auxiliary OOD data. The network parameters are updated to minimize the classification loss LLA (Equation of the ID samples, regularized by their reprosentation norms through the RNA loss LRNA (Equation to the updating of the model parameters, as being used to update the running statistics of the BN layers. (b) This illustration represents the latent values of ID (red) and OOD (blue) samples before the last BN layer of the RNAA-triiedd model. The error bars denote the maximuun and minimum values among the latent vectors averaged over ID and OOD data, respectively. Through the training, the running mean of the BN layer (black) converges between the ID and OOD values. After passing through the last BN and ReLU layers (before the classifier), the shaded region beneath the BN rumning mean is deactivated. RNA effectively generates noticeable gap in the activation ratio at the last ReLU layer and the representation norm between ID vs. OOD data, which serves as the basis for our OOD score.
5 The OOD data only indirectly contribute
where ERNA represents the empirical estimation of RNA loss, LHNA= == E log(1 (1 + (f))))( with batch size B, and hú denotes the projected representation of sample di, h5 = he (f:(()))). The norm of this gradient is computed as:
@LRNA
1
(7)
(1 12,12)
The gradient norms are inversely proportional to representation norms. Consequently, during the training process, the magnitude updates to model parameters decreases as ID representation norms increase. This behavior promotes convergence of the training procedure. Further details on training dynamics can be found
in Appendix E.9 We then combine this loss with the logit adjustment (LA) loss LLA (Menon et al. 2021 tailored for long-tail learning, and define our final loss function as below:
L= + ALRNA,
where À is hyperparameter balancing the two objectives. We set X = 0.5 for all the experiments in Section 1
It is important to note that we only use ID data to optimize the loss function (Equation by gradient descent. OOD training data do not provide gradients for model parameter updates, but instead are simply fed into the model as shown in Figure 3a] We use OOD data to regularize the model by updating the rumning statistics of the BN layers from both ID and OOD data. OOD samples contribute to updating only the running mean and variance of BN layers, causing changes OOD representations do not dircetly participate in the loss gradient for the classifier weights. Thus, OOD dala only indirectly influences parameter updates. In the rest of this section, will explain why forwarding OOD data is sufficient to regularize the model to produce relatively smaller representation norms of OOD data compared to those of ID) data, when the total loss is regularized by LRNA-
the latent vectors of ID data. However, the
Effect of auxiliary OOD data: regularizing the activation ratio In common deep neural network architoctures, BN layers are often followed by ReLU layers (Nair Hinton 2010). BN layers normalize inputs by estimating the mean and variance of all the given data. Consequently, the coordinates of input vectors with values below the running mean of the BN layer are deactivated they pass through the next ReLU
8

---

Published in Transactions on Machine Learning Research (08/2024)
0.175
ID Test OOD Test
ID Test
0.6
0.150
OOD Test
0.5
Density
0.125
Density
0.4
0.100
0.075
0.3
0.050
0.2
0.025
0.1
0.000
0.0
10
15
20
25
30
10
20
30
40
50
70 80
Representation norm
Representation norm
(a) RN histogram of LA
(b) RN histogram of RNA
Figure 4: (a) The histogram of representation norms of ID/OOD test data on LA-trained models on CIFAR10- LT. The red bars represent the density of representation norms of ID data and the blue bars represent that of OOD data (SVHN). (b) The histogram representation norms ID/OOD test data on RNA-trained models on CIFAR10-LT. The evident gap the distributions of representation norms of ID and OOD data enables effective OOD detection using representation norms.
layer (Figure 3b.. In Figure 2b we compare the activation ratio (the fraction of activated coordinates) the last ReLU layer (prior the last linear layer) for ID data (CIFAR-10), and OOD data (SVHN), using models trained by RNA, LA, and OF. RNA exhibits distinct activation ratio patterns, clearly separating ID and OOD data, RNA activates the majority of the coordinates of representation vectors of ID data (66.08% on CIFAR10) while dcactivating those of OOD data (4.17% on SVHN). This indicates that BN layer regularization ion using both ID and OOD data, along with enlarging the ID representation norms through CANA, results in a substantial discrepancy in the activation ratio at the last ReLU layer. Morcover, since the activation ratio corresponds to the fraction of non-zero entries in the representation vector, a notable gap in the activation ratio signifies a difference in representation norms between ID and OOD data, facilitating reliable OOD detection by the RN score (Equation 4b. Figure 4a and 4b present histograms of representation norms of ID vs. OOD data when trained by LA and RNA, respectively. The evident gap between ID and OOD test sets is shown only for the RNA-trained models, enabling elfective OOD detection with represent ion norms. Additional RN histograms for RNA-trained models on other pairs of ID and OOD test sets are available in Appendix E.10 The training dynamics RNA, including loss dynamics of LA loss (Equation 2 norm dynamics of head and tail data ID training/lest
and RNA loss (Equation as well as represent ation ation datasets, are provided in Appendix E.9
6 Experimental results
6.1 Experimental setup
and ImageNet-1k Deng et al. 2009). The long-tailed training sets, CIFAR10/100-I.T, are built by downsampling CIFAR10/100 Cuu et al. 2019), making the imbalance ratio, max, (c)/ min, (c), equal to 100, where (c)) is the number of samples in class c. The results for various imbalance ratios, inchuding balanced scenario, are reported in Table We use 300K random images (Hendryeks et al.] 2019a) as an auxiliary OOD training set for CIFAR10/ 100. For ImageNet, we use ImageNet-LT (Liu al. (2199) as the long-tailed training set and ImageNet-Exxee (Wang et al., 2022b) as the auxiliary OOD training set.
Datasets and training setup For ID datasets, use 11FAR10/100 (Kizheevkyy 2009
For experiments CIFAR10/I00 100, we use the semantically coherent out-of-distribution (SC-OOD) benchmark
datascts Yang et al. 2021 as our OOD test sets. For CIFAR10 (respectively, CIFAR100), we use Textures (Cimpoi el al. 2014), 1), SVHN (Netzer el al. 2011). CIFAR100 (respectively, CIFAR10), Tiny ImageNet (Le & Yang 2015), LSUN Yu et al. 2015), and Places365 (Zhou et al. 2018 from the SC-OOD benchmark. For ImageNet, we use ImageNet-Ik-Ool
Wang et al. 2022b), as our OOD test set. All the OOD training sets and OOD test sets are disjoint. Further details on datasets can be found in Appendix D.3 We train ResNet18
(He et al. 2015 CIFAR10/100-11 datasets for 200 epochs with batch size of 128, and ResNet50 HHe

---

Published in Transactions on Machine Learning Research (08/2024)
Table 2: OOD detection performance (AUC, AUPR, and FPR95) and ID classification performance (ACC, Many, Medium, and Few) (%) on CIFAR10/100-IA and ImageNet-LT. Means and standard deviations are reported based on six runs. Bold indicates the best performance, and underline indicates the second best performance. FT represents that the models are fine-tuned with ID samples for LTR tasks.
Method
AUC (1)
AUPR (t)
FPR95 (L)
ACC (t)
Many (1)
Medium (t)
Few (1)
Dataset: CIFARIO-LT
MSP
72.68+1.04
70.48+0.89
6..67+1.82
72.65+0.119
94.65+
72.19+0.49
51.32+00.7
OECC
88.0810.19
88.65:10.09
48.48-0.88
14.48:10.19
94.2910.20
72.69:0.68
56.7940..66
EnergyOE
88. 88.78+0.48
88.91+0.61
04.78
76.25+0061
93.94+0.51
74.37+0.34
61.01+1.8 11777
89. 59.6910.55
86.471441
33.51=0051
74.8510.57
93.90:10.35
1.13
57.60:11.78
PASCL
91.00+0.26
89.35+0.522
33.51 +0.87
75.86+00.6
92.50+0.22
72.84+0.94
63.67+1.76
BEL
92.49110.00 06
91.83110.66
00.88-10.21
76.30:10.17
91.5810.14
74.54::.29
63. 63.38:10.50
RNA (ours)
92.90+0..27
92.14+0.36
29.18+0.90
78.73+0.22
93.72+0.17
76.40
-0.85
66.90+1.00
PASCL-FT
88.0310.79
84.6612.00
40.43:11.22
76.48:10.22
92.9310.71
74.2011.11
62.8411.00 1.01
BEL-FT
90.80+0.18
90.57+0.22
36.79+00.65
81.50-0.19
89.63+0.40
76.33+0.58
00.26-0.400
ID Dataset:
CIFAR100-LT
MSP
61.39+0.40
37.86+0.30
82.17+0.38
40.61++0.20
71.02=0.67
38.59+0.45
11 11.06+0..88
OECC
69.64110447
66.5510.440
76.74+0.58
41.32:10.37
69.8810.31
39.4310.78
13.97:00.55
EnergyOE
69.14+0.53
66.67+0.21
8..01+1.13
39.17+00.31
08.53+0.52
39.54+0.17
8.03++0.82
OE
73.3710.22
67.26:10.41
67.83+1.882
39. 39.83:10.32
68.2810063
37.11+0.44
12.89
PASCL
73.14+0.37
66.77+0.50
67.36++0.66
40.01+00.31
68.9740.48
37.1840.38
12.35-+0.46
BEL
77.66=0.005
73.0110.06
61.1810014
40.87:10044
61.85:10.14
47.98110116
12.56+0.18
RNA (ours)
75.061:..49
70.71
66.90+11.05
44.8010.25
68.35
44.97
19.84+0.48
PASCL-FT
73.5910.58
67.69:10.22
67.66410.49
43.44:10.34
65.8310.38
40.76::0.00
22.70+00.94
BEL-FT
75.07+0.31
70.71++0.3
07.96
45.53-0.14
61.30+0.388
41.11+0.35
31.22=0.44
ID Dataset:
ImageNet-LT
MSP
54.22-+0.66
51.85:10.54
89.67
41.69:11.45
59.01+1.90
35.54+
14.29
OECC
62.8240.28
63.89:1033
87.83+0.22
41.03:::.23
59.36+10.36
34.32+0.255
12.78+0.34
EnergyOE
63.38+0..0
64.5110..23
88.3140.19
38.47++1.00
58.82400.99
30.61+1.17
8.56++0.52
OE
67. 07.12::0.35
69.20:10.34
87.65+0.22
40.87:+0.22
60.20=0.26
34.07:0042
10.11+0.65
PASCL
68.174:..24
70.26:+0.31
0..32:10.32
40.9440.94
60.00+0.92
34.24+1.04
10.59:t0.73
BEL
64.90:10.14
65.92110.18
47.91
40.5441.88
60.2740.91
33.40+10.98
9.84110.56
RNA (ours)
7..55+0022
74.60=0.27
78.16:0081
47.81=0.31
58.39
44.58:10.38
28.67=0.31
PASCL-FT
61.83:00.33
62.8610..59
88 88.9510.20
44.97+1.01
55.98+1.08
42.29
23. 23.26:4..8
BEL-FT
58.26:1.28
57.29+0.30
90.33+0..36
45.08
55.11+1.011
42.15::0.96
27.01++1.25
et al. 2015) on ImageNet-LT for 100 epochs with batch size of 256. We set À = 0.5 in 8 More details on the implementation are available in Appendix D.I
Evaluation metrics To evaluate OOD detection, we use three metrics -the area under the receiver operating characteristic curve (AUC), the area under the precision-recall curve (AUPR), and the false positive rate of ID samples at the threshold of true positive rate of 95% (FPR95). For ID classification, the total accuracy (ACC) and also the accuracy averaged for a different subset of classes categorized 'Many,, 'Medium', and 'Few' in terms of the number of training samples. All the reported numbers are the average of six runs from different random seeds.
measure
6.2 Results
Results on CIFAR10/100 The results on LIFAR10/100-L1 are summarized in Table 2 where the OOD detection performances are averaged over different OOD test sets. We compare against several basclines, including MSP (Hendrycks & Gimpel 2017), OECC (Papadopoulos al. 2021). Engery (Liu al. 2020
OE (Hendrycks et al. 2019a). and two recently published methods tailored for T.T-ODD tasks, PASCT.
(Wang
10

---

Published in Transactions on Machine Learning Research (08/2024)
Table 3: Performance metrics (%) for OOD detection and ID classification, along with the representation norm and the activation ratio at the last ReLU layer, comparing variants of RNA loss on CIFAR1O-LT. Results for our original proposed method (RNA) are highlighted in gray.
Training loss
Representation norm
Activation ratio
AUC(1) FLR95(L)
ACC(1)
ACC-Few(1)
CLA
LOOD-ast.
LiD-amp.
ID test
OOD test
ID test
OOD test
82.68
55.05
77.82
65.49
28.84
14.55
0.55
0.55
/
/
84.03
41.59
72.92
69.62
43.65
13.
0.74
0.18
92.90
29.18
78.73
66.90
43.79
0.67
0.66
0.02
et al. 2022b) and BEL CCho et al., 2023). p. We categorize the models based on whether they incorporate separate fine-tuning for LTR after OOD detection training. Certain baselines perform OOD detection and ID classification with two separate models, where one is trained for OOD detection (PASCL and BEL) while the other is fine-tuned for LTR (PASCL-FT and BEL-FT). Notably, the OOD detection and ID classification with single model. RNA outperforms the bascline methods for OOD detection on CIFAR10-LT and ranks second on CIFAR100-LT. In particular, on CIFAR10-LT, RNA improves AUC, AUPR, and FPR95 by 0.41%, 0.31% and 1.70%, respectively, compared to the state-of-the-ar method, BEL. RNA also shows improvements of 2.43% and 3.23% over the accuracy for the toal and 'Few' class groups, respectively, compared to the second best models among those without fine-tuning. When trained on CIFAR100-LT, RNA achieves the second-best AUC, AUPR, and FPR95 with values of 75.06%, 70.71%, and 66.90%, respectively. Moreover, RNA exhibits the highest overall classification accuracy of 44.80%, except for the BEL-FT. The results of OOD detection performance on each OOD test set are reported in Appendix E.
models perform
Results on ImageNet Table 2 also shows the results on ImageNet-LLT, which is large scale dataset. Our method significantly improves the OOD detection performance by 7.38%, 4.34%, and 9.46% for AUC, AUPR, and FPR95, respectively, compared to the second best model, PASCL. Furthermore, RNA outperforms BEL-FT in terms of the classification accuracy, increased by 7.7 and 1.66% for total and 'Few' cases, respectively. It is noteworthy that the performance improvement of our method on ImageNet-LTT is greater than those on CIFAR10/ CIFAR10/100-LT, 100-LT, indicating that our method demonstrates effective performance on large scale datasets, which is considered as a more challenging case for OOD detection. Furthermore, pairs of models, such PASCL and PASCL-FT, and BEL and BEL-FT, still exhibit trade-offs between OOD detection and ID classification. PASCL-FT improves in ACC and 'Few' by 4.03% and 12.67%, respectively, compared to PASCL. However, PASCL-FT underperforms PASCL in OOD detection by 6.34%, 7.40%, and 1.33% in AUC, AUPR, and FPR95, respectively. Similarly, BEL-FT outperforms BEL in ACC and 'Few' by 4.54% and 17.17%, respectively, but lags in AUC, AUPR, and FPR95 by 6.64%, 8.63%, and 2.42%, respectively. In contrast, RNA achieves the state-of brate-of-the-e of-the-rrt performance both OOD detection and ID classification with a single model, overcoming these tradc-offs.
6.3 Additional experiments
We perform a scries of ablation study to investigate our method from multiple perspectives. Our evaluations involve variants of RNA training loss, diverse combinations of OOD training and scoring methods, different. imbalance ratios, RNA-trained models with different configurations of auxiliary OOD sets, and calibration performance. Further ablation studics are available in Appendix E where we include the results about the robustness A in Equation OOD sets ($F.4), the training batch ratíos of ID and OOD samples ($F.5). alternative auxiliary OOD sets E.6 the structure of the projection function hs ($ E.7 and OOD fine-tuning task EEE.)
EE,2h, trade-offs in models trained by LA+OE (SE.)), the size of auxiliary
6.3.1 Variants of RNA training loss
We conduct an ablation study on variants of the RNA loss. Our proposed RNA loss (LID-amp.) amplifies ID representation norms during training. We examine two alternative variants: loss to attemnate OOD representation norms, denoted to as LOOD-att., and combination of both losses. The OOD-attemuation loss is defined LoOD-att. EzzDoop [log(1 + (fox))()))) which is the negative counterpart of the RNA
11

---

Published in Transactions on Machine Learning Research (08/2024)
Table AUC (%) of various combinations of training methods and OOD scores on CIFARIO-LT, CIFARI00- LT and ImageNet-LT. The gray regions indicate the results for our proposed methods. MSP and Energy are confidence- based OOD scoring methods, while RN is evaluated in the representation space.
OOD Sooring Method
Training
CTFARIO-LT
CTFARI00-TT
ImageNNt--TT
Method
MSP
Energy
RN (ouns)
MSP
Energy
RN (ours)
MSP
Euergy
RN (ours)
CE
72.68
72.77
70.98
61.39
61.55
55.67
54.22
54.07
56.73
OE
89.69
89.66
86.63
73.37
72.98
71.81
67.12
67.77
77.99
PASCL
91.00
91.09
81.18
T3.14
72.56
70.68
68.17
68.70
77.98
BEL
87.96
92.49
92.30
70.86
77.66
76.43
59.53
64.90
75.37
RNA (ours)
91.73
91.96
92.90
73.93
74.00
75.06
61.27
61.50
75.55
loss in Equation As shown in Table 3 these variants are not as effective as the original RNA loss on CIFARIO-LT. As shown in Table 3 the combination of attenuation loss and amplification loss widens the gap in reprosentation norm and activation ratio compared to the case of using only the attenuation loss. However, the amplification loss alone creates a more substantial gap than the combination loss. In particular, the combination loss succeeds in enlarging the ID norms, but whenever attenuation loss is used (either alone or with amplification loss), the resulting OOD norm is not as low as the case of using only the amplification loss.
The rationale behind this result lies in the optimization process. When solely employing the amplification loss (RNA loss), the optimizer amplifies the norm (fo(()) for ID data, necessarily by promoting high activation ratios for ID data while reducing the activation ratio for OOD data (as low as 2% in our experiment), due to the BN regularization using both ID and OOD data. However, applying the attenuation loss (either alone or in combination form) does not necessarily suppress activations for OOD samples in the feature coordinates to minimize the loss. Instead, the loss can be minimized by mapping the OOD features into its null space. Consequently, the resulting activation ralios and the representation norms for OOD samples are not as low as the case of using only the amplification loss.
6.3.2 Scoring methods
We evaluate the effects of our training method (Equation 8 and scoring method (Equation 4 separately. Table presents the mean AUC over six OOD test sets for models trainod with different objectives: CE, OE, PASCL, BEL and RNA, and evaluated with various OOD scoring methods: MSP, Energy, and RN. On CIFARIO-LT, the RNA-trained model achieves the highest AUC for MSP and RN scoring, and the second-best for Energy scoring, On CIFAR100-LT. RNA ranks the first with MSP and second with Energy and RN. This can be attributed to the fact that RNA training indirectly increases the confidence of ID data by amplifying representation norms. Consequently. RNA maintains competitive AUC performance even with MSP or Energy scoring, which rely on the confidence level. Additionally, our RN scoring method performs comparably to MSP or Energy scoring for the models trained by OE, PASCL, and BEL. While these methods regularize the confidence of auxiliary OOD samples, they lead a significant gap in representation norms between ID vs. OOD data (as shown in Figure 2b). which allows RN score to detect OOD samples.
The experiments with ImageNet-LT show different trends. The RNA-trained model exhibits lower AUC with confidence- baseed scores (MSP and Energy) compared to the models trained using OE or PASCL. However, when RN scoring is used, all the training methods (OE, PASCL, BEL and RNA) achieve significant improvements of 10.22%, 9.28%, 10.47%, and 14.05% over Energy scoring, respectively. This highlights the superiority of the representation--osse OOD scoring method (RN), especially for a large scale dataset.
6.3.3 Imbalance ratio
In Table 5 we present the AUC and ACC metrics for the models trained on CIFAR10/100-L11 with different imbalance ratios. Higher ratios indicate greater imbalance, posing a more challenging training scenario. A ratio of represents a balanced dataset, not a long-tailed one. Our proposed method consistently outperforms PASCL in both AUC and ACC across the imbalance ratios of 10, 50, 100, and 1000, including balanced
12

---

Published in Transactions on Machine Learning Research (08/2024)
Table AUC (%) and classification accuracy (%) of models trained on CIFARIO-LT and CIFAR100-LI with various imbalance ratio p, including balanced training sets (p = 1).
Method
ATC ACC
10
50
100
1000
10
50
100
OE
96.75 96.78/93.65
94.32/87 447
91.35/ 1/.5//99/62
89.60/74 /74.86
82.98/57.22 21
84.56/71.32
78.67/66.78
74.72/44.13
T3.37/39.88 83
PASCL
96.67/93.67
91.68/87.62 62
92.25/80..1 /80.15
91.00/766 76.18
86.62/39.18 13
84.64/71.99
78.59/37.89
75.02/17.16
T3.14/13.11
RNA (ours)
97.07 97.07/94.24 /94.24 96.86 28
93.87/ 82,
92.90/78.73 77.73
68.93
81.79/ 74.10
78. 10.6.77
75.67 (48.81
74.33/44.39
Table 6: OOD detection and ID clsasifica- tion performance (%) on CIFARIO-LT with different usage settings of auxiliary set.
Table 7: Calibration performance measured by Expected Calibration Error (ECE) (%) on CIFARIO, CIFARI00, and ImageNet.
Auxiliary set
Method
AUC
ACC
Method
CIFAR10
CIFAR100
ImageNet
Aux. OOD Aug. ID
CE
20.76
32.06.10.25
21.55
MSP
72.68
72.65
OE
17.56
16.53
16.8310.44
75.57
79.32
PASCL
18.78+0.63
17.17+0.25
16.37+0.48
RNA
83.57
78.13
RNA (ours)
12.56110.49
17.67+0.41
12.6510.42
92.90
78.73
training set. On the other hand, when trained on CIFAR100-LT, RNA improves ACC across all the imbalance ratios, but it exhibits relatively lower AUC when the imbalance ratio is 1 or 10. These results shows the superiority of our model for various imbalance ratios of the training set, especially for high imbalance ratios.
6.3.4 Use of auxiliary OOD data
In our ablation study, we investigate the impact of incorporating auxiliary OOD data in our method. The results in Table reveal that employing the RNA loss without any auxiliary data leads to a modest AUC improvement of merely 2. 2.80% over the bascline MSP approach. In contrast, the combination of RNA loss with BN regularization, including auxiliary OOD samples, yields a substantial gain of 20. 10.22.. This clearly highlights the significant role of BN regularization using auxiliary OOD data, coupled with the RNA loss.
We also explore the potential for climinating the need for auxiliary samples. We conduct an experiment wherein the auxiliary OOD set is replaced with augmented images derived from the original training samples. We employ a random cropping augmentation strategy spanning 5% to 25% of the original image size. This strategy yields notable AUC gain of 10.89% over the MSP baseline and an 8.00% gain compared to not utilizing any auxiliary samples but applying RNA loss. While not matching the performance gains of our original method with the auxiliary OOD set, this suggests the potential for replacing auxiliary OOD samples with augmented training samples in applying our method.
6.3.5 Calibration
We assess the calibration performance of our method on CIFAR10/1000 100- and ImageNet-LT. To measure cali- bration, utilize Expected Calibration Error (ECE), defined ECE := >M [B.. (ACC(Bm) Conf( Bm)), where M and n are the number of bins and test data, respectively, Bn, := {z (m-1)/M < Conf(x) S m/M} represents the confidence bin, and ACC(B... and Conf(Bm.) denote average accuracy and confidence within the bin Bm, respectively. In Table 7 we report the measured ECE values for models trained with CE, OE, PASCL, and RNA. The models are trained on the long-tailed datasets, and the ECE values are evaluated on the balanced test sets. RNA achieves the highest ECE values for CIFAR10 and ImageNet. This result demonstrates that our method exhibits strong calibration performance for ID data.
7 Discussion
We propose simple yet effective method for OOD detection in long-tail learning, Representation Norm Amplification (RNA). Our method regularizes the representation norm of only ID data, while updating
13

---

Published in Transactions on Machine Learning Research (08/2024)
the BN statistics using both ID and OOD data, which effectively generates noticeable discrepancy in the representation norm between TD and OOD data. The effectiveness of our method is demonstrated with experimental results on various OOD test sets and long-tailed training sets. Broader impacts, limitations and further discussion of our work are available in Appendix B
8
Acknowledgement
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government. (MSIT) (No. RS-2024-00408000 and No. 2021RIC1C1100533),
References
Yong Hyun Ahn, Gyeong-Moon Park, and Seong Tae Kim. Line: Out-of-distributioo detection by leveraging important neurons. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2023.
Abhijit Bendalc and Terrance E Boult. Towards open set deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1563-1572, 2016.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with
label-disstrbuutior--waae margin loss. In Advances in Newral Information Processing Systems, 2019.
Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357 2002.
Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. Atom: Robustifying cattof-ddisributiinn detection using outlier mining. In Machine Learning and Knowledge Discovery in Dalabases. Research Track: European Conference, ECML PKDD 2021, Bilbao, Spain, September 13-17, 2021, Proceedings, Part III 21, pp. 430-445. Springer, 2021.
Hyunjun Choi, Hawook Jeong, and Jin Young Choi. Balanced energy regularization loss for out-of-distribution detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15691-15700 June 2023.
Sungik Choi and Sae- Young Chung, Novelty detection via blurring. In International Conference on Learning Representations, 2020.
Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. A downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprinl Xrxi 08819, 2017.
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2014. doi: 10.1109/ 10.1109/CVPR.20 CVPR.2014..4.....111
Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. Parametric contrastive learning. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 715 724, 2021.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. balance loss based on effective number of samples. In CVPR, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248 255, 2009. doi: Wttp///V..0000.2222..
Akshay Raj Dhamija, Manuel Günther, and Terrance Boult. Reducing network agnostophobia. Advances in
Neural Information Processing Systems, 31, 2018.
Andrija Djurisic, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. Extremely simple activation shaping for out-of-distribution detection. Tn The Fleventh International Conference on Learning Representations, 2023.
14

---

Published in Transactions on Machine Learning Research (08/2024)
Xin Dong, Junfeng Guo, Ang Li, Wei-Te Mark Ting, Cong Liu, and H. T. Kung. Neural mean discrepancy for efficient out-of-distribution detection. 2022 IEFE/CV CVF Conference on Compuler Vision and Pottern Recognition (CVPR), pp. 19195 19205, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition 2015.
Matthias Hein, Maksym Andriushchenkoo and Julian Bitterwolf. Why relu networks yield high-confidence predictions far away from the training data and how mitigate the problem. 2019.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural nelworks. Proceedings of International Conference on Learning Represenlations, 2017.
Dan Hendrycks, Mantas Mazcika, and Thomas Dictterich. Deep anomaly detection with outlier exposure. Proceedings of the International Conference on Learning Regresentations, 2019a.
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using sell-supervised learning can improve model robustness and uncertainty. Advances in neural information processing systems, 32, 2019b.
Y. C. Hsu, Y. Shen, H. Jin, and Z. Kira. Generalized odin: Detecting out-of-distribution image without learning from out-of-distribution data. In 2020 IEEE/CVP Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10948 10957, 2020. doi: 10.1109/CVPPP SPPR4426002020001420
Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distributional shifts in the wild. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vanghan (eds.), Advances in Neural Information Processing Systems, 2021.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift, 2015.
Xue Jiang, Feng Liu, Zhen Fang, Hong Chen, Tongliang Liu, Feng Zheng, and Bo Han. Detecting out-of- distribution data through in-distribution class prior. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 15067-15088 PMLR, 23-29 Jul 2023.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling reprosentation and classifier for loig-tailed recognition. In Eighth International Conference Learning Representations (ICLR), 2020.
Bingyi Kang, Yu Li, Sa Xie, Zehuan Yuan, and Jiashi Feng. Exploring balanced feature spaces for representation learning. In International Conference on Learning Representations, 2021.
Julian Katz- Samuuls, Julia B Nakhleh, Robert Nowak, and Yixuan Li. Training OOD detectors in their natural habitats. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp.
PMLR, 17-23 Jul 2022.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. arXiv preprint ar arx::22044.1 1 1362, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 1rXia:1412.6980 2014.
arXiv preprint
Alex Krizhevsky. Learning multiple layers of features from tiny images, 2009.
Miroslav Kubat, Stan Matwin, et al. Addressing the curse of imbalanced training sets: one-sided selection. In Teml, volume 97, pp. 179. Citeseer, 1997.
Ya Le and Xuan S. Yang. Tiny imagenet visual recognition challenge, 2015.
15

---

Published in Transactions on Machine Learning Research (08/2024)
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidencc--alibratted classifiers for detecting out-of-distribution samples. Tn 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018a.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of- distribution samples and adversarial attacks. In Advances in Newral Information Processing Systems, volume 31. Curran Associates, Inc., 2018b.
Tianhong Li, Peng Cao, Yuan Yuan, Lijie Fan, Yuzhe Yang, Rogerio S Feris, Piotr Indyk, and Dina Katabi. Targeted supervised contrastive learning for long-tailed recognition. In Proceedings of the IEEE/CVE 2022.
Conference on Computer Vision and Pottern Recognition, pp.
Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In International Conference on Learning Representations, 2018.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy- based out-of-distribution detection. Advances Newral Information Processing Systems, 2020.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X. Yu. Large-scale long-tailed recognition in an open world. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017.
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. Advances in neural information processing systems, 31, 2018.
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In International Conference on Learning Representations, 2021.
Wenjun Miao, Guansong Pang. Tianqi Li, Xiao Bai, and Jin Zheng. Out-of-distributiin detection in long-tailed recognition with calibrated outlier class learning. ar preprint arXiv:2312. 2023.
Yifei Ming, Ying Fan, and Yixuan Li. Poem: Out-of-distribution detection with postcrior sampling. In International Conference on Machine Learning. PMLR, 2022.
Yifci Ming, Yiyou Sun, Ousmane Dia, and Yixuan Li. How to exploit hyperspherical embeddings for out-of-distribution delection? In The Eleventh International Conference on Learning Represenlations, 2023.
Sina Mohseni, Mandar Pitale, JBS Yadawa, and Zhangyang Wang. Self-supervseed learning for generalizable out-of-distribution detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5216 5223, 2020.
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In International Conference on Machine Learning, 2010.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupercised Feature Learning 2011, 2011.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 427-436, 2015.
Aristotelis-Anglls Papadopoulos, Mohammad Reza Rajati, Nazim Shaikh, and Jiamian Wang. Outlier exposure with confidence control for oft-t--distributiin detection. Nearocomputing. 441:138-150, jun 2021. doi: 10.1016/j.neucom.. .0..2022.000 .22.007.
16

---

Published in Transactions on Machine Learning Research (08/2024)
Jaewoo Park, Jacky Chen Long Chai, Jacho Yoon, and Andrew Beng Jin Teoh. Understanding the feature norm for out-of-distribution detection. In Proceedings of the TEEE/CVT International Conference on Computer Vision, pp. 1557 1667,1 2023.
Samira Pouyanfar, Yudong Tao, Amup Mohan, Haiman Tian, Ahmed S Kaseb, Kent Gauen, Ryan Dailey, Sarah Aghajanzadeh, Yung-Hsiann Lu, Shu-Ching Chen, et al. Dynamic sampling in convolutional neural networks for imbalanced data classification. In 2018 IEEE conference multimedia information processing and retricual (MIPR), pp- 112-117. IEFF, 2018.
Vikash Sehwag, Mung Chiang, and Prateek Mittal. Ssd: A unified framework for self-supe outlier detection. In International Conference on Learning Representations, 2021.
Yue Song, Nicu Sebe, and Wei Wang. Rankfeat: Rank-1 feature removal for out-of-distribution detection. Advances in Newral Information Processing Systems, 35:17885-17899, 2022.
Yiyou Sun and Yixuan Li. Dice: Leveraging sparsification for out-of-distribution detection. In Europeon Conference on Computer Vision, 2022.
Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activations. In Advances in Neural Information Processing Systems, 2021.
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. ICML, 2022.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive Icarning on distributionally shifted instances. In Advances in Neural Information Processing Systems, 2020.
Jingru Tan, Changbao Wang, Buyu Li, Quanquan Li, Wanli Ouyang, Changqing Yin, and Junjie Yan. Equalization loss for long-tailed object recognition. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 11662-11671, 2020.
Antonio Torralba, Rob Fergus, and William T. Freeman. 80 million tiny images: A large data set for nouparametric object and scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(11):1958 1970, 2008 doi: 10.1109/TPAMT.2000 200......0...2
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: good closed-set classifier is all you need? In International Conference on Learning Representations, 2022.
Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: -istof--istiibution with virtual-logit matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022a.
Haoran Wang, Weitang Liu, Alex Bocchieri, and Yixuan Li. Can multi-label classification networks know what they don't know? In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021.
Haotao Wang, Aston Zhang, Yi Zhu, Shuai Zheng, Mu Li, Alex J Smola, and Zhangyang Wang. Partial and asymmetric contrastive learning for out-of-distribution detection in long-tailed recoguition. In International Conference on Machine Learning (ICML), pp. 23446 23458, 2022b.
Hongxin Wei, Reuchunzi Xie, Hao Cheng, Lei Feng, Bo An, and Yixuan Li. Mitigating neural work overconfidence with logit normalization. In International Conference on Machine Learning (ICML), 2022.
Tong Wei, Bo-Lin Wang, and Min-Ling Zhang. Fat: Towards long-tailed out-of-distribation detection. ar preprint XrXi::2312.00 08939, 2023.
Jingkang Yang, Haoqi Wang. Litong Feng, Xiaopeng Yan, Huabin Zheng, Wayne Zhang, and Ziwei Liu. Semantically coherent out-of-distributio detection. In Proceedings of the IEEE International Conference Computer Vision, 2021.
17

---

Published in Transactions on Machine Learning Research (08/2024)
Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. orXiv preprint or Xiv: 03366, 2015.
Yeonguk Yu, Sungho Shin, Seongju Lee, Changhyun Jun, and Kyoobin Lee. Block selection method for using feature norm in out-of-distributioo detection. In Proceedings of the TEER/CV Conference Computer Vision and Pattern Recognition (CVPR), pp. 15701-15711, June 2023.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF internalional conference on compuler vision, pp. 6023 6032, 2019.
Bolei Zhou, Agala Lapedriza, Aditya Khosla, Aude Oliva, and Anlonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(6): 1452-1464, 2018. doi: 101109/TPAM..21777
Jianggang Zhu, Zheng Wang, Jingjing Chen, Yi-Ping Phoebe Chen, and Yu-Gang Jiang, Balanced contrastive learning for long-tailed visual recognition. In Proceedings of the IEEE/CVE Conference on Compuler Vision and Pattern Recognition, pp. 6908-696 -617, 2022a.
Yao Zhu, YueFeng Chen, Chuanlong Xie, Xiaodan Li, Rong Zhang, Hui Xue', Xiang Tian, bolun zheng, and Yaowu Chen. Boosting out-of-distibbution detection with typical features. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 20758- 20769. Curran Associates, Inc., 2022b.
18

---

Published in Transactions on Machine Learning Research (08/2024)
A Detailed review on related works
In this section, we provide more detailed review on some of the related works, discussed in Sec. 2
OOD detection OOD scores are typically evaluated based on output confidence (Bendale & Boult
2016
Hendryeks & Gimpel 2017 Liang et al.
2018: Liu et al., 2020), feature space properties (Lec ct al.
2018b
Dong et al.,
2021 Sun el al. 2021 Song et al.
2022
Sun & Li 2022 Sun el al.
2022: Vaze et al., 2022
Wang
iet al.
2022a Ahn et al. 2023
Djurisic et al.
2023 Yu et al.
2023 Zhu et al.
2022b or gradients Huang
jet al.
2021). Many methods have been developed to encourage the separability of ID and OOD data by these
scores, for example by exposing auxiliary OOD data during training and regularizing their confidence levels
(Malinin & Gales 2018; Hein et al.] 2019
Liu et al.]
2020 Mohseni et al. 2020: Papadopoulos et al., 2021
Yang et al.) 2021 Katz-Samuels et al.
2022
Ming et al.
2022
by promoting the ID-OOD separability in
the embedding space (Dhamija et al.
2018
Lee el al.
2018a Hendrycks el al. 2019b; Choi & Chung 2020
Hsu et al.) 2020; Tack et al.) 2020 Sehwag et al. 2021
Wang et al. 2021 Ming et al. 2023). However, these
methods often suffer from performance degradation in a long-tail learning sctup (Wang ct al. 2022b
Long-tail learning Long-tail learning addresses the classification problem when trained on imbalanced or long-tailed class distributions. When trained on imbalanced datasets, the model is often biased towards the majority classes, generating underconfident predictions on the minority or tail classes. There have been many
methods to fix this problem by over- or under-sampling (Kubat et al.) [997 Chawla al.
2002: Pouyanfar
ct al.] 2018
Kang ct al., 2021). adjusting margins decision thresholds (Cao et al. 2019
Tan ct al. 2020:
Menon et al.
20 or modifying the loss functions (Menon et al. 2021 Cui et al.
2021 Li et al. 2022 Zhu
In particular, some methods attempt to increase the value of the logit for the tail classes by the post-hoc hoc normalization of the weight of the classifier (the last linear layer) (Kang et al. 2020), or by applying a label frequency-dependent offset to each logit during training (Menon et 2021). While these approaches have shown empirical effectiveness and statistical robustness the context of long-tail learning, they often conflict with the principle of out-of-distribution (OOD) detection methods, which regulate the confidence of models rare samples to prevent overconfident prodictions (Hendrycks et al. 2019a Papadopoulos et al. 2021
et al.
2022a)
OOD detection and controlling overconfidence There exist OOD detection methods that do not use auxiliary outlier data during training, but effectively mitigate the overconfidence issue of the models. For example, LogitNorm (Wei et al., 2022) characterizes that training with cross-entropy loss causes logit norms to keep increasing as the training progresses, and this often leads to overconfident predictions of the models on both in-distribution and OOD test data. Therefore, the Logit method normalizes the logit, of every input data during training to prevent the logit norm from increasing too much. Some recently published works, Rectified Activations (ReAct) (Sun et al., 2021) and Directed Sparisification (DICE) (Sun) & Li 2022), on the other hand, focus the fact that a model trained only on in-distribution data, OOD data still activate non-negligible fraction of units in the penultimate layer, where the mean activation is biased towards having sharp positive values and there are "noisy units" with high variances of contribution to the class output. Thus, these methods use post-hoc activation truncation (Sun et al. 2021 or weight. sparsification (Sun & Li 2022). to avoid overconfident predictions on OOD data.
OOD detection with abstention class OOD detection methods can be categorized into score-based methods and methods incorporating abstention classes. Score-based methods employ specific OOD scoring metrics to measure the OOD-ness of test samples such as MSP (Hendrycks & Gimpel 2017). Energy (Liu et al. 2020), ; and our proposed method, RN. RNA and all the baseline methods including MSP (Hendryeks & Gimpel 2017), OECC (Papadopoulos et al. 2021), EnergyOE (Liu et al. 2020), OE (Hendrycks et al. 2019ab, PASCL WWag et al. 2022b), and BEL CCho et al. 2023) belong to the score-based OOD detection category. On the other hand, OOD detection meth with an abstention class use networks with additional class for OOD data, referred to as abstention classes. Consequently, the network outputs (C+ + -)-dimensional vectors, where C' is the number of classes. During training, these methods often employ classification loss to categorize auxiliary OOD samples into the abstention class. These methods faced challenges that OOD representations tended to collapse, being treated as the same class, despite dissimilarities in image semantics
19

---

Published in Transactions on Machine Learning Research (08/2024)
among OOD samples. However, recent studies (Mohseni et al 2020 Chen al. 2021 strive to overcome
these shortcomings and achieve improved OOD detection performance in balanced scenarios.
B Broader impacts and limitations
Mitigating OOD uncertainty is crucial, particularly in safety-critical applications such as medical diagnosis and autonomous driving. In these domains, the ability to effectively identify and handle OOD samples is essential to ensure the reliability and safety of the systems. By addressing the challenges associated with OOD detection, we can significantly enhance the trustworthiness and applicability of deep learning models in these critical domains.
Our proposed method specifically focuses on (ackling the challenging scenarios where reliable OOD detection needs to be performed on models trained with long-tailed distributions, which is a common occurrence in practical settings but has received limited attention in previous research. By introducing our method, we enhance the safety and reliability of deploying deep models, even when trained on imbalanced datasets that have not been extensively preprocessed for class balance. This enables the application of deep learning models in real-world scenarios, where long-tailed distributions are prevalent, without compromising their OOD detection capabilitics.
Our proposed method does have some limitations. Firstly, it relies on the availability of an auxiliary OOD datasot for effective performance. We address this issuc by investigating the possibility of replacing the auxiliary OOD dalaset by augmented training dataset as reported in Table
Additionally, as shown in Table S our method demonstrates suboptimal performance on the near OOD test set, where the OOD data is semantically similar the training data. This limitation ariscs because the representation distributions of the near OOD Lest data are closer to the in-distribution training data rather than the auxiliary OOD data, resulting in some test OOD samples having large representation norms. This vulncrability to ncar OOD data can be alleviated by employing confidence-based OOD scoring methods. However, it entails trade-off in OOD detection performance for far OOD data, as discussed in Sec. address this limitation, future research should focus on developing training methods that can enhance the OOD detection performance specifically for near OOD test sets in long-tail learning scenarios. By overcoming this challenge, we can further strengthen the robustness and practical applicability of OOD detection methods in more broad real-world settings.
To
C Pseudocode
The training scheme of Representation Norm Amplification (RNA) is shown in Algorithm [ and evaluation scheme of Representation Norm (RN) is shown in Algorithm 2
D Details on the experimental setup
D.1 Implementation details
We train ResNet18 (He et al. 2015) on CIFAR10/100-LT datasets for 200 epochs with a batch size of 128. We optimize the model parameters with Adam optimizer (Kingma & Ba, 2014] with an initial learning rate of 0.001 and we decay the learning rate with the cosine learning scheduler (Loshchilov & Hutter 2017). The weight decay parameter is set to 0.0005.
We train ResNet50 (He et al. 2015 ImageNet-LT for 100 epochs with a batch size of 256. We optimize the model parameters with SGD optimizer with momentum value of 0.9 and an initial learning rate of 0.1, and we decay the learning rate with the cosine learning scheduler. The weight decay parameter is set to 0.0005.
We set the balancing hyperparameter À = 0.5 as the default value. We do not use the learnable affine parameters 8 and y of all BN layers in the model, since they are trained to amplify both ID and OOD data by converging to large values. During training, the batch consists of an equal number of ID and OOD data samples. For example, batch size of 256 means that contains 256 ID samples and 256 OOD samples.
20

---

Published in Transactions on Machine Learning Research (08/2024)
Algorithm 1: RNA: Representation Norm Amplification
Input: In-distribution training DID, auxiliary OOD training set DOOD, batch size b, number of classes C, representation extractor fe, linear classifier W =[w1, W2..... wc], projection function hg, learning rate 7) for epoch 1, 1, ..... do for iter = 1, .,., do
// Sample a batch of ID and ODD samples.
BID = (( w))l= DID & BOOD = (-; 12 DOOD
B BID BoOD
// Feed forward the batch into So.
[21, 22.,, = fe
1/ Calculate the loss function with only ID representations.
LLA = Ei= |og/E EeElcT exp (w. 2 + log - (w,y + log Ty.)
CRNA = 1- [[)((=))]))
C LLA + ALRNA
// Update the model parameters.
< A VoL
10
W - W- WW
11
Ø t V
Algorithm 2: RN: Representation Norm Score
Input: Test set Drest, representation extractor fe, linear classifier W [Wi, W2..... ..., wc for Zrost Diest do
// Feed forward test sample into fe.
Pestt = fo(
// Calculate the RN score for 00D detection.
SRN(2teet) -1estett ll2
// Obtain the softmax prediction for ID classification.
Prest = arg max S(WT Stest)
For baseline methods, we implement the methods following the training setting reported in the original papers.
D.2 Computational resource and time
We run the experiments on NVIDIA A6000 GPUs. For CIPARI0/100-LT datasets, single GPU for each experimental run, and the entire training process takes about an hour. For ImageNet-LT, we use GPUs for each experimental run, and the entire training process takes about 4 and a half hours.
D.3 Datasets
For the CIFAR benchmark, we employ 300K random images an auxiliary OOD training set following Hendrycks et al. (2019a). 300K random images is subset of 80M Tiny Images dataset (Torralba et al. 2008). The selection process for the 300K random images ensures that the image classes are disjoint with those in the CIFARIO and CIFAR100 datasets.
For the experiments on ImageNet-lk, we use ImageNet-Exee Extra WWag et al. 2022b as an auxiliary OOD training set, and ImageNet-1kkO0
published in Wang al. (2022b), as our OOD test set. ImageNot-Extraa is created by sampling 317,711 data points from 500 randomly chosen classes ImageNet- where the classes disjoint from the classes in TmageNet- ImageNet-Il-OOO contains 50,000 OOD test images
21

---

Published in Transactions on Machine Learning Research (08/2024)
evenly sampled from 1,000 randomly selected classes of ImageNet-22kk, where the 1,000 classes do not overlap with those of ImageNet-1k and ImageNet-Extra.
D.4 Baseline methods
In this section, we summarize the main baseline methods that we compare to our method in the experiments.
Outlier Exposure (OE) Henndrycks et al.; 2019a): OE is training method specifically designed for out-of-distribution (OOD) detection, with the objective of amplifying the discrepancy in softmax confidence between in-distribution (ID) and OOD data. It leverages an auxiliary OOD dataset effectively during the training process to enhance the model's ability to distinguish between ID and OOD samples. The OE method involves constructing training batches that consist of both ID training samples and auxiliary OOD samples. By combining these samples, the method aims to minimize& composite loss function that includes both the classification loss for ID samples and an additional loss term for OOD samples. This additional loss term is defined as the cross-entropy between the uniform distribution and the softmax probabilities of the OOD samples.
Energy OE (Liu et al., 2020): EnergyOE (Liu et al 2020 is variant of the OE method that offers alternative approach to controlling the confidence levels of OOD samples. Instead of regularizing the cross-entropy loss of auxiliary OOD samples, EnergyOE maximizes the free energy of OOD samples. By maximizing the free energy, EnergyOF. aims to increase the uncertainty and reduce the confidence associated with OOD samples, facilitating their detection and differentiation from in-distribution samples.
Outlier Exposure with Confidence Control (OECC) (Papadopoulos et al., 2021): : OECC
is another variant of the OE method, which controls the total variation distance between the network
output for OOD training samples and the uniform distribution, regularized by Euclidean distance between the training accuracy and the average confidence on the model's prediction on the training set.
Partial and Asymmetric Supervised Contrastive Learning (PASCL) (Wang et al.! 2022b):
PASCT. (Wang et al. 2022b), which is a recently developed method for OOD detection in long-tailed recognition, uses the ideas from supervised contrastive learning (Khosla al. 2020 combined with Logit Adjustment (LA) (Menon et. al. 2021) and outlier exposure (OE) (Hendrycks al. 2019a), to achieve both high classification accuracy and reliable OOD detection performance. In particular, this method applics the partial and asymmetric contrastive learning that pushes the representations of tail-class in-distribution samples and those of OOD training samples, while pulling only tail-class in-distribution samples of the same class. In addition, in the second stage, the Batch Normalization (BN) (loffe & Szegedy 2015) layers and the last linear layer are fine-tuned using only ID training data order for the running mean and standard deviation of BN layers to be re-fit the ID data distribution only, since the BN layers are fitted to the union of ID and auxiliary OOD data distribution before the second stage.
Balanced Energy Learning (BEL) (Choi et al. 2023): : BEL (Choi al. 2023 is an enhanced algorithm derived from Energy OE LLiu al. 2020 Energy OE (Liu et al. 2020) trains the model to output smaller energy for ID samples and larger energy for auxiliary OOD samples. While both methods share the scheme of energy regularization for OOD detection, BEL CCoi et al. 2023) weighs auxiliary OOD samples individually based on the pscudo-label of cach sample. Initially, the prior label distribution is estimated on the auxiliary OOD set by the model undergoing training. Subsequently, in the loss term and energy margin, auxiliary OOD samples estimated as head class are more emphasized. This approach effoctively achieves superior OOD detection performance, but still requires auxiliary fine-tuning stage for achieving high classification accuracy in long-tailed scenarios, akin PASCL (Wang et al. 2022b
22

---

Published in Transactions on Machine Learning Research (08/2024)
E
Additional experimental results
E.1
OOD detection performance on six different OOD test sets
Table 8: AUC (%) for six different OOD test sets on models trained with CIFARIO/100-IT..
OOD test
Method
Average
Texture
SVHN
CIFAR
Tiny ImageNet
LSUUN
Places365
ID Dataset: CIFARIO-LT
MSP
1..91
71.98:2.82
70.94
72.8610.66 66
74.6110.89
72.8010.72
72. 72.68:11.04
OECC
91.26:10.59
95.86111000
80.20-0.24
83.6910.00
90.38:1045
87. 87.02:10.11
88.0810.119
EnergyOE
91.73:0..6
02.3912.01
81.70-0.51
84.5710.53
92.2610.26
90.02:10.17
88.7810.48
OF
91.66+0.72
95.02+1.13
83.43+0.32
86.10+0.33
91.81++0622
90.15+0.44
89.69+0051
PASCL
93.35+00.7
96.65+00554
84.44+0.31
87.34+0.30
92.84+0.29
91.37+0.199
91.00+00.26
BEL
95.56:0..22
97.44-0.27
85.1410.05
88.7610.00
94.66110.08
93.39:10.08
92.49-10.06
RNA (ouns)
95.97+00.3
97.36+0.65
85.47+0.30
89.33 = 0.25
95.48+00.17
93.76=0.11
92.90++0.27
II) Dataset:
CIFARTOU-L:T
MSP
35.68410447
63.45522.32
60.42-0.29
62,60:10228
62.4840.34
63.70:10.18
61.39:10.40
OECC
69.74+1.32
1..41:1.22
60.2110.51
67.5910.32
75.7910.61
74.1111.40
69.6410.47
EnergyOE
69.96++0.91
73.46++3.93
61.25+0.43
66.29+0.23
72.21+0.46
71.66+0.20
09. 09.14:00.33
OE
76.56:t0.79
1..3
0.54
+0.34
77.11
75.99++0.34
73.37+0.22
PASCL
76.15:10.0
19.2341139
62.26-0017
68.37:10.25
77.02-10.11
75.79:1.24
73.14100.37
BEL
81.99-0.00
88.32=0.18
59.26-1021
71.33=0.07
83.9210.10
81.11=0.09
77.6610.05
RNA (ours)
08.38:00.56
85.8542..61
58.0120.53
68.2040.27
80.99+0.37
78.89:00.35
05.06:00.99
Table 9: AUPR (%) for six different OOD Test sets on models trained with CIFARIO/100-LTT.
OOD test
Method
Average
Texture
SVHN
CIFAR
Tiny ImageNet
LSUN
Places365
ID Dataset:
CIFAR10-LT
MSP OECC
54.2411.84
82.52.1..50 1.6
66.11:1..66 65
62.76:10.71
70.20:10.96
87.05:10.42
70.48:10.89
87.56=00.75
08.28+0.45
80.11=0.22
79.83++0.14
90.91+0046
95.22++0.09
88.65+00.09
EnergyOE
86.71+1.111
1..75+1111
81.73+0.62
80.85+0072
92.22+0.31
96.22++012
88.91+0.61
OE
79.5241..63
1..7411..26
80.1811.04
78.15:10.99
89.03:11.70
05.23110.5
86.471141
PASCL
8..94
98,1410.00
83.05=0.50
81.82:1..0
91.1110.58
96.2410.09
89.35:10.52
BEL
92.12100.09
98.66-0.13
$4.8510.05
84.78110.10 10
93.32-10.12
97.23:10.04
91.83:10.06
RNA (ours)
90.0
98.6110.33
85.0410.41
85.10=0.48
94.65-10.26
97.40-0.07
92.1410.36
ID Dataset:
CIFAR100-LT
MSP
38.86+0.31
77.80+1.56
57.82+0.33
46.50+0.26
46.42+0.54
79.78+00.13
57.86+0.30
OECC
5..26+1.59
82.67+0.99
56.15+0.34
52.76+0..9
63.62+0.86
86.82+0.23
66.55+0.40
EnergyOR
58.34+1.38
86.224-2.22
57.73+0.55
52.02++0.42
59.85++0.55
85.89+0.16
66.67+0.21
OE
58.70:22.1
87.5410.966
57. 07.33
52.26:10.99
61.25:1..99
86.5311.22
67.26110.41
PASCL
5..50+11885
87. 87.09:10.75
$7.02-0.15
51.85110.44
60.8610.65
86.3110.18
66.7710..0
BEL
7..92+0014
22.69-0.16
54.86+0.15
56.41-0.11
0.
89.94-0.04
73.01+0.06
RNA (ours)
6..5
91.9742.01
53.9320.35
53.6840.41
18.15
88.97:00.27
70.71
Table 8 9 and 10 present the AUC, AUPR and FPR95 metrics, respectively, for the models trained using RNA and other bascline approaches across the six OOD test sets. These results collectively demonstrate the
OOD detection performance. Specifically, in Table g RNA achieves the best. results for all the six OOD test. sets when the model is trained on CIFARIO-LT. For CIFAR100-LT, RNA achieves the best results on four OOD test sets as well as on average, except for CIFARIÓ and Tiny ImageNet, which are semantically more similar to the training set (CIFAR100-LT) than the other OOD test sets. In Table g our proposed method exhibits the highest AUPR values across all the OOD test sets when trained on CIFARIO-ITT Similarly, for
23

---

Published in Transactions on Machine Learning Research (08/2024)
Table 10: FPR95 (%) for six different OOD test sets on models trained with CIFARI0/100-LT:
OOD test
Method
Average
Texture
SVHN
CIFAR
Tiny ImageNet
LSUN
Places365
ID Dataset:
CIFARIO-LT
MSP OFCC
67.27-3.88
61.49.55.34
70.52+1.56 156
65.70:1..97
1..
66.93:11.13
63.67:11.82
43.70+1.45
24.62+3.89
63.02+0.79
56.39+0.71
47.77+1.26
55.36+0066
48.48+0.88
EnergyOE
37.89+1.19
29.84+-3.62
63.42+1.81
55.82+1.41
36.37+1.40
43.54+0.60
44.48+0.78
OE
23.57-00.67
15.7412.37
56.30-0.46
46.26:10.33
27.8110.60
33.1940.59
33.8110.51
PASCL
22.91+1.74
13.40
57.20-0.85
47 07.26100211
26.91
33.35:00.7
33.51
BEL
22.22+0.39
10.88-1.088 1.03
57.94+00119
43.04+00.13
23.13++039
28.10+0.42
30.88+0.21
RNA (ours)
18.02+1.38
10.9442.91
15.82-11.36
42.11=0.27
20.50+0.57
27.72=0.42
29.18=0.90
ID) Dataset:
CIFART00-LT
MSP
89.44+0.88
25.22:2..5
86.00=0.53
81.59+0.23
80.85+0.47
09.9
0..17
OECC
80.37+2.18
70.63:22.75
8..32=0.377
78.42:+0.41
0..9
73.28+0.24
0.57
OnergyOE
84.40=1.004
75.12:5.22
81.45-0.48
81.15::0.64
79.64+1.20
78.50+00.6
80.0411.13
OE
17.08:22.13
55.36433.2
79.85=0.77
76.17=0..3
63.2440.31
65.31+0.40
67.83+0.82
PASCL
6..50+11.64
53.5713.22
79.81=00335
76.37++0.38
64.09::0.88
64.83:+0.41
0..36:+0.46
BEL
6..16:10.23
34.06=0.16
85.27=0.53
7..54=0.16
51.5410.13
57.52:00.32
61.18:10.14
RNA (ours)
69.60:11.75
45.4711.97
03.4310.53
79.32:10.33
60.36:1.10
03.23:10.47
66.90:11.55
Table 11: The AUROC performance (%) of the model trained with RNA using different OOD scoring method. The OOD test sets for near OOD task are CIFAR10 and Tiny ImageNet, while those for far OOD task are Texture, SVHN, LSUN, and Places365.
Training method
Scoring method
Near OOD
Far OOD
Average
OE
MSP
65.48
77.32
73.37
RNA RNA
MSP
64.81
78.49
73.93
RN
62.63
80.19
74.33
all but the CIFAR10 test set, RNA achicves the highest AUPR values when trained CIFAR100-LT. In
Table 10 our proposed method shows the best FPR95 values across all but the CIFARI00 test sets when trained on CIFARIO-LT. On the other hand, while RNA achieves the lowest FPR95 values for SVHN, LSUN, and Places365 when trained on CIFAR100-LT, it is outperformod by PASCL in terms of the average FPR95 value across the OOD Lest sets.
Near OOD detection poses a significant challenge in the field of OOD detection. In these scenarios, the OOD test sets closcly resemble the ID datasets. RNA method exhibits suboptimal performance for near OOD detection as shown in Table 9 and 10 This is particularly noticeable when using CIFAR100-LT7 training set and CIFARIO as the OOD test set. In contrast, other methods like OE with MSP score exhibit. relatively robust performance in such settings.
the ID
We attribute this observed trend to the inherent characteristics of the CIFAR10 and CIFAR100 datasets. While these datasets consist of disjoint classos, certain classes share overlapping features. For instance, presenting data from the "dog" class of CIFAR10 OOD test set CIFAR100-trained model may activate features associated with CIFAR100 classes like "fox", "raccoon", "skunk", "otter", and "beaver". Consequently, the logits corresponding to these multiple classes may simultaneously have high values, potentially leading to low MSP value. However, the representation norm might be large due to these related features. For such an example, the near OOD sample may be detected by MSP but incorrectly categorized as an ID sample by the RN score. To support this hypothesis, Table II illustrates the OOD detection performance (AUC) for two OOD scoring methods, MSP and RN, when applied to the RNA-trained model on CIFAR100-LT. As anticipated, using MSP leads an improvement in near OOD detection performance, but at the cost of worsening far OOD detection.
24

---

Published in Transactions on Machine Learning Research (08/2024)
155
750
14
400
18.2
340
1E.0
330
(a) CIFARIO-LT
(b) CIFARIOO-LT
(c) ImageNet-L1
Figure 5: AUC and ACC (%) performance of RNA-trained model with varying the balancing hyperparameter
X. Blue (x-marked) lines indicate the AUC values and red (circle-markdd) lines indicate the ACC values.
33
43
69
76.0
37
75.5
39
75.0
73
74.5
74.0
39
77
A
(a) CIFARIO-LT
(b) CIFARI00-LT
Figure 6: The ID classification performance (ACC) and OOD detection performance (FPR95) of models trained by LA+OE. FPR95 is averaged over six OOD test sets as in Section 6. The models are trained with the training loss, LLA + XLOE, where À denotes the balancing hyperparaneter. The results illustrate the trade-offs inherent combining long-tail learning methods (LA) and OOD detection methods (OE).
E.2 Hyperparameter robustness
Our proposed training method uses the combination of the two loss terms, the classification loss and the RNA loss regularizing the representation norm input data, as shown Equation The relative importance of the two loss terms can be luned with the hyperparameter X. Although all the results in the main paper are obtained with the fixed value of 0.5, we extensively explore the performance results when models are trained with varying the hyperparameter À for CIFARIO-LT, CIFAR100-LT, and ImageNet-LT, In Figure 5 we plot the AUC and ACC performances for 10 values of À = 0.1, 0.2. 1.0.. For CIFAR10/1000 10000T and ImageNet-L.T, the AUC values (blue lines) are robust to the change in X when À is greater than 0.2, although they are much lower when is small. On the other hand, the classification accuracy (red lines) decreases as À increases when trained on CIFARIO-LT and CIFAR100-LT, and this tendency is more dramatic when trained on CIFAR100-LT, For ImageNet-LT, the accuracy does not change significantly À values change.
E.3 Trade-offs in LA+ models
We explore the influence of LA and OE in LA+OE models by adjusting the balancing parameter À in the loss function, LLA + XLOE. Figure 0 presents the results of LA+OE models trained on CIFARIO-LT and CIFAR100-LT with various values 0.01, 0.05, 0.1, 0.5, and 1.0. Across both datasets, LA+OE models exhibit superior ID classification performance but inferior OOD detection performance with small X, while showing superior OOD detection performance but inferior ID classification performance with large X. These findings highlight the trade-offs between ID classification and OOD detection performance.
25

---

Published in Transactions on Machine Learning Research (08/2024)
E.4 Size of auxiliary OOD sets
Table 12: OOD delection and ID classification performance (%) across different number of auxiliary OOD samples CIFARIO-LT.
The number of ID samples
The number of OOD samples
AUC(1)
FPR95(1)
ACC(t)
ACC-Few(1)
300
67.75
74.26
45.82
39.47
3k
91.10
37.54
80.18
74.98
12406
30k
92.90
29.46
79.01
68.32
300k
92.90
29.18
78.73
66.90
We present the performance results of RNA-trained models varying the size of auxiliary OOD training sets on CIFAR10-LT Table 12 The OOD detection and ID classification performance remain comparable when the number of OOD samples is 3k, 30k, or 300k. However, when the number of OOD samples is reduced to 300, which is approximately 2.42% of the number of ID training samples, the AUC and ACC drop rapidly by 23.35% and 34.36%, respectively. In conclusion, RNA-trained models achieve comparable performance in both OOD detection and ID classification when the number of OOD samples is at least. quarter of the ID samples. However, the OOD samples are too few, the performance of the RNA-ttriied model degrades significantly.
E.5 Training batch ratio of ID and OOD samples
Table 13: Performance metrics (%) for OOD detection and ID classification, along with the representation norm and the activation ratio at the last. ReLU layer, comparing the training batch ratio on CIFAR10-I.T.
Batch ratio (ID:OOD)
Representation norm
Activation ratio
AUC(t)
FPR95(L)
ACC(t)
FIP PBN MBN #OOOD
ID test
OOD test
ID test
OOD test.
128:32
91.87
35.37
79.00
36.55
2.56
54.55
6.10
0.13
128:64
92.62
31.50
79.36
38.28
1.89
56.31
3.46
0.23
128:128
92.90
29.18
78.73
43.79
0.67
62.64
2.08
0.37
128:256
93.04
27.89
79.02
55.08
0.70
75.34
1.43
0.80
128:512
92.95
27.64
79.19
64.12
0.45
72.26
1.12
0.90
In Table 13 we present experiments varying the OOD sample size. Our method demonstrates worse FPR95
performances with smaller OOD) sample sizes.
Our approach detects OOD data based representation norms, which proportional to the gap between latent. vectors before the last BN layer and the running mean of the last BN layer. So, we will check the gaps with varying the OOD sample size. Let PID and POOD denote the mean of the latent vectors before the last BN layer for ID and OOD samples, respectively, and JBN denote the rumning mcan of the last BN layer. Afler training with RNA, we observe that μID > MBN > FOOD, as shown in Figure 3b, which implies that many ID representations are activated, while many OOD representations are deactivated after RNA training.
For superior OOD detection based on representation norms, ID representation norms should be relatively is proportional to the OOD detection performance.
larger than OOD representation norms. Thus, jnx- Max--Poo Poop At different ratio of ID to OOD, different number of ID or OOD samples accounts for calculating μBN, since PBn is the mean of all the ID and OOD samples in the batch. Therefore, as the OOD sample size decreases, PBn gets closer to An than POOD: so the ratio _ip- decreases, as shown in the rightmost column of μBN-WOOD
Table 13 This decreased ratio -m==Pa leads smaller gap of representation norms between ID and OOD
AgN-HOOD and declined OOD detection performance.
26

---

Published in Transactions on Machine Learning Research (08/2024)
Table 14: OOD detection and ID classification performance (%) on CIFARIO-LT with ImageNet-RC
Chrabaszez et al. 2017) as the auxiliary OOD training set.
Method
AUC (1)
AUPR (t)
FPR95 (4)
ACC (1)
Many (t)
Medium (t)
Few (t)
OE
92.29
91.85
28.16
74.67
94.59
73.64
56.10
PASCL
92.05
91.70
29.72
73.58
94.43
72.76
53.88
RNA (ours)
92.53
91.78
23.61
78.67
94.13
75.44
67.67
PASCL-FT
90.66
89.44
33.88
77.78
93.53
73.88
67.12
E.6 Alternative auxiliary OOD training set
We additionally conduct experiments employing ImageNet-RC as an auxiliary OOD dataset for CIFARIO-LT ID dataset. The results are presented in Table 14 Our results demonstrate the effectiveness of the RNA, which consistently outperforms both the OE and PASCL, in terms of OOD detection and ID classification. While the AUC and AUPR metrics exhibited comparable performance across the three methods, the FPR95 metric indicated RNA's 's superiority, surpassing OF and PASCL by 4.55% and 6.11 respectively. Furthermore, RNA demonstrats 4.00% enhancement in ID classification accuracy compared to OE, and 0.89% improvement over PASCL. These results validate RNA's effectiveness not only when trained with the 300K random images but also when employed with other OOD auxiliary sets.
E.7 Structure of projection function
Table 15: OOD detection and ID classification performance (%) on CIFARIO-LT with varions structure of projection function hg. The gray regions indicate the results for our original proposed method (RNA).
Training objective
AUC(t)
AUPR(1)
FPR95(L)
ACC(t)
Many(t)
Medium(t)
Few(f)
[lf(())l
92.18
92.27
33.54
80.11
94.07
77.25
69.87
W fo(x)]
92.78
91.89
29.48
77.63
93.67
74.02
66.10
((((()))
92.90
92.14
29.18
78.73
93.72
76.40
66.90
Table 16: The average norm of representations of ID training set (CIFARIO-LT)), OOD training set (300K random images), ID test set (CIFAR10), and OOD test set (SVHN) with various strueture of projection function hg. The gray regions indicate the results for our original proposed method (RNA).
Training objective
Norm of ID train
OOD train
ID test
OOD test
[=)[))) [W, fo(#)
79.89
7.93
59.31
4.61
IIfo(=))l
66.29
2.59
46.32
1.30
[ha(J(z))]
65.14
2.53
45.90
0.93
We conduct an ablation study evaluate the impact of architectural variations, specifically focusing on the structure of the projection function has In the proposed approach, we employ 2-layer MLP configuration for train the model using RNA loss CIFAR10-LT, where in Equation 5]
h4- As alternative configurations, [fx))) is replaced either by the feature norm [ or the linearly projected norm ||W, fe(x))]. The results presented in Table 15 indicate that the structure of the projection function he is not a critical element in our method, as the OOD detection and ID classification performances remain comparable across the model variants. The reason we use the 2-layer MLP projection function in our proposed approach is to regulate the situations where the fcature norm of ID samples increases excessively during training. Nevertheloss, our findings indicate that employing the feature norm itself within the logarithmic term adequately prevents the divergence of [o (x)]], shown in Table 16 and Section 5.2
27

---

Published in Transactions on Machine Learning Research (08/2024)
Table 17: The results of fine-tuning with auxiliary OOD data on pre-trained models by CIFARIO-LT, CIFAR100-LT, and ImageNet-LT, /FF" refer to pre-training and fine-tuning resp., where the fine- tuning loss is LID ACoOD.
PT
FT (CID)
FT (Cood)
AUC (1)
AUPR (t)
FPR95 (1)
ACC (1)
CE
LA
CE
LA
ID dataset: CIFARIO-LT
90.23
90.15
39.51
76.82
S
OE
90.36
90.03
36.70
76.26
90.29
36.69
76.21
88.78
88.91
44.48
76.25
EnergyOE
85.10
83.36
49.01
70.28
87.08
86.73
48.97
71.45
90.34
90.56
30.87
74.28
RNA (ours)
90.52
90.74
39.39
78.78
90.53
90.85
36.70
78.53
ID datasct: CIPAR100-LT
71.26
66.82
72.58
40.27
OE
70.91
66.52
72.57
41.09
71.82
67.27
71.08
41.06
69.18
66.24
77.23
39.46
EnergyOE
68. 73
65.93
78.14
38.32
69.13
66.35
77.61
38.24
73.02
68.93
70.65
40.69
RNA (ours)
72.58
68.72
71.99
44.82
72.56
68.85
71.67
44.65
dataset: TmageNett
63.65
65.34
89.63
32.28
OE
64.68
65.66
88.05
37.23
63.80
65.45
89.17
35.78
63.87
65.13
88.29
38.43
EnergyOF
47.63
46.77
92.82
31.79
36.47
40.31
95.59
19.70
68.95
68.94
86.05
34.66
RNA (ours)
69.27
69.22
85.37
38.91
72.20
71.45
83.25
37.66
E.8 Fine-tuning task with auxiliary OOD data
In the fine-tuning task for OOD detection (Hendrycks et al. 2019a Liu et al. 2020 Papadopoulos al. 2021], classification model is initially trained without the OOD set and then later fine-tuned with the OOD set to improve its OOD detection ability.
We evaluate the effectiveness of our training method when applied to the fine-tuning task. The pre-trained models are obtained by training a model on CTFAR10/100-LL and ImageNet-L.,, respectively, using only the classification loss, either CE or LA. We then train this model for 10 (respectively 3) more epochs CIFAR10/100-DD and 300K random images (respectively ImageNet-LT and ImageNet- with objective of LID + XCOOD. Here,
LCE LLA as LID, and COE, LEnergyOE, LRNA LOOD. Table 17 summarizes the results. "PT" and "FT" stand for pre-training and finc-tuning methods, respectivcly. We can oberve that RNA allains the most optimal performances in both OOD detection and ID classification. Notably, when the classification loss during pre-training or fine-tuning is LA, the classification accuracy of models fine-tuned using OE or EnergyOE is significantly lower than that of RNA. This discrepancy could be attributed to the inherent trade-offs between OOD detection and ID classification, arising from the application of OF discussed in Section 5.1
28

---

Published in Transactions on Machine Learning Research (08/2024)
Training loss dynamics
Training loss dynamics
4.0
355
7-
-7
3.0
1.0
2.5
-8-
2000
00
2.0
0.6
9-
1.5
0.5
-31
11
25
50
75
100 epachs
125
150
175
200
25
50
75
100
125
150
175
200
(a) CIFAR10-TT
(b) CIFAR100-IT
Representation ner dynamics
Representation norm dynamics
60
Train(Hedd
TraincTall
40
TestiHead!
Test(Tail)
30
000 (30K)
020 TesH)SAHHN)
TrainHHeaci
20
ID Train(ball
Test|Head!
00D Tii (30kk
0
DO0 TestISHHN)
50
100
175
25
75
125
200
spochs
(c) CIFAR1O-LT
(d) CIFAR100-LT
Figure 7: The training dynamics of the models trained with RNA on CIFARIO-IT and CIFAR100-LTT The training loss dynamics of LA loss (red) and RNA loss (blue) are in (a) and (b), and the representation norm dynamics of head and tail data in ID training and test dataset, OOD training set (300k random images), and OOD test (SVHS) are presented in (c) and (d).
E.9 Training dynamics of RNA
Figure and 7b depict the changes of training losses (LA loss LLA and RNA loss LRNA), respectively, over the training. Figure 7c and 7d illustrate the dynamics of representation norms of training ID, test ID, training OOD, and test OOD data, over the training. These figures demonstrate that the training effectively works to widen the gap between ID and OOD representation norms.
E.10 Distributions of representation norms
We present additional figures similar to Figure with different ID OOD datasets in Figure We observe large gap between ID and OOD representation norms when CIFAR10 is used as the ID set, and a smaller but still distinct gap when CIFAR100 is the ID set, and a very small gap when CIFAR100 the ID set and CIFARIO is the OOD set, which correspond to the OOD detection performance results for each OOD test set in Table Table and Table 10
29

---

Published in Transactions on Machine Learning Research (08/2024)
0.7
Test
ID Test
Test
OOD Test
Density
Density
0.4
0.2
0.2
0.1
0.0
30
80
30
70
Representation
Representation norm
(a) CIFARIO Texture
(b) CIFARIO SVHN
0.35
0.30
OOD Test
OOD Test
Density
0.25
Density
0.25
0.20
0.15
0.10
0.10
0.05
0.00
20
20
40
50
70
80
0.00
10
20
30
40
50
70
80
Representation norm
Representation norm
(c) CIFAR10
CIAFAR100
(d) CIFAR10 vs. Tiny ImageNet
Test
OOD Test
0.150
OOD Test
0.4-
Density
Density
0.100
0.050
0.1
0.025
0.0
10
20
30
40
5D
60
7a nD
0.000
20
40
60
nD
Representation norm
Representation norm
(e) CIFAR10 vs. Places365
(f) CIFAR100 vs. Texture
0.200
ID Test
0.035
ID Test
0.175
OOD Test
0.030
0OD Test
0.150
Density
Density
0.020
0.010
0.005
20
60 'Representation narm
40
80
20
40
60
80
Representation nor
(g) CIFAR100 vs. SVHN
(h) CIFAR100 vs. CIFAR10
ID Test
ID Test
OOD Test
OOD Test
Density
Density
20 40
60
Representation norm
Representation nom
(i) CIFAR100 vs. Tiny ImageNet
(j) CIFAR100 vs. Places365
Figure 8: The histogram of representation norms of ID/OOD test data on RNA-trained models on CIFAR10 LT and CIFAR100-LT, The red bars represent the density of representation norms of ID data and the blue bars represent that of OOD data (Texture, SVHN, CIFAR, Tiny ImageNet, and Places365).
30


Reviews:
[{"review_id": "T92UNw8w5H", "reviewer": "Reviewer_LZ9f", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. This paper is well-organized and easy to follow.\n2. The authors are inspired by the trade-off between OOD detection and long-tailed recognition which is well-motivated and indeed novel.\n3. The experimental results demonstrate the superior performance of RNA compared to other baseline approaches across various OOD test sets.\n\nCons:\n1. The claim that ''The OOD data does not contribute to model parameter updates'' is wrong.\n2. The authors mentioned calibration in Sec. 5.3.5. However, ECE is not a ''perfect'' metric for the task. It is non-differentiable. Other metrics should be discussed here.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper aims to solve the OOD detection in long-tail learning and focuses on the balance between OOD detection and long-tail recognition. The authors propose Representation Norm Amplification (RNA) to solve this challenge. The main contributions are as follows:\n1. Utilizing the norm of the representation as a new dimension for OOD detection, thus generating a noticeable discrepancy in the representation norm between ID and OOD data during the training process without perturbing the feature learning for ID classification.\n2. The authors conduct a series of experiments and validate the effectiveness of the proposed method.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The authors need a more detailed explanation of the claim that ''The OOD data does not contribute to model parameter updates''.\n2. The proposed idea is general for OOD detection task. It is not specifically designed for long-tail recognition. The authors should provide extra explanation and clarify this.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mY5ueNmT74", "reviewer": "Reviewer_gNpe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n1. The logic of the whole article is clear and well-written.\n2. The observation of the trade-offs between OOD detection and long-tailed recognition and motivation sound reasonable. The proposed solution in this paper is just right and fits the motivation.\n3. The experiments presented in the manuscript are valuable especially for the ablation study.\n\n**Weaknesses or Questions:**\n\n1. Although the authors explain in the methodology section that the truncation effect of the ReLU is helpful in detecting the OOD samples, I wonder why the results of reducing the representation norm of the OOD samples would be lower than only performing above ID samples.\n2. Since the success in detecting OOD samples in the manuscript contributes to the ReLU truncation effect and RNA loss, I don't know if the performance of the method is sensitive to the ratio of ID samples to OOD samples within each batch.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is proposed to address the problem, the conflict between OOD detection and long-tailed classification. The authors analysed in detail the contribution of the Logit Adjustment loss, which prefers the long-tailed classification, and the Outlier Exposure loss, which prefers the OOD detection, to the classifier weights, and observed the bad impact of the OE loss on the weights of the tailed classes, which would affect the performance of the ID samples. Therefore, they propose a novel method called Representation Norm Amplification, which disentangles ID classification and OOD detection in long-tail learning by detecting the norm of the samples' feature to find the OOD samples. The experiments conducting on CIFAR and ImageNet datasets demonstrate the effectiveness of the proposed method especially for the tailed classes.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Further discussion and analysis is required as to why it is optimal to perform RNA loss only on ID samples, with no restrictions on OOD samples.\n2. Sensitive analysis of the ratio of ID samples and OOD samples within each batch is required.", "broader_impact_concerns": "The work is proposed to tackle the challenging scenarios where reliable OOD detection needs to be performed on models trained with long-tailed distributions. This is a crucial and necessary safety issue that neural networks need to address before being deployed in the real world.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WGBHnrehLn", "reviewer": "Reviewer_aJme", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The paper is easy to follow. It critically evaluates existing methods, suggests a new method, and provides a comprehensive evaluation. The suggestion is straightforward but nontrivial and is was actually very surprising to me that it works at all.\n\nWeaknesses: Introduction/background could use reorganizing to improve clarity for non-experts.\n\nOverall, it seems a very good fit for TMLR and can be accepted after minor revisions. However, note that I am not an expert and so my confidence is low.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In their paper \"Representation norm amplification for out-of-distribution detection in long-tail learning\", the authors suggest a new method for training a classifier that simultaneously (i) performs well on rare classes; and (ii) allows out-of-distribution detection. The authors argue that existing methods show a trade-off between these two goals, whereas their new method achieves good performance on both metrics.\n\nCaveat: I am not an expert in this field, am not familiar with the relevant literature, and am not sure why I was selected as a reviewer for this paper. So my confidence is low, but I nevertheless tried to provide a helpful review.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "MAJOR COMMENTS\n\n* I started reading the paper without knowing what \"long-tail learning\" means, what is \"outlier exposure\" or \"logit adjustement\". It took me some time to figure out what is going on. I'd recommend to more explicitly define \"long-tail learning\" and \"long-tailed datasets\" in the beginning. More importantly, it would be good to have a Background section in the beginning that would introduce OE and LA with all the relevant formulas. Currently OE is introduced in Section 4.1, after Table 1 is presented and discussed, and the equation for LA only appears several pages later in Section 4.2. The text would be clearer if all of that is introduced before Section 3.\n\n* You say in Section 4.2 (page 6) that scaling the norm of ||f(x)|| does not affect class prediction. That's correct. But it does affect the logits and hence the softmax probabilities (which you don't say very clearly). You want OOD samples to have small norm, but small norm implies more uniform probabilities -- which is precisely the goal of OE (uniform probabilities for OOD samples), which you have criticized on top of the same page 6. I think you could make it more explicit here, why what you are doing is different.\n\n* In Eq (3) you have ||h(f(x))|| instead of ||f(x)||, where h() is a 2-layer MLP. It was completely unclear to me why you used h() there at all -- this should be better motivated, and I would like to see an experiment where you remove h(). Does the method perform worse then? Why? In the text you always talk about the norm of f(x), not of some MLP of f(x).\n\n* Fig 4b -- does this show the norms of f(x) or of h(f(x))? See previous question.\n\n* It took me some time to understand the argument on top of page 8 about how OOD samples have any effect during training if they are ONLY used to update the BN layers. Here is my understanding: you are saying that BN+ReLU will lead to half of the data producing a representation with near-zero norm. But your loss encourages high norm (on ID data). This means that OOD data will _have_ to have near-zero norm. Is that the argument? Given that it's very non-intuitive and suprising, I suggest you make this argument more clear / more explicit. Also: Fig 4b is really striking; do you observe the same with all different ID and OOD datasets? Could you show the distributions? Also: does your method require that OOD sample size is as large as ID sample size? Maybe comment on that.\n\n\nMINOR COMMENTS\n\n* Eq 7 -- please explain the rationale behind LA in more detail, for those who are unfamiliar with it.\n\n* Section 5.3 -- it's my terminology pet peeve, but \"ablation\" literally means \"turning something off\". If you are trying out different loss functions, it's not an \"ablation study\", it's \"sensitivity analysis\" or \"hyperparameter choice\" or something. Ablation study would be e.g. to remove the MLP h() from the loss.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: There was a consensus among the reviewers that the paper not only supports its claims with clear and convincing evidence but is of interest to the TMLR community.

The paper also benefited from the review process with the addition of some further background sections at the start of the paper as requested by reviewer [aJme].

For my part, I'd encourage the authors to lookup some of the literature on 'evidential networks' such as: https://arxiv.org/abs/2311.11367, https://arxiv.org/abs/1806.01768, and https://arxiv.org/abs/1802.10501, which I think is relevant to the approach here.  Those works try to reinterpret the output of a classifier network as the parameters of a Dirichlet distribution rather than a Categorical.  There the norm of the output has a direct operational meaning tied with the certainty of the prediction.

Given the consensus among reviewers and the clear audience appeal, I'll accept as is.
Decision: Accept (poster)