SELFOOD: Self-Supervised Out-Of-Distribution Detection via Learning to Rank
Authors: Dheeraj Mekala
Adithya Samavedhi
Chengyu Dong
Jingbo Shang
Venue: EMNLP 2023 Findings (2023)
Keywords: OOD detection
self-supervised
ranking
TL;DR: We present a self-supervised OOD detection method that requires only in-distribution samples and no OOD samples as supervision.

Abstract: Deep neural classifiers trained with cross-entropy loss (CE loss) often suffer from poor calibration, necessitating the task of out-of-distribution (OOD) detection. Traditional supervised OOD detection methods require expensive manual annotation of in-distribution and OOD samples. To address the annotation bottleneck, we introduce SELFOOD, a self-supervised OOD detection method that requires only in-distribution samples as supervision. We cast OOD detection as an inter-document intra-label (IDIL) ranking problem and train the classifier with our pairwise ranking loss, referred to as IDIL loss. Specifically, given a set of in-distribution documents and their labels, for each label, we train the classifier to rank the softmax scores of documents belonging to that label to be higher than the scores of documents that belong to other labels. Unlike CE loss, our IDIL loss function reaches zero when the desired confidence ranking is achieved and gradients are backpropagated to decrease probabilities associated with incorrect labels rather than continuously increasing the probability of the correct label. Extensive experiments with several classifiers on multiple classification datasets demonstrate the effectiveness of our method in both coarse- and fine-grained settings.

SELFOOD: Self-Supervised Out-Of-Distribution Detection via Learning to Rank
Dheeraj Mekala
Adithya Samavedhi Chengyu Dong"
Jingbo Shang 09.
University of California San Diego
Halcooglu Data Science Institute, University of California San Diego
{dmekala, asamavedhi, cdong, [ihhng]&ucse edu
Abstract
Sports
Art
aa wakix
Spart
PiSports [D)] Pisports D.)
Deep neural classifiers trained with cross- entropy (CE) loss often suffer from poor calibration, necessitating the task of out-of- distribution (OOD) detection. Traditional su- pervised OOD detection methods require ex- pensive manual annotation of in-distribution and OOD samples. To address the annotation bottleneck, we introduce SELFOOD, self- supervised OOD detection method that requires only in-distribution samples as supervision. cast OOD detection inter-document intra- label (IDIL) ranking problem and train the clas- sifier with our pairwise ranking loss, referred as IDIL loss. Specifically, given set of in-distribution documents and their labels, for each label, train the classifier to rank the softmax scores of documents belonging to that label to be higher than the scores documents that belong other labels. Unlike CE loss, our IDLL loss reaches zero when the desired confidence ranking is achieved and gradients backpropagated to decrease probabilities associated with incorrect labels rather than con- tinuously increasing the probability the cor- rect label. Extensive experiments with several classifiers on multiple classification datasets demonstrate the effectiveness both coarse-and fine-gra ine-grained settings.
Due
P|Arts| Bgl 911 P[Ants 0y)
Softmax Scores
(a) CE Loss
D Sports
0.2 Arts
5.3 Follttes
P|sports Dj) PiSports DJ
P(Sports (y) v
Dye Arts
0.7
0.15
P(Arts D,) P(Arts D2)
P(Arts 02) &
Softmax Scores
(b) IDIL Loss
Figure 1: CE Loss and IDII. loss optimization for docu- ments D1 Sports and D2 Arts. CE loss increases scores corresponding to the Sports for D, and Arts for D2, implying an intra-document comparison. Instead, IDIL loss compares softmax scores in an inter-document intra-label fashion where it reduces scores correspond- ing to Sports for D2 to be less than that 1),
deep learning models in safety-critical applica- tions (Moon et al., 2020).
Traditional supervised OOD detection methods (Hendrycks al., 2018; Larson et al., 2019; Ka- math et al., 2020; Zeng et al., 2021b) assume access to high-quality manually annotated in-distribution and OOD samples. However, this requires exten- sive annotation OOD samples belonging to di- verse distributions, which is expensive to obtain. Moreover, text classifiers are ideally desired to be more confident on in-distribution samples than OOD samples. However, the poor calibration of DNN precludes this phenomenon.
method in
1 Introduction
Deep neural networks (DNNs) are ubiquitously used for text classification (Liu et al., 2019; Devlin et al., 2019; Yang et al., 2019; Brown et al., 2020). However, they are generally poorly calibrated, re- sulting in erroneously high-confidence scores for both in-distribution and out-of-distribution (OOD) samples (Szegedy al., 2013; Nguyen al., 2015; Guo al., 2017: Mekala al., 2022a). Such poor calibration makes DNNs unreliable, and OOD detection task vital for the safe deployment of
To address these problems, we propose SELFOOD, self-supervised OOD detection framework that requires only in-distribution sam- ples supervision. To adhere to the aforemen- tioned ranking constraint, we formulate OOD de- tection as an inter-document intra-label (IDIL. ranking problem and train the classifier using our pairwise ranking loss, referred to as IDIL loss. As
Equal Contribution
* Jingho Shang is the corresponding author.

---

Text
Label
62
63
0.5 Sports
Sports
puaer was replaced
Spors
Stery Night post- Impressionist masterplece.
Art
Pullis
Sports
It's fiaily election
Pollics
Arts
Ans
0.8 Poltiios
Sports
The lana cheared as The gool was scortd.
Sports
The play reperved divense meveon from the criicss
Arta
IDIL Loss
blockads on the reads
Profesters created
Polidio
Politics
OOD Classifier
Labeled In-Distribution Data
Label Buckets
Figure 2: SELFOOD is seff-super framework that requires only annotated in-distribution data train the OOD classifier. Firstly, we bucket documents based on their annotated label (in dotted lines). Then, we compare each document bucket with all documents in other buckets to compute IDIL loss (in solid lines). Finally, we backpropagate gradients to decrease scores associated with incorrect labels during the training of the OOD classifier.
shown in Figure (a), classifiers are gener- ally trained using cross-entropy (CE) loss (Good, 1952) an intra-document fashion where for each document, the classifier is trained to distinguish between different labels by maximizing the score corresponding to the correct label. Instead, in method, shown in Figure 1(b), we propose train in inter-document, intra-label fashion where for each label, we train the model to rank the considered label probability score in documents belonging to the label higher compared to those not belonging to it. As OOD documents generally do not belong to any in-distribution label, we hypothe- size such explicit learning to rank translates to ac- curately distinguishing OOD from in-distribution.
labels as input, we bucket documents belonging to each label and train the classifier to rank the probabilities using our IDIL loss function. Specifi- cally, for each document in a label bucket, we pair up with all documents in every other bucket and compute IDIL loss.
Our contributions are summarized follows: We propose SELFOOD, a novel self-supervised method to train an OOD detection model without any OOD samples as supervision.
We formulate OOD detection as an inter- document intra-label ranking problem and op- timize it using IDIL ranking loss.
We perform extensive experiments on multiple text classification datasets to demonstrate the ef- fectiveness of our method OOD detection. We release the code on Github'.
Moreover, minimizing CE loss involves continu-
ous optimization increase the probability of the
correct label over the other labels, making the clas- sifier overconfident (Wei et al., 2022). Instead, in our method, our IDIL loss function becomes zero once the desired ranking is achieved, and during training, we backpropagate gradients decrease probabilities associated with incorrect labels rather than increasing the probability of the correct label. Theoretically, the perfect model trained using the CE loss with 0 training loss is solution our ranking problem, however, in Section 5.1, we em- pirically show that our ranking objective leads to a different solution, demonstrating the importance of the optimization procedure. Finally, it is important to note that our ranking formulation, loss function, and the self-supervised training strategy specifi- cally designed improve the performance of OOD detection rather than classification accuracy.
Related Work
Traditional supervised methods cast the OOD de- tection classification with binary labels (Kamath et al., 2020), with one additional label for unseen classes (Fei and Liu, 2016). Since manually anno- tating documents is expensive (Mekala and Shang, 2020; Mekala et al., 2022b), recent works have de- viated from requiring such extensive annotations and leveraging various distance metrics to detect OOD samples. Leys al. (2018); Xu et al. (2020) use Mahalanobis distance post-processing tech- nique identify OOD samples. These methods use distance-based scoring functions along with the intermediate model layer features determine an OOD score. Lee et al. (2018); Hsu et al. (2020) use similar distance metric called ODIN to detect
We present our framework in Figure 2. Given a set of in di-dstiibbution documents and corresponding
httpp:: http:://ggttuuu..cce/eeeerea /ott://///ceeeer3/////SLLL FELFOOD

---

OOD images. An alternate approach to compen- sate for the lack of OOD training data involves generating pseudo-OOD data for training. Ouyang et al. (2021) propose a framework to generate high- quality OOD utterances and importance weights by selectively replacing phrases in in-domain sam- ples. Zhan et al. (2021) generate pseudo-OOD samples for the task of intent detection using self-supervision. Zhou et al. (2021); Zeng al. (2021a) introduce self-supervised approaches to OOD detection using a contrastive learning frame- work. They suggest fine-tuning transformers using a masein-based contrastive loss to learn text repre- sentations (Mekala et al., 2021) for OOD classifi- cation. Vyas et al. (2018); Li et al. (2021) treat a part of in-domain data as OOD samples as an al- ternate self-supervised approach. Further, Wu et al. (2022a) use a Reassigned Contrastive Loss (RCL) threshold
OOD documents required for training.
3.2 Motivation
Numerous neural text classifiers have been pro- posed, incorporating multiple hidden layers (Rosen- blatt, 1957), convolutional layers (Kim, 2014), and various types of attention mechanisms (De- vlin et al., 2019; Liu et al., 2019; Radford et al., 2019). All these models culminate in a softmax head, which produces probabilities corresponding to each class. These classifiers generally trained with CE loss in an intra-document fashion i.e. each document is considered independently and the soft- max score of the true label is maximized. Such training of neural text classifiers is known to in- crease the magnitude of logit vectors even when most training examples are correctly classified (Wei et al., 2022), making them poorly calibrated, pro- ducing unreasonably high probabilities even for in- correct predictions (Szegedy et al., 2013; Nguyen et al., 2015; Guo et al., 2017). This diminishes their ability to maintain the desired attribute of ordinal ranking for predictions based on confidence levels. wherein a prediction exhibiting a higher confidence value should be considered more likely to be accu- rate than one with lower confidence value (Moon et al., 2020). Intuitively, text classifier possessing such quality would be perfect OOD detector.
along with an adaptive class-dependee mechanism to separate in-domain and OOD intents. Ren et al. (2019); Gangal et al. (2020) leverage likelihood ratios crafted by generative models to classify OOD samples. Wei et al. (2022) observe that the norm of the logits keeps increasing dur- ing training. leading to overconfident outputs, and propose LogitNorm as a solution to decoupling the output norm during training optimization. Moon al. (2020) introduce a novel Correctness Ranking Loss function in order to regularize output probabil- ities to produce well-ranked confidence estimates. Other calibration techniques include "Top-label" calibration which is used to regularize the reported probability for the predicted class (Gupta and Ram- das, 2021).
3.3 OOD Detection as Inter-Document Intra-Label Ranking
In order to align with the aforementioned character- istic, we propose formulating the OOD detection as an inter-document, intra-label ranking problem. Specifically, given a set of in-distribution docu- ments, we compare across documents but within the same label and train our model to generate higher probability score for documents belonging to the label than for documents not belonging the label. We consider the same model architec- ture as any text classifier with a softmax head that generates scores corresponding to each label, how- ever, we train it using our IDIL. loss instead of CE loss. Our assumption is that an OOD docu- ment does not fall under any specific label in the in-distribution space. Hence, we anticipate that the trained model would produce lower scores for OOD documents compared in-distribution doc- uments. This distinction in scores is expected facilitate easy separation between OOD and in- distribution documents.
3 SELFOOD: Self-Supervised OOD Detection
We present the problem statement, the motivation for our ranking formulation, and our method includ- ing the loss function and its optimization strategy.
3.1 Problem Statement
In this paper, we work on the OOD detection task with only in nn-distribution samples and no OOD sam- ples as supervision. Specifically, given a labeled dataset InD {211 B1), (I2, V2),... ..))...(.,,, (I vnn) sampled from a distribution space (X,C) where documents x; X' and labels y: E C as input, our goal to train an OOD detector M that accurately distinguishes in nn-distribution documents InD and OOD documents DOD $ (X,C) without any
IDIL Loss is pairwise-rankinn loss that enforces

---

desired ordinal ranking of confidence estimates. This loss function reaches its minimum value for particular label when the probability of that label being the annotated label is greater than its prob- ability when it is not the annotated label. Specifi- cally, for documents 21, E D'DDD and their cor- responding annotated labels y1, M2 where /1 # y2- IDIL loss corresponding to label y1 is mathemati- cally computed follows:
Table Dataset statistics.
Dataset
Domain
Criteria
# Does
# labels
NYT
News
Topic
13081
Yelp
Reviews
Sentiment
70000
5
Clima
Climate
Question Type
17175
TREC
General
Question Type
5952
of the difference and exclusively propagate the gradients through the minuend. In Equation 1, for instance, we detach the gradients for p(vn (2;) and solely backpropagate the gradients through ply [x2). This detachment allows for more con- trolled and selective gradient flow, aiding in the optimization process, and improvement in perfor- mance as shown in Scction 4.5.
(VI SiLU - p(v1l=))) (1)
where (i)U(x)= (x) is the Sigmoid Linear Unit (SiLU) function (Elfwing et al., 2018). To ensure stable training and enhance performance, we incorporate the SiLU function, a continuous variant of the Rectified Linear Unit (ReLU) (Hahn- loser al., 2000), in conjunction with the ranking loss. The SiLU function introduces smooth gradi- ents around zero, effectively mitigating potential instability issues during training. We observe that this inclusion contributes to the overall stability in training and improved performance of the model as shown in Section 4.5. Note that, in contrast to CE loss, IDIL loss becomes zero once desired ranking is achieved, addressing the overconfidence issue.
It is important to note that our optimization fo- cuses solely on the inter-document ranking loss. Consequently, while the trained model would serve as reliable OOD detector, it may not perform effectively classifier.
Experiments
We evaluate our OOD detection method against state-of-the-art baselines with two classifiers on multiple datasets belonging to different domains.
3.4 Implementation
4.1 Datasets
Ideally, the loss has to be computed over all pos- sible pairs of documents for each model update. However, it is computationally expensive. There- fore, following (Toneva et al.. 2019; Moon et al., 2020), we approximate the computation by consid- ering only documents in each mini-batch. Specifi- cally, we bucket the documents in the mini-batch based on their annotated label and pair each docu- ment in bucket with all documents in other buck- ets and compute the loss. Mathematically, the loss for mini-batch b is computed as follows:
We evaluate our method and baselines on four pub- licly available English text classification datasets belonging to different domains. In particular, we consider the news topic classification dataset New York Times (NYT) restaurant review senti- ment classification dataset Yelp and question-type classification datasets related to climate: Clima- Insurance+ (Clima) (Laud et al., 2023), and a gen- eral domain: TREC (Li and Roth, 2002; Hovy et al.. 2001). The documents within the New York Times dataset are labeled with both coarse and fine- grained labels. For our training and testing process, we utilize fine- fineggrained labels. The dataset statistics are provided in Table 1.
C c= Σ (ID12(1(11.2))
21 Eby #2Eb...
where by denotes the set of training data points x in this batch whose label are I, and b_t denotes the set of training data points æ in this batch b whose label is not 1.
4.2 Compared Methods
Cross Entropy Loss (CE Loss) trains a classi- fier using CE loss on in-distribution documents. The predicted probabilities from the classifier are used confidence estimates for OOD detection. Correctness Ranking Loss (CRL) (Moon et al.. 2020) is a regularization term added to the CE-
In contrast to CE loss, where the optimization
involves increasing the score corresponding to
the correct label, backpropagate gradients to decrease scores associated with incorrect labels.
Specifically, during the backpropagation of gradi- ents, we detach the gradients for the subtrahend
httt:://develaper. www.pppe...ttttt.toooommmm
hhttps://www.yet http://www..ell...ooomm////a.e cow/dataset.

---

Table OOD detection results with BERT RoBERTa classifiers. Each experiment repeated with three random seeds and the scores reported. The dalse-positive-rate 95% true-positive-ratt (FPR95), minimum detection error over all thresholds (ERR), the area under the risk-coverage curve (AURC), and the area under the precision- recolll curve (AUPR) using in-distribution samples as the positives are used as evaluation metrics.
BERT
RoBERTa
In-dist
0OD
Method
FPR95(1)
ERR(1)
AUROC(T)
AUPR(T)
FPR95(L)
ERR())
AUROC(T)
AUPR(T)
LogitNorm
82.0
33.2
69.4
57.9
84.4
33.9
66.6
55.0
OReSsRest
85.1
35.8
62.4
49.3
88.5
29.6
69.8
56.5
RCL
56.9
31.2
73.2
55.2
87.7
34.5
56.7
45.0
NYT
Bayes Approx
64.3
35.9
66.5
50.3
79.6
36.4
54.1
40.8
CB Loss
82.1
34,9
63.4
50.7
79.8
32.9
66.8
58.0
CRL
99.9
36,9
43.9
28.1
64.8
60.5
SELFOOD
63.2
19.6
79.4
82.7
69.5
21.6
72.6
78.8
LogitNorm
79.2
25.5
72.5
54.8
82.6
25.4
70.7
54.0
OneVsRest
83.0
25.1
69.5
53.2
86.7
28.0
$3.8
33.7
Yelp
RCL
59.9
27.8
67.4
42.1
22.5
70.7
56.3
Clima
Bayes Approx
74.7
28.7
56.7
33.6
63.0
28.0
66.8
40,9
CE Loss
83.2
27.8
48.6
37.0
76.4
22.0
78.2
68.4
CRL
99.6
29.0
49.1
38.8
99.9
20.6
63.0
56.9
SELFOOD
17.6
4.8
97.9
96.4
65.6
16.6
73.2
77.2
LogitNorm
68.2
27.0
79.8
82.1
23.0
83,9
84.4
OneVsRest
84,8
40.0
63.9
69.7
75.5
33.0
57.9
61.2
RCL
89.8
43.2
40.1
51.5
77.4
32.4
65.0
68.6
TREC
Bayes Approx
92.5
44.7
44.7
25.9
72.7
69.4
CE
74.2
33.7
52.8
62.7
58.5
27.5
75.3
78.3
CRL
33.3
15.5
66.7
77.2
100.0
37.3
56.7
73.9
SELFOOD
0.0
LogitNorm
32.3
0.3
95.5
78.1
0.5
95.7
44.7
OneVsRest
4.2
0.3
97.9
83.4
39.3
0,4
63.0
RCL
78.5
0.6
80.2
20.2
67.7
0.6
83.0
24.2
Yelp
Bayes Approx
46.6
93.0
61.6
63.7
0.7
87.7
21.3
CE Loss
27.2
71.2
0.4
62.3
CRL
96.6
0.7
70.9
16.5
99.8
0.7
24.3
9.7
SELFOOD
0.0
0.0
100.0
100.0
0.0
0.0
100.0
100.0
LogitNorm
9.3
1.0
97.7
91.2
10.9
2.3
97.2
77.3
VseSsRest
1.8
1.8
93.2
74,8
RCL
63.6
3.3
84.1
45.7
74.2
3.0
80.4
45.1
TREC
NYT
Bayes Approx
39.1
2.5
93.5
68.1
64.9
3.0
88.4
$4.4
87.1
CRI.
97.2
3.3
73.0
35.2
99.9
3.8
22.9
12.2
SELFOOD
0.0
0.0
100.0
100.0
0.0
0.0
100.0
100.0
LogitNorm
11.3
0.8
97.6
88,5
10.4
1.6
97.3
75.9
OneVsRest
3.4
98.7
93.2
36.3
1.8
91.7
60.1
RCL
73.5
60.7
2.0
86.4
48.2
Clima
Bayes Approx
51.7
2.0
90.5
54.4
61.1
2,4
87.6
40.4
CBLoss Loss
15.1
1.7
96.2
69.5
12.2
1.8
96.4
67.8
CRL
86.3
2.2
80.7
39.5
98.9
2.7
30.1
13.0
SELFOOD
0.0
0.0
100.0
100.0
0.0
0.0
100.0
100.0
loss to make class probabilities better confidence estimates. It estimates the true class probabil- ity to be proportional to the number of times
One trains a one vs rest binary classifier per label. The probability scores of each label from its corresponding classifier are normalized and used as their respective confidence estimates.
sample is classified correctly during training. LogitNorm (Wei et al., 2022) is variant of CE loss that normalizes the logit vector to have constant norm during training.
4.3 Experimental Settings
We experiment with BERT (Devlin al., 2019) and RoBERTa (Liu et al., 2019) as text classifiers. For SELFOOD, we train the classifier for 5 epochs with batch size of 16 using an AdamW optimizer. We use learning rate of 5e-5 using linear scheduler with no warm-up steps. For all baselines, we train the classifier for the same number of steps.
Bayesian Approximation (Bayes Approx) (Wu et al., 2022b) calibrates distribution uncertainty for OOD detection using Monte-Carla dropout. Reassigned Contrastive Learning (RCL) (Wu et al., 2022a) discriminates over-confident OOD samples using adaptive class-dependent local threshold mechanism.
our evaluation, for each dataset as in-

---

distribution, treat all other datasets OOD and compute the performance. Our evaluation follows a standard approach for each in-distribution dataset where we begin by splitting the in-distribution dataset into three subsets: 80% for train, 10% for val, and 10% for test. The model is trained using the train split, and its performance is evaluated on both the test split of the in-distribution dataset and the entire OOD dataset.
Table Ablation Study.
Method
FPR95
ERR
AUROC
AUPR
Clima OOD: NYT
SELFOOD
17.2
2,6
97.8
94,5
Grad Sub
67.9
82.4
Grad Sub & Min
98.4
10.7
49.2
32.3
Intra-Doc
92.7
SiLu
14.6
3.9
96.7
89.3
Dist: NYT
0OD: Clima
SELFOOD
10.5
1.7
98.4
901.5
+ Grad Sub only
83.5
77.4
44.8
Evaluation Metries, We utilize evaluation metries from (Hendrycks and Gimpel, 2017; De Verrses and Taylor, 2018; Moon et al., 2020) such as the false positive rate at 95% true positive rate (FPR95), min- imum detection error over all thresholds (ERR), the area under the risk-coverage curve (AURC), and the area under the precision-recall curve (AUPR) using an-distribution samples the positives.
Grad Sub & Min
96.5
5.2
60.3
36.0
Intra-Doc
48.3
93.2
80.2
SiLU
19.0
2.3
97.4
86.0
ents through the subtrahend instead of the minuend, (2) SELFOOD + Grad Sub & Min represents method with gradients backpropagating through both minuend and subtrahend, (3) SELFOOD + Inira-Doc considers intra-document comparison similar to CE loss, in addition to inter-document intra-label comparison for loss computation, and (4) SELFOOD- SiLU excludes SiLU function from the IDIL loss formulation. We also present the per- formance of SELFOOD for reference. SELFOOD performs better than SELFOOD Grad Sub & Min demonstrating that backpropagating through one part of the difference is more beneficial than both, and the comparison between SELFOOD and SELFOOD + Grad Sub indicates that backpropa- gating through minuend is better than subtrahend. We observe that incorporating intra-document com- parison into loss formulation leads to decrement in the ranking ability of the model. Finally, we observe that removing the SiLU function from the IDIL loss leads a decrease in most of the metrics.
4.4 Results
We summarize the evaluation results with BERT and RoBERTa as classifiers on Yelp. TREC as in-distribution in Table and NYT, Clima as in- distribution in Appendix A.1. All experiments are run on three random seeds and the mean per- formance scores are reported. As shown in Ta- ble 2, we observe that SELFOOD performs better than the baselines most of the in-distribution, OOD dataset pairs for both classifiers. A low FPR95 value indicates that the top-95% confi- dent in-distribution samples, selected based on their probability scores, predominantly rank higher than a majority of OOD samples and SELFOOD achieves improvements of up to 82 points in FPR95 with Yelp as in-distribution and Clima as OOD datasets when compared to CRL with BERT clas- sifier. SELFOOD also exhibits substantial im- provements of up to 33 points in Detection Er- ror, 48 points in AUROC, and 58 points in AUPR when compared CE-Loss with BERT classifier. SELFOOD achieves a perfect OOD detection score for some settings such as TREC as in-distribution dataset for both BERT and RoBERTa classifiers. These results highlight the effectiveness of our ranking formulation with self-supervised training using IDIL loss.
Analysis & Case Studies
In this section, we present comprehensive analy- sis of our proposed method from different perspec- tives to understand its effectiveness.
5.1 CE Loss vs SELFOOD: Training Trajectory Analysis
Mathematically, the model trained using CE loss is solution to our IDIL loss optimization. However, our empirical results reveal a significant disparity in OOD detection performance between SELFOOD and model trained CE loss. To gain a deeper un- derstanding, we plot the CE Loss and IDIL losses for models optimized with IDIL and CE loss, as shown in Figure 3(a). We consider Clima as the in-distribution dataset and BERT as the classifier.
4.5 Ablation Study
To understand the impact of each component in our IDIL loss design and implementation, compare our method with four ablated versions with BERT classifier in Table 3: (1) SELFOOD + Grad Sub represents our method with backpropagating gradi-

---

IDIL, vs Training Stogs
Validation FPR)%) vs Training Steps
100
Mww
0.006
CE Loss SELFOOD
Validazion FPR (%)
CE Loss SELFOOD
0.4
0.004
80
PC
CE SELFOOD
0.0
40
DEL--ELFOODU
0.002
0.73
0
0.000
25
0
1000 2000 3000
4000
1000
2000
3000
4000
PC
Faaiiing steps
Training nigingg Steps
PC
0.25
(a) vs Training Steps
(b) FPR vs Training Steps
(c) Learned weights
Figure 3: understand training trajectories of CE Loss SELFOOD, we plot their respective losses, validation FPR, and learned weights while training. We observe IDIL loss minimization is more correlated with performance
and the learned weights of CE-Loss and SELFOOD show divergent paths since the beginning.
100
SELFOOD
Naximam Sohmax Soore 161
8b 86
Mastmam Softmux Score Phi
8.000005
80
Resmmm Soh.meoomee..ooo Scoee
70-
Clmma NYT
Clima
Cimm
NYT
16
40
96.
30
16
20
50
70
90
10
10
50
70
90
Percentie (%)
Pertentile (1))
Percentile (16)
(a)
(b) CRL
(c) SELFOOD
Figure Maximum softmax score vs percentile in-distribution OOD data associated with that score. By using the least in-distribution maximum softmax score as the threshold (dotted line) for OOD classification, CE loss considers more than 50% and CRL considers almost 100% the OOD data as in-distribution (red region). However, in the case SELFOOD, we observe clear margin in maximum softmax scores that separate OOD and in-distribution.
Additionally, we plot the validation FPR95 using Yelp as the OOD dataset to analyze the correlation between loss minimization and performance Fig- ure 3()). Figure 3(a) illustrates that while minimiz- ing CE loss leads to fluctuations in the IDIL loss, ultimately reaching zero, Figure 3(b) shows that the performance is more closely associated with plicit IDIL minimization. Specifically, minimizing the IDIL loss results in improved FPR95. The find- ings indicate that the minimization CE loss and IDIL loss follows distinct trajectories. validate this hypothesis, we conduct an experiment using a toy 3-dimensional dataset comprising five isomet- ric Gaussian blobs and train a logistic regression classifier initialized with same weights, minimiz- ing IDIL and CE-loss. We flatten the weight matrix and plot the top-3 principal components. As shown in Figure 3(c), the learned weights of CE-Loss and SELFOOD progress different directions, thereby confirming our hypothesis.
with BERT classifier on Clima as in-distribution and NYT as OOD datasets in Figure 4. When com- paring the maximum softmax score with the per- centile data associated with that score, interesting observations can be made regarding its distribution. Specifically, when using the least in-distribution maximum softmax classification, find that the CE loss considers over 50% of the OOD data and CRL considers almost 100% of the OOD data as in-distribution. However, the case of SELFOOD, we observe a clear margin the maximum softmax scores that effectively separates OOD and in-distribution data. This suggests that the threshold needs to be care- fully tuned for CE loss and CRL, requiring more effort and annotated data, whereas in the case of SELFOOD, do not require such tuning. This demonstrates that SELFOOD is capable of accu- rately classifying between OOD and in-distribution samples based on their respective maximum soft- max scores, resulting in superior performance.
as the threshold for OOD
5.2 Maximum softmax score Analysis
5.3 Batch size Analysis
We plot the maximum softmax score with the per- centile in-distribution and OOD data associated with that score for CE loss, CRL, and SELFOOD
We compare documents within the bati- when computing the loss. Consequently. the num-

---

NYT
Yelo
Table 4: Fine-grained OOD detection results with NYT as the in-distribution dataset with BERT classifier. We choose two datasets AGNews, 20News from the news domain, the same as NYT, and consider them as OOD datasets. The results show that SELFODD can accu- rately detect OOD samples within the same domain.
1.0
0.8
FPR55
PPA35
BRR
2.8
AUROC
ALHOC
MPP
ALPS
0.0
0.D
16
Batch Size
O0D
Method
FPR95
ERR
AUROC
AUPR
(a) NYT
(b) Yelp
LogitNorm
45.9
0.9
86.7
15.8
Figure 5: Performance of SELFOOD w.r.t. batch size. We consider BERT classifier, Clima as n--distrbuttin, and NYT, Yelp OOD datasets. We observe an in- crease in performance with an increase in batch size.
Bayes Approx
57.5
0.9
87.7
23,9
AGNews
CE-Loss
45.5
0.9
86.5
16.6
CRI.
77.2
0.9
80.6
9.5
SELFOOD
0.0
0.1
100.0
99.4
LogitNorm
24.5
6.2
91.7
39.2
Bayes Approx
35.0
93.7
66.6
ber of document pairs over which the loss is com- puted depends on the batch size used during train- ing. To investigate the impact batch size on per- formance, we vary the batch size and evaluate the corresponding performance in Figure 5 with BERT classifier. We consider Clima as in-distribution, and NYT, Yclp as OOD datasets. As the batch size increases, the performance of the model also improves, until batch size of 16, where the perfor- mance reaches plateau. This observation aligns with our intuition that a larger batch size allows for more in-batch document pairs to be compared, lead- ing to more accurate loss computation. Based on these findings, we recommend utilizing a batch size of 16 or higher to achieve optimal performance.
20News
CE-Loss
25.4
91.9
39.6
CRI. 59.9
6.4
87.4
36.9
SELFOOD
28.2
4.9
94.2
61.5
and 1.8%. This highlights that while SELFOOD serves as reliable OOD detector, its performance as a text classifier is subpar. This observation be attributed to our IDIL loss formulation, which focuses on comparing confidence levels across doc- uments for each label rather than across labels for each document. As a result, the IDIL loss primarily promotes the ordinal ranking of confidence lev- els across documents, which enhances the model's 's OOD detection capabilities. However, this empha- sis inter-document ranking comes at the expense of inter-label ranking, resulting in limited classifi- cation capabilities. Moreover, when we introduce the intra-document comparison to the IDIL loss, as discussed in Section 4.5, we observe a decline in the model's 's ranking ability. Podolskiy et al. (2021) use mahalanobis distance for OOD detection and observe that although mahalanobis distance along- side ceasffomerbbbsce models is effective, such methods sensitive to the geometric features in the embedding space and can be spoilt if the embedder is used for classification and is overfit. This matches with our observation, upon adding intra-document comparison to the loss, the OOD detection performance drops. This further supports the notion that balancing the inter-document and intra-document comparisons is crucial for achiev- ing optimal performance in both OOD detection and text classification tasks.
5.4 Fine-grained OOD Detection
To explore the limits of SELFOOD, we consider m--istrribtton and OOD datasets from the same domain. The objective to assess its ability to dif- ferentiate between samples that are closely related but still OOD, which is challenging task. Specif- ically, we consider the news domain and choose NYT the in-distribution, and AGNews (Zhang al., 2015), 20News* 4 OOD datasets and train BERT classifier. As shown in Table 4, SELFOOD performs significantly better than the baselines on most of the metrics. Morcover, it also achieves near-perfect scores for the AGNews, highlighting its ability to accurately identify OOD samples even when they belong to the same domain as the in- distribution dataset.
5.5 Classification performance of SELFOOD
5.6 SELFOOD Mahalanobis distance
The BERT classifier trained with CE-Loss demon- strates exceptional classification performance, achieving accuracy scores of 97%, 85%, and 97% on NYT, Clima, and TREC datasets respectively. In contrast, SELFOOD achieves only 0.3%, 9.5%.
Mahalanobis cittace--assd estimation of OOD scores is an effective post-processing method used on trained OOD classifiers (Leys et al., 2018; Xu et al., 2020). We investigate whether this post- processing further improves the performance of our method. OOD scores are estimated as the dis-
httt:///wwonee.cmmrrssss////////

---

Table 5: OOD detection results with Mahalanobis Dis- tance post-procsssing technique. We choose two in- distribution, OOD pairs with BERT classifier. The re- sults show that SELFOOD's s OOD detection capabilities are enhanced with the post-processing technique.
Table 6: Qualitative analysis of CE-Loss vs SELFOOD with Yelp as in-distribution and TREC as OOD datasets. Wrong predictions are in Red and correct predictions are in Green.
Text
CE-Loss
SELFOOD
Who said "What contemptible scoundrel stole the cark from my lunch?"
InD
0OD
Method
FPR95
ERR
AUROC
AUPR
IaD
00D
Clima
Yelp
SELFOOD
14.7
0.6
98.0
90.8
Who said "Give me liberty give death"?
SELFOOD Maha
0.8
99.2
97.2
InD
0OD
SELFODD
63.2
19.6
79.4
82.7
Yelp
NYT
SELFOOD+
wouldn't I you want your staff to be knowlodgaable
tequila
about tequila?? Call me silly
So Gregg and are wondering Rourhon
Street, looking for the next "place"..
tance between test data point and the distribution of in-distribution samples using the Mahalanobis distance metric. Following (Xu al., 2020), consider the intermediate layer encodings of the OOD classifier trained on in-distribution samples as its representative distribution. We apply this post-processing on top of SELFOOD and exper- iment with two in-distribution, OOD pairs with BERT classifier. As shown in Table 5, we observe further improvement in performance, demonstrat- ing the quality of learned representative distribution using SELFOOD.
reads: "whol's
than
mean girl cheerleadens? how about resurrected girl cheerieulers with supernatural powers
this place could he farther away from my house, Camden... the Vonnisorium, everytime i grace the hipster-stained.
OOD
InD
also acknowledge that it comes at the expense of text classification performance. Future research can focus on developing techniques that effectively balance inter-document and intra-document com- parisons, enabling improved performance in both OOD detection and text classification tasks.
5.7 Qualitative Analysis of SELFOOD
7 Limitations
We present few examples with Yelp as in- distribution and TREC as OOD where SELFOOD accurately distinguishes and CE-loss fails to dis- tinguish between in-distribution and OOD sam- ples in Table 6. Although the first example has food-related words like lunch, cork and sentiment- indicative words such as contemptible scoundrel, SELFOOD accurately identifies it as OOD and CE loss model fails do so. Examples and have questions that revolve around eateries. CE- Loss model confuses and predicts them as OOD as TREC dataset contains only questions. How- ever. SELFOOD was able to rank them higher and accurately identify them in-distribution.
As mentioned in the previous sections, the OOD detection performance of our method comes at the cost of classification performance. This limitation needs to be addressed.
Ethics Statement
This paper proposes a self-supervised OOD detec- tion method. The aim of the paper is to minimize the human effort in annotating OOD documents and using only in-distribution documents for OOD detection. OOD detection is helpful in detection potential harmful content. Hence, we do not antici- pate any major ethical concerns.
6 Conclusion
Acknowledgments
this paper, we present SELFOOD, a novel frame- work for OOD detection that leverages only in- distribution samples as supervision. Building upon the insight that OOD samples typically do not be- long to any in-distribution labels, we formulate the OOD detection problem inter-document intra-label ranking task. To address this challenge. we propose IDIL loss which guides the training process. Through extensive experiments on multi- ple datasets, we demonstrate the effectiveness of SELFOOD on OOD detection task. However, we
We thank the anonymous reviewers and our col- leagues from Shang Data Lab, especially Bill Hogan and Zihan Wang for their helpful feed- back. Our work is sponsored in part by NSF CA- REER Award 2239440, NSF Proto-OKN Award 2333790, NIH Bridge2AI Center Program under award IU54HG012510-01. Cisco-UCSD Spon- sored Research Project, as well as generous gifts from Google, Adobe, and Teradata. Any opinions, findings, and conclusions or recommendations ex- pressed herein are those of the authors and should

---

be interpreted as necessarily representing the views, either expressed or implied, of the U.S. Gov- ernment. The U.S. Government is authorized to reproduce and distribute reprints for government purposes not withstanding any copyright annota- tion hereon.
Richard HR Hahnloser, Rahul Sarpeshkar, Misha Ma- howald, Rodney J Douglas, and H Sebastian Seung. 2000. Digital selection and analogue amplification coexist in cortex-inspired silicon circuit. nature, 405(6789):947-951
Dan Hendrycks and Kevin Gimpel. 2017. A baseline for detecting misclassified and out-of-distribution exam- ples in neural networks. In International Conference Learning Representations.
References
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever. and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Dan Hendrycks, Mantas Mazeika, and Thomas Diet- terich. 2018. Deep anomaly detection with outlier exposure. arXiv preprint arXiv: 181..04682.45
Eduard Hovy, Laurie Gerber, Ulf Hermjakob, Chin- Yew Lin, and Deepak Ravichandran. 2001. Toward semantics- baasd answer pinpointing. In Proceedings of the First International Conference on Human Lan- guage Technology Research.
Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. 2020. Generalized odin: Detecting out-of- distribution image without learning from out-of- distribution data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 10951-10960.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019, Bert: Pre-training of deep bidirectional transformers for language understand- ing. ArXiv, abs/18 1118.08805....
Amita Kamath, Robin Jia, and Percy Liang. 2020. Se- lective question answering under domain shift. Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 5684- 5696.
Terrance De and Graham W Taylor. 2018. Learn- ing confidence for out-of-distribution detection in neural networks. arXiv preprint arXiv:1802.0488
Stefan Elfwing. Eiji Uchibe, and Kenji Doya. 2018. Sigmoid- Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural Networks, 107:3-11.
Yoon Kim. 2014, Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 1746-1751, Doha, Qatar. Association for Computational Linguis- tics.
Geli Fei and Bing Liu. 2016. Breaking the closed world assumption in text classification. In Proceedings of the 2016 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 506-514.
Stefan Larson, Anish Mahendran, Joseph J Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K Kummerfeld, Kevin Leach, Michael A Laurenzano, Lingjia Tang, and et al. 2019. An evalu- ation dataset for intent classification and out-of-scope prediction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing 1311-1316.
Varun Gangal, Abhinav Arora. Arash Einolghozati, and Sonal Gupta. 2020. Likelihood ratios and genera- tive classifiers for unsupervised out-of-domain de- tection in task orientrie dialog. Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 34, pages 7764-7771.
pages
Irving John Good. 1952. Rational decisions. Journal of the Royal Statistical Society: Series B (Methodologi- cal), 14(1):107-14.
Tanmay Laud, Daniel Spokoyny, Tom Corringham, and Taylor Berg- KirgKKirkaatrick. 2023. Climabench: A henchmark dataset for climate change text understanding in english. arXiv preprint arXiv:2301.0423
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Wein- berger. 2017. On calibration of modern neural net- works. In International conference on machine learn- ing, pages 1321-1330. PMLR.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. 2018. A simple unified framework for detecting out- of-distritution samples and adversarial attacks. Ad- vances in neural information processing systems, 31.
Chirag Gupta and Aaditya Ramdas. 2021. Top-label cal- ibration and multiclass-to-biinee binbarry reductions. arXiy preprint arXiv:2107.0083.

---

Christophe Leys, Olivier Klein, Yves Dominicy, and Christophe Ley. 2018. Detecting multivariate out- liers: Use robust variant of the mahalanobis dis- tance. Journal of experimental social psychology, 74:150-156.
Alexander Podolskiy, Dmitry Lipin, Andrey Bout, Eka- terina Artemova, and Irina Piontkovskaya. 2021. Re- visiting mahalanobis distance for transformer- out-of-domain detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 13675-13682.
Xiaoya Li, Jiwei Li, Xiaofei Sun, Chun Fan, Tianwei Zhang. Fei Wu, Yuxian Meng, and Jun Zhang. 2021. k folden: k-fold ensemble for out-of -ft--ff--isributiinn de- tection. arXiv preprint 2xXi:2208. 12731.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Nlya Sutskever. 2019. Language models are unsupervised multitask learners.
Xin Li and Dan Roth. 2002. Learning question clas- sifiers. In COLING 2002: The 19th International Conference on Computational Linguistics.
Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. 2019, Likelihood ratios for out- of-distribution detection. Advances in neural infor- mation processing systems. 32.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint 1XXxv::907.. 1669..
Frank Rosenblatt. 1957. The perceptron, perceiving and recognizing automaton Project Para. Cornell Acronautical Laboratory.
Dheeraj Mekala, Chengyu Dong. and Jingbo Shang. 2022a. LOPS: Learning order inspired pseudo-label selection for weakly supervised text classification. In Findings of the Association for Computational Linguistics: EMNLP 2022. pages 4894-4908, 4908, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2013. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geof- frey J. Gordon. 2019. An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representa- tions.
Dheeraj Mekala, Varun Gangal, and Jingho Shang. 2.21. Coarse2Fine: Fine-grained text classification on cousell-grainne annotated data. In Proceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing, pages 583-594, Online and Punta Cana, Dominican Republic, Association for Computational Linguistics.
Apoorv Vyas, Nataraj Jammalamadaka, Xia Zhu, Di- pankar Das, Bharat Kaul, and Theodore Willke. 2018. Out-of-distribuutio detection using an ensem- ble of self supervised leave-out classifiers. In Pro- ceedings of the European Conference on Computer Vision (ECCV), pages 250-564.
Dheeraj Mekala and Jingbo Shang. 2020. Contextu- alized weak supervision for text classification. In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, pages 323- 333.
Hongxin Wei, Renchunzi Xie, Hao Cheng. Lei Feng. Bo An, and Yixuan Li. 2022. Mitigating neural net- work overconfidence with logit normalization. In In- ternational Conference on Machine Learning, pages 23631-23644, PMLR.
Dheeraj Mekala, Tu Vu, Timo Schick, and Jingbo Shang. 2022b. Leveraging qa datasets to improve generative data augmentation. In Proceedings of the 2022 Con- ference Empirical Methods in Natural Language Processing.
Yanan Wu, Keqing He, Yuanmeng Yan, QiXiang Gao. Zhiyuan Zeng, Fujia Zheng, Lulu Zhao, Huixing Jiang, Wei Wu, and Weiran Xu, 2022a. Revisit over- confidence for ood detection; Reassigned contrastive learning with adaptive class-dependent threshold. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 4 4165-4179.
Jooyoung Moon, Jihyo Kim, Younghak Shin, and Sangheum Hwang. 2020. Confidence-aware learning for deep neural networks. In international conference on machine learning. pages 7034-7044. PMLR.
Anh Nguyen, Jason Yosinski, and Jeff Clune. 2015. Deep neural networks easily fooled: High con- fidence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 427-436.
Yanan Wu, Zhiyuan Zeng, Keqing He, Yutao Mou, Pei Wang, and Weiran Xu. 2022b. Distribution cali- bration for oft-o--domain detection with Bayesian approximation. In Proceedings of the 29th Inter- national Conference Computational Linguistics, pages 608 615, Gyeongju. Republic of Korea, Inter- national Committee on Computational Linguistics.
Yawen Ouyang, Jiasheng Ye, Yu Chen, Xinyu Dai, Shu- jian Huang, and Jiajun Chen. 2021. Energy-based un- known intent detection with data manipulation. arXiv preprint 2107 12542.

---

Hong Xu, Keqing He, Yuanmeng Yan, Sihong Liu, Zi- jun Liu, and Weiran Xu. 2020. A deep generative distance-based classifier for out-of-domain detection with mahalanobis space. Proceedings of the 28th International Conference on Computational Linguis- tics, pages 1452-1460.
Zhilin Yang. Zihang Dai, Yiming Yang. Jaime G. Car- bonell, Ruslan Salakhutdinov, and Quoc 2019. Xlnet: Generalized autoregressive pretraining for lan- guage understanding. In Neural Information Process- ing Systems.
Zhiyuan Zeng, Keqing He, Yuammeng Yan, Zijun Liu, Yaman Wu, Hong Xu, Huixing Jiang, and Weiran Xu. 2021a, Modeling discriminative representations for out-of-domain detection with supervised contrastive learning. arXiv preprint arXiv:2105. 14289.
Zhiyuan Zeng, Hong Xu, Keqing Hc, Yuanmeng Yan, Sihong Liu, Zijun Liu, and Weiran Xu. 2021b. Ad- versarial generative distance-based classifier for ro- bust out-of-domain detection. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7658-7662. IEEE.
Li- Li-Ming Zhan, Haowen Liang, Bo Liu, Lu Fan, Xiao- Ming Wu, and Albert Lam. 2021. Out-of-scope in- tent detection with self supersupevision and discrimina- tive training. arXiv preprint 1.6::2006.086/66
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text clas- sification. C. Cortes, N. D. Lawrence, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 649-657. Curran Associates, Inc.
D. Lee,
Wenxuan Zhou, Fangyu Liu, and Muhao Chen. 2021. Contrastive ouittf-distribution detection for pretrained transformers. arXiv preprint 104.022104.08812.

---

A Appendix
A.1 Results with NYT & Clima as in-distribution datasets
We present experimental results Clima and NYT in-distribution datasets with BERT and RoBERTa classifiers in Table 7.
A.2 Experimental Settings
Training trajectory analysis experiment. We compare two experiments showcasing empirical evidence on how the trajectory CE-loss mod- els and IDI loss models differ. In 3(b) we plot the change in validation FPR on BERT-classiferss optimized on CE-loss and IDIL loss. Here, we cal- culate the FPR across training steps a validation dataset consisting of 500 in-distribution and OOD data points each compare the training trajectories the two models.
We generate the toy dataset by creating five isometric Gaussian blobs of 3-dimensional data points, with each cluster containing 1000 samples. The centers of all the clusters are bounded by the range [-10.0, 10.0] and the cluster standard de- viation is set 2.0 account for limited over- lap between cluster datapoints. We consider four clusters as being in-distribution and the remain- ing cluster as OOD, We train logistic regression classifier on the in-domain datapoints initialized with the same weights and minimizing IDIL and CE-loss. The model is optimized using Stochas- tic Gradient Descent with learning rate of 0.01 for 300 epochs. The weights visualized at the begining of each epoch. In order to facilitate visual- ization, the weight matrices of both approaches of dimension are flattened and Principal Component Analysis (PCA) is applied to reduce the dimension- ality of the weight matrix. We plot the top-3 prin- cipal components after eacch epoch to conclude that the learned weights of Co-loss and SELFOOD progress in different maner.

---

BERT
RoBERTa
In-dist
OOD
Method
[PR95(1)
(RR(()
AUROC(T)
AUPR(T)
(PR95(L)
ERR(U)
AUROC(T)
AUPR(T)
LogitNorm
20.6
1.0
96.8
92.3
18.4
1.0
97.3
93.6
OneSsRest
84.7
1.7
87.3
82.6
92.7
1.9
91.1
85.1
RCL
86.7
2.3
Clima
Bayes Approx
40.6
1.2
95.4
90.0
80.0
2,4
88.4
76.5
CE Loss
22.1
1.0
96.9
92.5
15.7
1.0
97.3
92.8
CRL
32.3
1.7
95.4
79.3
3.2
79.8
67.0
SELFOOD
10.5
1.6
98.4
90.5
4.6
1.0
99.3
95.4
LogitNorm
13.3
5.5
97.2
87.9
6.4
84.1
OsessRest
96.0
6.2
84.0
79.4
97.2
6.3
90,5
82.8
RCI.
62.9
92.2
85.8
73.9
8.8
88.4
73.1
NYT
TREC
Bayes Approx
38.8
94.8
89.9
67.5
8.3
88.7
74.7
Loss
14.5
5.8
97.0
87.1
21.9
6.7
95.5
83.5
CRI.
67.9
6.5
89.9
79.1
85.6
9.6
71.4
61.4
SELFOOD
0.0
100.0
100.0
0.0
100.0
100.0
LogitNorm
0.6
98.5
87.9
OneVsRest
15.1
97.4
78.1
8.7
0.6
97.6
78.8
RCL
45.1
0.5
94.9
84.7
76.7
1.0
90.2
60.0
Yelp
Bayes Approx
28.7
96.4
86.8
83.0
1.1
87.7
56.6
CF Loss
3.4
0.7
99.2
82.3
0.6
99.3
85.4
CRL
69.5
0.5
89.3
77.8
73.3
1.0
83.8
58.3
SELFOOD
22.5
57.8
0.2
97.5
LogitNorm
37.8
5.5
94.6
91.9
60.5
7.0
91.6
88.0
OneVsResl
67.7
RCL
80.1
18.4
75.7
55.3
91.2
18.4
66.6
47.7
TREC
Bayes Approx
83.7
15.8
77.5
63.8
65.7
13.9
83.6
70.7
34,9
9.1
CRL
83.0
12.7
82.0
72.2
60.9
12.6
79,2
63.2
SELFOOD
0.3
1.1
99.4
99.4
38,9
8.8
90.1
84.7
LogitNorm
57.7
0.8
93.0
81.3
54.7
1.2
92.6
72.3
OneVsRest
43.7
1.0
92.7
72.5
58.8
1.8
73.5
34.7
RCL
91.0
60.7
12.2
98.0
2.3
12.7
Yelp
Rayes Approx
81.4
2.2
76.5
22.7
96.8
2.2
52.3
16.3
Loss
46.3
0.9
93.8
79.5
1.9
47.2
CRL
98.7
L.4
67.3
53.3
72.9
2.1
19.0
SELFOOD
14.7
0.6
98.0
90.8
37,6
1.5
72,3
$4.9
LogitNorm
37.6
4.2
94.4
87.3
46.1
4.5
93,1
85.4
OneVsRest
34.6
4.8
94.3
85.3
40.6
9.4
83,1
$1.3
RCI.
NYT Bayes Approx
87.4
11.2
70.4
38.3
96.8
11.3
55.8
29.2
CB Loss
44.9
4.8
93.2
84.4
62.5
6.7
89.6
75.2
CRI
SELFOOD
17.2
2.6
97.8
94.5
55.3
6.7
68.6
61.5
Table 7: OOD detection results with BERT & RoBERTa classifiers. Each experiment is repeated with three random seeds and the mean scores are reported. The false-positive-rate at 95% tree-positive-rate (FPR95), minimum detection enror over all thresholds (ERR), the area under the risk-coverage curve (AURC), and the area under the precision-recall curve (AUPR) using in-distribution samples as the positives used as evaluation metrics.


Reviews:
[{"review_id": "dvYX8AEqiT", "reviewer": "Reviewer_NgSb", "summary": "", "questions": "\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the review. Below, we address the reviewer’s main concerns:\n\n__More discussion on self-supervised approaches__\nZhou et al propose an unsupervised training method to fine-tune transformer models on a combination of contrastive loss and cross entropy loss. They experiment with several scoring functions such as maximum softmax probability, energy score, mahalanobis distance, and cosine similarity. Zeng et al. also propose an unsupervised training approach using supervised contrastive Loss and cross Entropy loss to train Bi-LSTM models for OOD Detection. They employ the BERT model only to obtain encodings that are fed into the BiLSTM model. SELFOOD follows a similar experimental setting as both, however during training, we do not use cross entropy loss and backpropagate gradients to decrease probabilities associated with incorrect labels rather than increasing the probability of the correct label, which results in faster convergence. Moreover, our method is also applicable to transformer-based architectures. We will add more discussion and comparison with these two papers.\n\n__Effect of Back propagation control on other losses__\nIn order to understand the importance of back propagation control on other losses, we experiment with Triplet Margin Loss. We train a BERT-based OOD detector with TREC as in-distribution datasets and evaluate on Clima, NYT, Yelp as the OOD datasets. The embedding vectors from the trained detector are used to identify the OOD samples using Mahalanobis distance. We run two settings i.e. with and without dropping the backpropagation through the positive term, to compare the improvement. The results are shown below.\n\n| **OOD Dataset**  | **Method**                         | **FPR95**    | **AUROC** |\n|------------------|------------------------------------|--------------|-----------|\n| Clima          | Triplet Marging Loss Grad Neg Only | 0.0698       | 97.916    |\n| Clima          | Triplet Marging Loss Grad Both     | 6.2067       | 99.219    |\n|                  |                                    |              |           |\n| NYT              | Triplet Marging Loss Grad Neg Only | 0.33208      | 99.1708   |\n| NYT              | Triplet Marging Loss Grad Both     | 1.7030e-2    | 99.9941   |\n\n\nWe do not notice any significant improvement by dropping the backpropagation through the positive term. We hypothesize that contrastive losses such as Triplet Margin Loss work on bringing together intra-label data points in the latent space while the IDIL loss aims to modify the softmax probabilities of the data points with an additional linear head.\n\n__OOD detector as a separate classifier__\nSeveral works previously trained a separate OOD classifier which is different from the classifier for the underlying task. For example, [1] propose a Random Forest OOD detector as an external calibrator in addition to a Question Answering model. [2] experiment with the mahalanobis distance for OOD detection and compare its effect across multiple models. They conclude that although mahalanobis distance alongside transformer based models is effective, such methods are sensitive to the geometric features in the embedding space and can be spoilt if the embedder is used for classification and is overfit. This matches with our observation in Lines 523-526, where upon adding intra-document comparison to the loss, the OOD detection performance drops. Therefore, we believe training an OOD classifier separately is a good design. Balancing the inter-document and intra-document comparisons is crucial for achieving optimal performance in both OOD detection and text classification tasks and we leave this for future work.\n\n[1] Selective Question Answering under Domain Shift ACL’20\n\n[2] Revisiting Mahalanobis Distance for Transformer-Based Out-of-Domain Detection AAAI’21", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This work is well-motivated, and the proposed IDIL training process is clearly defined. It seems simple and effective, providing a promising solution for OOD detection.\nThe ablation study is comprehensive, including different scenarios. The limitations of the method are also clearly discussed and analyzed.\nThe paper is well-written and easy to follow. The author also claims to release the code.", "reasons_to_reject": "Previous work that having similar self-supervised setting should be discussed with more details. \ne.g. “Zhou et al.; Zeng et al. introduce self-supervised approaches to OOD detection using contrastive learning”. \nFrom the ablation study, I can see the effectiveness affected by how to perform backward propagation a lot. How do previous work performs if their method also involving backward propagation control? For example, what if rather than pairwise learning to rank loss, the model is trained by listwise infoNCE loss but drop the backward propagation of the positive example term? It is worth to know how important the “pairwise” setting is.\n", "questions_to_authors": "\n", "paper_topic_and_main_contributions": "This work proposes to use the learning to rank method for out-of-distribution detection with only in-distribution data. Specifically, this work designed an IDIL training loss (as well as modified backward propagation) based on pairwise ranking loss. Experiments conducted on well-known datasets such as Yelp and TREC show that this method obtains significant effectiveness improvement compared to existing methods.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 137 ood→OOD\n\nIs the OOD detector usually expected to be an effective classifier at the same time? maybe a little bit more discussion on this.\nIt would help reader better understand limitation & future direction."}}, {"review_id": "AZq8SrLSs7", "reviewer": "Reviewer_MNGK", "summary": "", "questions": "+ Apart from discrete classification, is there any way to adopt SelfODD for continuous labels? This can make SelfODD more general in various usages.\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__Effectiveness of IDIL Loss__\nAs mentioned in Lines 76–84, minimizing CE loss involves continuously increasing the probability of the correct label over the other labels, making the classifier overconfident. Such overconfidence makes it an unreliable OOD detector. Therefore, this motivated us to design a loss function that (1.) encourages desired ranking, and (2.) becomes zero once desired ranking is achieved, to avoid overconfidence. \n\nMoreover, we empirically analyze CE-loss and IDIL loss in section 5.1. We plot FPR vs training steps in figure 3b for CE-loss and IDIL loss and we observe IDIL loss minimization is more correlated with OOD detection performance than CE loss minimization. We also plot their training trajectories in figure 3c and observe that they diverge since the beginning. We leave the theoretical comparison between CE-loss and IDIL loss for future work.\n\n__Qualitative Analysis__\nWe present a few examples where SELFOOD accurately detected OOD samples and CE-loss failed. We consider Yelp as in-distribution and TREC as the OOD dataset with BERT classifier. Yelp is a sentiment classification dataset with reviews as text and TREC is a question classification dataset.\n\nA sample from Yelp dataset where CE-loss model gave low scores and SELFOOD gave high scores:\n\n*Its a tequila house - wouldn't you want your staff to be knowledgeable about tequila? Call me silly So Gregg and I are wondering Bourbon Street, looking for the next \"place\". I don't normally do tequila by itself (bad memories), but I thought, why not. We walk in, and i ask the bartender what the best tequila was. I got the deer in the headlights look. I asked what the most expensive tequila was. She told me it was 80 and pointed to a top shelf with a really cook looking bottle, but she couldn't tell me the name. I then asked what a 35-40 tequila was. She got down a bottle (which took awhile), Gregg and I tried it, I can't say that it was so good and memorable that I can tell you the name. You might want to stick to the frozen margarita machines at this place. I think the bottles are just for show.*\n\nThe sentiment of the above review is difficult to predict and the CE-loss model returns low maximum softmax probability scores. However, the SELFOOD model accurately detects its content and generates high scores.\n\nFew samples from TREC dataset where CE-loss gave high scores and SELFOOD gave low scores:\n1. *Who said \"What contemptible scoundrel stole the cork from my lunch ?\"*\n\n2. *Who were the \"filthiest people alive ?\"*\n\nThese samples have strong sentiment-indicative words such as “contemptible scoundrel” and “filthiest people” and CE-loss model predicts high scores. However, SELFOOD recognizes it as not relevant to Yelp and predicts OOD by generating low scores. We will add this discussion to the paper.\n\n__SELFOOD for continuous labels__\nThe proposed SELFOOD is not applicable for continuous labels and we leave this for future work. We agree that it would make it more generalised.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ This paper is well-written and easy to follow.\n+ The proposed ranking loss has the potential to be a new loss function for general classification tasks.\n+ The proposed SelfODD shows significant improvements on both in- and out-domain classifications across various datasets.\n", "reasons_to_reject": "+ Though having promising results, there is a lack of motivation or theoretical discussion on why this ranking can be that effective. From my understanding, the standard CE loss will also decrease the incorrect probabilities (though not that actively).\n+ It can be better to present some qualitative examples and demonstrate how SelfODD achieves the correct prediction compared to the baseline.\n", "questions_to_authors": "+ Apart from discrete classification, is there any way to adopt SelfODD for continuous labels? This can make SelfODD more general in various usages.\n", "paper_topic_and_main_contributions": "This paper aims at the out-of-distribution (OOD) detection issue for classification. They present SelfOOD to address this annotation bottleneck and require only in-distribution labels. They train the target classifier to rank softmax scores instead of the standard cross-entropy loss. This ranking score then can help decrease probabilities associated with incorrect labels.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Nbe2RYlsyR", "reviewer": "Reviewer_6CvH", "summary": "", "questions": "Could the authors justify why SELFOOD is only evaluated on the task of text classification?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__Intent classification datasets__\nThanks for the suggestion! We experiment with SNIPS as in-distribution dataset and Clinic 150 as OOD dataset with BERT classifier. The minimum detection error of CE-loss model is 2.0 and SELFOOD is 1.76. The results demonstrate improvement with SELFOOD over CE-loss. We will add more experiment results on intent classification datasets in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed SELFOOD method is well-motivated and sound. \n\n- The paper conducted a comprehensive comparison with SOTA baselines. \n- The performance improvements brought by SELFOOD are quite significant on all four datasets. \n- The paper is well-written and easy to follow. ", "reasons_to_reject": "- The major concern is that while the paper claimed to propose a general self-supervised method for OOD detection, it was only evaluated on the task of text classification. There are other common tasks and datasets for OOD detection, such as intent classification, sentiment analysis, and topic prediction. However, the paper did not evaluate on them. \n- Intent classification for dialogue systems is the most common application for OOD detection, and there are many datasets such as CLINIC150, STAR, ROSTD, SNIPS, ATIS for intent detection are proposed. Without evaluating on them, it is a bit hard to demonstrate the generalizability of SELFOOD. ", "questions_to_authors": "Could the authors justify why SELFOOD is only evaluated on the task of text classification?", "paper_topic_and_main_contributions": "This paper proposes SELFOOD, a self-supervised OOD detection framework that requires only in-distribution samples as supervision. SELFOOD formulates OOD detection as an inter-document intra-label ranking problem and optimizes it using the proposed IDIL ranking loss. Compared with traditional CE loss where optimizing the probability of the correct label often leads to overconfidence, the IDIL ranking loss trains the model to rank the considered label probability score in documents belonging to the label higher than those not belonging to it. The paper evaluates SELFOOD and baselines on four text classification datasets: NYT, Yelp, Clima, and TREC.  Extensive experiments and analyses demonstrate the effectiveness of the proposed method. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "NA"}, "extra_text": {}}, {"review_id": "vylqCTNoNg", "reviewer": "Reviewer_At17", "summary": "", "questions": "In Table 2, it’s observed the proposed method outperforms baselines by a large margin. Are all methods using the same backbone model? Is the comparison fair for all the baselines and the designed approach?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__Calibrated Model__\nWe wish to emphasize that our method does not yield a calibrated model and we never stated so in the paper. Instead, it yields a model that conforms to the desired ordinal ranking, wherein the predicted probabilities align with the intended hierarchy, i.e., the probability of the correct label surpasses that of incorrect labels.  An ideally calibrated model produces probabilities whose magnitudes accurately represent their associated confidences, resulting in the aforementioned ranking. However, the probability scores generated by the model trained with the IDIL loss function do not reliably signify its level of confidence. Given that accurately ranked probabilities are sufficient for precise OOD detection, our approach delivers a high-quality OOD classifier.\n\n__Accuracy Results__\nWe discuss and analyse the classification performance of SELFOOD with BERT classifier on NYT, Clima, and TREC datasets in section 5.5. SELFOOD performs poorly compared to other methods in classification. This is due to our IDIL loss formulation which promotes the ordinal ranking of confidence levels. It compares confidence levels across documents for each label rather than across labels for each document, which is required for high accuracy. Therefore, we emphasize in lines 90-94 that our method is specifically designed to improve the performance of OOD detection rather than classification accuracy.\n\n__Backbone Model & Fair Comparison__\nAs stated in lines 331-337, we consider two backbone models BERT & RoBERTa for all baselines and similar experimental settings such as epochs and learning rate for a fair comparison.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The task of self-supervised OOD detection is valuable for reducing the annotation cost and is under-investigated.\n\nThe idea of inter-document intra-label ranking for calibration is novel and reasonable. \n\nThe proposed method is simple and effective on the OOD detection task.", "reasons_to_reject": "It’s claimed that training with the proposed IDIL loss results in a more calibrated model, are there any experiments results to support the statement? Additional empirical results or theoretical analysis besides OOD detection metrics (FPR 95, AUROC) is suggested to support the statement and show the effectiveness of the proposed method.\n\nIt seems that the results on the ACC@FPRn metric are not reported in the paper. As mentioned, the detection performance of the proposed method comes at the cost of classification performance, it’s unclear about the ID accuracy compared to baselines. It’s strongly advised to report ACC@FPRn in the main text, especially ACC@FPR0.", "questions_to_authors": "In Table 2, it’s observed the proposed method outperforms baselines by a large margin. Are all methods using the same backbone model? Is the comparison fair for all the baselines and the designed approach?", "paper_topic_and_main_contributions": "The paper focus on self-supervised out-of-distribution detection, and proposes a novel method named SELFOOD, which only utilizes annotated in-distribution data as supervision. The main contribution is formulating the task as an inter-document intra-label (IDIL) ranking problem, and proposing the IDIL loss for backpropagation. The proposed method outperforms most of the baseline methods on various datasets.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "I6IFVRMXXV", "reviewer": "Reviewer_bvkd", "summary": "", "questions": "1. In the paper, do you assume that OOD samples do not belong to any in-distribution labels?\n2. Have you done any study on replacing LLM objective with the proposed loss?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__Contributions__\nIn this paper, we present SELFOOD, a novel framework for OOD detection that leverages only in-distribution samples as supervision. Building upon the insight that OOD samples typically do not belong to any in-distribution labels, we formulate the OOD detection problem as an inter-document intra-label ranking task. To address this challenge, we propose IDIL loss which guides the training process. Through extensive experiments on multiple datasets, we demonstrate the effectiveness of SELFOOD on OOD detection task. We believe our paper has significant contributions and appreciate it if you reconsider your comment.\n\n__Assumption__\nYes. As stated in lines 217-218, we assume that an OOD document does not fall under any specific label in the in-distribution space.\n\n__LLM with IDIL loss__\nWe cannot adapt SELFOOD to language modeling because our method is only for OOD detection and not for classification whereas language modeling invovles token classification at each step.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper introduces a promising approach to OOD detection that can significantly reduce the need for labeled OOD data. This could have practical implications for real-world NLP applications where labeled OOD data is often scarce or expensive to obtain.", "reasons_to_reject": "The paper is well-structured, but the contribution isn't sufficient for a long paper. An interesting follow-up study would be exploring how the proposed loss can be utilized in LLM training - for instance, improving the current LLM training objective using this proposed idea.", "questions_to_authors": "1. In the paper, do you assume that OOD samples do not belong to any in-distribution labels?\n2. Have you done any study on replacing LLM objective with the proposed loss?", "paper_topic_and_main_contributions": "This paper introduces SELFOOD, a self-supervised out-of-distribution (OOD) detection method that addresses the annotation bottleneck and achieves effective OOD detection through inter-document intra-label ranking. The authors simplify the pair-wise computation by only considering documents in each mini-batch. SELFOOD requires only in-distribution samples as supervision and does not rely on any external datasets or pre-trained models. The proposed method is evaluated on several benchmark datasets and outperforms state-of-the-art supervised and unsupervised methods. Additionally, SELFOOD is shown to be effective in both coarse and fine-grained classification settings. Overall, SELFOOD presents a promising approach to OOD detection that can significantly reduce the need for labeled OOD data.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes to use the learning-to-rank method for out-of-distribution detection with only in-distribution data. The method being proposed is mainly used to lower the desired confidence ranking by decreasing the probabilities associated with incorrect labels. The paper is well-written and well-motivated. However, as all the reviewers pointed out, the paper is restricted to a single task of text classification without the generalization to any other NLP tasks. This limitation decreases the impact of this paper.
Decision: Accept-Findings