FLatS: Principled Out-of-Distribution Detection with Feature-Based Likelihood Ratio Score
Authors: Haowei Lin
Yuntian Gu
Venue: EMNLP 2023 Main (2023)
Keywords: OOD Detection
Likelihood Ratio
Intent Classification

Abstract: Detecting out-of-distribution (OOD) instances is crucial for NLP models in practical applications. Although numerous OOD detection methods exist, most of them are empirical. Backed by theoretical analysis, this paper advocates for the measurement of the "OOD-ness" of a test case $\boldsymbol{x}$ through the \emph{likelihood ratio} between out-distribution $\mathcal P_{\textit{out}}$ and in-distribution $\mathcal P_{\textit{in}}$. We argue that the state-of-the-art (SOTA) feature-based OOD detection methods, such as Maha and KNN, are suboptimal since they only estimate in-distribution density $p_{\textit{in}}(\boldsymbol{x})$. To address this issue, we propose \textbf{FLATS}, a principled solution for OOD detection based on likelihood ratio. Moreover, we demonstrate that FLATS can serve as a general framework capable of enhancing other OOD detection methods by incorporating out-distribution density $p_{\textit{out}}(\boldsymbol{x})$ estimation. Experiments show that FLATS establishes a new SOTA on popular benchmarks.

FLatS: Principled Out-of-Distribution Detection with Feature-Based Likelihood Ratio Score
Haowei Lin 12,3+ Yuntian Gu
Institute for Artificial Intelligence, Peking University
School of Intelligence Science and Technology, Peking University
Yuanpei College, Peking University
1 Linhaowei@pkuu edu. cn guyuntianêstu, pku. edu. cn
Abstract
Where does the black hole come from?
Detecting out-of-distribution (OOD) instances is crucial for NLP models practical appli- cations. Although numerous OOD detection methods exist, most of them are empirical. Backed by theoretical analysis, this paper advo- cates for the measurement of the "OOD-ness" of a test case æ through the likelihood ratio be- tween out- distributiiion Powr and in di-iistiibution Pu. We argue that the state-of-the -iee-of-thh-rtttt (SOTA) feature-based OOD detection methods, such as Maha (Lee et al., 2018) and KNN (Sun et al.. 2022), are suboptimal since they only estimate in-distribution density Pin (x)). To address this issue, we propose FLatS, principled solution for OOD detection based on likelihood ratio. Moreover, we demonstrate that FLattS can serve general framework capable of enhancing other OOD detection methods by incorporating out-distribution density Pow (2:) estimation. Ex- periments show that FLatS establishes new SOTA on popular benchmarks.'
Feature Distribution
Human
Sman((x) pon(x)) / pa((x)
out-distribatin
Indisttiiution
Model
Robs)
feature icx))
The framework of OOD detection with feature- based likelihood ratio score (FLatS), The model extracts the feature of input text, and then outputs the OOD score SFtais az) that takes the form of likelihood ratio between out-distribution Pour and in-distribution Pin-
Figure I:
representation h(æ; 0) of test case æ encoded by the tested NLP model. For example, Maha (Lee et al., 2018) estimates the Mahalanobis distance between (2;; 0) to the in-distribution (IND), while KNN (Sun et al., 2022) estimates the distance to the k-nearest IND neighbor. These techniques have demonstrated remarkable performance in recent benchmark studies (Yang et al., 2022: Zhang al., 2023).
1 Introduction
Natural language processing systems deployed in real-world scenarios frequently encounter out-of- distribution (OOD) instances that fall outside the training corpus distribution. For instance, it is hard to cover all potential user intents during the train- ing of a task-oriented dialogue model. Therefore, it becomes crucial for practical systems to detect these OOD intents or classes during the testing phase. The ability detect OOD instances enables appropriate future handling, including additional labeling and utilization for system updates, ensur- ing the system's 's continued improvement (Ke et al., 2022, 2023).
However, these methods were proposed without principled guidance. To address this, our paper first formulates OOD detection test problem and derives that the principled solution towards OOD detection is to estimate the likelihood ratio Paua Pon((2) (x)/Ptn (zx). Under this framework, we show that Maha and KNN only estimates IND den- sity Pma (x) and assumes OOD distribution Pour to be uniform distribution, which is sub-optimal. This paper then proposes principled solution for OOD detection with feature- likelihood ratio score, namely FLatS. In FLatS, the IND density Pm (x) is also estimated with KNN the training corpus, while the OOD density Pour is estimated with KNN on OOD data. Though we are not access the real OOD data, we leverage public corpus (c.g., Wiki, BookCorpus) as auxiliary OOD data. Apart
binary hypothesis
A rich line of work has been proposed to tackle OOD detection. Among them, the best-performing methods exploit the information of feature hidden
*Corresponding author:
'Our code is publicly available at l linhaowei //FLatS.
com/

---

from KNN, we further demonstrate that the idea of FLatS to incorporate OOD distribution information is applicable to other OOD detection techniques. Experiments demonstrate the effectiveness of the proposed FLatS..
3 Method
3.1 A Principled Solution for OOD Detection In his seminal work, Bishop (1994) framed OOD detection as a selection problem between the in- distribution Po. and an out-of-distribution Pood- From a frequentist perspective, the objective of OOD detection can be formulated as binary hy- pothesis test (Zhang and Wischik, 2022):
2 Background
This paper focuses supervised multi-class classi- fication, a widely studied setting in OOD detection. The formal definition is given follows:
Ho : x Pour U.S. H1 : x ~ Pin
(3)
Definition 1 (OOD detection) Given an input space X C & and label space y = (1,..., K), Pxy is a joint in-distribution (IND) over X x y. drawn i...d from Pxy, OOD detection aims to decide whether test case a E X is drawn from the IND data distribution Pin (the marginal IND distribu- tion on X some OOD data distribution Pout-
By leveraging the Neyman- man and Pearson, 1933), Theorem demonstrates that likelihood ratio is a principled solution for OOD detection (the proof is given in Appendix A);
lemma (Ney-
Given a training set D = [(I,,
Theorem 1 A test with rejection region R defined as follows is a unique uniformly most powerful (UMP) test for the test problem defined in Equa- tion (3);
OOD detection has been studied extensively. For example, using the maximum softmux probability (MSP) (Hendrycks and Gimpel, 2016) to measure IND-ness is popular in literature. There more advanced methods like maximum logit (Hendrycks et al., 2019) and energy score (Liu et al., 2020). Among the existing OOD detection methods, distance-based Mahalanobis (Maha) score and K- nearest neighbor (KNN) score achieve remarkable performance on common OOD detection bench- marks. These methods first extract latent feature z h(:; 0) of test case x with the pre-trained lan- guage model 0. For Maha and KNN, the OOD-ness of x are measured by the two 2
{x: Pout (2)) < dot;
where Xo is a threshold that can be chosen to obtain a specified significance level.
Theorem highlights the importance of detect- ing OOD samples based on both low IND den- sity Pin (2) and high OOD density (w)) How- ever, most cestance-basse OOD detectors are ba- sically probability density estimators that only es- timate IND density Pae (=r(x) with training data, and assume OOD distribution Pout as uniform distribu- tion (see Appendix B for justifications).
Assuming a uniform OOD distribution Pour may lead to potential risks. For instance, consider a sce- nario where Pou NN( 0.01) and Pm - (0, 1). It is apparent that O has higher IND density than 1: Pa (0) > Pie but 0 is indeed more OOD-like (1mr(1)/Poo/ 10 e -49.5 This toy case illustrates that OOD de- tection cannot be based solely on IND density but should incorporate both IND and OOD densities. Although we derive the principled solution for OOD detection with likelihood ratio, it is notewor- thy that typically have access to genuine OOD data in real application, thus the OOD den- sity Powr ( (x) hard estimate. To address this, we follow recent works (Xu et al., 2021) to make use of a public corpus (e.g., Wiki, BookCorpus (Zhu et al., 2015)) to serve as auxiliary OOD data.
(1) = min(z TA -Ma): (1) cE)
SkNw( D) = -INN== 12)
(2)
than 1: 10 Poma (P)) (0)/pax(0) (0)
In Equation (1), Me is the class centroid for class e and E is the global covariance matrix, which are estimated on IND training corpus D. In Equa- tion (2). || 1l is Euclidean norm, x = 2 2|||=||2 denotes the normalized feature 2, and D' denotes the set of normalized features from training set D.
KNN(2* 2=" DD) denotes the k-nearest neighbor of x²
in set D More details are given in Appendix B.
"Note that S(x:) in this paper measures OOD-ness of æ. which means OOD sample will have high S(x). Many litera- ture define S(z) to measure the IND-ness of æ.

---

3.2 Feature-based Likelihood Ratio Score
4 Experimental Setup
This subsection designs an OOD score based on the likelihood ratio Pour(2)) /Pin (2x) motivated by Theorem 1. Since it is challenging to directly estimate the raw data distribution within the high- dimensional text space, we consider estimation in the low-dimensional feature space. As Appendix B suggests, (a) and (x) defined in Equa- tion (1) and Equation (2) essentially function as density estimators that estimate the IND distribu- tion P/ in the feature space. We will also exploit them to estimate OOD distribution Pow in our pro- posed method.
4.1 Datasets and Baselines
Datasets. We utilize 4 intent classification datasets CLINC150 (Larson et al., 2019), ROSTD (Gangal et al., 2020), Banking77 (Casanueva et al., 2020), and Snips (Coucke et al., 2018) for our experiments, which are commonly used in OOD detection litera- ture. For each dataset, we use some classes as IND and the remaining classes as OOD classes. More details can be found in Appendix C.
Choice of auxiliary OOD corpus Daux- We adopt English Wikipedia, 3 which is the source used in common by RoBERTa for pre-training.
To connect the normalized probability densi- ties with unnormalized OOD scores, we lever- age energy-based models (EBMs) to parameter- ize Pin and Pow: Given test case x, it has density Pin(z)
Baselines. We compare the proposed FLatS with 9 popular OOD detection methods. (1) For confidence-based methods that leverages output probabilitics classifiers trained on IND data to detect OOD samples, we evaluate MSP (Lee et al., 2018), energy score (Liu et al., 2020), ODIN (Liang et al., 2017), D2U (Chen et al., 2023), MLS (Hendrycks et al., 2019); (2) For distance- based methods, we test LOF (Breunig et al., 2000),
( in Pn, and density Pood(a:) = expl- Fno /Z) in Pouts where Z1, Z2 are noramlizing constants that en- sure the integral of densities Pia(æt) and Pon((a) equal 1, and Eou(()) are called energy func- tions. Then we can derive the OOD scores in the form of likelihood ratio with energy func- tions: S(x) - log(Pema (z)
Maha (Lee et al., 2018), KNN (Sun al., 2022), and GNOME (Chen et al., 2023).
(x)) (Prrr((r))))))) = (x) ( Bon (x) + log(Zz/ZZ (1). Since log( 22/ Z1) is a con- stant, can be omitted in the OOD score definition:
Evaluation Metrics. We adopt two widely- used metrics AUROC and FPR @95 following prior works (Yang al., 2022). Higher AUROC and lower FPR @95 indicate better performance.
SFLoIs (x) -En(x:) Eoou (x))
(4)
4.2 Implementation Details
Since the energy function (-) and E-m(.) do not need to be normalized, we can estimate them with OOD scores. For IND energy Eina(x)) we simply adopt the OOD score SxNv(x:). For OOD energy (z), we replace the training corpus D in Equation (2) with an auxiliary OOD corpus D Daux:
Architecture. We adopt ROBERTaBASE as our backbone model. The model is fine-tuned on IND training datasets before OOD detection evaluation. The fine-tuning follows the standard practice (Ken- ton and Toutanova, 2019), where we pass the final layer 
 token representation to a feed-forward classifier with softmax output for label prediction, together trained with cross-entropy loss.
SFLas(x) Sxvv(w;;)) D) (x; Daux).
(5)
Hyperparameters. We use KNN following (Chen et al., 2023). Searching from {0.1, 0.2, 0.5, 1.0, 2.0]. we adopt a = 0.5 for Equation (5). We use Adam optimizer with learning rate of 2c 5, batch size of 16 and 5 fine-tuning epochs. We evaluate the model on IND validation set after every epoch and choose the best checkpoint with the highest IND classification accuracy.
10 for
Since (xNv(>>; D) and (xNw(æ;; Daux) may be in different scales, a is a scaling hyper- parameter to make the two scores comparable. To the best of our knowledge, this is the first feature-based OOD score that follows the principled likelihood ratio so- lution. Also, KNN in Equation (5) only an exam- ple, which can be replaced by other feature-based OOD scores such as SMaha (z) (see Section 4.3 for ablation studies on different estimation methods).
https://ddmppswwiimmcia...g

---

CLINC150
ROSTD
Banking77
Snips
AUROC 1
FPR@95
AUROC
FPR@95 L
AUROC
FPR@95
AUROC1 f
FPR
MSP
95.7240.18
19.0840.55
75.42-0:05
51.244021
83.3540.20
56,2040032
79.1740.22
56.15-0.6
Energy
16.1810.12
15.7640.43
76.52-0.10
52.5340.32
82.6440.22
51.02
75. 75.10-0.32
40.64-0.63
ODIN
96.2010.11
15.9010.12
75.71-0.09
52.1540.33
83.05-10.4
50.7440.59
80.65
$1.34-0.61
D2U
96.2610.15
15.66-10.45
75.72-0:11
52.1440.39
83.08
10.19-10.5
80.65 10.36
51,33-0.66
MI.S
96.3640.13
16.40-1..44
76.54-0.11
52.35-0.33
82.62-10.22
50.6540.58
75.1 10.32
40.65-0:61
LOF
97.17-0.10
14.58-0.45
97.49-10.05
4.69 1:29-
92.73-10.12
10.25
94.13 1..21
13.37-0054
Maha
97.57-1..09
12.26-10.43
99.66
1.06--0221
92.64-0.15
41.54 10.31
94.33 94.33-0.18
13.82 -3.52
KNN
97.53-0.11
13.50.10.45
99.67-0.03
0.71 10.18
92.7440.15
42.04-0.22
94.44 10.19
13.38-0.54
GNOME
96.8440.14
14.94-0..55
99.63-10.10
99.63
1.47 1.47--022
91.43-0.000
44.23 10.24
92.58 10:25
14.45
FL.tS
97.80
9.90
99.83=
0.21 00:03
93.85
40.02
97.98#0.17
9.62
Table 1: OOD detection performance (higher AUROC 1 and lower FPR@95 J is better) on the 4 benchmark datasets.
All values are percentages averaged over 5 different random seeds, and the best results highlighted in bold.
20
18
wlo Pout Pout
pin uniform Maha KNN
Paut
KNN
16
Maha
14
uniform
12
10
MSP
Energy
ODIN
D2U
MLS
LOF
Mana
KNN
GNOME
Figure 2: Ablation Setting I: Average FPR @95 (%) for baselines on CLINC150 with (w/) or without (w/)) incorporation OOD density estimation.
Figure 3: Ablation Setting 2: Average FPR @95 (%)
for baselines on CLINCI50 with different IND OOD density estimation methods (uniform, Maha, KNN).
4.3 Ablation Settings
the superiority of our proposed FLatS.
FLatS enhances other baselines. Figure 2 shows the FPR 995 results CLINC150 under ablation setting We observe that all the baselines achieve lower FPR @95 results by incorporating OOD density estimation. Therefore, FLatS not only single method, but can serve as a general framework to improve other SOTA OOD methods.
Note that SFLas(x) in Equation (5) only an il- lustrative method based on KNN. The concept of principled likelihood ratio can be extended within a broader framework to develop more OOD scores. To comprehensively assess the potential of this idea, conduet two additional ablation studies: Setting 1: In this setting, we aim to enhance the existing baselines by incorporating OOD den- sity estimation. We replace Ein (a) in Equation (4) with baseline OOD scores. Meanwhile, we main- tain Four(a) (x) as Skow Daur), thus exploring the impact of incorporating OOD density estimation on performance improvement.
FLatS adopt different E,, and Eout- Fig- ure 3 shows the FPR results CLINC150 with different ways (uniform, Maha, KNN) es- timate Pn, and Po.. The results reveal that the incorporation of OOD distribution estimation (no matter KNN or Maha) beneficial compared to assuming Pour as uniform distribution.
Setting 2: In this setting, we aim study the ef- fects of different estimation methods for both OOD density Pour( and IND density (x)) Specif- ically, we replace (x)(x) and (x) in Equa- tion (4) with Suniform (x) = const., SMaha(x)) and (xwv(x).
Related Work
OOD detection is crucial for NLP applica- tions (Ryu et al., 2018; Borjali et al., 2021). In test time, the key difference of OOD detection methods is the OOD score design, which can be roughly categorized into two branches: confidence-basse -hased methods (Hendrycks and Gimpel, 2016; Liu et al., 2020; Hendrycks al., 2019), and distance-based methods (Lee al.. 2018; Sun et al., 2022: Bre- unig et al., 2000). Some textual OOD detection methods (Arora et al., 2021) also exploit perplexity of auto-regressive language models (Arora et al.,
5 Results and Analysis
FLatS establishes new SOTA. As shown in Ta- ble 1, FLatS achieves the best performance on the four benchmark datasets. The second best methods are KNN and Maha, whose average FPR @95 are 17.71% 1% and 17.41% %. They are higher than the aver- age FPR @95 of FLatS (14.94%), which confirms

---

2021). Leveraging auxiliary OOD data (collected public corpus or synthesized OOD data) for train- ing has been considered in literature (Xu et al., 2021; Wang et al., 2022). However, none of the works use auxiliary OOD data to estimate OOD dis- tribution Pour. which is a key novelty of our paper. More related work can be found in this excellent survey (Lang al., 2023).
efficiency of KNN in real-world applications has been proved in previous works (Ming et al., 2022; Yang et al.. 2022). Therefore, our method FLatS that adopts KNN is scalable and efficient in real- world applications.
Also, though we use public corpus for the estima- tion Pf in the experiments. FLatS is compatible with any desired OOD data when they are available. As FLatS does not require model re-training, it has great potential in test-time adaptation to tackle distribution shifting in real-world scenarios.
There are also some previous works (Ren et al., 2019; Xiao et al.. 2020) that use "likelihood ra- tio" to detect OOD samples. However, our FLatS framework is very different from these works in the following aspects: (1) They used probabilistic gen- erative models, e.g., VAËs (Kingma and Welling, 2013), estimate likelihood, which is hard to train and difficult to scale up (in visual domains), and (2) The "likeli- hood ratio" they used is not between Pour and Pm- and thus neither of them is a principled OOD de- tection method. For example, Ren et al. (2019) exploits a "background generative model" trained using random perturbation and Xiao et al. (2020) leverages variational posterior distribution for test samples. They can also be viewed as special cases of FLatS which are estimated with different proxy distributions.
Conclusion
This paper proposes to solve OOD detection with feature-based likelihood ratio score, which is princi- pled (juttffedd by Theorem I). The proposed FLatS is simple and effective, which not only establishes new SOTA, but can serve as a general framework to improve other OOD detection methods.
less effective in OOD detection.
Limitations
We list two limitations of this work. First, this paper mainly focuses on the post-hoc OOD detec- tion approaches. Post-hoc OOD detection meth- ods compute the OOD score without any special training-time regularization for models. Although a large group of OOD detection methods are post- hoc, there are also some regularized fine-tuning schemes to improve the OOD detection capability of NLP models. Since FLatS can enhance other post-hoc OOD detection baselines as shown in Sec- tion 5, it is exciting to see if our FLatS can also improve those training time techniques in the fu- ture work. Second, our proposed FLatS is based on the existing OOD score (KNN), and we do not propose any new score with novel estimation tech- niques. The main contribution of this paper is to solve OOD detection with principled likelihood ratio, and we will see if better scores can be devel- oped to further improve the likelihood ratio estima- tion in the future.
7 Discussion
the derivation our FLatS framework, We ex- ploit the energy-based models (EBMs) for param- eterization. EBMs are known for their flexibility with sacrifice to their tractability. But in our case, we leverage their flexibility to derive principled OOD scores (following Theorem 1) while keep the tractability via approximation with traditional OOD scores (e.g., KNN) in real-world applications. The detailed explanation is shown as follows.
Flexibility: Since Theorem 1 suggests that we should design OOD scores under the form of likelihood ratio between Powr and Pm,, adopt EBMs to model the two probability distri- butions Pout and Pin due to the flexibility of EBMs. Thanks to EBMs, we transform the computation of likelihood-ratii into two unnormalized energy func- tions (x) and (x) as shown in Section 3.2. Tractability: Contrary to the traditional works that directly optimize EBMs via MCMC (Grath- wohl et al., 2019; Lafon et al., 2023) which may face the problem of computational inefficiency, we approximate the energy functions using traditional feature-bact based OOD scores (KNN or Maha). The
Ethics Statement
Since this research involves only classification of the existing datasets which are downloaded from the public domain, we do not see any direct ethi- cal issue of this work. In this work, we provide a theoretically principled framework solve OOD detection in NLP, and we believe this study will lead to intellectual merits that benefit from a reli- able application of NLU models.

---

References
Zixuan Ke, Haowei Lin. Yijia Shao, Hu Xu, Lei Shu, and Bing Liu. 2022. Continual training of language models for few-shot learning. In Empirical Methods in Natural Language Processing (EMNLP).
Udit Arora, William Huang, and He He. 2021. Types of out-of-distribution texts and how to detect them. arXiv preprint arXiv:2109.0682. 068.7777..
Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu, 2023. Continual pre- training of language models. In The Eleventh Inter- national Conference on Learning Representations.
Christopher M Bishop. 1994. Novelty detection and neural network validation. IEE Proceedings viviiiiiinssiisonnn Image and Signal processing, 141(4):217-22
Alireza Borjali, Martin Magnéli, David Shin, Hen- rik Malchau, Orhun Muratoglu, and Kartik M Varadarajan. 2021. Natural language processing with deep learning for medical adverse event detection from free-text medical narratives: A case study of de- tecting total hip replacement dislocation. Computers in biology and medicine, 129:10414 104140.
Jacob Devlin Ming-Wer Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirec- tional transformers for language understanding. In Proceedings of NAACL-HLI, pages 4171-4186.
Diederik Kingma and Max Welling. 2013. Auto- encoding variational bayes. arXiv:1312.611
arXiv preprint
Markus M Breunig, Hans-Peter Kriegel, Raymond Ng, and lörg Sander. 2000. Lof: identifying density- based local outliers. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pages 93-104.
Marc Lafon, Elias Ramzi, Clément Rambour, and Nico- las Thome. 2023. Hybrid energy based model in the feature space for out-of-distribution detection. arXiv preprint arXiiv2330. 16966.
lñigo Casanueva, Tadas Temčinas. Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020. Efficient intent detection with dual sentence encoders. arXiv preprint Xrv: (eX:::2:030000 200.00077
Hao Lang, Yinhe Zheng, Yixuan Li, Jian Sun, Fei Huang, and Yongbin Li. 2023. A survey on out- of-distribution detection in nip. arXiv preprint arXiv:2305.03226.
Stefan Larson, Anish Mahendran, Joseph Peper. Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K Kummerfeld, Kevin Leach, Michael A Laurenzano, Lingjia Tang, et al. 2019, An evaluation dataset for intent classification and out-of-scope pre- diction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLL) pages 1311
Sishuo Chen, Wenkai Yang, Xiaohan Bi, and Xu Sun. 2023. Fine-tuning deteriorates general textual out- of-distribution detection by distorting task-agnostic features. arXiv preprint arXiv:2301. 12715.
Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Calta- girone, Thibaut Lavril, et al. 2018. Snips voice plat- form: embedded spoken language understanding system for -dyiiin voice interfaces. arXiv preprint arXiv: 100e:
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. 2018. A simple unified framework for detecting out- of of-distribution samples and adversarial attacks. Ad- vances in neural information processing systems, 31.
Varun Gangal, Abhinav Arora, Arash Einolghozati, and Sonal Gupta. 2020. Likelihood ratios and genera- tive classifiers for unsupervised out-of-domain de- tection in task oriented dialog. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 34, pages 7764-7771.
Shiyu Liang. Yixuan Li, and Rayadurgam Srikant. 2017. Enhaneing the reliability of out-of-distriutiio im- age detection in neural networks. arXiv preprint arXiv: 7706.02690
Will Grathwohl, Kuan-Chieh Wang, Jörn-Herii Ja- cobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. 2019. Your classifier is secretly an energy based model and you should treat it like one. arXiv preprint arxiv.19/2.03263.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan 2020. Energy- duitti--ditiiiuttion detection. Advances in Neural Information Processing Systems,
33:21464-21475.
Yifei Ming, Yiyou Sun, Ousmane Dia, and Yixuan Li. 2022. How to exploit hyperspherical embeddings for out-of-distribution detection? arXiv preprint M:Xiv:2203.04450
Dan Hendrycks, Steven Basart, Mantas Mazeika, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. 2019. Scaling out-of-distribttion de- tection for real-world settings. arXiv preprint arXiv:1911.11132.
Jerzy Neyman and Egon Sharpe Pearson. 1933. Ix. on the problem of the most efficient tests of statis- tical hypotheses. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, (694 706):289-337.
Dan Hendrycks and Kevin Gimpel. 2016. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.0213..

---

Jie Ren, Peter Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. 2019. Likelihood ratios for out- of-distribution detection. Advances in neural infor- mation processing systems, 32.
statistic T(X), which is a function of the data a. and rejection region R, which is subset of the range of If the observed value i of T falls in R. we reject Ho-
Definition 3 (UMP test) Denote the power func- tion (R(0) (x( & R),, where Pe denotes the probability measure when O is the true parame- ter. A test with test statistic T and rejection region R is called uniformly most powerful (UMP) test at significance level if it satisfies conditions:
Seonghan Ryu, Sangjun Koo, Hwanjo Yu, and Gary Ge- unbae Lee. 2018. Out-of-domain detection based on generative adversarial network. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 714-718.
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. 2022. Out-of-distrbbution detection with deep nearest neighbors. arXiv preprint arXiv:2204.06507
1. supaee, 3T S a.
2. v0 11 PR (0) 2 Bre(() for every other test T' with rejection region R' satisfying the first condition.
Mengyu Wang. Yijia Shao, Haowei Lin, Wenpeng Hu, and Bing Liu. 2022. Cmg: A class-mixed generation approach -ut-of-distribution detection. Proceed- ings of ECML/PKDD-2002.
A.2 Proof of Theorem
Zhisheng Xiao, Qing Yan, and Yali Amit. 2020. Like- lihood regret: An out-of-distribution detection score for variational auto-encoder. Advances in neural in- formation processing systems, 14-20685-20699
Lemma 1 (Neyman and Pearson, 1933) Let [X1, X2..., Xn) be a random sample with like- lihood function L(0). The UMP test of the simple hypothesis Hn 8 fg against the simple hypo- thetis Ha : = Og at level a has a rejection region of the form:
Keyang Xu, Tongzheng Ren, Shikun Zhang, Yihao Feng, and Caiming Xiong. 2021. Unsupervised out-of-domain detection via pre- pre-trained transform- ers. arXiv preprint 1rXiv:216600094
L(Og)
< k
L(0.)
Jingkang Yang, Pengyun Wang, Dejian Zou, Zitang Zhou, Kunyuan Ding, Wenxuan Peng, Haoqi Wang,
where is chosen so that the probability of type error is a.
Guangyao Chen, Li, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, Dan Hendrycks, Yix- uan Li, and Ziwei Liu, 2022. Openood: Benchmark- ing generalized diitr-idistriutiion detection.
Now the proof of Theorem is straightforward.
From Lemma 1, the UMP test for Equation (3) has rejection region of the form:
Andi Zhang and Damon Wischik. 2022. Falsehoods that ml researchers believe about ood detection. arXiv preprint 2rXiv:2210.12766
Pout (x)
λo
Pin (x)
Jingyang Zhang, Jingkang Yang, Pengyun Wang, Haoqi Wang, Yueqian Lin, Haoran Zhang, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, et al. 2023. Openood vl. 5: Enhanced benchmark for out-of-distribution detection. arXiv preprint arXiv:2306.0930..
where An is is chosen that the probability of type I error is a.
A.3 UMP test achieves optimal AUROC
In this section, we show that the UMP test for Equa- tion (3) also achieves the optimal AUROC, which popular metric used in OOD detection. From the definition of AUROC, we have:
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhut- dinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In The IEEÉ International Con- ference on Computer Vision (ICCV).
AUROC ('' FPR d(TPR)
Theoretical Analysis of Theorem
(r(0mm) - 33z (Oout)
A.1 Preliminary
[ (33r((()))
Definition 2 (Statistical hypothesis testing) Consider testing null hypothesis Ho : 0 E Oo against an alternative hypothesis H1 A O₁, where Oo and O₁ are subsets of the parameter space and O, n 1, = A test consists of a test
where FPR and TPR are false positive rate and true positive rate. Therefore, an optimal AUROC requires UMP test of any given level a = P except on a null set.

---

B Distance-Basel OOD Detectors are IND Density Estimators
CLINC150 (Larson et al., 2019) is dataset specifically designed for OOD intent detection. It comprises 150 individual intent classes from di- verse domains. The dataset contains a total of 22,500 IND queries and 1,200 OOD queries. The IND data is split into three subsets: 15,000 for training, 3,000 for validation, and 4,500 for testing. Additionally, the dataset includes 1,000 carefully curated OOD test data for evaluating performance on -ft-of-domain queries.
In this section, we will show that (Maha(z) and SkNv(æ) defined in Equation (1) and Equation (2) are IND density estimators under different sumptions. Assume have a feature encoder Φ : X' + RT, and in training time we empirically observe n IND samples (0(21)) ((x))...((x.
Analysis of Maha score. Denote E to be the covariance matrix of ((x). The final feature we extract from data is:
ROSTD (Gangal et al., 2020) large-scale in- tent classification dataset comprising 43,000 intents distributed across 13 intent classes. The dataset also includes carefully curated OOD intents. Fol- lowing the dataset split, we obtain 30,521 samples for IND training, 4,181 samples for IND validation, 8,621 samples for IND testing, and 3,090 samples for OOD testing.
xx) A ())))
where AAT Σ. Note that the covariance of is I.
Given a class label e, we assume the distribution (xxcc) follows a gaussian ((1 Mee I). Immedi- ately we have u to be the class centroid for class e under the maximum likelihood estimation. We can now clearly address the relation between Maha score and IND density:
Banking77 (Casanueva al., 2020) is fine- grained intent classification dataset focused on the banking domain. It consists of 9,003 user queries in the training set, 1,000 queries in the validation set, and 3,080 queries in the test set. The dataset encompasses 77 intent classes, of which 50 classes are used as IND classes, while the remaining 22 classes designated as OOD classes.
SMaha (x) max (In (=)()) - m In 2n cE)
Analysis of KNN score. We use normalized fea- ture z <<>)\/|<<<>>)] for OOD detection. The probability function can be attained by:
Snips (Coucke et al., 2018) is dataset con- taining annotated utterances gathered from diverse domains. Each utterance is assigned an intent la- bel such as "Rate Book", "Play Music", or "Get Weather." The dataset encompasses 7 intent classes, of which 5 classes are used as IND classes, and the remaining 2 classes are used OOD classes. After splitting the dataset, obtain 9,361 IND training samples, 500 IND validation samples, 513 IND test samples, and 187 OOD test samples.
p(z' B(z,r))
Pmn(2)= = lim + r)
where B(z,r) = {z' 1|z'==11 I TA 1 - 1} Assuming each sample z(x;)) is i.i.d with probability mass 1/n, the density can be esti- mated by k-NN distance. Specifically,
= ANN(=)]l2. p(z'= B(z,r)) k/n and x (m 1)/2
D Hardware and Software
B(2,r)
-( +) n p- + m-1mm11) where
We run all the experiments on NVIDIA GeForce RTX-2080Ti GPU. Our implementations are based on Ubuntu Linux 16.04 with Python 3.6.
I is Euler's gamma function. When is large and k/n is small, we have the following equations:
AT(M1 m- + 1) (m 1)/2mppm- π
(xn(x) ~
2
AI m-1 +1) T (mm-1)/2
SkNN(x)) x
m- (pm(x))
C Datasets Details
We use four publicly available intent classification datasets as benchmark datasets:


Reviews:
[{"review_id": "UYRIg2cONc", "reviewer": "Reviewer_UcFx", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer UcFx,\n\nWe sincerely appreciate the time and effort you have devoted to evaluating our paper! We are grateful for the constructive feedback provided and would like to address the concerns raised in the review as follows. \n\n>  The method's reliance on likelihood ratio-based OOD scores, as justified by Theorem 1, warrants closer scrutiny. \n\nThanks for your suggestion. Theorem 1 establishes the sound theoretical foundation of our approach, demonstrating its robustness. The theorem indicates that the likelihood ratio-based OOD scores are principled, and our empirical study further demonstrates the superiority. We will clarify on this to ensure a rigorous scrutiny in our revised paper.\n\n> The application of energy-based models (EBMs) for parameterization introduces an inherent complexity, raising questions about the scalability and efficiency of the proposed framework, particularly in real-world applications.\n\nWe acknowledge the concern regarding the use of energy-based models (EBMs) for parameterization. EBMs are known for its **flexibility** with sacrifice to its **tractability**. But actually, we exploit its **flexibility** to derive principled OOD scores following Theorem 1 while keep the **tractability** via approximation with traditional OOD scores (e.g., KNN) in real-world applications. The detailed explanation is shown as follows.\n\n**Flexibility**: Since Theorem 1 suggests that we should design OOD scores under the form of likelihood ratio between $\\mathcal P_{out}$ and $\\mathcal P_{in}$, we adopt energy-based models (EBMs) to model the two probability distributions $\\mathcal P_{out}$ and $\\mathcal P_{in}$ due to the **flexibility** of EBMs. Thanks to EBMs, we transform the computation of likelihood-ratio $p_{out}(x)/p_{in}(x)$ into two unnormalized energy functions $E_{out}(x)$ and $E_{in}(x)$ (see line 158-174).\n\n**Tractability**: Contrary to the traditional works that directly optimize EBMs via MCMC [1, 2] which may face the problem of intractability, we approximate the energy functions using *traditional feature-based OOD scores* (KNN, Maha, or Uniform. see line 248-252). The efficiency of KNN in real-world applications has been proved in previous works [3, 4, 5]. Therefore, our method FLATS that adopts KNN is scalable and efficient in real-world applications. \n\nThanks for this valuable feedback. We will include this discussion in our revised paper which will significantly improve the clarity and highlight the contribution of our paper.\n\n> The presented comparison with the KNN and Maha approaches is indicative, but a more detailed analysis of the strengths and weaknesses of FLATS in comparison to a broader spectrum of methods, including both traditional and contemporary alternatives, would contribute to a more nuanced understanding of its efficacy.\n\nWe acknowledge the need for a more comprehensive analysis of FLATS against a wider array of methods. Apart from the quantitative analysis in section 5, it's important to add more qualitative discussion. We will take this advice to add detailed analysis of the strengths and weaknesses of FLATS in our revised paper. Since we cannot edit the paper at this stage, we provide a brief summary of the strengths and weaknesses as follows:\n\n**Strengths**: \n\n- **Effectiveness:** Our approach capitalizes on feature-based information, a facet overlooked by both traditional and contemporary methods like MSP, Energy, ODIN, MLS. Research underscores the superiority of feature-based OOD detection techniques [4].\n- **Principled**. Traditional and contemporary feature-based methods KNN, Maha, LOF, GNOME are not principled as analyzed in line 123-128. FLATS is grounded in theoretical rigor, making it a more principled option.\n- **Flexibility:** While contemporaneous methods tend to be specialized and intricate, our FLATS framework is characterized by its simplicity and broad applicability, as evidenced in section 5. For example, GNOME devises to aggregate normalized scores produced by different models, and it has a trade-off between distribution shift types.\n\n**weaknesses**:\n\n- **Post-hoc only**. It's worth noting that our OOD detection methodology operates in a post-hoc manner, without specific training-time regularization. We are intrigued by the potential for future work to explore integrating training-time techniques into the FLATS framework.\n- **Auxiliary Data Selection**. Some contemporary methods such as [6] designed advanced techniques to choose auxiliary OOD data. While currently FLATS did not elaborate on this, it's exciting to see if this will further enhance detection performance.\n\n[1] Lafon, Marc, et al. \"Hybrid Energy Based Model in the Feature Space for Out-of-Distribution Detection.\" (ICML 2023)\n\n[2] Grathwohl, Will, et al. \"Your classifier is secretly an energy based model and you should treat it like one.\" (ICLR 2020)\n\n[3] Sun, Yiyou, et al. \"Out-of-distribution detection with deep nearest neighbors.\" (ICML 2022)\n\n[4] Yang, Jingkang, et al. \"Openood: Benchmarking generalized out-of-distribution detection.\" (NeurIPS 2022)\n\n[5] Ming, Yifei, et al. \"How to Exploit Hyperspherical Embeddings for Out-of-Distribution Detection?.\" (ICLR 2023)\n\n[6] Ming, Yifei, et al. \"Poem: Out-of-distribution detection with posterior sampling.\" (ICML 2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper's strength lies in its rigorous approach to addressing out-of-distribution (OOD) detection, introducing the well-grounded feature-based likelihood ratio score substantiated by Theorem 1. The extension of FLATS to diverse feature-based OOD scores highlights its versatility and departure from existing methods, showing potential for real-world applications.\n\n2. The experimental results underscore its contributions, as FLATS establishes itself as the new OOD detection state-of-the-art (SOTA), surpassing benchmarks like KNN and Maha. Experiments with various Pin and Pout estimation approaches further emphasize FLATS' adaptability and robustness, significantly boosting the paper's impact on the NLP community.", "reasons_to_reject": "1. The method's reliance on likelihood ratio-based OOD scores, as justified by Theorem 1, warrants closer scrutiny. The application of energy-based models (EBMs) for parameterization introduces an inherent complexity, raising questions about the scalability and efficiency of the proposed framework, particularly in real-world applications.\n\n2. The presented comparison with the KNN and Maha approaches is indicative, but a more detailed analysis of the strengths and weaknesses of FLATS in comparison to a broader spectrum of methods, including both traditional and contemporary alternatives, would contribute to a more nuanced understanding of its efficacy.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper addresses the issue of detecting out-of-distribution (OOD) instances within the realm of Natural Language Processing (NLP) models. The paper's primary contributions are centered around the introduction of a novel methodology designed to assess the \"OOD-ness\" of a given test case. This is achieved through the application of a likelihood ratio, which juxtaposes the probabilities associated with out-of-distribution and in-distribution contexts. The paper posits that current state-of-the-art (SOTA) methods exhibit suboptimal performance due to their exclusive focus on estimating in-distribution density. In response, the paper presents a principled solution for OOD detection rooted in the concept of a likelihood ratio, denoted as FLATS. This framework holds the potential to serve as a versatile foundational structure, effectively bolstering various other OOD detection methodologies through the incorporation of out-distribution density.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YLAuVQWLr8", "reviewer": "Reviewer_RnjR", "summary": "", "questions": "1. Have you conducted experiments on other backbones or other than NLP classification tasks?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer RnjR,\n\nWe sincerely appreciate the time and effort you have devoted to evaluating our paper! We are grateful for the constructive feedback provided and would like to address the concerns raised in the review as follows. \n\n> Experimental validation is weak, only considering Roberta as the backbone.\n\nThanks for this suggestion! We also tried other backbones to validate our proposed FLATS approach and the results (AUROC / FPR@95) are shown as follows.\n\n|            | Bert-base-uncased   | Bert-large-uncased  | Roberta-base        | Roberta-large       | T5-3b (LoRA)        | T5-11b (LoRA)       | LLaMA-3b (LoRA)     | LLaMA-7b (LoRA)     |\n| ---------- | ------------------- | ------------------- | ------------------- | ------------------- | ------------------- | ------------------- | ------------------- | ------------------- |\n| #parameter | 110M                | 340M                | 125M                | 335M                | 3B                  | 11B                 | 3B                  | 7B                  |\n| KNN        | 97.3 / 13.5         | 98.0 / 8.00         | 97.5 / 13.5         | 98.1 / 7.50         | 98.1 / 8.90         | 98.3 / 7.01         | 97.0 / 13.6         | 97.5 / 12.3         |\n| FLATS      | **97.6** / **9.70** | **98.2** / **6.70** | **97.8** / **9.90** | **98.3** / **6.50** | **98.4** / **6.45** | **98.8** / **5.90** | **97.8** / **9.90** | **98.2** / **7.15** |\n\n**Table 1. The AUROC$\\uparrow$ (left) and FPR@95$\\downarrow$ (right) results of KNN and FLATS on CLINC150 with different backbones.**\n\nHere we include Bert-base-uncased, Bert-large-uncased, Roberta-base, Roberta-large, T5-3b, T5-11b, LLaMA-3b, LLaMA-7b as the tested backbones. We conducted experiments on CLINC150 dataset as introduced in Appendix Section C. For LLMs with sizes larger than 3B, we adopt LoRA [1] to do parameter-efficient tuning to avoid out-of-memory issue. From the table we could see, **FLATS consistently improves the OOD detection performance across different backbones**, which demonstrate its empirical superiority.\n\n>  Have you conducted experiments on other backbones or other than NLP classification tasks?\n\nYes. The experiments on other backbones are shown in the response to your last comment. \n\nWe also tried NLP **non-classification** tasks and results are shown in **Table 2** below. While classification tasks are the most popular settings for OOD detection in NLP, we also noticed that there are works [2] focusing on non-classification tasks. In this setting, no class label is available and we can only train the model via unsupervised learning on IND data. We follow [2] to conduct experiments on SST and CLINC150 datasets with **Roberta-base** as the backbone. EDF, MDF, MDF (BCAD), MDF (IMLM), MDF(BCAD, IMLM) are methods proposed by [2]. Notice that other baselines in our paper are not suitable for non-classification settings, thus only KNN and FLATS are compared as they don't need class label in inference. To adopt KNN and FLATS to non-classification setting, the backbone is trained using unsupervised mask-language-model (MLM) loss on IND data.\n\n|                         | EDF  | MDF  | MDF (BCAD) | MDF (IMLM) | MDF (BCAD, IMLM) | KNN  | FLATS     |\n| ----------------------- | ---- | ---- | ---------- | ---------- | ---------------- | ---- | --------- |\n| SST (unsupervised)      | 99.5 | 99.8 | 99.2       | 99.9       | 99.9             | 99.7 | **100.0** |\n| CLINC150 (unsupervised) | 56.9 | 78.6 | 80.5       | 80.1       | 84.4             | 89.3 | **92.9**  |\n\n**Table 2. The AUROC$\\uparrow$ results of different unsupervised OOD detection methods on SST and CLINC150 with Roberta-base.**\n\nFrom the table we can see, our FLATS also establishes a new state-of-the-art on non-classification NLP tasks. As non-classification tasks have become more and more popular in the LLM-era, we believe that the proposed FLATS has the potential to be widely adopted in many real-world applications.\n\nWe genuinely appreciate your feedback, which has prompted us to develop a more comprehensive analysis of FLATS' comparative strengths, thereby elevating the overall quality and depth of our revised manuscript. If you have further questions, don't hesitate to contact us.\n\n[1] Hu, Edward J., et al. \"Lora: Low-rank adaptation of large language models.\" (ICLR 2021).\n\n[2] Xu, Keyang, et al. \"Unsupervised out-of-domain detection via pre-trained transformers.\" (ACL 2021).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is well-written and organized.\n2. This paper has contributions from both theoretical and empirical perspectives: Not only provides a theoretical understanding and analyses of previous methods but also proposes a novel method that outperforms previous methods.", "reasons_to_reject": "1. Experimental validation is weak, only considering Roberta as the backbone.", "questions_to_authors": "1. Have you conducted experiments on other backbones or other than NLP classification tasks?", "paper_topic_and_main_contributions": "OOD detection is an important topic but most of the previous works are empirical. This paper demonstrates a theoretical understanding that uses the likelihood ratio between out-distribution and in-distribution to measure the extent of OOD. Based on this, the authors claim that SOTA methods are suboptimal because they only estimate the in-distribution density. The authors propose a method named FLATS, which outperform previous methods on several OOD detection benchmark.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "no", "Typos_Grammar_Style_and_Presentation_Improvements": "no"}, "extra_text": {}}, {"review_id": "A7GhRxaWJY", "reviewer": "Reviewer_RYgf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer RYgf,\n\nWe sincerely appreciate the time and effort you have devoted to evaluating our paper! We are grateful for the constructive feedback provided and would like to address the concerns raised in the review as follows. \n\n> The rationale behind opting for RoBERTa, as opposed to other language models with varying parameters like T5-3b, 11b, llama-3b, and 7b. It is crucial to present a comprehensive comparison with these different language models.\n\nThanks for this suggestion! We also tried other backbones to validate our proposed FLATS approach and the results (AUROC / FPR@95) are shown as follows.\n\n|            | Bert-base  | Bert-large | Roberta-base        | Roberta-large       | T5-3b (LoRA)        | T5-11b (LoRA)       | llama-3b (LoRA)     | llama-7b (LoRA)     |\n| ---------- | ------------------- | ------------------- | ------------------- | ------------------- | ------------------- | ------------------- | ------------------- | ------------------- |\n| #params | 110M                | 340M                | 125M                | 335M                | 3B                  | 11B                 | 3B                  | 7B                  |\n| KNN        | 97.3 / 13.5         | 98.0 / 8.00         | 97.5 / 13.5         | 98.1 / 7.50         | 98.1 / 8.90         | 98.3 / 7.01         | 97.0 / 13.6         | 97.5 / 12.3         |\n| FLATS      | **97.6** / **9.70** | **98.2** / **6.70** | **97.8** / **9.90** | **98.3** / **6.50** | **98.4** / **6.45** | **98.8** / **5.90** | **97.8** / **9.90** | **98.2** / **7.15** |\n\n**Table 1. The AUROC$\\uparrow$ (left) and FPR@95$\\downarrow$ (right) results of KNN and FLATS on CLINC150 with different backbones.**\n\nHere we include Bert-base, Bert-large, Roberta-base, Roberta-large, T5-3b, T5-11b, LLaMA-3b, LLaMA-7b as the tested backbones. We conducted experiments on CLINC150 dataset as introduced in Appendix Section C. For LLMs with sizes larger than 3B, we adopt LoRA [1] to do parameter-efficient tuning to avoid out-of-memory issue. From the table we could see, **FLATS consistently improves the OOD detection performance across different backbones**, which demonstrate its empirical superiority.\n\n> In Section 5, it is recommended to provide a short explanation for the superiority of this method.\n\nThanks for this advice. Apart from the quantitative analysis in section 5, we acknowledge that it's important to add more qualitative discussion. We will take this advice to add detailed analysis of the strengths of FLATS in our revised paper. Since we cannot edit the paper at this stage, we provide a brief summary of the superiority of FLATS as follows:\n\n- **Effectiveness:** Our approach capitalizes on feature-based information, a facet overlooked by both traditional and contemporary methods like MSP, Energy, ODIN, MLS. Research underscores the superiority of feature-based OOD detection techniques [2].\n- **Principled**. Traditional and contemporary feature-based methods KNN, Maha, LOF, GNOME are not principled as analyzed in line 123-128. FLATS is grounded in theoretical rigor, making it a more principled option.\n- **Flexibility:** While contemporaneous methods tend to be specialized and intricate, our FLATS framework is characterized by its simplicity and broad applicability, as evidenced in section 5. For example, GNOME devises to aggregate normalized scores produced by different models, and it has a trade-off between distribution shift types.\n\nWe genuinely appreciate your feedback, which has prompted us to develop a more comprehensive analysis of FLATS' comparative strengths, thereby elevating the overall quality and depth of our revised manuscript. \n\n[1] Hu, Edward J., et al. \"Lora: Low-rank adaptation of large language models.\" (ICLR 2021).\n\n[2] Yang, Jingkang, et al. \"Openood: Benchmarking generalized out-of-distribution detection.\" (NeurIPS 2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": " they propose FLATS, which can incorporate out-distribution density.", "reasons_to_reject": "1) The rationale behind opting for RoBERTa, as opposed to other language models with varying parameters like T5-3b, 11b, llama-3b, and 7b. It is crucial to present a comprehensive comparison with these different language models.\n2) In Section 5, it is recommended to provide a short explanation for the superiority of this method.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "topic: detecting out-of-distribution (OOD) instances in NLP models\nmain contribution: they think the Sota feather-based OOD detection methods such as MAha, and KNN is suboptimal since they only estimate in-distribution density, to address this issue, they propose FLATS, which can incorporate out-distribution density.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LChcaDLGVm", "reviewer": "Reviewer_4HcV", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 4HcV,\n\nWe sincerely appreciate the time and effort you have devoted to evaluating our paper! We are grateful for the constructive feedback provided and would like to address the concerns raised in the review as follows. \n\n> The relation between this work and \"Likelihood Ratios for Out-of-Distribution Detection\" should be discussed. Likelihood ratio is used in both of these works.\n\nThanks for this advice! We acknowledge that it's important to discuss the relation between our paper and \"Likelihood ratios for OOD detection\" to improve clarity of our contributions and avoid potential misunderstandings. \n\nFirstly, there are some prior works [1-2] that use \"likelihood ratios\" to detect OOD samples. These works shared some similarity with ours:\n\n1. They also noticed that the likelihood of IND distribution alone is not a good OOD score.\n\n2. They also adopted \"likelihood ratio\" to address it, though the likelihood ratio is not between $\\mathcal P_{out}$ and $\\mathcal P_{in}$ as we did.\n\nHowever, our paper is very different from these works in the following aspects:\n\n1. These works used **probabilistic generative models** to estimate likelihood, which is hard to train and difficult to scale up (in visual domains), and less effective in OOD detection. \n2. The \"likelihood ratio\" they used is not between $\\mathcal P_{out}$ and $\\mathcal P_{in}$, and thus **neither of them is a principled OOD detection method**. For example, [1] exploits a \"background generative model\" trained using random perturbation and [2] leverages a variational posterior distribution for test samples. They can also be viewed as special cases of FLATS where $\\mathcal P_{out}$ is estimated with different proxy distributions.\n\nWe will add this discussion in our revised paper. We genuinely appreciate your feedback, which has prompted us to develop a more comprehensive analysis of FLATS' comparative strengths, thereby elevating the overall quality and depth of our revised manuscript. \n\n[1] Ren, Jie, et al. \"Likelihood ratios for out-of-distribution detection.\" (NeurIPS 2019)\n\n[2] Xiao, Zhisheng, et al. \"Likelihood regret: An out-of-distribution detection score for variational auto-encoder.\" (NeurIPS 2020)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Backed by theoretical analysis, this paper introduce a novel principled OOD detection method FLATS that measures OOD-ness of an input through the likelihood ratio between OOD distribution and IND distribution.\n\n2. Moreover, FLATS can serve as a general framework which is able to enhance other OOD detection methods by considering OOD density estimation.\n\n3. A new SOTA on popular benchmarks is established. And the experiments and analysis are comprehensive and convincing.", "reasons_to_reject": "1. The relation between this work and \"Likelihood Ratios for Out-of-Distribution Detection\" should be discussed. Likelihood ratio is used in both of these works.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes FLATS, a principled method for OOD detection based on likelihood ratio. FLATS is shown to enhance other OOD detection methods by incorporating out-distribution density estimation.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This papers tackles the problem of out-of-distribution detection. The reviewers appreciated the combination of theoretical foundations and empirical evaluations that outperform the proposed baselines on the given datasets. The authors were very responsive to the questions and feedback from the reviewers and the additional insights should certainly improve even more this paper.
Decision: Accept-Main