Beyond the Known: An Unknown-Aware Large Language Model for Open-Set Text Classification
Authors: Xi Chen
Chuan Qin
Ziqi Wang
Shasha Hu
Chao Wang
Hengshu Zhu
Hui Xiong
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Open-set text classification
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: An Unknown Aware LLM for Open-Set Text Classification

Abstract: Open-set text classification (OSTC) requires models to correctly classify in-distribution (ID) samples while reliably rejecting out-of-distribution (OOD) inputs—an essential capability for real-world NLP systems. Most OSTC methods train on ID data under the closed assumption that all outputs belong to the known label space and then perform OOD detection with the biased representations, which inherently lack awareness of unknowns and thus yield overconfident predictions on OOD inputs. In this work, we present UnLLM, an Unknown-aware Large Language Model for OSTC. Instead of fixing classification to the entire known label space, we reformulate it into a subset-conditioned text generation task: the LLM is prompted with sampled subsets of known labels, and any instance outside the candidate set is explicitly assigned as “unknown”. This reformulation transforms OOD detection from a post-hoc procedure into an intrinsic modeling capability. More importantly, our approach is the first to explicitly incorporate the unknown into classification, enabling systematic modeling of unknowns through a unified representation–logits–inference optimization, which progressively strengthens the model’s capacity to capture open-set risk. Extensive experiments across six benchmarks show that UnLLM consistently outperforms state-of-the-art (SOTA) baselines. Code and datasets are available at https://github.com/cx9941/UnLLM.

Published as a conference paper at ICLR 2026
BEYOND THE KNOWN: AN LIKNOWN-AWARE LARGE LANGUAGE MODEL FOR OPEN-SET TEXT CLASSIFI- CATION
Xi Chent:2, Chuan Qin Ziqi Wang', Shasha Hu', Chao Wang", Hengshu Zhu Hui Xiong School of Computer Science and Technology, University of Science and Technology of China
2 Computer Network Information Center, Chinese Academy of Sciences
3 University of Chinese Academy of Sciences
School of Artificial Intelligence and Data Science, University of Science and Technology of China
5 AI Thrust, The Hong Kong University of Science and Technology (Guangzhou)
6 Department of Computer Science and Engineering.
The Hong Kong University of Science and Technology, Hong Kong SAR (chenxi040) wzq1 42857, shashahu) [chuanqin@42) zhuhengshu]@@gaail com, wangchaoal @ustc. edu. cn, xionghui@ust.hk
austc.edu...
ABSTRACT
Open-set text classification (OSTC) requires models to correctly classify in- distribution (ID) samples while reliably rejecting out-of-distribution (OOD) in- puts an essential capability for real-world NLP systems. However, existing approaches largely follow the post-hoc OOD detection paradigm after a closed- world training, optimizing predictive distributions solely over known label spaces and thus producing overconfident and biased predictions on OOD inputs. this work, present UnLLM, an Unknown-awarc Large Language Model for OSTC. Instead of fixing classification to the entire known label space, we re- formulate it into subset-conditioned classification task: the LLM is prompted with sampled subsets of known labels, and any instance outside the candidate set is explicitly assigned an "unknown" class. This reformulation transforms OOD detection from a post-hoc procedure into an intrinsic modeling capability. Grounded in this formulation, the modeling of the "unknown" further systemati- cally realized through unified tion, which progressively strengthens the model's capacity to capture open-set risk. Extensive experiments across six benchmarks show that UnLLM consistently outperforms state-of-the-art (SOTA) basclines. Code and datasets are available at https: thub. com cX9941/UnLLM
optimiza-
INTRODUCTION
Text classification a cornerstone task in natural language processing (NLP), underpinning diverse applications such as topic categorization (Prakhya al.] (017). document management (Shu al., 2017). and intent recognition (Zhou et al., 2023] Qin et al. 2025b). Most conventional models, however, operate under closed-world assumption (Fei Liu| 2016). presuming that all classes en- countered at inference time are known during training. This unrealistic assumption restricts the ap- plicability of these models in real- world, dynamic scenarios where encountering OOD instances is common. Consequently, the paradigm of OSTC (Prakhya et al. 2017) emerged, necessitating mod- els capable of accurately classifying ID samples and concurrently identifying OOD samples.
A common approach is to first train neural network on ID data and then apply post-hoc OOD detection techniques such as MSP (Hendrycks & Gimpel| 2016) and OpenMax (Bendale & Boult 2016). More recent efforts fine-tune pretrained language models (PLMs) to establish compact deci- sion boundaries through contrastive and prototype-based learning, as in ADB (Zhang et al. 202I).
"Correspondigg Authors

---

Published as a conference paper at ICLR 2026
Imput
ID Label
OOD Label
Sub Label Set
K-dimension Classifier
1LM Head with closed-sct training
LLM Head with training
OOOO0O
COOOOOO
C
Input
Text
Input
OOD
Friliing
00000
Eatroyy
Loss
Aufogromive
Gencrlive
Astograsiiv
Generztive
Proi-bre OOD Dolectios
Post-bee OOD Deteeiion
clussify, Input
Inforence
6) Closed-sct disrrminative finc-tuning
b) Closed-set generative fine-tuning
Open-set generative fine-tuming
Figure/ 1: Illustration of different fine-tuning methods. Representation visualizations show signifi- cant overlap between ID and OOD (red) in a) and b), whereas c) demonstrates clear separation.
KNNCon (Zhou et al. 2022), and CLAP (Liu et al. 2023). However, these PLM-based methods following discriminative paradigm often produce narrowly concentrated embeddings that hinder OOD detection. To address this issue, LLM-OOD (Liu et al.) 2024) introduced generative fine- tuning framework that reformulates classification as text generation and leverages final-token repre- sentations, which yield more distinguishable embeddings, improving OOD separability.
Nevertheless, both discriminative and generative fine-tuning remain confined to the closed-set paradigm, where models are optimized exclusively on ID labels and lack explicit open- set super- vision (Scheirer et al. 2012). In computer vision, several methods (c.g.. VOS (Du et al. 2022), NPO (Tao et al. ))23)) attempt to synthesize virtual outliers to regularize decision boundaries. How- ever, these synthetic outliers cannot be guaranteed to represent true OOD instances -articcuaart under sparse ID goverage-introdueee
label noise and constraining generalization. This limitation raises a critical question: Is genuine open-set training-whers OOD detection is integrated with guaranteed-correct
unaltainable?
To explore this question, we first conduct systematic analysis and visualization of the differences between the two existing
airffigmm-dddiiiiinaiiiie and generative fine-tuning-as shown in Fig- ure a and [Jb. This analysis yields two key insights. First, the generative fine-tuning paradigm inherently benefits from a broader output space in the LLM'' classification head. This expanded di- mensionality, combined with the extensive general knowledge acquired during pretraining, enables the model to learn more generalized and contextually rich representations, thereby improving its resilience to open-set risk. However, generative fine-tuning still assumes that the label space is iden- tical to the entire label set, restricting the model to optimize final-token representations only over ID label tokens. The gap between label spaces at training and testing further degrades predictive per- formance, raising a natural idea: If we explicitly model this gap during training, can we effectively eliminate such bias? Motivated by this, are the first to propose an open-set training paradigm tailored for LLMss. Specifically, by conditioning the L.MM on partial subsets of labels, we construct conditional-OOl -OD candidate label set. This formulation enables the model to explicitly perceive open-set risk during training. Unlike prior methods that treat OOD detection as a post-hoc decision process, our ap- proach opens the LLM head to directly optimize parameters associated with OOD-relevant tokens. This design allows the model to go beyond implicit uncertainty estimation and explicitly model the semantic space of the unknown, thereby enhancing its capacity to recognize OOD inputs.
commples-instancee whose ground-truth labels are deliberately excluded from the
However, this design still faces several key challenges in practice: (1) Distribution gap between conditional-OOD and real OOD samples: Conditional-OOD samples are constructed from known classes with clear semantic boundaries. In contrast, real OOD samples often lie near the decision boundaries between multiple known classes. (2) Misalignment between internal knowledge and output: Even though "open-set training" can separate ID and OOD samples at the representation level, standard generative strategies may fail to faithfully reflect the model's internal knowledge. (3) Overconfidence in label-similar OODs: LLMs often assign overly confident predictions to OOD inputs that semantically close to known labels, leading erroneous decisions.
2

---

Published as a conference paper at ICLR 2026
To overcome these, we propose UnLLM based open-set training paradigm that strengthens the modeling of the unknown across three tightly connected levels: (1) Representation model- ing, where open-set generative fine-tuning, combined with contrastive learning and orthogonality constraints, produces compact ID embeddings and well-separated known/unknown representations, thereby reducing the distribution gap; (2) Probability calibration, which leverages OOD parameter calibration to align internal activations with unknown semantics, ensuring consistency between in- ternal knowledge and probabílistic outputs; and (3) Reflective inference, which employs analogy- augmented self-reflection during inference to mitigate overconfidence on semantically confusing OOD cases. Comprehensive experiments conducted across six benchmark datasets demonstrate that our proposed UnLLM, consistently surpasses SOTA baselines. Additional visualization analyses further substantiate our model's capability to distinctly separate ID and OOD representations, suc- cessfully addressing the persistent bottleneck posed by "closed-sel training" paradigms.
2 RELATED WORKS
Open-Set Text Classification Early OSTC methods relied maximum softmax probability
(Henyykend & Gimpel, 2016) to reject uncertain predictions. However, researchers (Ben-
dale & Boult (016) argued that MSP merely rejects uncertain predictions, Open Sppenaax & Boult 2016) improved over MSP by fitting a Weibull distribution to logits. DOC (Shu et (017)
introduced one-vs-rest sigmoid classifiers to reduce open space risk, while DeepUnk (Lin & Xu
2019) leveraged large-margin losses for better separation. Nevertheless, they still lack explicit de-
cision boundaries for unknown classes. Recent works exploit PLMs to refine decision boundaries. ADB (Zhang et al.] (202T) proposed adaptive spherical margins, CLAP (Liu et al. (202)) optimized boundary scaling, and KNNCon (Zhou et al. 2022) combined contrastive training with local out- lier detection. Researchers (Zhou et al., 2023) noted that PLMs often "overthink" semantic features, leading to overfitting on labels. To address this, they proposed DyEn, which employs confidence- based early exits to mitigate overfitting. However, the overthinking problem remains fundamentally unresolved, as is not explicitly optimized during training.
OOD Detection OOD detection is closely tied to OSTC and has been widely studied. Confidence- based methods such as MSP (Hendrycks & Gimpell 2016). OpenMax (Bendale Boult 2016). and Energy (Liu et al., 2020) dominated early research, with extensions incorporating auxiliary OOD data (Mohseni et al.] 20200 20.0 often impractical in real settings. To avoid external supervision, ODIN (Hsu et al. 2020) introduced input perturbations, while VOS (Du et al., (022) and NPO (Tao) jet al. 2023) synthesized virtual outliers to regularize decision boundaries. However, such syn- thetic samples poorly approximate real unknowns, especially under sparse ID coverage, resulting in limited generalization. In recent years, LLMS have advanced rapidly, demonstrating remarkable capabilities (Qin al. 2025a) and achieving strong performance across wide range of downstream tasks (Chen et al. 2023: Tong et al. (2025). LLM-OOD (Liu et al.) 2024) demonstrated that discrim- inative PLMs suffer from low-isotropy embeddings that hinder OOD separation. They proposed a generative fine-tuning paradigm, reformulating classification as text generation and exploiting token-level representations, which improved embedding isotropy and OOD detection performance. VALID (Emde et al.) 2025) also pointed out that LLMs often lack explicit awareness of domain boundaries and used rikelihood-ratii-basee OOD detection to restrict off-oind generations.
Nonetheless, both discriminative and generative approaches still rely closed-set training with host-hoc detection, leaving models fundamentally unaware of unknown classes during training.
PRELIMINARIES
Problem Definition Following the open-set learning setting (Scheirer et al.) 2012), the training set ((x, y) with N textual input instances are drawn from product space D = X )r, where X' is the input space, and yd = {1, K) } is the label space of known classes. During the test period, some samples might belong to none of the known classes, which can be allocated to one super unknown class K + 1. These OOD samples are drawn from product space D" X' )", where ) = {K- + 1} is the label space of unknown classes. OSTC aims to classify ID inputs into one of the known classes yd and identify OOD inputs by assigning them the unknown classes )
3

---

Published as a conference paper at ICLR 2026
Stage I: Open-set generative fine-tuning
Stage 3: Analogy-Augmentee Self Reflection
A sanpls
+
+
+
[[
OOD
Text
Prediction
Example
Retrival
Contrastive Learaing
Orthugonality Construints
+
+
+
Stage OOD Purameter Cubliration
alja qusiffal
Reflection
hP
W
2 Pénng
Ah
AN
Ral
LLM
h
Ah
W
No
Yes
R+1
Result Refinement
Direction
Orthogonal Projection
Parameter Calibration
WK-1
Construction
OOD
Prediction
Figure 2: An illustration of our proposed UnLLM.
Generative Fine-tuning with LLMs We outline the generative fine-tuning used in LLM- OOD (Liu et al. 2024). which leverages the inherent generative capabilities of LLMSs for text clas- sification. This approach formulates the classification task conditional text generation problem. Specifically, given an input z, we expand it using a template to streamline output extraction, such as: "## Input: æ. ## Output: y". Subsequently, we maximize the likelihood of generating the la- bel tokens y: max P (y x) = - Po (ya x. ye)). where 0 represents the parameters, ly] is the length of the label in tokens, and y y refers to the tokens preceding the current prediction token yı-
METHOD
As illustrated in Figure 2 UnLLM comprises three stages. Stage Open-Set Generative Fine- tuning: We employ nabel-partitioning strategy to generate both ID and pseudo-OOD inputs for generative fine-tuning, allowing the model to develop K+1-class classifier without relying on ex- ternal data. This stage integrates contrastive learning to enhance intra-class representation compact- ness. Additionally, we synthesize a virtual OOD representation plane and apply an orthogonality constraint loss enforce clear separation between ID and OOD representations. Stage OOD Parameter Calibration: We calibrate the parameter of the K+l-class tokens in the LLM's Head guided by validation-set bias. This ensures alignment between the model's internal representations and its external outputs. Stage 3: Analogy-Augmented Self-Reflection: To mitigate overconfi- dence in ambiguous cases, we retrieve relevant examples in the training set during inference to facilitate analogical reasoning, enabling the model to verify whether given text is OOD.
4.1 OPEN-SET GENERATIVE FINE-TUNING
Traditional methods often suffer the vulnerability OOD inputs due to the lack of explicit knowl-
edge of unknowns during training time (Du et al. 2022). To allow LLM to perceive the task OOD detection, reformulate the training objective from max (y)]z) to max P (2 )²), where y represents a subset of the y* and is a partition-conditional label defined as ÿ = y if y E "" and K 1, allowing
ū K otherwise. This formulation constructs OOD samples labeled
the model to recognize the boundaries of each label and mitigate overfitting on OOD inputs.
Label Partition We introduce a label partition strategy to construct sub-label sets. Specifically, for given ID training instance 24, the label set is evenly divided into partitions, | i [1, s]). ensuring that labels in each partition are mutually exclusive. For each label partition ) E D. the corresponding prrrition-conitiiona label is denoted as Di.j. Let C1 c.... represent the candidate labels within y We construct the following textual input for the model:
You are an expert in text classification.
Text to classify: zs.
Given condidate categories: (NID(e)) Text(c))) NID(es).exx((I)) NID(K + 1).Tex((K + 1)]. Which category does the text belong to?
Answer: NID(ğ.s) N.D(B..)..ex( Tex((
4

---

Published as a conference paper at ICLR 2026
Here, Text(k) is the textual content of label lo and NID(-) is numbering method for consistent label formatting, as detailed in Appendix A.1
Generative Fine-tuning We follow the generative fine-tuning (Liu et 2024). with the objective to maximize the likelihood of the label Øi.5:
P
yenn
log Po (Fijjk Ii, Wi Fis,ek)
k=1
where Vi.j denotes the tokens of (ID(O Fiyl is the length of the token sequence, and Vijisk refers to the tokens preceding the current token Yi,j.--
Contrastive Learning To further model discriminative representations, we utilize contrastive learning (Zeng et al. (02T) to maximize inter-class variance and minimize intra-class variance. Specifically, we extract the representations of the Va.5 tokens at the last layer of LLM as h E R where is the length of Yı,, token sequence and is the embedding size of LLM. We normalize the
bis as b and define the contrastive loss with N N as the number of examples of ik.t
exp(ha.j hy
L
Σ
log E Eisi joy exp(hs, hprr")
N
Orthogonality Constraints To sharpen the decision boundaries of ID labels, we construct vir- tual OOD subspace by sampling outliers from low-likelihood regions of each class distribution (Du jet al. 2022 Tao et al.) 2023). Unlike VOS and DPO, which directly regularize on such synthetic samples, we note their inherent estimation bias and limited fidelity to real OOD data. Drawing inspi- ration from VIM (Wang et al. 2022), we project virtual OOD features onto the principal component plane and enforce orthogonality with the ID subspace. This orthogonalization emphasizes the salient structure of the OOD subspace while suppressing noise.
Specifically, we assume that the representation for each class [1, K] follows class- conditoon Gaussian distribution: Po Vi.j = k) ~ -N (Ns,, o.), where μı E Rixd and σ₁ E Rt To estimate (4 and Øs. maintain ctass-conditional queue Qs- The empirical class mean fag and variance of are caleulated as: f [e.1 En,jeR. b.j, of Te. En.,,o. (h 7
For each k, we sample virtual outliers E RR from the e-likelihood region of the estimated class- conditional distribution:
exp(- (v -3)) e, where is a dynamic value chosen based on /2n0z
20
the smallest likelihood in 2k. ensuring that the sampled outliers are located near the boundaries.
The sampled virtual outliers are aggregated to approximate the critical regions near decision bound- aries. Rather than imposing pairwise constraints, we generalize them into a principal subspace O E Rdde obtained via principal component analysis (PCA), where the number of critical characteristics. ID features are flattened into HiD RD where NID is the number of ID sam- ples the batch. To promote better separation, we minimize Lonh to ensure the orthogonality:
Lonh HHD
(3)
where || - denotes the Frobenius norm. The details of PCA are put in Appendix A.2
Learning Objectives We jointly fine-tune the LLM with composite loss: L AaLa + Aorth Loeth Lgen- where λs and Aoth are the hyperparameters. By jointly optimizing L, the LLM achieves compact intra-class representations while maintaining distinct OOD boundaries.
4.2 OOD PARAMETER CALIBRATION
After training, the LLM gains an awareness of the OOD detection task and can effectively distin- guish between ID and OOD representations. However, we observe a misalignment between the LLM's internal knowledge (representation space) and its outputs. This discrepancy arises due the standard generation method, which relies on token-level probabilities and fails to provide meaning- ful OOD confidence estimates (Kapoor al. 2024). Previous studies suggest that the activation
5

---

Published as a conference paper at ICLR 2026
space of many LLMs contains interpretable directions (Li et al. 2024). Motivated by this, pro- pose identifying a calibration direction in the activation space and calibrating the OOD token weight, WK+1 E Rd in the LLM's output layer to produce calibrated probabilities for K + class.
Calibration Direction Construction First, we evaluate the fine-tuned LLM on the label- partitioned validation set to identify false ID samples X false OOD samples X and correctly predicted samples X The corresponding representations are extracted as HP E B H E gl**|xxxd, and H" E R| Next, we derive the average representative vectors for false pos- itives, false negatives, and correct predictions, denoted as h" R he Bixd and h" Rixd To ensure the OOD token weight of the LLM head W K+1 E R aligns with OOD samples while being distant from ID samples, define the calibration direction as Ah h" h E Rexd which indicates how W T K+1 should be adjusted.
Orthogonal Projection To ensure that adjustments do not disrupt correct predictions, project Ah onto the orthogonal subspace of the representations derived from true predictions h". This orthogonal projection problem has the following closed-fome solution (Zhang al. 2018) as:
Ah (h' (h" "h""""""T> ("""""""
To isolate the OOD-specific adjustment direction while preserving the model's ability to predict correctly, we subtract Åh from Ah, obtaining the calibration vector: Ah'
Ah Ah Rxd
Parameter Calibration Finally, we calibrate W K++ by shifting it in the Ah' direction: W K+1
WT K++ + A',h', where A, a hyperparameter controlling the calibration magnitude. By calibrating W K+11 which represents the OOD mapping function, this approach aligns the model's internal knowledge for OOD detection, without additional training.
4.3 ANALOGY-AUGMENTEN SELF-REFLECH
During the inference phase, the fine-tuned LLM processes test samples using the same label partition strategies. For each test sample, the model sequentially evaluates it against label subsets, stopping once an ID label is identified. If no ID label is found, the sample is classified as OOD. However, we observe that LLMs often exhibit overconfidence in their predictions, especially for texts with high semantic similarity to known labels.
Inspired by -ugmened generation (Roth al., 2024). strategy that leverages past expe- riences to address unfamiliar problems, we propose an analogy-augmentee self- reff-reflection approach address this challenge. Specifically, given an instance z; and its generated label Di.j. we retrieve examples most similar to .i based on embeddings computed by PLM associated with iij. These retrieved examples, denoted as {aı, a2,... }. are ranked by similarity as follows:
Sim(z;, a1) 2 Sim(x;, 02) 2 ... Sim (r;; az) (Mr(L))(r;) LM(a))), where Cos(..)) denotes the cosine similarity function and LM(-) represents the embedding function. The retrieved analogical examples are then feed to the LLM for further reflection as follows:
(5)
Recall relevant exemplars of Ør.z 01,02,"" Does the strictly align with the specified scope? Please start by answering Yes No.
Answer:
Samples receiving "No" response are classified as OOD. This analogy-enhanced reflection allows the LLM to better understand label semantics, reducing biases from superficial semantic similarity.
EXPERIMENT
5.1
EXPERIMENTAL SETTINGS
Datasets We evaluated our method on six see text classification benchmark datasets: BANK- ING (Casanueva et al. 2020), CLINC (Larson et al. 2019), StackOverflow (Xu et al. 2015). News-

---

Published as a conference paper at ICLR 2026
groups (Schneider 2003). Reviews (Jindal & Liu. 2008), and THUCNews (Li al. 2006). In particular, THUCNews is a Chinese dataset, while the others are English datasets and was randomly sampled to match the size of other datasets. Detailed descriptions are put in Appendix B.1
Dataset Split To ensure fair comparison with prior methods, we aligned datasct splits as closely as possible with existing work. For the BANKING, CLINC, and StackOverflow benchmarks, we di- rectly used the provided splits in previous studies (Zhou et al. 2023). For Newsgroups and Reviews, followed the methodology in prior
where pre-defined splits are unavailable (Shu et al. 2017), work to divide the data; the same procedure was applied to THUCNeWS. Consistent with (Zhou al., 2023). we retained 25%, 50%, or 75% of the classes as ID. while the remaining classes were treated as OOD. OOD samples were excluded from training and validation, and reserved for testing. Detailed dataset statisties are reported Appendix B.2
Baselines We compared UnLLM against mainstream OSTC methods, which have been carefully baselines: DOC (Shu al.) 2017). Unk (Lin & Xu 22119), ADB (Zhang et al., (2021). CLAP (Liu et al., 2023). KnnCon (Zhou
discussed in Section 2] including discrimitive training-baae
et al. 2022). and DyEn (Zhou et al.] [l.[(223) To the best of our knowledge, no prior work has systematically applied LLMs to OSTC. For a fair comparison, we construct LLM-based baselines by fine-tuning LLMs on the ID training set and adapting existing OOD detection methods to this setting. For generative fine-tuning, scoring (Liu et al. 2020). VOS (Du et al.) 2022), and NPO (Tao et al. (023). where VOS and NPO leverage virtual OOD distributions to improve detection. More
employ LLM-OOD (Liu et al.) 2024). For discriminative
fine-tuning, we adapt Energy-
baseline implement details and OOD detection baselines, which perform less competitively, are provided in Appendix [
Evaluation Metrics Following the evaluation approach in prior studies (Liu et al. 2023 Zhou al. 2023), we treat all OOD classes as the K+1 class. K-FI and N-FI represent the macro FI- scores for ID and OOD classes, respectively, effectively capturing the model's performance on ID classification and OOD detection. Aligning with previous works, we also compute Fl and ACC, provided in Appendix C.6 due to page limitations.
5.2 OVERALL PERFORMANCE
Table presennss the performance comparison of our method against baselines across six datasets. The results clearly indicate that UnLLM achieves substantial and consistent improvements over SOTA baselines, demonstrating its effectiveness. Specifically, our method improves the K-FI scores across all datasets by average of 4.40%. 2.80%, and 2.55% and the N-F1 scores by 1.63%, 1.53%, and 5.09% under 25%, 50%, and 75% known class settings, respectively.
Several key insights can be drawn from the results. First, while UnLLM achieves the best over- all performance, -ased baselines frequently rank second, underscoring the strong potential of large language models in OOD detection. However, it is noteworthy that generative training approaches (e.g., LLM-OOD) do not outperform discriminative counterparts (e.g., Energy Based), suggesting that prior generative strategies fall short in capturing discriminative decision boundaries. In contrast, our conditional subset training paradigm enriches sample diversity during optimization, significantly boosting both classification accuracy and OOD detection ability.
Second, the results on Reviews and Newsgroups reveal that BERT-based methods struggle with long-text classification, whereas generative LLMs are more adept at modeling complex semantic structures. This semantic advantage contributes to stronger classification and OOD detection ca- pabilities. Nevertheless, LLM-based approaches remain vulnerable to overfitting, as they rely on the conventional K-class classification paradigm. Our method addresses this limitation by explicitly modeling unknown-class awareness during training through a K+1 classification framework. This design effectively exploits the representational power of LLMs while aligning training and inference objectives, yielding both significant and stable performance improvements.
Finally, performance variations across different known class ratios reveal a consistent trend: as the ratio of known classes increases, K-F1 improves due to the greater availability of ID labels, facilitating more accurate ID classification. However, N-F1 generally declines, reflecting increased challenges in OOD detection. This decline occurs because models tend to overfit known labels more labeled data becomes available. Notably, our approach maintains strong OOD detection

---

Published as a conference paper at ICLR 2026
Table 1: Performance of various methods across 6 datasets at different ratios. Metries include K-F1 and N-F1. The best results are highlighted in bold, while the second-best results are underscored. Each result represents the mean value of four repetitive experiments.
BANKING
CLINC
StuckOverflow
Reviews
Newsgroups
TheUCnewx
Ratio
Backbone
Method
K-F1
N-F1
K-F1
N-F1
K-F1
N-F1
K-F1
N-F1
K-F1
N-F1
K-F1
N-F1
CNN
DOC
66.40
81.57
75.46
85.63
69.81
83.61
59.59
79:57
66.21
89:59
56,43
82.27
LSTM
DeepUnk
70.87
87.74
75.31
91.36
67.10
$8.93
45.18
51.59
56.19
71.54
41.40
46.72
BERT
ADB
51.07
61.84
59.00
66.67
75.91
88.37
48.27
68.
28.70
34.02
58.98
62,20
CLAP
58 38.69
72.75
69.75
54.45
78.20
35.79
49.17
KNNCon
65.97
70.63
79.53
$7.85
66.83
75.63
53.75
71.26
55.71
60.18
47.91
42.33
DyEa
65.53
73.71
77.16
86.85
60.17
55.89
44.28
42.11
57.74
67. 67.04
43.99
17.90
LLMMA3.1-B
LLM-OOD
79.91
78.54
90.67
60.09
6:4.17
80.08
EnergyBBased
71.07
90.82
83. 13.10
91.46
80.91
94.57
48.21
89 69
60.45
83.35
61.72
92.41
VOS
74.39
89.82
61.79
90.95
62.06
89.64
61.35
89.09
61.28
84.66
51.59
89.72
NPO
73.83
89.73
82.60
90.29
86.01
95
54.25
89.78
59.19
85,
92.79
UaLLM
75.04
92.02
83,90
93.58
88.65
96.00
62.16
91,94
68.33
91.82
83,63
94.46
0.5
CNN
DDC
74.50
77.92
84.10
84.38
77.96
79.57
55.42
67.52
79.
74.
46.63
64.58
LSTM
DeepUnk
59.75
74.91
69 80
$1.54
76.79
81.86
41.86
20.12
69.07
59.93
62.55
24.58
BERT
ADB
63.09
56.92
71,14
61 61
83.95
83.41
51.56
49.
33.99
22.
76.67
69.88 88
CLAP
67.15
$4.99
55.00
50.37
71.49
62.51
KNNCon
80.35
75.13
82.71
62.30
85.1
83.12
61.31
46.13
77.60
63.58
76.44
48.49
DyEu
78.55
68.34
88,38
82,46
77.06
60:25
60.47
27.66
77.83
62.19
71.79
34.68
LLaMA3.1-BB
LLM-OOD
75.06
78.85
89.71
89.54
65.55
77.11
72.95
78.66
EnergyBased
74,96
82.76
88.39
90.50
$9.58
90.57
47.32
74,93
75.39
76.77
80.66
80.41
VOS
78.93
83.94
66.38
80.99
58.88
76.33
61.71
27.04
76.88
74.99
77.78
80.22
NPO
78.70
84.12
91.31
91. 91.53
90.
90.77
58.
76.82
78.11
76.73
82.57
79.00 79.90
UnLLM
82.74
85.72
93:42
93.00
91.91
92.60
68.81
80,59
83.95
77.20
84,83
80.75
CNN
DOC
78.85
87.37
72.81
84.90
72.30
52.53
67.67
$7.58
41.32
LSTM
DeepUnk
59.27
54.85
58.57
67.62
$2.57
68.93
27.31
11.81
65.95
40.93
70.76
26.25
BERT
ADB
72.07
43.48
78, /4/40
$1.69
87.65
75. 10
47.
35,55
35.007
17.35
79:59
$4.59
40.37
71.19
75.98
53.00
72.82
50.93
KNNCOn
86.90
61.61
93.91
80.67
89.38 38
74.55
65.22
49.02
86.36
62.53
90.49
63.10
DyEn
86.79
54.74
93.38
78.02
85,43
54,09
64.24
21.81
85.24
47.06
87,88
28.73
MLMMA31-88
LLM-OOD
96
60.82
77.79
83:29
67.96
71.94
EnergyBased
76.84
63.80
90.18
79.11
85.94
74.07
30.08
44.55
70.78
56.69
79.41
70.34
VOS
79.68
84
72.68
83.54
67.85
48,
18
75.72
83.85
74.05
NPO
79.89
65.79
93.24
83
90. 10
79.22
39. 08
46.744
76.68
60.36
78.67
70.35
UnLLM
88.58
72.60
96.94
89.73
92.36
83.08
67.84
56.65
89.06
72.18
93.55
78.11
performance even under highly known-class settings, demonstrating its ability to mitigate overfitting and maintain stable performance across diverse scenarios.
5.3 RESULTS ON UNLLM vs. LLMS WITH DIFFERENT PROMPTING STRATEGIES
We further compared UnLLM with LLMs that were not fine-tuned, under various prompting strategies including shoo-shot, few-shot, chain- of-thought (CoT), and mmalogy-augmente self- reflection (Analogy) in Section4.3 In par- ticular, we evaluated three different back- bones, namely LLaMA3.1-8B, Qwen2.5-32B, and DeepSeek- 0324 is one of SOTA open-source LLMs.
Table 2: Performance comparison on CLINC dataset across different prompting strategies.
LLaMA3.1-8B
Qwen2.5-32B
V3-0222
Ratio
Method
K-F1
N-F1
K-F1
N-FI
K-F1
N-F1
Zero
58.70
Fewshoo
49.45
63.01
17.52
34.30
58.81
57.87
0.25
Cot
55.00
54.43
56.86
56.30
51.03
65.54
where DeepSeek-V3-
Amalogy
UnL.LM
83,90
61.26
93.58
56.16
90,00
78.46
92.67
91.90
$1.06
91,50
Zero-shot
43.65
62.93
68.09
37.66
$2.36
As shown in Table 2 across all ratios, in- context prompting methods (Zero/Few/CoT) yield relatively unstable performance. Few- shot even collapses under certain backbones (e.g., Qwen2.5-32B at 0.25 and 0.5 ratios), whereas CoT sometimes provides modest im- provements but remains inconsistent. Analogy is generally stronger than other in-context strategies, showing clear gains, highlighting that the de- signed gallogy-augmentee inference is effective complement to prompting. By contrast, Un- LLM achieves consistently superior K-F1 and N-FI scores on both LLaMA3.1-8B and Qwen2.5- 32B, regardless of the ratio, and significantly outperforms all mompting-based methods. We do not
Oit
63.62
50.88
48
74.93
71
Andlogy
56.35
36,52
22.87
$4.55
K3.53
87.60
UnL.1M
93.42
93,00
96,63
95.63
Zero-shot
57.24
31.56
67.00
50.47
75.11
$1.93
Few-shot
44.57
34.87
15.39
42.20
75.87
46.00
0.75
65.62
77.44
Analogy
47.60
50.13
21.27
83.68
73.61
UnLLM
96.94
39.73
96.58
89.17
8

---

Published as a conference paper at ICLR 2026
100
ACCOS)
100)
F1(5)
100
K F109)
100
N NIF1SS)
70
70
70
6.2
0.
0.22
125
6:72
325
-
6.5
-4
0.2
03
0.55
1.0
Figure 3: Parameter sensitivity analysis of Nonh on the BANKING dataset.
fine-tune DeepSeek due to its extremely high GPU requirements. For completeness, we report other evaluation metries (ACC, F1) in the Appendix C.3
5.4 QUALITATIVE ANALYSIS
Parameter Sensitivity Analysis To assess the impact of Aonth conduct experiments the BANKING dataset across three proportions, varying Aorh from 0.1 to 1.0 (Figure 3). The results indicate that the orthogonal loss effectively improves both ID classification and OOD detection, with Anth = 0.4 yielding relatively balanced performance across various evaluation metrics and proportions. Overall, the model demonstrates robustness this parameter selection, showing that not only enhances OOD detection but also benefits ID classification. More sensitivity analyses on other parameters are provided in Appendix B.4
Ablation Study To evaluate the effectiveness
of our proposed modules, we compared Un-
UnLLLM wio fu
UsLLM wio calibration
UnLLM
LLM with several variants: "w/o Ca" that ex-
UnL.LM wie Cor
UsLLM wio anslogy
cludes La during training. "w/o Lorth" " that re- moves Cunh. "w/o calibration" that eliminates OOD parameter calibration, and "w/o analogy" performs infrence without shalogy--uugeenter self reflection. As shown in Figure 44] Un- LLM achieves the best performance across all ratios. Below, we analyze the role and signifi- cance of each component:
100
0.25
90
100
Ro
90
80
First, removing the contrastive loss leads significant performance drop, particularly when the ratio 0.25 or 0.5. This result highlights the importance of contrastive learning in pro- moting tighter intra-class clustering, which is crucial for distinguishing ID and OOD repre- sentations. Second, as the ratio increases, the impact of orthogonal constraint loss becomes more pronouneed. A higher ratio introduces more labels, increasing the likelihood of overfitting on known categories. The orthogonal constraint loss effectively mitigates this risk by enhancing the separation between ID and OOD representations. Finally, OOD parameter calibration and analogy-augmenttd self-reflection prove to be indispens- able, as their removal results in substantial performance decline. This finding underscores the importance of addressing Knowledge-output misalignment and overconfidence in semantically sim- ilar labels for improving OOD detection. Due to page limitations, additional ablation study results on other datasets are provided in Appendix C.I
100
Raiio 0.75
90
ACC
K-FI
N-FI
Figure 4: Ablation study CLINC dataset.
5.5
CASE STUDY
We compare UnLLM with the LLM-OOD baseline on StackOverflow (known-class ratio 0.75). Fig- visuaaisss the learned representations via T-SNE: UnLLM forms more compact ID clusters and maintains clearer margin OOD samples, indicating improved representation-level separability.

---

Published as a conference paper at ICLR 2026
Table Performance on Intra-Inter Ratio with different methods. Higher is better.
Dataset
Ratio
LLM-OOD
UnLLM
StackOverflow
0.25
0.99
1.98
0.50
0.88
5.62
0.75
0.68
4.56
CLINC
0.25
0.90
1.15
LLM-ODD
by USLLM
0.50
0.94
21.30
0.75
1.09
6.43
Figure 5: Representation visualization with red
denoting OOD and other colors representing ID.
To complement this qualitative evidence, we further report the Intra-Inte Ratio, which quantifies ID 10-OOD separation in the representation space:
Intra-Inter Ratio Inter-cluster Distance
Avg. cosine distance between ID and OOD clusters Avg. cosine distance within the II) cluster
Intra-cluster Distance
higher ratio suggests that ID samples are tightly clustered while remaining far from OOD samples. UnLLM consistently yields higher ratios than LLM-OOD, confirming its improved ID-OOD separability across datasets and settings.
As summarized in Table 3
6 CONCLUSION
In this work, we proposed UnLLM, for OSTC, addressing key limitations stemming from the ab- sence of supervised signals for unknown instances during training. First, introduced open-set generative fine-tuning to develop K+1 classifier. Additionally, we proposed an OOD parameter calibration method align the model's internal cognition of the unknown with its outputs. Dur- ing inference, we integrated an analogy-augmenntd self-reflection mechanism to mitigate overcon- fidence. Experimental results across six datasets demonstrate that UnLLM consistently achieves SOTA performance in OOD detection while maintaining competitive accuracy in ID classification.
7
ETHICS STATEMENT
We acknowledge and adhere to the ICLR Code of Ethics) We commit to promoting positive societal impact while considering and mitigating potential harms (e.g., privacy, bias, unfairness). We uphold scientific integrity by ensuring transparency, reproducibility, and honest reporting methods and limitations, and we disclose any ethical review status (e.g., IRB approval) when applicable. We further disclose conflicts of interest, respect fairness and non-discrimination, and give proper credit prior work while honoring confidentiality and data usage rights.
REPRODUCIBILITY STATEMENT
We summarize our efforts below to facilitate reproducible results:
Datasets. We use publicly available datasets, which are described in detail in Section 5.1 ti1 and Appendix B.1
Sec-
Baselines. The description and hyperparameters of the OOD detection baselines are explained in Section and Appendix B.3
Methodology, Our method fully documented in Section 4 with the pseudo algorithm detailed
in Algorithm and Algorithm S2 Hyperparameters are specified in Section B.4 with thorough ablation study provided Section 5.4 Appendix C.I
Open Source. Code and datasets publicly available at http:///gg..tuucccc//////11////1
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported in part by the National Natural Science Foundation of China (NSFC) (Grant No. 62506352), in part by the Strategic Priority Research Program of Chinese Academy of Sciences (Grant No. XDB1350102), in part by the National Natural Science Foundation of China (Grant No.92370204), in part by the National Key R&D Program of China (Grant No. 20022237F072500,,,, in part by the guangdong Basic and Applied Basic Research Foundation (Grant 2222233115112205577 in part by the Key-Area Special Project of Guangdong Provin- cial Ordinary Universities (2024ZDZX007), in part by the Natural Science Foundation of Anhui Province (Grant No. 2508085QF211 I), the National Natural Science Foundation of China (Grant No. 62506348), the Opening Foundation of State Key Laboratory of Cognitive Intelligence, iFLYTEK (COGOS-2025HE02).
REFERENCES
Abhijit Bendale and Terrance E. Boult. Towards open set deep networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition ( (CVPR), 2016.
Thomas F Burns and Tomoki Fukai. Simplicial hopfield networks. In The Eleventh International Conference on Learning Representations, 2023. URL https://openrevier/ wtw.:iewwwe//ooummiiiii
HLsgaww
Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. Effi cient intent detection with dual sentence encoders. In Tsung-Hssen Wen, Asli Celikyilmaz, Zhou Yu, Alexandros Papangelis, Mihail Eric, Anuj Kumar, Iñigo Casanueva, and Rushin Shah (cds.), Proceedings of the 2nd Workshop on Natural Language Processing for Conver- sational AI, pp. 38-45, Online, July 2020. Association for Computational Linguistics. doi: URL peatts://aalaanthoooog wwg.//ggg////// http://or/.......oocoaaaaa......
Xi Chen, Xinjiang Lu, Haoran Xin, Wenjun Peng, Haoyang Duan, Feihu Jiang, Jingho Zhou, and Hui Xiong. A table-to-text framework with heterogeneous multidominance attention and self- evaluated multi-pass deliheration. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 607-620, Singapore, De- cember 2023, Association for Computational Linguistics. doi: 10.18653/v 00533/1120222 1/2023. 2i023.iindings-emml.. 44. URL
httph/oooloo.ogg//2233333/ findings-emninn emp1p. 44
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized lms.. Advances in Neural Information Processing Systems, 36, 2024.
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0000 04805, 2018
Xuefeng Du, Zhaoning Wang, Mu Cai, and Sharon Li. Vos: Learning what you don't know by virtual outlier synthesis. In International Conference on Learning Representations, 2022. URL
Comellus Emde, Alasdair Paren, Preetham Arvind, Maxime Guillaume Kayser, Tom Rainforth, Thomas Lukasiewicz, Philip Torr, and Adel Bibi. Shh, don't say that! domain certification in LLMs. In The Thirteenth International Conference on Learning Representations, 2025. URL https::/openrest
nwt//orummiddFo...... vQBum)
Geli Fei and Bing Liu. Breaking the closed world assumption in text classification. In Kevin Knight, Ani Nenkova, and Owen Rambow (eds.), Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 506-514, San Diego California, June 2016. Association for Computational Linguistics. doi: 10. 1118866//111111.11 1061. UtL/htttps://acaanntoooe 1twprhhhoooooooo////-1--11 or66
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of distriibuttnnno examples in neural networks. In International Conference on Learning Representations, 2016.
11

---

Published as a conference paper at ICLR 2026
Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized odin: Detecting out- of-distribution image without learning from out ditt---distribttion data. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10951-10960, 2020.
Nitin Jindal and Bing Liu, Opinion spam and analysis. In Proceedings of the 2008 international conference on web search and data mining, pp. 219-230, 2008.
Sanyam Kapoor, Nate Gruver, Manley Roberts, Arka Pal, Samuel Dooley, Micah Goldblum, and Andrew Wilson. Calibration-tuning Teaching large language models to know what they don't know. In Raúl Vázquez. Hande Celikkanat, Dennis Ulmer, Jörg Tiedemann, Swabha Swayamdipta, Wilker Aziz, Barbara Plank, Joris Baan, and Marie-Catherine de Marneffe (eds.), Proceedings of the 1st Workshop Uncertainty-Aware NLP (UncertaiNLP 2024), pp. 1- 14, St Julians, Malta, March 2024. Association for Computational Linguistics. URL https: 7/aclanthology wttp://///.......r...... unccerrii.cr
Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. An evaluation dataset for intent classification and out-of-scope prediction. In Kentaro Inui, Jing Jiang. Vincent Ng. and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empir- ical Methods Natural Language Processing and the 9th International Joint Conference on Natural Language P Processing (EMNLP-IJCNLP), pp. 1311-1316, Hong Kong, China, Novem- ber 2019. Association for Computational Linguistics. doi: 10.18653/v1/11--1133 URL https:///allanchooooyyoo///////
Hakyung Lee, Keon-Hee Park, Hoyoon Byun, Jeyoon Yeom, Jihee Kim, Gyeong-Mon Moon Park, and Kyungwoo Song. CED: Comparing embedding differences for detecting out-o--distribution and hallucinated text. In Findings of the Association for Computational Linguistics: EMNLP 2024, 2024.
Jingyang Li, Maosong Sun, and Xian Zhang. A comparison and semi-quantitative analysis of words and saaracter-bigrant as features in Chinese text categorization. Nicoletta Calzolari, Claire Cardie, and Pierre Isabelle (eds.), Proceedings of the 2 International Conference on Com- putational Linguistics and 44th Annual Meeting of the Association for Computational Linguis- fics, pp. 545-552, Sydney, Australia, July 2006. Association for Computational Linguisties. doi: 10.3115/122015. 117775..2200 1200444. URL
org www.oogg///Po wwwrr//////1100 1069.
Junyi Li, Xiaoxue Cheng, Xin Zhao, Jian- Yun Nie, and Ji-Rong Wen. HaluEval: A large-scale hal- lucination evaluation benchmark for large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 6449-6464, Singapore, December 2023. Association for Computational URL https:///aaaantolooo.yooo
Linguistics. doi: 10. http:/222222.mmpmmmmmm.. emnlp-main. 397
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. inference-time intervention: Eliciting truthful answers from language model. Advances in Neural Information Processing Systems, 36, 2024.
Ting-En En Lin and Hua Xu. Deep unknown intent detection with margin loss. In Anna Korhonen, David Traum, and Lluís Märquez (eds.), Proceedings of the 57th Annual Meeting of the Associ- ation for Computational Linguistics, pp. 5491-5496, Florence, Italy, July 2019. Association for URL https://aclanthhnt .rgg
Computational Linguistics. doi: 10. 10.188653/1//11--15 P19-1543
Bo Liu, Li-Ming Zhan, Zexin Lu, Yujie Feng, Lei Xue, and Xiao-Ming Wu. How good are LLMs at ousto--disttibution detection? In Nicoletta Calzolari, Min- Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 8221 Torino, Italia, May 2024. ELRA and ICCL. URL https://aclanttoolggy hrw.:aannthooooy.or///222// wtt.2822...eee--e Irec-main. 720
Weitang Liu, Xiaoyun Wang. John Owens, and Yixuan Li. Energy-basee distribbutiin detec- tion. Advances in neural information processing systems, 33:21464-21475 2020.
12

---

Published as a conference paper at ICLR 2026
Xiaokang Liu, Jianquan Li, Jingjing Mu, Min Yang, Ruifeng Xu, and Benyou Wang. Effective open intent classification with k-center contrastive learning and adjustable decision boundary. In Proceedings of the AAAI Conference Artificial Intelligence, 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL httpp:://penreevien. wttp:/iwww.ntfooummiiiiiim
Bkg6RiCqY7
Sina Mohseni, Mandar Pitale, JBS Yadawa, and Zhangyang Wang. Self-supervised learning for generalizable out-of-ditributtion detection. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 5216-5223, 2020.
Sridhama Prakhya, Vinodini Venkataram, and Jugal Kalita. Open set text classification using CNNs. In Sivaji Bandyopadhyay (ed.), Proceedingy of the 14th International Conference Natural Lan- guage Processing (ICONN20177, pp. 466-475, Kolkata, India, December 2017. NLP Association of India URL Utt/httts//acclanhhoooo
Chuan Qin, Xin Chen, Chengrui Wang, Pengmin Wu, Xi Chen, Yihang Cheng, Jingyi Zhao, Meng Xiao, Xiangchao Dong. Qingqing Long, et al. Scihorizon: Benchmarking ai-for-science readi- ness from scientific data to large language models. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pp. 5754-5765, 2025a.
Chuan Qin, Chuyu Fang, Kaichun Yao, Xi Chen, Fuzhen Zhuang, and Hengshu Zhu. Cotr: Efficient job task recognition for occupational information systems with class-incremental learning. ACM Trans. Manage. Inf. Syst., 16(2), March 2025b. ISSN 2158-656X. doi: 10. 105134//77 URL
https://doi.org: 10. 1145/3712305
K Roth, Rushil Gupta, Simon Halle, and Bang Liu. Pairing analogy-augmented generation with procedural memory for procedural q&a. arXiv preprint WwXiv:2409.013444 2024.
Walter Scheirer, Anderson de Rezende Rocha, Archana Sapkota, and Terrance E Boult. Toward open set recognition. IEEE transactions on paltern analysis and machine intelligence, 35(7):
7757-1772,2011
Karl-Michael Schneider. A comparison of event models for naive Bayes anti-spam E-mail filtering. In Ann Copestake and Jan Hajič (eds.), 10th Conference of the European Chapter of the Associ- ation for Computational Linguistics, Budapest, Hungary, April 2003. Association for Computa- tional Linguistics. URL https:: : /ttttsss///caatthoooooooor//////
Lei Shu, Hu Xu, and Bing Liu. DOC: Deep open classification of text documents. Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Conference Empirical Methods in Natural Language Processing, pp. 2911-2916, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. 10.18653////D171133 URL https: www.aaattooooo.oor////D/0
Gaurang Sriramanan, Siddhant Bharti, Vinu Sankar Sadasivan, Shoumik Saha, Priyatham Kat- takinda, and Soheil Feizi. LLM-check: Investigating detection of hallucinations in large language models. The Tiihth Annual Conference on Neural Information Processing Systems, 2024. wtw.ne//fooummmdd_Hklpp...Z.
Leitian Tao, Xuefeng Du, Jerry Zhu, and Yixuan Li. Non-parametric outlier synthesis. In The Eleventh International Conference on Learning Representations, 2023. URL https://
@penreview.net/ooruuue
Zhenyu Tong, Chuan Qin, Chuyu Fang, Kaichun Yao, Xi Chen, Jingshuai Zhang, Chen Zhu, and Hengshu Zhu. From missteps mastery: Enhancing low-resource dense retrieval through adap- tive query generation. In Proceedings of the 3 ACM SIGKDD Conference on Knowledge Dis- covery and Data Mining V.I. KDD '25, pp. 1373-1384, New York, NY, USA, 2025. Associa- tion for Computing Machinery, ISBN 97984007 12456. doi: 10.1145/3600624.770222 URL https:://oi, 000 02455/5666644... .012244..7999 3709225
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. /3971, 2023.
13

---

Published as a conference paper at ICLR 2026
Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of- distrribbutiinn with virtual- logit matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4921-4930, 2022.
Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, Bo An, and Yixuan Li. Mitigating neural net- work overconfidence with logit normalization. In International conference on machine learning, pp. 23631-23644, PMLR, 2022.
Jiaming Xu, Peng Wang. Guanhua Tian, Bo Xu, Jun Zhao, Fangyuan Wang, and Hongwei Hao. Short text clustering via convolutional neural networks. In Phil Blunsom, Shay Cohen, Paramveer Dhillon, and Percy Liang (eds.), Proceedings of the 1st Workshop on Vector Spuce Modeling for Natural Language Processing, pp. 62-69, Denver, Colorado. June 2015, Association for Compu- tational Linguistics. doi: 10.3115/v1/115- 1509. URL MRLLIttts///aaaaathooooo .rg /W////15---1 5- 1509
Zhiyuan Zeng, Keqing He, Yuanmeng Yan, Zijun Liu, Yanan Wu, Hong Xu, Huixing Jiang, and Weiran Xu. Modeling discriminative representations for out-of-domain detection with super- vised contrastive learning. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing ( (Volume 2: Short Papers), pp. 870-878, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v 1/2021 acl--scort.tlii 10thortt110,, 10. /tBL/ttts///alllntooloyg// /ttpo//////////////ooooooooooooo httpp//ww.....22....aoooorrrrr acl-shortt 110
Hanlei Zhang, Hua Xu, and Ting- En Lin. Deep open intent classification with adaptive decision boundary. In Proceedings of the AAAI Conference on Artificial Intelligene, 2021.
Liheng Zhang, Marzieh Edraki, and Guo-Jun Qi. Cappronet: Deep feature learning via orthogonal projections onto capsule subspaces. Advances in neural information processing systems, 31, 2018.
Yunhua Zhou, Peiju Liu, and Xipeng Qiu. ENN-contrastive learning for out-of-domain intent clas- sification. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 5129-5141, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v 1/c.2/..c2l---.e.. long..5...322..... URL https: http:::/aclanttooogg..oom hwwptthhoooggyoo//222222 acl- 352
Yunhua Zhou, Jianqiang Yang, Pengyu Wang. and Xipeng Qiu. Two birds one stone: Dynamic ensemble for OOD intent classification. In Anna Rogers, Jordan Boyd-Graber, and Naoaki
Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10659-10677 Toronto, Canada, July 2023, Asso-
ciation for Computational Linguistics. doi: 11118653/112022 595- URL https:
s/aclantholoogy.org /rg/ 2023. ael-long..ee 595
14

---

Published as a conference paper at ICLR 2026
A MORE DETAILS OF UNLLM
A.1 LABEL NUMBERING METHODOLOGY
To standardize label formatting, we employ zero-padded numbering scheme with three digits. Specifically, the numerical identifier (ID(z) for label computed as: str.zfill(a,, 3), where str.zfill(ar, i) converts the label into zero-padded string of length i. In this work, we set i 3. For example, given the label {1, 2.3. the corresponding identifiers are:
[D(1 str.zfill(,,3) "001",
ID(2) str.zfill = "002",
ID(3) str.zfill 3) "003".
This numbering scheme ensures consistent label representation, facilitating structured data pro- cessing and representation extraction.
A.2 PRINCIPAL COMPONENT ANALYSIS FOR OOD SUBSPACE CONSTRUCTION
We formalize the construction of the OOD subspace using PCA. Let the virtual outliers sampled across all classes form the set V, whose flattened feature embeddings are stacked into the matrix M E R xtd The covariance matrix is computed as E M M E Ridxtd, Applying singu- lar value decomposition (SVD), we obtain Eneg UAU where columns of U are orthogonal eigenvectors representing the dominant directions of variation in V. Selecting the top e eigenvectors yields the OOD subspace O U[:: el E ! ID features HID xtd are projected onto O, and the orthogonality loss of the main text penalizes their alignment, thereby promoting a clearer separation between ID and OOD spaces.
A.3 EXAMPLES FOR LLM INPUTS
To better help readers understand the prompt, give two specific instances for LLM training and inference with self-reflection as follows:
Training Instance
You are an expert in text classification with focus on Out-of-Distribution (OOD) detection.
Your task to accurately classify a given piece of text into one of the provided categories only if it strictly matches the meaning and scope of the category definition. If the text does not match any category definition, it should be identified as "Out-of- Distribution (OOD)"
Text to Classify: Typed FP: Tuple Arguments and Curriable Arguments.
Questions: Given the candidate categories: "000..t is OOD", "002.haskell", "003.ma- gento", "005.apache") J, which category does the text belong to?
Answer: haskell
15

---

end for end for Published as a conference paper at ICLR 2026
Self- Reff-Reflectin Inference Instance
You are an expert in text classification with a focus on cut-of-Distribution (OOD) detection.
Your fask categories only if it strictly matches the meaning and scope of the category definition. If the text does not match any category definition, it should be identified as "Out-of- Distribution (OOD)"
accurately classify a given piece of text into one of the provided
Text to Classify:
Think someone has took money out with my card. What shall do?
Questions: Given the candidate categories: [001..top reverted", "010.declined card payment", "019.compromise" card", "018.cash withdrawal charge", "011 rreveredd card paymen!?", "000.1i is 0OD", "007.failed transfer", "008. edit personal details", "005.card arrival", "016.card not working". "002.balane not updated after cheque or cash deposit"), which category does the text belong to?
Answer: 019. oo9poompod card..
Recall relevant exemplars of 019..0.cmmm comprommmd card.: ["I sees some suspicious spending on my credit card that don't recall I had made. What should do?", "Someone might using my card. What should do?". Does the text strictly align with the specified scope of 019.compromised card.? Please start by answering Yes No. Answer: No.
A.4 ALGORITHM PSEUDOCODE
In this paper, designed novel method to implement the OSTC. In order to make our algorithm procedure easier to understand, the pseudocode for the fine-tuning pipeline of open-set generative fine-tuning is available in Algorithm ST and the inference pipeline with suggleee self- reflection provided in Algorithm $2
Algorithm S1 LLM Fine-Tuning Procedure
Require: Pre-trained LLM, Training set {(z, M Number of partitions s, Hyperparameters
det, Aonhh
1: Initialize model parameters 0
2: for each batch {(,, 3)) hi) do
3: for each training instance x; in the batch do
4: Partition label set y into mutually exclusive subsets (VP k.j 1
5: for each label partition y do
6:
Generate partition-cone coniiiiionnnniinnlll label Ti Compute generative loss fgen (Eq. 1) Extract representation h;
7:
8:
9:
10:
Compute contrastive loss Let (Eq.
12: Estimate class-conditional distribution parameters Po. 0%
Sample virtual outliers Vy from low-likelihood regions Compute orthogonality constraint loss Lont
14:
end for
16:
Optimize LLM parameters / using composite loss L.
17: return Fine-tuned LLM
16

---

Published as a conference paper at ICLR 2026
Algorithm S2 LLM Inference Procedure
Require: Fine-tuned LLM, Test instance I;
1: Partition label set into a mutually exclusive subsets [D
2: for each label partition Dr, do
3: Construct textual input with candidate labels
4: Generate predicted label ji.j
5: if Úi. + K + then
6: return Øi.y (ID prediction) end if
8: end for
9: Retrieve analogical examples {a1 a2, based on similarity
10: Perform analogy-augmented self-reflection
If response is "No", classify as K
12:
return Vi.j (ID prediction)
Table S1: Statistics of datasets. |||| denotes the total number of instances. average length of each instance in the dataset.
Length indicates the
Dataset
N
[Trinn]
[Val]
||Tsst|
Length
BANKING
77
9,003
1,000
3,078
11.77
CLINC
150
17,995
2,250
2,250
8.31
StackOverflow
20
11,996
1,998
5,991
8.34
Reviews
50
29,823
4,942
14,794
143.04
Newsgroups
20
11,291
1880
5,657
306.75
THUCNews
25,200
2,800
5,600
7.15
Table S2: Dataset Splits and Statistics.
|||| denotes the total number of instances.
Ratio
0.25
0.50
0.75
Dataset
|
Val||
|est
[Triin|
[Val]]
|
Train
Vall
|
BANKING
2,222
247
3,078
4,592
511
3,078
6,784
755
3,078
CLINC
4,439
555
2,250
8,996
1,125
2.250
13,436
1,680
2,250
StackOverflow
2,998
499
5,991
5,999
999
5,991
8,998
1,499
5,991
Reviews
7,176
1.192
14,794
14,935
2,465
14,794
22,649
3,750
14,794
Newsgroups
2,835
472
5,657
5,876
979
5,657
8,578
1,427
5,657
THUCNeWs
5,400
600
5,600
12,600
1,400
5,600
19,800
2,200
5,600
B
MORE DETAILS ON EXPERIMENTAL SETTINGS
B.1 DATASET DESCRIPTION
To verify the effectiveness and universality of our proposed method, we conducted exhaustive ex- periments six widely used text classification datasets. These datasets are summarized as:
Newsgroups (Schneider 2003 consists of 18,828 documents partitioned evenly across 20 mutu- ally exclusive classes.
Reviews (Jindal & Liu 2008 consists of 50 classes of products or domains, each with 1,000 review documents.
CLINC (Larson et al. 2019) is a very popular dataset, which encompasses a broad range of intents, totaling 150 across 10 domains. The entire dataset consists of 22500 in-domain samples and 1200 Out-of-domain samples.
BANKING (Casanueva al. 2020 is kind of dataset about the banking business, with 77 categories. The data is characterized by the imbalance of samples in different categories. The training set, validation and test set contain 9003, 1000, and 3080 samples respectively.
17

---

Published as a conference paper at ICLR 2026
StackOverflow (Xu al. 2015) is dataset about programming languages released by Kag- gle.com. The dataset is subdivided into 20 categories and has 2 samples. The number of samples in the training set, validation and test set is 12000, 2000, and 6000 respectively.
THUCNews (Li et al. 2006) is Chinese news text dataset, with 10 categories: finance, real estate, stocks, education, technology, society, politics, sports, games, and entertainment.
These datasets are only for scientific research and are available for all members of the NLP research community. We have adhered to the typical method of utilizing these resources.
B.2 MORE DETAILS ON DATASET SPLIT
We present the dataset split statistics across different training ratios in Table S2 The training and validation sets contain only ID samples, while the test set includes both ID and OOD samples.
B.3 MORE DETAILS ON BASELINES
We categorize the baselines into three groups: backbone-specife classification models, OOD de- tection methods, and confidence calibration methods. The former rely on model-specific training objectives, while the latter two are monoel- techniques that can applied on top different backbones (CNN, LSTM, PLMs, and LLMs)
Backbone-specific methods.
DOC (Shu al. 2017: A CNN-based classifier replacing softmax with one-vs-rest sig- moid outputs to reduce open-space risk.
DeepUnk (Lin Xu 2019): An LSTM-baased model that learns deep intent features using
large-margin cosine loss.
2021): ; A BERT-based method that learns adaptive spherical decision boundaries for open intent detection.
ADB (Zhang et al.
2023): A BERT-based method that inflates and shrinks decision bound- CLAP (Liu et al.
aries to balance ID/OOD separation.
KNNCon (Zhou al. 2022 A BERT-based method that leverages KNN-based con- trastive learning during training and applies LOF for OOD scoring.
DyEn (Zhou et al. 2023): A BERT-based method that dynamically ensembles internal classifiers and uses early exits mitigate PLM overthinking.
LLM-OOD (Liu et al. 2024): An LLM-bbsed method that reformulates classification as text generation, leveraging LLM fine-tuning to produce more isotropic embeddings for OOD detection.
OOD detection methods (model-agnostic)). These methods provide scoring mechanisms that can be applied on top of different backbones:
OpenMax Bendale & Boult 2016): Fits Weibull distributions to logits to estimate open- set probabilities.
Energy (Liu et al. dence.
2020): Uses energy-based confidence scoring to mitigate overconfi-
VOS (Du et al.
2022):
Generates virtual Gaussian-based outliers for decision boundary
regularization.
y Improves on VOS by synthesizing non non-parametric outliers without distributional assumptions.
NPO (Tao et al.
ViM Wang et al. 2022): Computes virtual logit margins to enhance ID/OOD separation.
SHE Buuns & Fukai 2023): Extends Hopfield networks with higher-order (setwise) connections encoded via simplicial complex, increasing memory storage capacity and strengthening attractor dynamics; also instantiated with continuous Hopfield networks, sug- gesting potential improvements Transformer attention.
18

---

Published as a conference paper at ICLR 2026
Confidence calibration methods (model-agnostic). These methods aim to improve the reliability of predictive probabilities and can be applied on top of different backbones:
Temperature Scaling (Guo et al. 2017): simple yet effective post-hoc calibration tech- nique that rescales logits with single temperature parameter to alleviate overconfidence.
LogitNorm (Wei et al. 2022): A training-time regularization method that normalizes logits across samples, encouraging more consistent confidence estimates and improving model calibration.
Implementation details. All backhone-specific methods reproduced strictly following the pa- rameter settings reported in their original papers. For OOD detection methods and confidence cal- ibration methods, we implement them within a unified framework to ensure consistency across backbones. It is worth noting that most of these methods were originally developed in the com- puter vision community and have not been directly applied open-set text classification (OSTC). Therefore, we adapt their formulations to the OSTC setting and re-implement them based on the pytorch lobrary for fair comparison. For these these methods, we first fine-tuned BERT or LLM on the training set and classify samples with an OOD score exceeding predefined threshold as OOD. Following prior work (Zeng al. (02T). we set this threshold to 0.5.
B.4 MORE IMPLEMENT DETAILS
Implement Details For a fair comparison, we used bert-base-uncased and bert-base-chinesd backbones for discriminative language models English and Chinese datasets, respectively, and LLaMA3.1-8B-mm.. Instrucf as the backbone for generative models. For BERT (Devlin 2018 we applied full-parameter fine-tuning method for training. For LLaMA (Touvron et 2023). we employed QLoRA (Dettmers et al., 2024) to minimize additional parameter requirements during training. We adopted AdamW optimizer (Loshchilov & Hutter 2019) and set learning rate as le-4, the training batch size as 16, the epochs 3, label partition number s as 2, and virtual OOD subspace dimension as 20. All experiments were conducted on a machine equipped with 8 NVIDIA A100 GPUs (each with 40GB memory). The training was performed for 3 epochs for LLM--based methods and 20 epochs for non non-LLM-bee -ased methods. Unless otherwise stated, training is conducted with a learning rate of 5 x 10-5, batch size of 16 for training, and batch size of 32 for evaluation. For LLM training, we utilized a parameter-efficient fine-tuning (PEFT) method to minimize additional parameter requirements. Specifically, we employed the QLoRA technique (Dettmers et al. 2024), which freezes the weights of pre-trained LLMs and integrates trainable low-rank decomposition matrices into each Transformer layer. During fine-tuning, the model predicted answers using only the class label tokens to compute the auto-regressive loss. During inference, we utilized the BERT pooler layer as the PLM embedding function in Eq
C MORE DETAILS ON EXPERIMENTAL RESULTS
C.I MORE DETAILS FOR ABLATION STUDIES
In addition to the ablation study on the CLINC dataset, we also conduct the same experiments on the StackOverflow and BANKING datasets. The results, summarized in Slaland Figure are consistent with our previous observations, further validating the effectiveness of the modules.
We provide additional ablation evidence to further verify that reformulation (Open-set SFT) is essential and serves as the foundation for subsequent modules. To avoid ambiguity, we distinguish two training paradigms:
Closed-set SFT fine-tunes the LLM only on ID labels, optimizing to generate class outputs exclusively; no OOD semantics are introduced during training.
"ntps://hugggnngfcc..c
bo/google-bertbbet-bbe
'https//hugginnfacccco waeeeeooooooeeeeeee co/google- bearteeeeeeee--b---cinee
hhtps://huggingface. wwwmeta- Ilama/ waw.m//lamaa..1111m1 1-t.llamma..-----ssmtt-t
19

---

Published as a conference paper at ICLR 2026
UiLLMM wi Cat
UMLLM wis callerotio aralogy
UNLLM
UsLLM cullbration
CoLLM
UnLLM wio Sent
UZLLM wio anology
160
5a0o 0.25
Raio-022
80
100
100
50
100
Rato 675
Reii=077
70
ACC
F1
K-F1
N-F1
ACC
F1
K-FI
N-FI
(a) Ablation study on BANKING dataset.
(b) Ablation study on StackOverflow dataset.
Figure S1: Ablation study on BANKING and StackOverflow datasets.
Table S3: Extended ablation study on CLINC with known-class ratio 0.25.
Category
Method
ACC(%)
F1(%)
K-FI(%)
N-FI(%)
Closed-act SFT
24.31
48.40
49.71
0.00
LLM Output
Contrastive Learning
24.53
48.50
0.35
Contrustive Learning Orthogonal Regularizer
24.58
47.53
48.79
0.82
Contrastive Learnin Orthogonal Regularizer Analogy
36.89
43.81
43.98
37.27
LLM-OOD
65.66
66.68
66.65
67. 89
Contrastive Learning
68.00
30.21
29.71
78,
Post-hoc
Contmisivee Learning Onthogonal Regularizer
81.02
74.33
73.68
86.28
Contrasive Learning Orthggnnl Regularizer Analogy
80,80
73.49
73.15
86.16
Open-set SFT
80.18
75.50
75.24
85.30
Contrastive Learning
82.22
77.57
86.88
LLM Output
Contrastive Learnin Orthogonal Regularizer
83.51
78.98
78.75
88.09
Contrastive Learnin Onthogonal Regularizer Calibration
85.09
90.04
Contrasivve Learnin Orthogonal Regularizer Calibration Analogy
89.87
83.28
83.01
93.15
Open-set SFT fine-tunes the LLM with both ID labels and conditional-OOD labels, en- abling the model to explicitly learn an additional OOD class during training.
Setup. Starting from either Closed-set SFT or Open-set SFT, we sequentially integrate the pro- posed modules (contrastive learning, orthogonal regularization, calibration, and maggeeddd self-reffe reff-reflection and compare against the post-hoc baseline (LLM-OOD). Table SS reports results on CLINC with known-class ratio 0.25.
Results and discussion. Table S3 shows that stacking modules top of Closed-set SFT does reliably alleviate the large-scale overfitting of LLM outputs to the ID label space; performance can even degrade (e.g., Fl from 48.40 43.81). In contrast, Open-set SFT alone yields a substantial gain (FI from 48.40 to 75.50), and the subsequent modules further improve performance, reaching 83.28 F1. These results support our claim that Open-set SFT provides the crucial mechanism that makes the later components effective.
C.2 MORE DETAILS FOR PARAMETER SENSITIVITY ANALYSIS
Besides Aunh, we also conducted parameter sensitive analysis of Ad, and [Vel.
The Influence of Aa We analyzed the impact of > on model performance by varying it from 0.1 to 1.0 (Figure S2a)). Results show that contrastive loss is effective for both ID classification and OOD detection, with the balanced performance at Ag = 0.8, enhancing ID-OOD separation. However, higher Xa values harm performance, orthogonal loss compresses ID representations, weakening intra-ID classification. This highlights the necessity balancing generative and con- trastive objectives for optimal results.
20

---

Published as a conference paper at ICLR 2026
K-FISS))
100
K-FNN
N-F1N)
90
90
90
80
70
70
323
677
022
375
025
0.2
0..
0.5
1.0
62
0.4
0.6
088
1.0
10
30
40
19
20
30
50
(a) Parameter sensitivity analysis of As the BANKING dataset.
(b) Parameter sensitivity analysis of [V]]] the BANKING dataset.
Figure S2: Ablation study on different datasets.
Table S4: Performance comparison on CLINC dataset across different prompting strategies.
Backbome
Method
0.75
ACC
F1
K-FI
N-FI
ACC
F1
K-FI
N.FI
ACC
F1
K-FI
N.FI
Zero-Shot
53.05
52.04
51.85
59.37
50.42
58.07
26
43.65
54.38
57.01
57.24
31.56
LLaMA3.1-SB
Few-shet
55,40
49.81
49.45
63.01
54.24
38.00
37.66
63.22
45.18
44.48
44.57
34,87
Cot
50.60
54.99
55.00
54.43
$7.09
63.45
63.62 63.62
50.88
62.64
65
65.62
40.10
Analogy
61.97
81.96
61,26
88.16
56.61
69.78
56.35
76.52
47.62
52.04
47.60
50.13
Zero-Shot
56.11
57.77
57.65
62.20
61.17
62,90
62.93
60.57
65.26
66.85
67. 00
50.47
Qwen2.5.32B
Fow-sher
74.32
19.27
17.52
84.30
55,40
19.29
62
69.26
34.25
15.63
15.39
42.20
Cat
52.42
56.84
56.86
56.30
56.86
62. 68
88
48.06
62.91
66.49
66.75
37.32
Analogy
78.81
88.22
78.46
91.90
73.02
81,38
72.87
84,55
71,21
72.09
71.27
64.58
Zero-Shot
57.11
60.33
58.70
61.90
64.53
69.47
68 69
59, .2
71.42
75.57
75.11
$1.93
DeepSeek-V3-Uk V3-0324
Few-sbet
54.71
57.87
62.74
76.27
75.87
46.00
Col
60.71
62.75
61.03
65 54.554
73.24
75.87
74.93
71.44
75.91
77.99
77. 74.44
62.28
Analogy
81.35
91.80
83.59
86.01
82.05
73.61
LLaMa3,1 L-8
UnLLM
90.58
84.16
83,90
93.58
92.96
93.42
93.42
93.00
94.87
96.88
96.94
89.73
Qwen2.5.7B
UnLLM
87.82
93.99 96
87,76
94.44
96.37
96.42
90.54
Qwen2.5. 3.BB
UnLLL
89.42
90.07
90.00
92.67
95.51
96.62
96.63
95.63
94,13
96.51
96.58
89.17
The Influence of V&l We examined the impact of the number of virtual outliers per class [Vx in Figure S2b where vary [Vel across {10, 20, 30, 40, 50). In general, larger [Vx] is advantageous as it allows for a more accurate estimation of Gaussian distribution parameters. However, in many cases, the model struggles to sample sufficient number of negative instances within the designated low-confidence region, leading to a decline in performance. Consequently, we set the queue size 20 in our experiments. Nevertheless, in some cases, a smaller queue size is necessary due the limited number of instances available for certain classes
C.3 MORE DETAILS OF RESULTS ON UNLLM Vs. LLMS WITH DIFFERENT PROMPTING STRATEGIES
Table $4 reports a detailed comparison between UnLLM and strong prompting baselines on CLINC under different known-class ratios. We consider standard prompting strategies, including zero-shot, few-shot, and chain-of-thought (CoT), as well as the analogy prompting used in our inference-time self-reflection. Overall, prompting alone is insufficient while analogy prompting can improve results for some backbones and ratios, its behavior is unstable across settings and may degrade as the known-class ratio increases. In contrast, UnLLM achieves consistently strong performance across backbones and ratios, demonstrating that the gains mainly come from our training time reformulation and learning objectives rather than prompt engineering.
provide robust open-set performance:
C.4 RESULTS ON UNLLM VS. HALLUCINATION DETECTION METHODS
Another line of research that appears related to OSTC is hallucination detection in LLMs. The relevance lies in their shared focus uncertainty estimation both tasks aim to identify model outputs that deviate from expected or reliable content. However, a fundamental distinction exists: hallucination detection primarily targets generation-based inconsistencies, often evaluating factual correctness in open-ended text generation, whereas OSTC is a classification task that jointly models in-distribution classification and OOD detection. Accurate class prediction is prerequisite for
21

---

Published as a conference paper at ICLR 2026
Table S5: Performance Comparison vs, hallucination detection methods
Ratio
Method
BANKING
CLINC
StackOverflow
ACC
Fi
K-FI
N-I1
ACC
F11
K-FI
N-I1
ACC
[T]
K.FI
N.F1
Check (Altn Score)
71.09
74
36.45
74.96
85.69
C-Checkk (Hidden Sccre)
69.82
16.55
15.78
81.8
69.46
15.41
16.66
81.45
75.00
14.72
0.44
85.7
0.25
CED Score
59.23
5.04
74.45
58.31
2.34
0.41
73.64
67.5
13.64
0.25
80.59
HaluEival (CCTT)
50.63
49.81
52.09
S6.32
50.60
$4.99
55,00
54.43
52.74
55.41
55.39
55.53
UnLLM
87.86
75.89
75.04
92.02
90.58
84.16
83.90
93.58
94,00
89.87
88.65
96.00
L.M. Check (Aitn Score)
50.16
16.78
16.6
64.68
50.89
19.84
20.70
64.57
49.91
6.17
0.12
66.57
M-Checkk (Hidden Score)
51.04
13.07
11,98
66.17
48.76
15.77
16.33
63.15
49.91
6.17
0.12
66.57
CED Score
45.61
2.07
0.47
39.16
1.02
0.29
56.24
44.52
5.65
0.05
61.63
HolaEval (COTT)
51.58
51.79
53.21
49.53
57.09
63.45
63.62
50.88
69.06
75.02
76.45
60.75
84.57
82.82
93.42
93.42
93.00
92.20
91.97
91.91
92.60
L.M.M Check (Attn Score)
31.84
21.25
21.37
40.01
29.6
14.67
14.58
39.94
25.11
2.75
0,25
40.05
LLM-Check (Hidden Score)
10.94
31.20
22.93
40.02
0.75
CED Score
22.32
1.67
1.07
36.23
24.04
0.34
0.00
38.77
18.69
2.01
0.04
31.48
HaluEval (COT)
56.87
61.78 78
63.4
62,64
65.40
65.62
40.10
80.52
84.27
85,73
62.42
UnLLLM
84.54
88.31
88.58
72.60
94.87
96.88
96.94
89.73
89.68
91.78
92.36
83.08
Table S6: Performance of EnergyBaset and UnLLM under different similarity thresholds on CLINC (KCR=0.25).
Threshold
Method
ACC(%) %)
F1(%)
K-F1(%)
N-F1(%)
EnergyBased
84.47
67.78
66.63
89 19.45
0.96
UnLLM
91.93
80.85
80.14
94.42
EnergyBased
76.92
53.65
49.21
84.75
0.97
UnLLM 97.44
94.19
93.33
98.46
OSTC, which necessitates a different kind of uncertainty modeling that is sensitive to categorical decision boundaries rather than semantic fidelity in free-form generation.
Despite these differences, we include hallucination detection baselines in our evaluation for com- pleteness. Specifically, we implemented representative scoring functions from LLM-Check (Srira- manan et al. 2024) (c.g., Attention Score, Hidden State Score), CED score (Lee et al. 2024). and HaluEval (Li al. 2023) (e.g., Chain-of-Thougee Consistency) on our fine-tuned LLM and evalu- ated them under the OSTC setting. Experimental results in Table |55 confirm that these methods are inadequate in OSTC scenarios: although some baselines can achieve up to 80% of our method's s per- formance on N-F1, their K-FI scores remain significantly lower. This gap highlights the difficulty of balancing classification accuracy and OOD detection, which is the unique objective and core challenge of OSTC. Moreover, these hallucination baselines fail consider knowledge injection and decision boundary formation during fine-tuning, both of which are crucial for OOD detection in classification tasks.
C.5 MORE DETAILS FOR CASE STUDY
We clarify that both overconfidence and representation-ouupil misalignment are well-established phenomena in LLMs, and we further verify them quantitatively in our OSTC setting.
Evidence for overconfidence. LLMs often make overly confident predictions when a test utter- ance semantically close to some known labels. To substantiate this, for each test sample we compute its similarity to all candidate labels and use the similarity associated with the predicted label as confusability score. We then apply different similarity thresholds and retain subsets of test samples with increasingly high confusability (i.e., harder cases where energy- post-hoc rejec- tion is expected to be less reliable). Table So shows that as the threshold increases (harder subsets), the Energy Based baseline degrades substantially, while UnLLM remains strong, directly evidencing both the existence of overconfidence and the effectiveness of our framework in mitigating it.
Evidence for representation-oufuut misalignment. Prior work has shown that LLMs may in- ternally encode correct distinctions while the output layer fails to express them (Li et al. 2024). To validate this in our setting, we compare model variants by measuring (i) AUC, which reflects ID- OOD separability in representation space, and (ii) K-F1/N-F1, which reflect output correctness
22

---

Published as a conference paper at ICLR 2026
Table S7: Calibration ablation on CLINC: AUC (representation separability) vs. K-FI/N-FI (output correctness).
0.25
0.5
0.75
AUC(S)
K-FI(S)
N-FI(())
AUC(%)
K K-F((S)
N-F1(S)
AUC(S)
K-FI(%)
N-F1(%)
Closed-set SFT
90.79
49.71
0.00
92.23
71.96
0.18
91.63
87.76
0.00
Closed SPT (LLM OOD)
90.79
66.65
67.89
92.23
78.85
46.07
91.63
91.96
60.82
Open-set SFT (no calibration)
94.12
77.32
86.83
97.29
91.29
86.61
98.70
94.
85.12
Open-se set SFT (only calibration)
96.33
79.30
88.61
97.47
91.25
87.21
98.75
96.62
88.50
UnLLI
97.29
83.90
93.58
98.42
93.42
93.00
99.13
96.94
89.73
ID and OOD, respectively, Results in (ablleSSrreeal clear mismatch under the closed-set training paradigm: Closed-set SFT yields high AUC yet nearly zero N-FI, indicating that the model can separate ID/OOD internally but cannot express "OOD" in its outputs. Open-set SFT substantially improves both separability and output expressiveness, and calibration further tightens this alignment, consistently increasing N-FI across ratios. Finally, UnLLM achieves the best overall alignment and performance.
C.6
MORE DETAILS FOR OVERALL PERFORMANCE
Beyond the main text, we additionally compare suite of BERT-based OOD detection baselines and confidence calibration methods (Temperature Scaling, LogitNorm) while temperature scaling and logit normalization are effective for probability calibration.
As shown in Tables S8 $9 model-agnostic OOD scoring helps BERT in open-set metrics but still trails end-to-end OWD-aware training, while calibration methods (Temperature Scaling, LogitNorm) primarily improve reliability of confidences with limited impact OSTC accuracy/F1. This rein- forces the need for an OOD-aware pipeline such as UnLLMfor robust OSTC.
D LIMITATIONS
Our proposed approach achieves substantial performance gains over SOTA baselines across multiple public OSTC benchmarks, even under constrained computational resources. Due to limited access to large-scale hardware, our experiments are conducted using the parameter-efficient QLoRA fine- tuning strategy. Remarkably, this lightweight setting already leads significant improvements existing methods, demonstrating the strong effectiveness and practicality of our approach. However, have not yet evaluated under full- full-parann parameterr fine-tuning, Exploring such directions constitutes an important part of our future work.
E THE USE OF LARGE LANGUAGE MODELS STATEMENT
The authors use Large Language Models (LLMs) assistive tool in the preparation of this manuscript, accordance with the ICLR 2026 policy. We LLMs proofread, check grammar, and refine the language in the manuscript for improved clarity and readability.
23

---

Published as a conference paper at ICLR 2026
Table S8: Performance of various methods across 6 datasets at different ratios. Metrics include ACC and FI. The best results are highlighted in bold, while the second-best results are underscored. Each result represents the mean value of four repetitive experiments.
BANKING
CLINC
StackOverdlow
Reviews
Newsgroups Fl
THUCnews
Ratão
Backhone
Method
ACC
F1
ACC
F1
ACC
F1
ACC
F1
ACC
ACC
F1
CNN LSTM
DOC
74:70
67.44
80.23
74.82
77.06
68.42
71.31
59.03
85.26
68.65
73.57
56, 29
DeepUnnk
82.20
71.71
87.36
75.74
83.22
70.75
47.992
45.60
64.86
58.91
43.86
42.73
0.25
BERT
OpenMax
59.02
33.72
64.28
33.89
32.49
43.30
31.52
38.26
32.79
40.22
49.33
48.16
Energy
60.16
57.43
74.04
60.63
61.71
58.97
39.67
45.92
VOS
38.76
37:03
51.14
54.00 $4.04
75.58
67.88
73.16
51.57
49.71
49.74
68.12
62.
VIM
52.66
52.26
60.73
61.24
54.36
56,72
57.95
50.19
42.53
16.78
75.46
68.55
52.45
36.38
39.81
51.02
TemperaturSeaalnn
50.80
37.45
$3.50
53.62
40.68
48.97 9/2
48, 88
43.08
34.70
40.26
37.72
39.99
LogilNonm
70.35
21.71
74,30
45. 92
51.44
55,78
57.92
39.
45.60
44,40
22.32
28.31
NPO
37.31
56.00
49.07
51.82
68.54
62.99
ADB
54.58
51.61
60.22
59. 19.20
83.12
77.99
58. 16
49.83 83
29.59
29.59
55.86
59.79
CLAP
60.64
59.17
65.14
62.17 17
75.37
71,76
68. 68.82
56, 28
39.70
37.69
44.12
41.72
KNNCon
66.21
83.65
79.75
62.97
56.32
42.44
46.51
DyEa
67.84
65.94
82,12
77.41
52.79
59.46
-11.06
44.12
61.98
59.29
28.22
37:47
LLMMA
LLM-OOD
73.97
66.33
65.66
66. 66.68
86,65
$0.56
80.03
61. 60
69.86
62.26
73.83
68.
OpenMax
64.24
89.
39.90
38.05
22.13
26.62
EnergyBased
86.02
72.05
88.93
83.32
91.60
83.19
83.19
$1.40
76.17
64.27
87.25
69.39
VOS
85.08
75.16
85.54
62.55
83.98
66.65
82.95
63..49
77.76
65.17
83.31
63.37
50.82
75.22 26
62.37
87.61
SHE
58.68
52.11
75.83
67.76
10.83
44,28
18.99
47.99
46.69
51.05
22.32
27.12
Temperaturecaalng
45.24
50.57
45.49
54.15
38.96
43.71
44.69
45.53
37.11
43,55
50.14
46.05
LogiNNom
71:73
66.34
82.84
78 28
48.41
49.53
49,26
19.19
30.31
38.87
45.87
43.36
NPO
84.87
74.63
82.74
82,81
93.29
87.61
83,56
56.98
78.55
63.57
87,83
68.
UnLLM
87.86
75.89
90.58
84.16
94,00
89.87
86,98
64.44
86.75
72.24
91.66
86.34
CNN
DOC
75.94
74.79
84.00
84.23
78.17
777.86
56.25
75.97
77.46
58.97
49,
LSTM
DoepUnk
67.67
60.14
76.36
69.96
79.65
77.26
34,80
40.87
65.53
68.28
50.01
57.81
0.5
BERT
OpenMax
57.42
44.16
62,53
53.81
55.47
66.38
-15.84
$4.83
56.52
64.78
58.78
69.
Energy
72.24
78.13
79.83
85.74
72.68
77.56
48.49
59.02
68.49
74.57
62.78
69.61
VOS
54:17
$1.20
63,82
72.31
84.27
84.17
50.14
62.64
39
68.
75,
VIM
63.17
65.92
67.61
77.34
79.53
81.96
64,77
62.27
61.36
67.84
74.81
79 10
SHE
61.54
72.65
65.99
78.58
63.91
72.32
54,08
61.99
58.56
68.15
61.32
70.52
TempenatueScaaling
67.26
68.
72.50
58.33
51.27
54.38
59.03
54.67
66.95
LogiNNor
55.60
38, 39
62,53
36,82
77.16
79. 33
59.54
42.46
54.24
44.33
58. 68
69,
NPO
56.14
44.04
58,90
70.84
85.38
84.92
66,23
51.12
61.70
56.13
67.66
75.16
ADB
59.13
62.93
65.75
48.38
51.47
32,90
72.89
75.82
CLAP
53.55
55.99
64.31
67. 11
84.72
85.02
53.91
55.005
46.07
49.54
66.28
70.37
KNNCon
76.54
80.21
73.10
$2.44
83.54
8-1.93
51.47
60.73
71.38
76.33
66.41
72.95
DyBa
78.29
84.34
47.78
59.21
71.16
58.97
67.15
LLaMA
LLM-OOD
79.39
75.23
64,92
78.42
89.43
89.70
61.80
65.29
75.51
76.74
80.88
82.80
OpenMax
72.41
64.22
86.
85, 34
70.52
75.89
44,53
57.89
67.18
7.3.15
53.87
66. 12
EnergyBased
79.78
75.16
89.76
90.06
75.98
75.52
82.00
80.25
VOS
81.74
79.06
76.67
66.57
70.74
60.47
21.93
62. 30
75,80
76.71
78.97
78.08
VIM
66.11
49.26
83.52
78. 36
88.41
87.23
68.54
62. 688
77 60
78. 94
81.59
82.60
79.90
77.48
72
$4.37
66.26
$7.25
67.77
TemperatueSeaaingg
60.52
72.17
64.57
78.09
62. 80
70.27
55.38
62.82
58.03
66.79
57.63
68.66
LogiNoom
80.96
80.90
90.34
91.92
69.74
75. 18
66.30
66.48
61.72
69.38
61.81
71.21
NPO
91.31
77.16
81.04
80.15
UnLLM
84.57
82.82
92.96
93.42
92.20
91.97
76.32
69.26
80.35
83.34
82.49
84.32
CNN
DOC
73.33
78.68
83.24
87.30 30
811 .40
84.55
55.75 77
14.17
67.51
08.08
36.87
27.97
LSTM
DeepUink
59.22
63.92
77.96
81.72
26.82
63.08
67.05
0.75
BERT
OpenMaxx
55.86
$7.50
65.84
69.09
70.95
76.60
$2.83
58.34
69.72
77400
73.60
79.20
Energy
77.04
85.27
87.74
92.70
83.49
86.82
56.99
63..3 34
78.21
82.83
80.27
83.37
VOS
$6.00
58.05
75.57
80.31
82.94
86.62
50.66
66.26
69.15
81.04
84.53
VIM
69.60
74.58
79.66 64.644
86,6
84.50
87.40
57.34
67
72,96
76.94
83.94
56.55
SHE
74.58
84.04
83.48
90.85
78.97
83.56
59.89
65.881
75.97
$0.89
77.54
H2.37
TemperaturScaalin
$3.86
53.40
74.07
79.85
78.08
82.70
53.15
54.93
63.08
65,45
73.37
79.00
LogifNom
39.48
28.06
06
43.8%
31.32
79.90
84.04
45. 34
39:24
54.
50.82
76.27
81.22
NPO
55.22
56.15
75.80
82.62
84.21
87.22
50.50
48.85
65.88
69.06
81.89
81.83
ADB
65.28
71.57
71.90
77.
83.73
86.87
43.81
47.05
30.83
34,00
72.59
77.51
CLAP
57.02
10
65.13
70.998
87.46
50.33
52.73
40.82
44.55
66. 10
70.85
KNNCon
80.19
86.46
90.17
93.80
85. 20
88.46
60.64
64.80
80.80
84.87
85.39
88.
DyEa
79.13
86.24
89.16
93.25
78.52
83.47
57.12
63.15
78..26
82.85
79.94
82.95
LLMMA
LLM-000
71.93
$4.69
85.72 72
87.92
79.48
85.02
87.96
OpenMax
65.94
70.63
83.21
87.25
82.59
87.09
55.72
62.11
77..51
81.13
70.41
76.52
EnergyBased
72.70
76.62
87,20
90.08
82.45
85.20
41.59
31.92
67.65
69.90
77.12
79.45 48
VOS
79.44
$1.33
78.79
71.13
74.69
80.52
82.96
VIM
41.23
31.52
63.62
62.56
76.23
78.81
59. 10
59.30
75.99
79.11
$3,80
85. 18
SHE
63.39
70.09
84.17
87.73
82.14
86.49
60.74
60.06
73.47
77.95
75.06
79. 79.74
TemperatureScaling
84.08
$1.20
90.07
78.56
60.57
74.08
79.06
74.27
79.60
LogitNom
74.01
77.31
91.48
94.11
84.29
88.41
60. 16
62.01
76.90
81.06
76.13
81.07
NPO
75.43
79.64
90.48
95.15
86.87
89.42
46,11
39.27
72.34
75.66
76.20
77.92
UnLLM
88.31
94.87
89.68
91.78
62.16
67.48
85.38
90.26
92.26
24

---

Published as a conference paper at ICLR 2026
Table S9: Performance of various methods across 6 datasets at different ratios. Metrics include K-F1 and N-FI. The best results are highlighted in bold, while the second-best results are underscored. Each result represents the mean value of four repetitive experiments.
BANKING
CLINC
StackOverdlow
Reviews K-F1
Newsgroups
THUCnews
Ratão
Backhone
Method
K-F1
N-F1
K-FI
N-F1
K-F1
N-F1
N-FI
K-F1
N-F1
K-F1
N-F1
CNN LSTM
DOC
66.40
81.57
75:46
85.63
69:81
83.61
59.59
79:57
66:21
89.59
56.43
82.27
DeepUnnk
70.87
87.74
75.31
91.36
67.10
88.93
15.118
$1.59
56.19
71.51
41.40
46.72
0.25
BERT
OpenMax
32.05
65.29
32.89
70.50
47.87
20.46
39.48
23.55
44.18
20.41
47.18
51.10
Energy
57.00
71.41
38.
57.43
38.90
VOS
36.72
42.80 86
54.07
53.21
65.05
82.17
49.03
82.08
48.99
53.49
57.58
75.
VIM
51.95
58.11
61.13
65.44
56.58
57.45
-1.005
63.86
18.03
40.52
64.06
82.02
53.10
40.21
56.13
48.92
38.71
48.58
34.37
46.60
55.77
TemperaturSeaalnn
36.24
60..4
53.52
57.35
51.011
38.50
41.98
56.37
42.68
28. 16
41.06
36.77
LogilNonm
18.57
81.24
44,94
82.25
55. 88
55 25
37,08
68.51
43.79
47.43
35.85
5.68
NPO
48.96
56.35
63.90
49.83
58.57
ADB
$1.07
61.84
59,00
66.67
75.91
88. 37
48.27
58. 56
28.70
34.02
59.98
62.20
CLAP
58.69
68.21
61. 88
72.75
69:75
81.81
54,45
78.30
35.79
47.22
39,23
49.17
KNNCon
70.63
79.53
53.75
55.71
60.18
47.91
42.33
DyEa
65.53
73.71
77.16
86.85
60.17
55.88
-44.28
42.11
57.74
67.04
43.99
17,90
LLMMA
LLM-OOD
65.62
79.91
66.65
67.89
78.54
90.67
60.09
80.68
59.33
76,93
64.17
80.08
OpenMax
85
13:76
33.06
7.29
EnergyBased
71.07
90.82
83
91.46
80,91
94.57
18.21
89:69
60.45
83.35
61.72
92.41
VOS
74.39
89.82
61.79
90.95
62.06
89.64
61.35
89.09
61.28
84.66
$4.59
89.72
48.95
86.32
72.41
60.41
64.55
70:41
92.62
SHE
51.41
65.38
67.38
81,77
45.82
36,60
47,52
53.54
51.84
17.08
33.53
7.91
Temperaturecallng
50.89
44.40
54.43
43.52
45.94
32.58
45.35
47.91
46.18
30.43
42.49
56.73
LogiNNom
65.73
77.89
78.04
87.29
49.79
24
43.48
15.87
42.66
45.48
NPO
73.83
89.73
82,60
90.29
86.01
60
54,25
89.78
59.19
85,49
06.06
92.79
UnLLM
75.04
92.02
83,90
93.58
88.65
96,00
62.16
91.94
68.33
91.82
83.63
94.46
CNN
DOC
74:50
77:92
84.10
84.38
79..57
55.42
02.522
77.
'14.67
46.63
64.58
LSTM
DoepUnk
59.75
74.91
69.80
81.54
76.79
81.86
-1..6
20.12
69.07
59.93
62.55
21.58
0.5
BERT
OpenMax
43.75
59.95
53.68
63.20
70.21
28. 10
56.02
25.16
68.00
32.49
71.31
36.53
Energy
78.40
68.00
85,88
75.78
78.59
67.22
60.22
29.17
76.34
56.83
73.23
44.34
VOS
$1.11
72.59
$1.59
12
84.71
72.85
65.17
56.54
73.79
59.
VIM
66.06
60:41
77.62
56.46
82.45
77.03
62.07
67.11
70,06
45.74
80.42
69.88
SHE
73.35
46.27
78.96
50,20
74.45
50.95
62.72
43.72
71.87
30.95
74.51
42.59
TempenatueScaainng
46.96
68.45
63.35
74.06
56.95
14
60:47
61.01
39.22
72.74
26.41
LogiNNor
37.74
63.08
36.37
70.60
79.72
76.01
-11.48
04.044
43.52
$2.43
74.08
38. 14
NPO
43.58
61.90
71,23
41.63
84.82
85,99
50,23
73.42
55.76
59.82
77.56
58.33
ADB
56.92
71.14
83.95
51.56
49.18
33,99
76.67
69.88
CLAP
56.04
54.13
67.15
65 63.94
99
85. 36
55.00
55.79
50,37
41.19
.9
62.51
KNNCon
80.35
75.13
82.71
62.30
85.11
83.12
61.31
46.13
77.60
63.58
76.44
-18.49
DyBa
34
88.38
77.06
77,88
62.19
71.79
LLaMA
LLM-OOD
75.06
81.65
78.85
46.07
89.77
89.54
65.55
59.18
77.11
72.95
83.34
78.66
OpenMax
63.87
77.49
85.31
88 DS
77.94
55.37
59,56
16.11
74.99
54.82
72.34
22.59
EnergyBased
82.76
74.93
75.39
76.77
82.11
80.26
VOS
78.93
83.94
66.38
$0.99
58.88
76.33
61.71
77.04
76.88
74.99
77.78
80.22
VIM
48.61
73.91
78.26
85.71
87.01
89.37
67.51
72.05
79. 14
76.92
84.09
80.21
$4.96
76.58
40:44
74.07
72.65
33,50
TemperatureScaling
72.90
44.64
78.51
46.01
72.79
45.00 06
63.47
46.54
70.23
32.41
73.52
34.69
LogiNoom
90.88
82.01
91.95
89 94
76.84
58 60
66.44
67.40
72.05
42.74
74.94
45. 14
NPO
91.31
23
76.82
78.17
766.3
82.37
79.90
UnLLM
82.74
85.72
93.42
93.00
91.91
92.60
68.81
80.59
83,95
77.20
84.83
80.75
CNN
DOC
78.85
61.25
87.37
72.81
84.90
72.30
53.95
52.53
67.66
57.588
29.77
41.32
LSTM
DeepUink
54.85
58.57
82.57
27.31
11.81
65.95
40.93
70.76
0.75
BERT
OpenMaxx
57.82
39.45
69.22
53.91
79.52
32.82
59,38
19.81
77.22
26.52
83.38
37.46
Energy
85:93
47.26
92.87
73.63
87.99
69.29
64.52
18.50
85.25
46.46
$8.15
30.87
VOS
58.38
39.45
80.51
58.20
72.19
48.49
47.16
70.65
46.54
86.14
69.39
VIM
75.04
48.39
86.89
$4.43
88.27
74.36
90
50.25
78.
47.80
87.76
74.51
SHE
84.96
31.57
91.15
56.93
85.43
55.57
66.51
38.91
84.06
33.25
85.32
$2.86
TemperaturScaalin
53.62
40.88
80.09
52.40
84.23
59.70
59.19
45.00 08
67.31
3.7.62
82.57
43.38
LogifNom
27.83
40.80
31.19
45.15
85.25
65.41 .1
122
43.79
51.27
44.17
83.83
55.11
NPO
56.45
39.05
82.92
49.72
88.02
75.09
41.89
47.05
70.67
44,89
86.14
71.73
ADB
72:07
43.48
78.40
51.69
87.65
75.10
47.35
35.5
35.00
17.93
79.59
54.59
CLAP
62.48
40.57
71.19
47.04
88.23
98
53.00)
42.41
46.16
20.47
72.82
50.93
KNNCon
86.90
61.61
93.91
80.67
89.38 38
74.55
65.22
49.02
86.36
62.53
90.19
63.
DyEa
86.79
54.74
93,38
78.02
85,43
$4,09
64,24
21.81
85.24
47.06
87,88
28.73
LLMMA
LLM-OOD
72.10
91.96
77.79
83.29
67.06
89.41
71.94
OpenMax
70.88
56.18
87.37
73.66
88.69 69
63.09
63.49
10.57
K3.14
$1.02
81.
27.75
EnergyBased
76.84
63.80
90.18
79.11
85.94
74.07
31.57
44.57
70.78
$6.69
80.33
71.01
VOS
65.65
$4.64
83.54
49.18
75.72
39.23
83.85
74.05
VIM
31.27
45.94
62.59
58.71
79.51
68.31
59.37
$6.56
80.12
63.99
86.40
78.06
SHE
70.70
35.08
87.87
71.67
88.19
60.97
67.09
55.55 85
79.89 79,80
50.14
82.78
49.35
TemperatureScaling
39.79
90.47
87.03
43.91
42.15
81.98
35.29
82.94
46.18
LogitNom
77.53
64.82
94.20
$4.11
89.83
67.13
62.29
$1.37
82.70
56.41
83.73
54.42
NPO
79.89
65.79
93.24
83, 14
90. 10
79.22
39.08
74
76.65
60.36
78.67
70.35
UnLLM
72.60
96.94
92.36
83.08
67.84
56.65
72.18
93.55
78.11
25


Reviews:
[{"review_id": "S6mZA8qG6X", "reviewer": "Reviewer_EwEz", "summary": "This paper proposes UnLLM, an unknown-aware large language model designed for open-set text classification. The authors introduce a 3-fold method, including:\n1) open-set generative fine-tuning to develop a K+1 classifier. \n2) OOD parameter calibration method to align the model’s internal cognitiion of the unknown with its outputs. \n3) analogy-augmented self-reflection mechanism to mitigate overconfidence.\n\nExperimental results across six datasets demonstrate that UnLLM consistently achieves SOTA performance in OOD detection while maintaining competitive accuracy in ID classification.", "questions": "Please refer to Weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "* This paper targets an important problem (Open-Set Text Classification), and the proposed method is novel.\n\n* Experimental results across six datasets demonstrate that UnLLM consistently achieves SOTA performance in OOD detection while maintaining competitive accuracy in ID classification.", "weaknesses": "1. The writing is not good enough, I find it hard to follow the insights of the proposed method and the details of the equations. For example,\n* What are the insights behind adding contrastive learning loss?\n* In Equation 2, the authors say $N_{y_{i,j}$ is the number of examples of $y_{i,j}$, but what happens when the model is optimized in batch?\n* Why authors could \"assume that the representation for each class k ∈ [1,K] follows a class-conditional\nGaussian distribution:\"?\n\n2. I have a concern about section `Calibration Direction Construction`, where authors propose to \"evaluate the fine-tuned LLM on the label partitioned validation set\". I don't think using statistics of validation set is a fair comparison to other baselines.\n\n3. The proposed 3-fold methods seem to have no relation, which makes me feel that LLM is too complicated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "goltLYxBic", "reviewer": "Reviewer_163y", "summary": "This paper investigates the open-set text classification (OSTC) problem, in which models must correctly classify in-distribution (ID) samples while identifying out-of-distribution (OOD) inputs. The authors propose UnLLM, an unknown-aware large language model that integrates open-set awareness directly into the fine-tuning process. The method reformulates classification as a subset-conditioned generation task, where random subsets of known labels are provided and instances whose true labels are excluded are treated as 'unknown'. To enhance representation quality and calibration, UnLLM incorporates contrastive learning, orthogonality constraints, and a post-hoc logit calibration step that aligns output weights with OOD-relevant representations. During inference, an analogy-augmented self-reflection mechanism enables the model to reassess uncertain cases and mitigate overconfidence. Experiments on six benchmark datasets demonstrate that UnLLM consistently outperforms both traditional PLM-based methods (e.g. ADB, CLAP, KNNCon) and recent LLM-based baselines (LLM-OOD, VOS, NPO), achieving strong results in both ID classification (K-F1) and OOD detection (N-F1).", "questions": "* At line 246, the authors state that the orthogonality loss minimises the projection of ID representations onto the principal subspace of outliers (Section 4.1, Eq. 8). However, minimising this projection only reduces the overlap between the two subspaces; it does not guarantee orthogonality or decorrelation in a strict sense. Could the authors clarify the theoretical rationale behind this formulation? Why is minimising the projection considered sufficient to ensure separation, rather than explicitly enlarging the null space of the outlier subspace or directly enforcing an orthogonality constraint between the ID and OOD representation spaces?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "* The main contribution of the paper lies in reformulating OSTC as a subset-conditioned text generation task, extending prior work that handled OOD detection only as a post-hoc step. This framing is new in the context of large language models and conceptually straightforward.\n\n* The authors evaluate their approach on six benchmark datasets and a broad range of baselines, covering both traditional PLM-based and more recent LLM-based methods. The experimental coverage is thorough and shows consistent empirical improvements.\n\n* Although the design is primarily empirical, the results suggest that the proposed paradigm can be effectively applied to large models and yields measurable performance gains in OSTC scenarios.", "weaknesses": "The main issue of the paper lies in its limited scientific depth. While the proposed framework achieves promising empirical results, it lacks sufficient theoretical or empirical analysis to explain why existing methods fail and why the proposed reformulation, which frames OSTC as a subset-conditioned text generation task, is essential or fundamentally different. As a result, the paper does not convincingly establish the underlying rationale or mechanisms that support its claimed contributions. The main concerns are summarised as follows:\n\n* **Lack of clear evidence that the proposed reformulation is essential.**  \n  The paper provides weak evidence that the subset-conditioned text generation reformulation is fundamentally important for open-set text classification. While the full framework shows improvements, the comparison setup is confusing and does not isolate the benefit of the reformulation itself.\n\n  * In particular, the representation-learning components (contrastive learning and orthogonality constraints) can be readily applied to existing methods, and their effectiveness has already been demonstrated in prior work. As shown in Table 1 and Figure 4, removing these components (‘w/o CL’ or ‘w/o Orth’) significantly reduces performance, often below other LLM-based baselines, suggesting that most of the gains may stem from these well-known techniques rather than from the proposed reformulation.  \n  * The Reflective Inference (Analogy-Augmented Self-Reflection) module also appears largely independent of the new formulation. It is introduced as an inference-time enhancement and could be attached to any model trained under different paradigms. Since this component does not interact with the subset-conditioned fine-tuning objective, its inclusion further obscures the specific contribution of the reformulation itself.  \n  * To substantiate the importance of the new formulation, the authors should include **control experiments** to isolate its contribution more clearly. *For example*, they could apply the same contrastive, orthogonality, and reflection modules to standard generative fine-tuning or other LLM-based baselines (such as LLM-OOD or VOS) without the subset-conditioning step. Comparing such models with and without access to the subset of known labels would help determine whether the reformulation itself contributes meaningfully beyond these auxiliary components.  \n  * Additionally, while the logit calibration component may depend on the new formulation, the paper also lacks a clear, controlled experimental analysis identifying which parts of the framework genuinely benefit from it.\n\n\n* **Unfair and inconclusive comparison between LLM-based and BERT-based baselines.**  \n  The paper emphasises that one of its main contributions is the application of LLMs to OSTC and reports that LLM-based baselines outperform BERT-based methods (Section 5.2). However, this comparison is not meaningful, as the models differ drastically in scale and architecture: the LLaMA-3.1-8B model contains billions of parameters, whereas the BERT-base models are two orders of magnitude smaller. The observed performance gap therefore likely reflects differences in model capacity and pretraining rather than an inherent advantage of the proposed generative fine-tuning paradigm.  \n\n* **Lack of evidence for consistency across model backbones.**  \n  The paper reports results on two LLM architectures (LLaMA 3.1-8B and Qwen 2.5-7B), but the baselines are only evaluated on LLaMA. As a result, it is unclear whether the proposed UnLLM framework consistently improves performance across different backbones or whether the observed gains are specific to a single model. To claim general applicability, the authors should include baseline comparisons on Qwen 2.5-7B or at least provide an analysis showing that the observed improvements are robust to architectural variations.\n\n* **Unclear and unsupported motivation, both theoretically and empirically.**  \n  The paper’s motivation is difficult to follow at both the conceptual and empirical levels. Several key statements, such as 'LLMs often exhibit overconfidence in their predictions' (Section 4.3) and 'we observe a misalignment between the LLM’s internal knowledge (representation space) and its outputs' (Section 4.2), are presented as empirical observations but are not supported by any systematic study or quantitative evidence. It remains unclear whether these issues actually occur under the presented OSTC setting, and no diagnostic analysis is provided to demonstrate their impact. Furthermore, the paper does not establish a clear theoretical or conceptual connection between these claimed problems and the proposed solutions. As a result, it is difficult to assess whether the introduced components are sufficient or even relevant to address the stated challenges.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iJGDPrTpx5", "reviewer": "Reviewer_rWfQ", "summary": "The paper tackles open-set text classification (OSTC) by making “unknown-class” awareness an explicit, trainable behavior for LLMs via UnLLM, a three-stage pipeline.", "questions": "- Consider a compact numeric ablation table ****(mean±std) and a reliability diagram (ECE).\n\n- What encoder, k, and index size are used for analogy; what’s the per-example latency?\n\n- Nice motivation—could you add one controlled test that separates *withheld in-domain labels* (conditional-OOD) from *cross-domain OOD*, to empirically reinforce generalization.\n\n- The experimental section should take into account advanced OOD detection methods that leverage additional auxiliary OOD data, as referenced in [1, 2, 3, 4].\n\n\n\n[1]: Ming, Yifei, et al. \"Poem: Out-of-distribution detection with posterior sampling.\" *International Conference on Machine Learning*. PMLR, 2022.\n\n[2]: Wang, Qizhou, et al. \"Out-of-distribution detection with implicit outlier transformation.\" *arXiv preprint arXiv:2303.05033* (2023).\n\n[3]: Zheng, Haotian, et al. \"Out-of-distribution Detection Learning with Unreliable Out-of-distribution Sources.\" *arXiv preprint arXiv:2311.03236* (2023).\n\n[4]: Wang, Qizhou, et al. \"Learning to augment distributions for out-of-distribution detection.\" *arXiv preprint arXiv:2311.01796* (2023).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The three-stage pipeline (contrastive + orthogonality regularization; K+1 logit calibration; analogy-augmented inference) is coherent and practical.\n\n- Shows consistent improvements on several benchmarks.\n\n- Simple, implementable training recipe that may make LLM classifiers less overconfident under partial label contexts.\n\n- The paper does not over-promise formal guarantees; instead it presents precise objectives/losses and a closed-form calibration projection (Eq. 4) with clear tensor-shape intent.", "weaknesses": "- This paper feels close in spirit to prior OE extensions and recent fine-tuning-with-auxiliary-data methods (WOODS, VOS, NPOS).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)