AP-OOD: Attention Pooling for Out-of- Distribution Detection
Authors: Claus Hofmann
Christian Huber
Bernhard Lehner
Daniel Klotz
Sepp Hochreiter
Werner Zellinger
Venue: ICLR 2026 Poster (2026)
Keywords: out-of-distribution detection
attention pooling
nlp
language models
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Out-of-distribution (OOD) detection, which maps high-dimensional data into a scalar OOD score, is critical for the reliable deployment of machine learning models. A key challenge in recent research is how to effectively leverage and aggregate token embeddings from language models to obtain the OOD score. In this work, we propose AP-OOD, a novel OOD detection method for natural language that goes beyond simple average-based aggregation by exploiting token-level information. AP-OOD is a semi-supervised approach that flexibly interpolates between unsupervised and supervised settings, enabling the use of limited auxiliary outlier data. Empirically, AP-OOD sets a new state of the art in OOD detection for text: in the unsupervised setting, it reduces the FPR95 (false positive rate at 95% true positives) from 27.77% to 5.91% on XSUM summarization, and from 75.19% to 68.13% on WMT15 En–Fr translation.

Published as a conference paper at ICLR 2026
AP-OOD: ATTENTION POOLING FOR OUT-OF- DISTRIBUTION DETECTION
Claus Hofmann' Christian Huber Bernhard Lehner?
Daniel Klotz" Sepp Hochreiter
Werner Zellinger*
1 Institute for Machine Learning, JKU LIT SAL IWS Lab,
Johannes Kepler University, Linz, Austria
Silicon Austria Labs, JKU LIT SAL IWS Lab, Linz, Austria
3 Interdisciplinary Transformation University Austria, Linz, Austria
ELLIS Unit, LIT AI Lab, Institute for Machine Learning, JKU Linz, Austria
A ABSTRACT
nuttof-distribution (OOD) detection, which maps high-dimensional data into scalar OOD score, is critical for the reliable deployment of machine learning models. A key challenge in recent research how to effectively leverage and aggregate token embeddings from language models to obtain the OOD score. In this work, we propose AP-OOD, a novel OOD detection method for natural language that goes beyond simple average- based aggregation by exploiting token-level information. AP-OOD a semi-supervised approach that flexibly interpolates between unsupervised and supervised settings, enabling the use of limited auxiliary outlier data. Empirically, AP-OOD sets a new state of the art in OOD detection for text: in the unsupervised setting, it reduces the FPR95 (false positive rate at 95% true positives) from 27. 7.8% to 4.67% on XSUM summarization, and from 77.08% to 70.37% on WMT15 En Fr translation. Code is available at https://ggthhu.ee.o milj/mm//uuaaa /apoood.
INTRODUCTION
Out-of-distribution (OOD) detection is essential for deploying machine learning models in the real world. In practical settings many models encounter inputs that deviate from the modcl's training distribution. For example, language model trained to summarize news articles might also receive a prompt with a cooking recipe. In such situations, models may assign unwarranted confidence to their predictions, leading to erroneous outputs and hallucination. A hallucination is a state in which the model generates output that is nonsensical or unfaithful (2023) observe that a common failure case in abstractive summarization is for the model to output "All images are copyrighted" when prompted to summarize news articles from a publisher (CNN) that differs from what it was trained on (BBC). Many authors attribute hallucination to model uncertainty (e.g., Yadkori et al., 2024; Aichberger which decomposes into aleatoric uncertainty (resulting from noise in the data) and epistemic uncertainty (resulting from a lack of training data). OOD prompts exhibit high epistemic uncertainty (Ren et al., 2023). The purpose of OOD detection is to classify these inputs as OOD such that the system can then, for instance, notify the user that no output can be generated. Many existing post-hoe OOD detection methods (e.g., Huang et al., 2021; Sun & Li, 2022; Wang et al., 2022) assume a classifier as the base model. In contrast, in language modeling, the base model is typically an autoregressive generative model without an explicit classification head. This necessitates the development of OOD detection methods specifically tailored for language modeling. Our contributions are as follows:
the prompt (Farquhar et al., 2024). For example, Ren et al.
al., 2025),
1. We propose AP-OOD, an OOD detection approach for natural language that leverages token-level information to detect. OOD sequences.

---

Published as a conference paper at ICLR 2026
Mahalerstto dodance with pooking
Figure 1: Illustrative example for the failure of mean pooling. (Left) ID and OOD sequences Z, E R2x2, where each sequence contains a pair of token embeddings with two features each. Token embeddings that belong to the same sequence The means of the ID and OOD sequences both cluster around the origin. (Right) mean pooling approach cannot discriminate between the ID and OOD sequences.
connected with lines. (Center)
uoe
Coond
Score of AP 00000000
Figure 2: Illustrative example for the mechanism that AP-OOD uses to correctly discriminate between IDD and OOD (as opposed to the mean pooling approaches). The setting is the same as in Figure 1. (Left) The loss landscape forms two basins at the locations of the ID token embeddings. (Center) After training AP-OOD with a single weight vector w, the learned w is located in one of the basins. (Right) AP-OOD achieves perfect discrimination between the ID and OOD sequences.
2. AP-OOD is a semi-supervised approach: It can be applied in unsupervised (i.e., when there exists no knowledge about OOD samples) and supervised scttings (i.c., when some OOD data of interest available to the practitioner), and smoothly interpolates between the two.
3. We show that AP-OOD can improve OOD detection for natural language in summa- rization and translation.
4. We provide a thcorctical motivation for the suitability of AP-OOD for OOD detection on tokenized data.
1.1 BACKGROUND
Conditional language models are typically trained given input sequences (x1, X2. σN) with æ; E x to autorogressively generate target sequences (y1, y2, .... yn) with y, X. The input sequences are drawn i.i.d.: I; ~ PID- We consider input sequences æ E X that deviate considerably from the data generation (T)(#) that defines the "normality" of our data as OOD. Following Ruff et al. (2021), an observed sequence is OOD if it is an element of the set
(x E X (ID(#) < <e) where 2 0,
(1)
and E IR is a density threshold. In practice it is common (e.g., Hendrycks & Gimpel, 2016; Lee ct al., 2018; Hofmann al., 2024) to define a score + R that uses an encoder
We X = Us21 8 for the of input sequences, and {e1.-.., vy is the vocabulary.
2

---

Published as a conference paper at ICLR 2026
6: + 3 (where 2 denotes an embedding space). Given and 6, OOD detection can be formulated as a binary classification task with the classes in-distribution (ID) and OOD:
ID if s()())) 22
B(x,7) = OOD if s()()) <
(2)
The outlier score should - in the best case preserve the density ranking, but does not have to fulfill all requirements of probability density (proper normalization nonnegativity). For evaluation, the threshold y E R is typically chosen such that 95% of ID samples from a previously unscen validation under the receiver operating characteristic (AUROC) can be directly computed on s(ő(x)) without fixing 7, since the AUROC sweeps over all possible thresholds.
are correctly classified ID. However, metrics like the area.
2
METHOD
AP-OOD is a semi-supervised method: It can trained without access to outlier data (unsupervised), and with access to outlier data (supervised), and can smoothly transition between those two scenarios as more outlier data becomes available for training. In the following, we first introduce AP-OOD in an unsupervised scenario (Section 2.1) and generalize to the supervised scenario (Section 2.2).
2.1 UNSUPERVISED OOD DETECTION
Background Ren et al. (2023) propose to detect OOD inputs using token embeddings obtained from a transformer encoder- model (Vaswani et al., 2017b) trained on the language modeling task. Given input sequence æ, they obtain sequence of token embeddings. They compare obtaining embeddings E E g² from the encoder dene X - Z and generating sequence of embeddings G 2 using the decoder ddec 2 2:
(3) For clarity, we write Z E Z for sequence of token embeddings, whether produced by the encoder or the decoder, and we call Z the sequence representation of æ. To obtain a single vector 2 E RD, Ren et al. (2023) perform mean pooling:
E : (x)
G := Ódec (
1 S
Σ 2
Then, they propose to measure whether & is OOD by first fitting a Gaussian distribution (H, E), a E R2, Σ DxD to the per-sequence mean embeddings computed from the training corpus, and then computing the squared Mahalanobis distance between and μc: dh (t). (5)
(2,
(z - p1) and SMaha (2)
Averaging hides anomalies. The key limitation of the approach described above is the use of the mean of the token embeddings Z: Averaging the entire sequence into the mean discards the token-level structure that would otherwise be informative for detecting whether sequence is OOD. Figure 1 shows toy example of this failure mode: The ID and OOD sequences are indistinguishable using their means, and therefore, the Mahalanobis distance with mean pooling fails to discriminate between them.
Mahalanobis decomposition. To address this limitation, we begin by expressing the Mahalanobis distance as directional decomposition:
D
L (z, 11) = (w]z= -- w 11)
(6)
j=1
The weight vectors wy E RRD form a basis of I and determine - via - wrwy One possibility to map a given E-1 to weight vectors wy is to select the directions of the w as the unit-norm cigenvectors of I-1, and to select the squared norms of the wy their corresponding eigenvalues (see Appendix B.2).
We use Z U821 T Dx8 for all finite-length sequences of D-dimensional token embeddings.
3

---

Published as a conference paper at ICLR 2026
Algorithm 1 AP-OOD
Require: (#1..... æN); Dene, Odec: B, M, nstepss
1: for == 1 to N do
2: Compute sequence embedding Z; using Z; (2n) or Z, (dne(
3: for step 1 to nsteps do
4: Randomly sample mini-batch indices B C [l......N}
5: Collect mini-batch {e,)
6: Form batch-local concatenation ZB + llieB Zj. Compute loss t [&S Eieg &f (Z,, Zz) -E log( w 11)))
7:
8:
Compute gradients of L w.r.t. (wi,..., WMr and perform a gradient update
9: Do mini-batch attention pooling to compute μj + Zsoftmax(B Z"wy) (Appendix C.1)
10:
(Z) + -dj 2) + log (11m,118)-
11:
return s(-)
Beyond mean pooling. To overcome the limitations of mean pooling, we generalize Equation (6) by using attention pooling (Bahdanau, 2014; Ramsauer et al., 2021):
(Z. == Zsoftmax($ 33 z"w) w) and z AttPoola w).
(7)
where 3 E Rz0 is the inverse temperature, and w RD is a learnable query. AP-OOD also uses attention for the corpus-wide pooling: Starting with the sequence representations (Z,...., ZN) with Z, := dene (2;), we construct Z E Z by concatenating the sequence = -Z, ... ZN). AP-OOD computes the global prototype using μ == AttPoolg (2, w). Given the & and f from the attention pooling, AP-OOD computes the squared distance dz, Z) analogous to Equation (6):
representations:
M d2 2) = Σ 1=1
M Σ (2, (8)
w max Z"w,, - w/ Zsoftmax 2Tw,))
We refer to M E N as the number of heads. In general, M does not need to equal the embodding dimension D. We show in Appendix B.4 that, when B = and M D, Equation (8) reduces to the Mahalanobis distance (Equations (5) and (6)). To the best of our knowledge, AP-OOD is the first approach to integrate attention pooling into the Mahalanobis distance via learnable directional decomposition. In Appendix B.1, show that 8min Z) min, -3 Z, Z) + log(I [og([1,;113) Il)) is score function as defined in Equation (2). Our score arises naturally as the upper bound
s()) Σ 1=1
(Z + log([[w;113) = -2(2, +
Σ log([ (l
(9)
In Appendix D.7, we empirically compare the min- baseb score Smin to its upper- bound variant s(Z) and find that s s(Z) yields stronger OOD discrimination. The choice of this score naturally leads to the loss function of AP-OOD:
M Σ j=1
L(w wM) = N
Σ 8 PZ. 2)
log (1|w,
(10)
1=1
We provide the pseudocode for AP-OOD in Algorithm 1. Scaling to large data sets requires efficient computation of μ = Zsoftmax(3 z"'w), the naive method loads the entire concate- nated sequence into memory, but we reduce the memory footprint by performing attention pooling mini-batches. We describe this procedure in Appendix C.I.
Multiple queries per head. We now extend AP-OOD and use multiple queries per head. We use set of stacked queries W; (w.l,.... W.T) DxT per head. For simplicity, consider single head with the queries W E RDT for now. We begin by extending the
4

---

Published as a conference paper at ICLR 2026
softmax notation from Ramsauer et al. (2021) to matrix-valued arguments. Given matrix A E RSXT
exp(3axt)
softmax(3A) = Ex=1 Ei= exp(Bagr)
(11)
In other words, the softmax normalizes over the rows and columns of A. Next, we extend the attention pooling process from Equation (7) with the matrix-valued softmax: AP-
OOD transforms the sequence representation Z E R with S tokens to new sequence
representation RDX with T tokens using = 2P. The updated attention pooling process is
AttPools (Z, W) == Zsoftmax($ w) and Z = -AttPoolg(2 Z W). (12)
To the best of our knowledge, this work the first use matrix-valued global softmax to transform a sequence Z into another sequence Z. though similar matrix-based memory updates have recently been introduced in recurrent architectures (Beck et al., 2024). Finally, AP-OOD uses W E I DxT to transform the Z E RDxT to scalar using (W, Z)F Tr(WT W Z) (where (+-) denotes the Frobenius inner product, and vea(.) denotes the flattening operation). To summarize, the extended squared distance is
vec( (e)"
dz( Σ (T(,"" Zsoftmax(B z Z)), Tr(W3" Zsoftmax(B Z"W,))) (13) 5=1
Finally, the regularizing term is - log( ()))) (where || || denotes the squared Frobenius norm). To summarize, the extended loss
L(WI.... WM)
1 N
Σ
(Z 2)
Σ log (IIW,
(14)
We provide PyTorch-style pseudocode implementing Equation (14) in Appendix C.2, and we analyze Equation (13) through the lens of kernel functions in Appendix B.3.
2.2 SUPERVISED OOD DETECTION
Background. Supplying an OOD detector with information about the distribution of the OOD examples at training time can improve the ID-OOD decision boundary (Hendryeks et al., 2018). In practice, it is hard to find OOD data for training that is fully indicative of the OOD distribution seen during inference. Outlier exposure (OE; Hendrycks et al., 2018) therefore uses a large and diverse auxiliary outlier set (AUX; c.g., C4 for text data) as stand-in for the OOD case. However, acquiring such large and diverse AUX datasets is not always possible. For example, consider a translation task with a less widely spoken source language. As another example, consider detecting defects in industrial machines using recordings of their sounds (Nishida et al., 2024). Practitioners can collect a relatively large amount of ID audio data from machines while they run without defects. However, it is much harder to collect diverse AUX examples from defective machines because defects are infrequent. such a casc, one might have to resort to smaller AUX data set. Therefore, an OOD detector should scale gracefully with the degrec of auxiliary supervision, adapting to the available number of AUX examples (c.g., Ruff et al., 2019; Liznerski et al., 2022; Yoon et al., 2023; Ivanov el al., 2024; Qiao el al., 2024).
Utilizing AUX data. To adapt AP-0OD to the supervised setting, we follow Ruff et al. (2019) and Liznerski et al. (2022): AP-OOD punishes large squared distances d² Z) for ID samples Z and encourages large squared distances for AUX samples Z. For- mally, AP-OOD minimizes the binary cross-entropy loss with the classes ID and AUX with p(y ID) Z) exp(-d' (Z, 2)). Given N ID examples (Zi,..., ZN); and N' AUX examples (ZN+1, ZN4N), AP-OOD minimizes the supervised loss
N
Lsur N N'
Σ d dZ, 1=1
1
Σ
log(1
2))), (15)
N+N
N=
where X E Roo If X 0, Lsur equals the unsupervised loss L without the regularizing term.
5

---

Published as a conference paper at ICLR 2026
Table 1: Unsupervised OOD detection performance on text summarization. We compare results from AP-OOD, Mahalanobis (Lee et al., 2018; Ren et al., 2023), KNN (Sun et al., 2022), Doep SVDD (Ruff et al., 2018), model perplexity (Ren al., 2023), and entropy (Malinin & Gales, 2020) on PEGASUSLARGE trained on XSUM as the ID data set. + indicates "lower is better" and 1 "higher is better". All values in %. We estimate standard deviations across five independent data set splits and training runs.
CNX:DM
Newsroom
Reddit
Samsum
Mean
Impul 0OD
AUROC 1
68. 95-9000
86. 00:00-014
98.6 18.6geo0z
99. 77:00
88, 3%.44
Mabalomobis
FPR95
92 12.16:0.20
64.02-0:53
2.43-012
0.1742m2
39.
KNN
FPR95
AUROCT
19. 30.30
51.3220.1
73.83
34.44
91.5339.10
21.20
98.8 88
3.gitou2
80.33
Toep SVDD
FPR55
AUROC
25. 25.7529946
24
91.36
36. 05
0.39
98,
98.557
0.72
27.84
91.01
AP-0OD (Ours)
AUROC
FPR95
12.88
97.4855.32
98. 60..54=-.000 3o
5.78
0.00
88:50 os
0. 0.00
1HL BHL76=000 os
4.67
9M8
Ousput OOD
Perpiexity
FPRS5
AUROC 1
77.83-9.16
41 41.65--0117
79. 90-40:044
52 86-80-99
82.3 82.50-20.77
45.25-1105
77 77.90--00x
47.35
63. 75
63.21
Eukropy
FPR95
AUROC
79:26-996
69.7629.10
64.65
76.92=0.05
30.45 30.45-
93. 83.4220011
50 50.83-1667
87.07=0.15
79.30
56.30
AUROC
22.6759.5
97.38
97. 17
96.99=9-s
Mahalanobis
FPR95
88.46
49 06
12: 22:2750
15 15.25=9-35
41.51
KNN
FPR95
AUROC
73.36
14.173933
54 54.30
86 69 69.669--11
95.82-9.12
17.28
37. 07.28-9905
10.99=103
38.98
88.40
Doep SVDD
FPR05
AUROC
31
66.10=1.5
10 (1:47:0a)
20 Lo
97. 28
9.31
21 1.3
915 06.61==021
88.31
32.31
AP-OOD (Ours)
AUROCT
FPR95
29.58
93 93.53#0054
94.41
28
3.18:2.44
98. 72
4.09
98. 08.599t0.00
96 96.39
16
EXPERIMENTS
Toy experiment. We present toy experiment illustrating the main intuitions behind AP-OOD. Figure demonstrates a simple failure mode of mean pooling approaches: First, we generate ID and OOD token embeddings Z, R2x2, Each ID sequence representation consists of one token sampled from (V), o²1) (where a := 0.1) and one token sampled from V((-1.-1) o² I.. The OOD sequences contain two tokens sampled from (V((-1,)), o²T) and ((1, I)), respectively. The left panel shows the generated sequences, where each sequence consists of two dots (representing the two tokens) connected by line. Because the means of the ID and OOD sequences both cluster around the origin (central panel), the Mahalanobis distance with mean pooling fails to discriminate between them (right panel). Figure 2 shows how AP-OOD overcomes this limitation: We set M = 1 and T = 1 and train AP-OOD as described in Section 2.1 on the ID data only, but we modify the pooling mechanism from Equation (7): We replace the dot product similarity in the softmax with the negative squared Euclidean distance, as is known to work better in low-dimensional spaces (we provide the formal definition for this modification in Appendix D.1). The left panel of Figure 2 shows that the loss landscape of w forms two basins at the locations of the ID tokens. The central panel shows that after training, is located in one of the basins. Finally, the right panel shows that AP-OOD perfectly discriminates ID and OOD.
Summarization. We follow Ren et al. (2023) and use a PEGASUSLARGGE (Zhang et al., 2020) fine-tuned on the ID data set XSUM (Narayan et al., 2018). We utilize the C4 training split as the AUX data set. We measure the OOD detection performance on the data sets CNN/ DDilaily Mail (CNN/DM; news articles from CNN and Daily Mail; Hermann et al., 2015; See al., 2017), Newsroom (articles and summaries written by authors and editors from 38 news publications; Grusky et al., 2018), Reddit TIFU (posts and summaries from the online discussion forum Reddit; Kim et al., 2018), and Samsum (summaries of casual dialogues; Gliwa et al., 2019). The ForumSum data set used in the experiments of Ren et al. (2023) has been retracted. Therefore, we do not use it in our experiments.
Translation. We train a Transformer (base) WMT15 En (Bojar et al., 2015). The model trains for 100,000 steps using AdamW (Loshchilov & Hutter, 2017) with a cosine schedule (Loshchilov & Hutter, 2016), linear warmup, and a peak learning rate of 5 10-4

---

Published as a conference paper at ICLR 2026
AUROC
(a) CNN/DA
(b) Newsroom
Figure 3: OOD detection performance on the input token embeddings of PEGASUSLARGE trained XSUM. We vary the number of AUX samples and compare AP-OOD, binary logits (Ren et al., 2023), Deep SAD (Ruff et al., 2019), and relative Mahalanobis (Ren et al., 2023). AP-OOD attains the highest AUROC independent of AUX sample count.
We set the batch size to 1024 and the context length to 512. Following Ren et al. (2023), the AUX data set is ParaCrawl En Fr, and the OOD data sets are newstes12014 2014,, newsdiscussdev2215 (ndd2015), and newsdiscusstes et al., 2015), and the Law, Koran, Medical, IT, and Subtitles subsets from OPUS (Tiedemann, 2012; Aulamo & Tiedemann, 2019).
(nd12015) from WMT15 (Bojar
Training. We extract 100,000 ID sequence representations (E or G) and use all extracted representations for training AP-OOD in all experiments. We also extract AUX sequence representations, and we vary the number of AUX sequences available from 0 (unsupervised) to 10,000 (fully supervised). While training AP-OOD, the transformer model remains frozen. We use the Adam optimizer (Kingma & Ba, 2014) without weight decay, set the learning rate to 0.01, and apply a cosine schedule (Loshchilov & Hutter, 2016). We train for 1,000 steps with a batch size of 512. We select M and T such that the parameter count of AP-OOD matches the parameter count of the Mahalanobis method (i.c., the size of Σ). For more information on hyperparameter selection, we refer to Appendix D.2. An additional scaling experiment on input sequence representations of the summarization task investigates larger parameter spaces: We train on the full XSUM data set (instead of the 100,000 ID) sequence representations used in the other experiments). We select the number of heads (M) from the set {1, 16, 128, 1024), the number of queries (T') from the set { 16), and 3 from (1/VD, 0.25, 0.5, 2). The largest configuration has parameter count 16 times greater than the Mahalanobis baseline.
Basclines. We compare AP-OOD to six unsupervised OOD dotection mothods: We apply the embedding- based methods Mahalanobis (Lee et al., 2018; Ren et al., 2023), KNN (Sun el al., 2022), and Deep SVDD (Ruff el al., 2018) to both the input and output sequence representations (E and G, respectively). and we apply Perplexity (Ren et al., 2023) and Entropy (Malinin & Gales, 2020) to the output of the decoder. We also compare AP-OOD to three supervised OOD detection methods: binary logits (Ren et al., 2023), relative Mahalanobis (Ren et al., 2023), and Deep SAD (Ruff et al., 2019). We evaluate the discriminative power of the methods in our comparison using the false positive rate at 95% true positives (FPR95) and AUROC.
Audio data. To demonstrate the effectiveness of AP-OOD on data modalities other than text, we apply the method to the MIMII-DG audio data set (Dohi et al., 2022). The data set comprises audio recordings of 15 different machines, ranging from 10 to 12 seconds in length. The dataset contains 990 samples per machine. During preprocessing, the raw andio waveforms are converted into audio spectrograms. We train a transformer to classify subset of 7 machines. The remaining 8 machines are considered as OOD. The architecture and training method for the network adopted from Huang et al. (2022). To adjust for the
7

---

Published as a conference paper at ICLR 2026
0.915
16-
05.12
90.02
2.90
C22
0.23
0.23
0.900
34
12
00.51
0.000
2.00
0.875
50.43
(a) Mean AUROC
(b) Best /
Figure 4: OOD detection performance on the input token embeddings of PEGASUSLARGE trained XSUM when scaling to large M and T. We vary M (1, 128, 1024), T E (1,4, 16), and B E (1/vD, 0.25, 0.5, 1, (Left) Mean AUROC in % for the best at each (M,T) combination. (Right) The best B selected at each (M, T)) combination.
small data set size, we decrease the size of the architecture: We increase the patch size to 32 32 pixels, decrease the embedding dimension to 32, and utilize only three attention blocks with four heads each. Consequently, the encoder of the network produces 128 tokens with D 32 features. We train AP-OOD on the encoder output in the unsupervised setting using M = 128 heads and T 8 queries.
4 RESULTS & DISCUSSION
Table 1 shows the results unsupervised OOD detection text summarization. AP-OOD surpasses methods with mean pooling by large margin for both input and output settings for most OOD data sets. Most notably, the mean FPR95 in the input setting on CNN CNN/DM DM improves from 74.20% for the best baseline Deep SVDD to 12.88% for AP-OOD. The table also shows that the embedding- based methods (Mahalanobis, KNN, Deep SVDD, and AP- OOD) perform better than the prediction-based baselines perplexity and entropy. Figure 3 shows the results of AP-OOD in the semi-supervised setting: supplying AUX data AP- OOD improves the AUROC, and more AUX data results in larger improvement. AP-OOD attains the highest AUROC independent of AUX sample count. We include the results on additional OOD data sets in the semi-supervised setting and results on fully supervised OOD detection on text summarization in Appendix D.3, and we present ablations on AP-OOD on text summarization in Appendix D.8. Furthermore, we verify the effectiveness of AP-OOD on the decoder-only language modeling paradigm using Pythia-160M in Appendix D.6.
Figure 4 presents the results when scaling AP-OOD to larger parameter counts. As increase the number of heads (M) and queries (T), we observe a steady increase in the mean AUROC on the summarization task. The highest mean AUROC of 99.40% is achieved by the largest configuration tested (M = 1024, T 16).
Table 2 shows the results on unsupervised OOD detection on the translation task. AP-OOD gives the best. average results for the input and output iegs We include results on fully supervised OOD detection for translation in Appendix D.5. Unlike in the summarization task, the prediction-base based methods in the translation task. Notably, perplexity outperforms all embedding- based methods evaluated on output token embeddings, with the exception of AP-OOD, We explain this discrepancy through the varying levels of aleatoric uncertainty inherent the two tasks. In translation, the source sentence imposes strict lexical and syntactic constraints, resulting in low-entropy output distributions and therefore low alcatoric uncertainty. In contrast, summarization yields multiple distinct yet equally valid outputs, thereby increasing output entropy and, consequently, the aleatoric uncertainty. Prediction- based uncertainty estimators inherently limited in settings with non-trivial aleatoric uncertainty (Tomov et. al., 2025).
methods (perplexity and entropy) are competitive with embedding-
8

---

Published as a conference paper at ICLR 2026
Table 2: Unsupervised OOD detection performance on English-to-Frendd translation. We compare results from AP-OOD, Mahalanobis (Lee et al., 2018; Ren et al., 2023), KNN (Sun ct al., 2022), Doep SVDD (Ruff et al., 2018), model perplexity (Ren et al., 2023), and entropy (Malinin & Gales, 2020) on Transformer (base) trained on WMT15 En Fr the ID data set. + indicates "lower is better" and t "higher is better". All values in %. We estimate standard deviations across five independent dala set splits and training runs.
IT
Koran
Law
Medical
Subtities
mdd2015
md12015
m12014
M
Inpect ODD
Mabulanobis
FPR95 L
AUROCT
RS 22-2258
37 77.9gee-1
74.5224137
58.08 17
00.7g2h
00-1--h:31
78.63
ov
70L733
67:27 14
62. 02.0240044
67.777
62,00
93.6020
49
80.49
70.96
KNN
AUROC1
FPRU5 1
H4,252001
71.5249551
59.91 60.47
91. 72.72
54.7553
08.95-004
77.86400
71.91
87.47
G3 sagae
91. 33.3
92.07
66.16-0
57 24
M11.27
71.00
Deep SVDD
AUROC1
FRR05 L
92 12.155
96 22.28-005
96. 18.1112000 140400
72.26
63.3 57 43.27
50-3--2
08.0
78, T8.08+444
85.00
93. 77:77
59 a0.g9-
04.8
60.004923
96 82
16.7046.42
69.72
AP-OOD (Ours)
FPR05 4
AUROC1
95.00
21.85
91.07
78.49-1.05
63
89.19
19:19 10:30
80. 10:9.6
61.35
60.35
57,00
67 40
87.85
88.07
95.10
74.81
75.25
Output 0OD
Perplesity
FPR05
AUROC1 1
44.9440.50
01,34
71.7549:4
91.7040.00
90.83 90.834491
45, 13-33233
70.47 44.36
91.278941
51.11 60.29
72 st
85: 05.05 48.c5
78.8340000
85,6749.M
96.67 96.6746.02
29:221
40.02
11.08
72.25
Eutrupy
FPR05 1
AUROC1
74 71.86-0.20
68 6g:0..0
03,
53 =3:47
50.51 14.35
00.21 80.2)
76.8491.1
70.36 90.21
75 78:40.1
76 21.35
77.63=1.0
72.57
85 36-339-977
01.01
60:05
Mohnlonobis
FPR05 1
AUROC1
54. 49.922
91 01.07700.0
90.8 00,8220014
75.5740.20
64
92.31 31
73.08-430
78.22
74.9799.4
56,880077
62 16
92 873332
00. 05.07799.
63.53
58, 82
49.70
84.28
71.46
KNN
AUROCT
FPR35
95 05.40.0
11
75. 75.31
65 01 19.10
82 30-331
85.8781111
79.04
65. 78-48.8
92. 02:9
66, 76.724913
03.90-
97. 02
58 90.90.11111
76.50
Deep SVDD
AUROCT
PRUG 1
57 1.
80.9749.44
70.96
614
91 12.54-1.33
72. Li
82.77
82.0gan
94. 74.77
60 r0aaya
91.91
61.4gt9
56. 56.50
NO.74
620.22
AP-OOD (Ours)
AUROC1
EPR9 1
26.42
B1. 86.4
81.01
88.24 28.22--9.0
37 $7.36
62.26
78.74 $1.56
91.39 1:41
49.36-7
T5. 40.90
80.4
75.41
$1.06-91
94 24.22255
77. 3.
70.37
Table 3: Unsupervised OOD detection performance on audio classification. We compare results from AP-OOD, Mahalanobis (Lee et al., 2018; Ren al., 2023), KNN (Sun et al., 2022), Deep SVDD (Ruff et al., 2018), MSP (Hendrycks & Gimpel, 2016), and EBO (Liu et al., 2020b) trained on MIMIIDD (Dohi et al., 2022) as the ID data set. $ indicates "lower is better" and "higher is better". All values in %. We estimate standard deviations across five independent training runs.
Mahalonobia KNN
Dorp SVDD
MSP
EBO
AP-0OD (Ours)
AUROC 1
64.96 66.96
81.21
53. 58 21000
88.08 90.00
90.75
19.000
92.86 02.86
FPR95 L R4.00
57.11 50000
89. #1.00
36. #0.000
61.80 29.005
22390
In the audio task, the network achieves an accuracy of 97.6% on the primary classification task. Table 3 presents the results of the unsupervised OOD detection methods AP-OOD, Mahalanobis (Lee et al., 2018), KNNN (Sun et al., 2022), and Deep SVDD (Ruff et al., 2018). Additionally, we compare AP-OOD to 2 met for classifiers, Maximum Softmax Probability (MSP; Hendrycks & Gimpel, 2016) and Energy-based OOD Detection (EBO; Liu et al., 2020b). In contrast to the other methods, MSP and EBO do not apply to transformer tokens, making them unsuitable for summarization and translation tasks. The results show that AP-OOD improves the FPR95 metric from 36.43% (MSP) to 22.35%.
We evaluate the runtime performance of AP-OOD by mcasuring the inference time of single batches the summarization task. We find that while AP-OOD is slower than the Mahalanobis baseline, it is still substantially faster than a forward pass through the transformer encoder. Because AP-OOD scales linearly, its relative overhead diminishes for longer sequences. For more details on the runtime behavior, we refer to Appendix D.9.
5 RELATED WORK
OOD detection. Some authors (c.g., Bishop, 1994; Roth et al., 2022; Yang et al., 2022) distinguish between anomalies, outliers, and novelties. These distinctions reflect different goals within applications (Ruff et al., 2021). For example, when an anomaly is found, it will usually be removed from the training pipeline. However, when a novelty is found, it should be studied. We focus on detecting samples that are not part of the training distribution and consider sample categorization as a downstream task. OOD detection methods can be categorized into three groups: Post-hoc, training-time, and OF methods. In the post-hoc
9

---

Published as a conference paper at ICLR 2026
approach, one employs statistics obtained from a classifier. Perhaps the most well-known approach is the maximum softmax probability (MSP; Hendrycks & Gimpel, 2016). A wide range of post-hoc OOD detection approaches have been proposed to address the shortcomings of MSP (e.g.. Lee et al., 2018; Hendrycks et. al., 2019a; Liu et al., 2020a; Sun et al., 2021; 2022; Wang et al., 2022; Zhang et al., 2023b; Djurisic et al., 2023; Lin et al., 2023; Xu et al., 2024; Guo et al., 2025). A commonly used post-hoe method is the Mahalanobis distance (e.g., Lee et al., 2018; Sehwag et al., 2021; Ren et al., 2023). Recently, Müller & Hein (2025) proposed feature normalization to improve Mahalanobis-basoe OOD detection, and Guo et al. (2025) show that the Mahalanobis distance benefits from dynamically adjusting the prior geomelry in response to new dała. In contrast to post-hoc methods, training- methods modify the training process of the encoder (e.g., Hendrycks et al., 2019c; Sehwag et al., 2021; Du et al., 2022; Hendrycks et al., 2022; Ming et al., 2023; Tao et al., 2023; Lu et al., 2024). Finally, the group of OE methods incorporates AUX data in the training process (e.g., Hendrycks et al., 2019b; Liu et al., 2020a; Ming et al., 2022; Zhang et al., 2023a; Wang ct al., 2023; Zhu et al., 2023; Jiang ct al., 2024; Hofmann et al., 2024).
OOD detection and natural language. Most. of the aforementioned OOD detection approaches target vision tasks, and many of them require a classificat ion model as the encoder 6. Applying these vision-based OOD methods to text is not straightforward due to the eequence-depenentt nature of natural language (e.g., in autoregressive language generation). OOD detection specifically tailored for natural language still underexplored. Ren et al. (2023) propose the model's log- log-perplexity of a generated sequence y as a simple bascline for OOD detection on conditional language modeling tasks: - + Σt= However, they show experimentally that model perplexity is inherently limited. Because of these shortcomings, Ren et al. (2023) propose embedding-basee based OOD detection methods for text data. Relatively few other works have explored OOD detection for generative language modeling. Notable applications include translation (e.g., Xiao et al., 2020; Malinin et al., 2021; Ren et al., 2023), summarization (Ren et al., 2023), and mathematical reasoning (Wang et al., 2024). A related field is hallucination detection (e.g.. Malinin & Gales, 2020; Farquhar el. al., 2024; Du et al., 2024; Aichberger et. al., 2025; Park el al., 2025). Unlike OOD detection (which flags inputs outside the training distribution), the goal of many hallucination delection methods is to identify prompts generative language model is unlikely to answer truthfully.
ar).
6 LIMITATIONS & FUTURE WORK
We would like to discuss two limitations that we found. First, the selection of the AUX data is crucial, since it determines the shape of the ID OOD decision boundary. If the AUX distribution diverges from the OOD examples faced at inference, the induced boundary may not be aligned with the task. Second, remains unclear how reliably the OOD detoction performance on specific data sets can indicate the general ability to detect OOD examples, as a large portion of plausible OOD inputs remains untested. An interesting avenue for future work is to apply OOD detection methods to large language models (LLMs; e.g., Abdin et al., 2024; Dubey et al., 2024; Yang et al., 2025). While we demonstrate the applicability of AP-OOD on the decoder-only language modeling paradigm of LLMs (Appendix D.6), further challenges include proprietary training data, finding OOD data for training and evaluation given the breadth of the ID data, and the ambiguity of PID arising from complex training pipelines involving multiple phases (e.g., Wei et al., 2022; Ouyang et al., 2022).
7 CONCLUSION
We introduce AP-OOD: an approach for OOD detection for natural language that can learn in supervised and unsupervised settings. In contrast to previous methods, AP-OOD learns how to pool token-level information without the explicit need for AUX data. Our experiments show that when supplied with AUX data during training, the performance of AP-OOD improves as more AUX data is provided. We compare AP-OOD to five unsupervised and three supervised OOD detection methods. Overall, AP-OOD shows the best results.
10

---

Published as a conference paper at ICLR 2026
ACHKNOWLEDGEMENTS
The ELLIS Unit Linz, the LIT AT Lab, the Institute for Machine Learning, are supported by the Federal State Upper Austria. We thank the projects FWF AIRI FG 9-N (10.5576//FG)) (FFG- 899943), StarrsWaters [OOIIONN-L6-DDR 01), FWF Bilateral Artificial Intelligence (10.55776/COE12) We thank NXAI GmbH, Audi AG, Silicon Austria Labs (SAL), Merck Healthcare KGaA, GLS (Univ. Waterloo), TÜV Holding GmbH, Software Competence Center Hagenberg GmbH, dSPACE GmbH, TRUMPF
Al4GreenHeatingGrin
SE + Co. KG. This work has been supported by the "University SAL Labs" initiative of Silicon Austria Labs (SAL) and its Austrian partner universitios for applicd fundamental research for dlectronic-based systems. We acknowledge EuroHPC Joint Undertaking for awarding us access to Karolina at ITAllnovations, Czech Republic and Leonardo at CINECA, Italy.
REPRODUCIBILITY STATEMENT
To ensure reproducibility, the source code of our implementation of AP-OOD in the unsuper- vised and supervised settings is available at https:/ //gggtth..ooo com. /ml-juu/ ftttuu..omm/mlljku/// information about data, the training process, and the hyperparameter selection in Section and Appendix D.2.
/ml-jku/appood. ap-ood. We provide
REFERENCES
Laurence F Abbott and Yair Arian. Storage capacity of generalized networks. Physical review A, 36(10)55091, 1987.
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. arXiv preprint orxiv:24 089005 2024.
Lukas Aichberger, Kajetan Schweighofer, Mykyta Iclanskyi, and Sepp Hochreiter. Improving uncertainty estimation through semantically diverse language generation. In The Thirteenth International Conference on Learning Represenlations, 2025.
Ghadi S Al Hajj, Aliaksandr Hubin, Chakravarthi Kanduri, Milena Pavlovic, Knut Dagestad Rand, Michael Widrich, Anne Schistad Solberg, Victor Greiff, Johan Pensar, Günter Klambauer, al. Incorporating probabilistic domain knowledge into deep multiple instance learning. In Forty-forst International Conference on Machine Learning, 2024.
Stuart Andrews, Ioannis Tsochantaridis, and Thomas Hofmann. Support vector machines for multiple-instance learning. Advances in neural information processing systems, 15, 2002.
Andreas Aucr, Martin Gauch, Daniel Klotz, and Scpp Hochreiter. Conformal prediction for lime series with modern hopfield networks. Advances in Neural Information Processing Systems, 36:56027-5660 56074, 2023.
Mikko Aulamo and Jörg Tiedemann. The OPUS resource repository: An open package for creating parallel corpora and machine translation services. In Mareike Hartmann and Barbara Plank (eds.), Proceedings of the 22nd Nordic Conference Compulational Lin- guistics, pp. 389 394, Turku, Finland, Septembera&&Seetober 2019. Linköping University Electronic Press.
Damitry Bahdanau. Neural machine translation by jointly learning to align and translate. arxie preprint 1rX.v:1409.041 0473, 2014.
Pierre Baldi and Santosh S Venkatesh. Number of stable points for spin-glasses and neural networks of higher orders. Physical Review Lellers, 58(9):913, 1987.
Maximilian Beck, Korbinian PÄūppel, Markus Spanring, Andreas Auer, Oleksandra Prud- níkova, Michael Kopp, GAijnter Klambaner, Johannes Brandstetter, and Sepp Hochreiter.
11

---

Published as a conference paper at ICLR 2026
xlstm: Extended long short-term memory. In ehirth- Conference Neural Infor- mation Processing Systems, 2024. URL https: http://xxvvvv............/ arggabs//4055555 0tlbs/2405045555 04517.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle OaAZBrien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Christopher M Bishop. Novelty detection and neural network validation. IEE Proceedings- Vision, Image and Signal processing. (41 222,, 1994.
Ondrcj Bojar, Rajen Chatterjoe, Christian Federmann, Barry Haddow, Matthias Huck, Chris Hokamp, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Carolina Scarton, Lucia Specia, and Marco Turchi. Findings of the 2015 workshop on statistical machine translation. In Ondrej Bojar, Rajan Chatterjee, Christian Fodermann, Barry Haddow, Chris Hokamp, Matthias Huck, Varvara Logacheva, and Pavel Pecina (eds.), Proceedings of the Tenth Workshop on Statistical Machine Translation, pp. 1-46, Lisbon, Portugal, September 2015. Association for Computational Linguistics.
Barbara Caputo and Heinrich Niemann. Storage capacity of kernel associative memories. 2002: International Conference Madrid, Spain, August 28-30, 30, 2002 Proceedings 12, pp. 51-56. Springer, 2002.
In Artificial Neural Vetworks&ATICANN
HH Chen, YC Lee, GZ Sun, HY Lee, Tom Maxwell, and C Lee Giles. High order correlation model for associative memory. In AIP Conference Proceedings, volume 151, pp. 86-99. American Institute of Physics, 1986.
Thomas G Dietterich, Richard H Lathrop, and Tomás Lozano-Pérez. Solving the multiple instance problem with axis-parallel rectangles. Arlificial intelligence, 89(1-2):31 71, 1997.
Andrija Djurisic, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. Extremely simple activa- tion shaping for out-of-distribution detection. In The Eleventh International Conference on Learning Represenlations, 2023.
Kota Dohi, Tomoya Nishida, Harsh Purohit, Ryo Tanabe, Takashi Endo, Masaaki Yamamoto, Yuki Nikaido, and Yohci Kawaguchi. MIMIT DG: Sound dataset for malfunctioning indus- trial machine investigation and inspection for domain gencralization task. In Proceedings of the 7th Delection and Classification of Acoustic Scenes and Events 2022 Workshop (DCASE2022). Nancy, France, November 2022.
Xucfong Du, Zhaoning Wang, Mu Cai, and Yixuan Li. Vos: Learning what you don't know by virtual outlier synthesis. arXw preprint 2r0:: 011007 2022.
Xuefeng Du, Chaowei Xiao, and Sharon Li. Haloscope: Harnessing unlabeled Ilm generations for hallucination detection. Advances in Neural Information Processing Systems, 37: 102948-102972 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arxiv e-prints, pp. arXiv 2407, 2024.
Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):6255 220-660,-66--6 2024.
Andreas Fürst, Elisabeth Rumetshofer, Johannes Lehner, Viet T Tran, Fei Tang, Hubert Ramsauer, David Kreil, Michael Kopp, Günter Klambaner, Angela Bitto, et al. CLOOB: Modern Hopfield networks with InfoL.OO outperform clip. Advances in neural information processing systems, 14220450220463 2022.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text. for language modeling. arXiv preprint Xr:: 100:2000..00027, 2020.
12

---

Published as a conference paper at ICLR 2026
E Gardner. Multiconnected neural network models. Journal of Physics A: Mathematical and General, 20(11):3453, 1987.
Bogdan Gliwa, Iwona Mochol, Macicj Biesck, and Aleksander Wawer Samsum corpus: A human-annotated dialogue dataset for abstractive summarization. ar preprint or 19:11. 2019.
Ary L Goldberger, Luis AN Amaral, Lcon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Chung-Kang Peng, and H Eugene Stanley. Physiobank, physioloolkit, and physionet: components of new research resource for complex physiologic signals. circulation, (01(23):e215-e22) 2000.
Max Grusky, Mor Naaman, and Yoav Artzi. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. arXiv preprint arXiv:1804. 11288, 2018
Kaiyu Guo, Zijian Wang, Tan Pan, Brian C Lovell, and Mahsa Baktashmotlagh, Improv- ing oft-of-distributiin detection via dynamic covariance calibration. In Forty-second International Conference on Machine Learning, 2025.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of- distribution examples in neural networks. arXi preprint 1rXi::161.001190 2016.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. Xrviv preprint arXiv: 1rX:o::812.00,606, 2018.
Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joe Kwon, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. Scaling out-of-distribution detection for real-world settings. Xiviv preprint or Nix::19111 2019a.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations, 2019b. URL https://openrevieu httt:/oooumiid-iyycthttee
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using supervipe learning can improve model robustness and uncertainty. Advances in neural information processing systems, 32, 2019c.
Dan Hendrycks, Andy Zou, Mantas Mazeika, Leonard Tang, Bo Li, Dawn Song, and Jacob Steinhardt. Pixmix: Dreamlike pictures comprehensively improve safety measures. In Proceedings of the IEEE/CV /CVF Conference on Computer Vision and Pattern Recognition, pp. 16783 16792, 2022.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines read and comprehend. Advances in neural information processing systems, 28, 2015.
Clans Hofmann, Simon Schmid, Bernhard Lehner, Daniel Klotz, and Sepp Hochreiter. Energy- based hopfield boosting for out-of-distribution detection. Xrxiv preprint arXi::205500870 08766 2024.
J. J. Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 79(8):2554 2558, 1982.
J. J. Hopfield. Neurons with graded response have collective computational properties like those of two-state neurons. Proceedings of the National Academy of Sciences, 81(10): 3088-3092, 1984. doi: 10.1073/ 8..0.8..
D Horn and M Usher. Capacities of multiconnected memory models. Journal de Physique, 39(3):389-33) 395, 1988.
Jerry Yao-Chich Hu, Pei-Hsuan Chang, Haozheng Luo, Hong- Yu Chen, Weijian Li, Wei-Po Wang, and Han Liu. Ouflier-effciien hopfield layers for large transformer based models. In Forty-first International Conference on Machine Learning, 2024.
13

---

Published as a conference paper at ICLR 2026
Po-Yao Huang, Hu Xu, Juncheng Li, Alexci Baevski, Michael Auli, Wojeiech Galuba, Florian Metze, and Christoph Feichtenhofer. Masked autoencoders that listen. In NeurIPS, 2022.
Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detocting distributional shifts in the wild. Advances in Neural Information Processing Systems, 34: 677 689, 2021.
Maximilian Isse, Jakub Tomczak, and Max Welling. Attention-based deep multiple instance learning. In International conference on machine learning, pp. 2127 2136. PMLR, 2018.
Viktor Ivanov, Maurizio Scaramuzza, and Richard C Wilson. Deep temporal semi-supervised one-class classification for gnss radio frequency interference detection. The Journal of Navigation, (7 ):59 81, 2024.
Wenyu Jiang, Hao Cheng, MingCai Chen, Chongjun Wang, and Hongxin Wei. DOS: Diverse outlier sampling for out-of-clstributiin detection. In The Twelfth International Conference on Learning Representations, 2024. URL https: http:r///ppereviee. wwwp:/reieew.ne/foorrooe net/forum? id=iriEgxFB4y.
Byeongchang Kim, Hyunwoo Kim, and Gunhee Kim. Abstractive summarization of reddit posts with multi-level memory networks. arXiv preprint 1rXi::181100078, 2018
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXi preprint Xiv: 1411. 1269900,2260 2014.
D. Krotov and J. J. Hopfield. Dense associative memory for pattern recognition. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Newral Information Processing Systems, pp. 1172 1180. Curran Associates, Inc., 2016.
Balaji Labshminarayaaan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Newral Information Processing Systems, volume 30, pp. 6402-6415 2017.
Lisa Langnickel, Johannes Darms, Katharina Heldt, Denise Ducks, and Juliane Fluck. Continuous development of the semantic search engine preVIEW: from COVID-19 to long COVID. Datalase, 2022, 07 2022. TSSN 1758-0463. doi: 10. 10.1093/d dat//daaaaeebbbeeee /bac048. URT. https://doi. org/ 10. 1093/
baac048.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for delect- ing out-of-disttibbution samples and adversarial attacks. Advances in neural information processing systems, 31, 2018.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distrriutiin detection. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. Associates, Inc., 2020a. URL https: pppoeedddnggc/ssssseoooo neurips ct/ccc////// cttppappe//222///11..... wttppiip..pppppp/rpeeeeee
Curran
1539222200044000a144444000000000 Paper, pdf.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based ofitt--disttituttinn detection. Advances in neural information processing systems, 33:21464-21476 21475, 2020b.
Xixi Liu, Yaroslava Lochman, and Christopher Zach. Gen: Pushing the limits of softmax- based out-of-distribution detection. In Proceedings of the IEEE/CC /CVF Conference on Compuler Vision and Pallern Recognilion, pp. 23946 23955, 2023.
Philipp Liznerski, Lukas Ruff, Robert A Vandermeulen, Billy Joe Franks, Klaus- Robert Müller, and Marius Kloft. Exposing outlier exposure: What can be learned from few, one, and zero outlier images. arXiv preprint xrXiv:2255.14474 2022.
llya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arxiv: 039883 2016.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint Xin: 771.005011, 2017.
14

---

Published as a conference paper at ICLR 2026
Lefteris Loukas, Manos Fergadiotis, lon Androutsopoulos, and Prodromos Malakasiotis. EDGAR-CORPUS: Billions of tokens make the world go round. In Proceedings of the Third Workshop Economics and Natural Language Processing, pp. 13-18, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. URL org 2rg/2021..oo www.lp-1..22..
https://aclanthhnt
Haodong Lu, Dong Gong, Shuo Wang, Jason Xue, Lina Yao, and Kristen Moore. Learning with mixture of prototypes for out out-of-distribution detection. In The Twelfth International Conference on Learning Representations, 2024. URL pottps://ppereeee wwwprrvvvieeeee//ruutnn.. id=uNNkKaD3MCs.
Tan Magnusson, Akshita Bhagia, Valentin Hofmann, Luca Soldaini, Ananya Harsh Jha, Oyvind Tafjord, Dustin Schwenk, Evan Walsh, Yanai Elazar, Kyle Lo, et al. Paloma: A benchmark for evaluating language model fit. Advances in Neural Information Processing Systems, 011-4338-64377 2024.
Andrey Malinin and Mark Gales. Uncertainty estimation in autoregressive structured prediction. aie preprint or Kkw:20022006633 2020.
Andrey Malinin, Neil Band, German Chesnokov, Yarin Gal, Mark JF Gales, Alexey Noskov, Andrey Ploskonosov, Liudmila Prokhorenkova, Ivan Provilkov, Vatsal Raina, et al. Shifts: A dataset of real distributional shift across multiple largo-scale tasks. arXw preprint ar 1xxx::210700755, 2021.
Oded Maron and Tomás Lozeno-Pérez. A framework for multiple-instance learning, Advances in neural information processing systems, 10, 1997.
Yifei Ming, Ying Fan, and Yixuan Li. POEM: Out-of-distribution detection with posterior sampling. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (ods.), Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 15650 15665. PMLR, 17 23 Jul 2022. URL fttts::/proceedingg
mlm www.prreeeeeee..e..... wwtppppssssseeeeee https////2mmmmm22a..mm html.
Yifci Ming, Yiyou Sun, Ousmanc Dia, and Yixuan Li. How to exploit hyperspherical embeddings for out-of-distribution detection? In The Eleventh International Conference on Learning Representations, 2023.
Maximilian Müller and Matthias Hein. Mahalanobis+-; Improving ood detection via. feature normalization. In Forty-second International Conference on Machine Learning, 2025.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint 4rXiv:1808.081 2018.
Tomoya Nishida, Noboru Harada, Daisuke Niizumi, Davide Albertini, Roberto Sannino, Simone Pradolini, Filippo Augusti, Keisuke Imoto, Kota Dohi, Harsh Purohit, et al. Description and discussion on dease 2024 challenge task 2: First-shot unsupervised anomalous sound detection for machine condition monitoring. arXiv e-prints, pp. arXiv- 2406, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Fabian Paischer, Thomas Adler, Vihang Patil, Angela Bilto-Nemling Markus Holzleitner, Sebastian Lehner, Hamid Eghbal-Zadeh, and Sepp Hochreiter. History compression via. language models in reinforcement learning. In International Conference on Machine Learning, pp. 17156 17185. PMLR, 2022.
Seongheon Park, Xuefeng Du, Min-HHsan Yeh, Haobo Wang, and Yixuan Li. Steer Ilm latents for hallucination detection. arXiv preprint Xic:2503300191 2025.
15

---

Published as a conference paper at ICLR 2026
Demetri Psaltis and Cheol Hoon Park. Nonlinear discriminant functions and associative memories. In AIP conference Proceedings, volume 151, pp. 370-375. American Institute of Physics, 1986.
Hezhe Qiao, Qingsong Wen, Xiaoli Li, Ee-Peng Lim, and Guansong Pang. Generative semi-supervised graph anomaly detection. Advances in neural information processing systems, 11-4660-4883 2024.
H. Ramsauer, B. Schäfl, J. Lehner, P. Seidl, M. Widrich, L. Gruber, M. Holzleitner, M. Pavlović, G. K. Sandve, V. Greiff, D. Kreil, M. Kopp, G. Klambauer, J. Brandstetter, and S. Hochreiter. Hopfield networks is all you need. In 9th International Conference Learning Representations (ICLR), 2021. URL https://openrevaatt wet/forum/fft t189RnzIiCd.
Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mohammad Saleh, Balaji Lakshmi- narayanan, and Peter J Liu. conditional language models. In The Eleventh International Conference on Learning Representations, 2023. URL /ttpr//////eeeeeerr..ee wwtneeeeouuur odw.ommiikkkuSSS00P2
detection and selective generation for
Karsten Roth, Latha Pemula, Joaquin Zepeda, Bernhard Schölkopf, Thomas Brox, and Peter Gehler. Towards total recall in industrial anomaly detection. In Proceedings of the IEEE/ CVF Conference on Computer Vision and Pattern Recognition, pp. 2022.
Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In International conference on machine learning, pp. 4393-4402. PMLR, 2018.
Lukas Ruff, Robert A Vandermeulen, Nico Görnitz, Alexander Binder, Emmanuel Müller, Klaus-Rober Robert Müller, and Marius Kloft. Deep semi-supervised anomaly detection. arXiv preprint arxiv:1906 02694, 2019.
Lukas Ruff, Jacob R Kauffmann, Robert A Vandermeulen, Grégoire Montavon, Wojciech Samek, Marius Kloft, Thomas G Dietterich, and Klbert Müller. A unifying review of deep and shallow anomaly detection. Proceedings of the IEEE, 109(5):756 795,, 2021.
Ana Sanchez-Fernenn Fernandez, Elisabeth Runetshofer, Sepp Hochreiter, and Guenter Klambauer. CLOOME: a new search engine unlocks bioimaging databases for queries with chemical structures. hioRriv, 2022.
Bernhard Schäfl, Lukas Gruber, Angela Bitto-Nemling, and Sepp Hochreiter. Hopular: Modern Hopfield networks for tabular data. orXis preprint arXiv:2206. 00664,, 2022.
Johannes Schimunek, Philipp Seidl, Lukas Friedrich, Daniel Kuhn, Friedrich Rippmann, Sepp Hochreiter, and Günter Klambauer. Context-enriched molecule representations improve few-shot drug discovery. In The Eleventh International Conference on Learning Representations, 2023.
Bernhard Schölkopf, Robert C Williamson, Alex Smola, John Shawe-Taylor, and John Platt. Support vector mothod for novelty detection. Advances in neurol information processing systems, 12, 1999.
Bernhard Schölkopf, John C Platt, John Shawe- Taylor, Alex J Smola, and Robert C Williamson. Estimating the support of high-dimensional distribution. Neural computation, (8(7):1443-1471 2001.
Abigail See, Peter Liu, and Christopher D Manning. Get to the point: Summarization with pointer-generator networks. arXiv preprint Xrx: 04368, 2017.
Vikash Sehwag, Mung Chiang, and Prateek Mittal. Ssd: A unified framework for self- supervised outlier detection. Xixio preprint orXix:2003. 12051, 2021.
16

---

Published as a conference paper at ICLR 2026
Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang, Jian Zhang, Xiangyang Ji, al. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. Advances in neural information processing systems, 34:2136-2144 2021.
Yiyou Sun and Yixuan Li. Dice: Leveraging sparsification for out-of-distribution detection. In European Conference on Compuler Vision, pp. 691 708. Springer, 2022.
Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activations. Advances in Neural Information Processing Systems, 34:144 157, 2021.
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In International Conference on Machine Learning, pp. PMLR, 2022.
Leitian Tao, Xuefeng Du, Xiaojin Zhu, and Yixuan Li. Non-parametric outlier synthesis. Xin preprinl or Xiv:23333 029666, 2023.
David MJ Tax and Robert PW Duin. Support vector data description. Machine learning, 54 (1):45 66, 2004.
Jörg Tiedemann. Parallel data, tools and interfaces in opus. In Lree, volume 2012, pp- 2214 2218. Citeseer, 2012.
Tim Tomov, Dominik Fuchsgruber, Tom Wollschläger, and Stephan Günnemann. The illusion of certainty: Uncertainty quantification for Imss fails under ambiguity. The 1st Workshop Epistemic Intelligence in Machine Learning (FurIPS), 2025.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 5998 6008. Curran Associates, Inc., 2017a.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances neural information processing systems, 30, 2017b.
Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of-distribution with virtual-logit matching. In Proceedings of the IEEE/CV conference on computer vision and pallern recognition, pp. 4921 4930, 2022.
Qizhou Wang, Zhen Fang, Yonggang Zhang, Feng Liu, Yixuan Li, and Bo Han. Learning to augment distributions for out-of-distribution detection. In NewrIPS, 2023. URL https::/oppnreet wwwpeeviiiiieeeefooeeeeeeeeeeeee
Yiming Wang, Pei Zhang, Baosong Yang. Derek Wong, Zhuosheng Zhang, and Rui Wang. Em- bedding trajectory for out-of-distribution detection mathematical reasoning. Aduances in Neural Information Processing Systems, 37:42965 42999, 2024.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022. URL https: //openreview. meet/forum?idee wit.mm/ddddgZZGgddgtt
M. Widrich, B. Schäfl, M. Pavlović, H. Ramsaner, L. Gruber, M. Holzleitner, J. Brandstetter, G. K. Sandve, V. Greiff, S. Hochreiter, and G. Klambauer. Modern Hopfield networks and attention for immune repertoire classification. Ar 2007. 2020a.
Michael Widrich, Bernhard Schäfl, Milena Pavlović, Hubert Ramsauer, Lukas Gruber, Markus Holzleitner, Johannes Brandstetter, Geir Kjetil Sandve, Victor Greiff, Sepp Hochreiter, et al. Modern hopfield networks and attention for immune repertoire classification. Advances in neural information processing systems, 13:18832-18844 2020b.
Tim Z Xiao, Aidan N Gomez, and Yarin Gal. Wat zei je? detecting out-of of---fiiibttttion translations with variational transformers. arXiv preprint ar Xiw:20006. Kiw:2006.08944, 2020.
17

---

Published as a conference paper at ICLR 2026
Kai Xu, Rongyu Chen, Gianni Franchi, and Angela Yao. Scaling for training time and post- hoc out-of-distributinn detection enhancement. In The Twelfth International Conference Learning Representations, 2024.
Yasin Abbasi Yadkori, Iiaa Kuzborskij, András György, and Csaba Szepesvári. To believe or not to believe your Ilm. arXiv preprint arXiv:2406 02543, 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint or arxi::200.0099 09388, 2025.
Jingkang Yang, Pengyun Wang, Dejian Zou, Zitang Zhou, Kunyuan Ding, Wenxuan Peng, Haoqi Wang. Guangyao Chen, Bo Li, Yiyou Sun, et al. Openood: Benchmarking gencralized out-of-distribution detection. Advances in Neural Information Processing Systems, 35: 32598 32611, 2022.
Jinsung Yoon, Kihyuk Sohn, Chun-Liang Li, Sercan O Arik, and Tomas Pfister. Spade: Semi-supervised anomaly detection under distribution mismatch. Transactions on Machine Learning Research, 2023.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International conference on machine learning, pp. 11328-11339. PMLR, 2020.
Jingyang Zhang, Nathan Inkawhich, Randolph Linderman, Yiran Chen, and Hai Li. Mixture outlier exposure: Towards out-o--distribution detection in fine-grained environments, In Proceedings of the IEEE/CVE /CVF Winter Conference on Applications of Computer Vision (WACV), pp. 5531-5540, Jamuary 2023a.
Jinsong Zhang. Qiang Fu, Xu Chen, Lun Du, Zelin Li, Gang Wang, Shi Han, Dongmei Zhang, et al. Out-of-distribution detection based on in-distribution data patterns memorization with modern Hopfield energy. In The Eleventh International Conference on Learning Representations, 2023b.
Jianing Zhu, Yu Geng, Jiangchao Yao, Tongliang Liu, Gang Niu, Masashi Sugiyama, and Bo Han. Diversified outlier exposure for dut-o--ditt detection via informative extrapolation. Advances in Neural Information Processing Systems, 36, 2023.
18

---

Published as a conference paper at ICLR 2026
TABLE OF CONTENTS
A Related Work
20
B Theoretical Notes
B.1 OOD Score Investigation
21
B.2 Mahalanobis Decomposition
B.3 AP-OOD: Kernel View
22
B.4 AP-OOD Reduces to Mahalanobis Distance with Mean Pooling for B =0
24
C Additional Algorithmic Details
25
C.1 Mini-Batch Attention Pooling
25
C.2 AP-0OD in PyTorch/Einae Einops--ike Pseudocode
25
C.3 On the Difference Between Heads and Queries
25
D Experiments
26
D.1 Additional Details for the Toy Experiment
26
D.2 Hyperparameter Selection
26
D.3 Supervised Experiments on Text Summarization
27
D.4 Visualizing AP-OOD's Attention Maps the Summarization Task
27
D.5 Supervised Experiments on Translation
29
D.6 Experiments on Decoder-Only Language Modeling
30
D.7 OOD Score Comparison
D.8 Ablations
31
D.9 Performance Measurements
33
E Details Continuous Modern Hopfield Networks
36
The Use of Large Language Models
36
19

---

Published as a conference paper at ICLR 2026
A RELATED WORK
Continuous modern Hopfield networks. Modern Hopfield networks (MHNs) are energy-based associative memory networks. They advance conventional Hopfield networks (Hopficld, 1984) by introducing continuous queries and states and a new energy function. MHNs have exponential storage capacity, while retrieval is possible with a one-step update (Ransauer et al., 2021). The update rule of MHNs coincides with attention as it is used in the Transformer (Vaswani et al., 2017a). Examples for successful applications of MHNs are Widrich et al. (2020a); Fürst et al. (2022); Sanchez-Fernand
et al. (2022); Paischer et al. (2022); Schäfl al. (2022); Schimunek et al. (2023); Auer al. (2023) and Hofmann et al. (2024).
Multiple instance learning (MIL). MIL (Dietterich et al., 1997; Maron & Lozano- Pérez, 1997; Andrews et al., 2002; Ilse el al., 2018) considers classifier that maps bag Z = (2)...,, =s) of instances 2s to bag-level label Y E {0, 1}. MIL also assumes that individual labels ys {0,1) exist for the instances, which remain unknown during training. By assumption, the bag-level label is positive once one the instance-level labels is positive (and negative if all instance-lovel labels negative), i.c., Y max, Us- Recent MIL methods use attention pooling (lise et al., 2018; Shao et al., 2021; Al Haji et al., 2024) and modern Hopfield networks (Widrich al., 2020b) to pool the features of the instances.
One-class classification (OCC). OCC (Schölkopf et al., 1999) is the problem of learning decision boundary separating the ID and OOD regions while having access to examples from the ID data set only. One-Class SVM (Schölkopf et al., 2001) learns maximum margin hyperplanc in the feature space that separates the ID data from the origin. Support Vector Data Description (SVDD; Tax & Duin, 2004) learns hypersphere which encapsulates the ID data. Most closely related to AP-OOD is Deep SVDD (Ruff et al., 2018). Deep SVDD learns encoder ob. (): RR - R2 by minimizing the volume of data-cnclosing hypersphere in the output space. Ruff et al. (2019) propose Deep SAD, an extension Deep SVDD that makes use of AUX data during training. However, Liznerski et al. (2022) show that the effectiveness of this extension degrades with increasing dimensionality.
20

---

Published as a conference paper at ICLR 2026
B THEORETICAL NOTES
B.1 OOD SCORE INVESTIGATION
In the following, show that
min JE(L....M)
df (z) Z) log(I/w, (0(1ww,112) 2log(e) 1og(2x)
T²² whenever 3j [w, 11 žj is normally distributed with probability density function
)イ(25) = |ewll2
(- (1100l11 2j -w, -03 12)2))
exp
(16)
22T
weight vectors wy E RD, encoder dene X + z, = Us21 RDx5, Z E 2, 2 = 2, 8j = Zp,,
Mj Zp, Pj A and Pj E A with
S-1 = ((P1.....,.)) E [0, 11s
ΣE Pr -=
i=1
Proof.
Note that the vene Sene-pushforwadd density Poemo of PID satisfies
Preen (Z) Σ pID()) (( Z) PID (I)
w²
Analogously, we get p5{2,) 2 Prlcu (Z) for 2; Zpj and (3((z) 2 po (23) for 2j T 1| Zj-
That is, for any j E [1....,M], we have that PID( I Pa
(Z) S Pj (2)) S pz(2)) As a PTD ( Moreover, the
consequence, for all (I....,M. M3 } it holds that (2()))
following equivalence holds:
pz(25) <
1 exp
(-2 (1 2j wí uz)²
<
2T
||w0|12 exp
1 [w] žj w n2)
T2
e
V2
2
[w] žj
w/ 23)² + log( ( (1:)
< 2 + log(2n)
(17)
As consequence, we have that O, if Equation (17) is satisfied for any (1,...,,M)
O
B.2 MAHALANOBIS DECOMPOSITION
We assume the D weight vectors w, are linearly independent. First, we start from the directional decomposition and show the relation to the Mahalanobis distance.
R =
Σ 1=1
(w z - un² ja)²
(18)
D
= (ž - 14)"
Σ w nou
(2 11)
(19)
(z
12) Σ - - 11)
(20)
Because the weight vectors are linearly independent, E-1 has full rank. Next, we go in the opposite direction and show that the eigenvectors V = (v...., and eigenvalues
D diag( (I,,..,App) of E can be used to define corresponding wi.
21

---

Published as a conference paper at ICLR 2026
Manaa (2, (z 11"E-((2 - -p)
(21)
(z - v² -Vi2z 14)
(22)
(VD-iVE VD-VVH) ' (VD-IV= VD-IVH)
(23)
D
2(wFz -w3 un)
(24)
1=1
where wy 0z E DVD and Σ- VTD-IV.
The relation between the Mahalanobis distance and the directional decomposition is as follows:
1. Any linearly independent sequence W1.-.., WD induces a positive defovite malria E-1 =
Ej=1 w,w., ; and hence a Mahalanobis distance satisfying
Σ
[ wJH²²²
122 (z 1I 1))
(25)
Conversely, ony full-rank covariance matria E admits sel of linearly independent veclors Wi...., WD such that Σ-1 = Σ}= w,w], and therefore Equation (25) holds.
Thus, our decompasition and the Mahalanobis form represent the same quadratic form; the cigen-decomposition is only one possible choice of wy.
B.3 AP-OOD: KERNEL VIEW
In this section, we express the distance function de using kernels to gain further insight into its properties. We use d² as defined in Equation (13):
M d( = Σ 1=1
(T(W"" Zsoftmax(3 zZ z"),)) - Tr(""" Zsoftmax(8 2" w))))²² (26)
where 2. Z, are the sequence representation and the concatenated sequence representa-
tion, respectively, W, IR DxT are the weights of head for j E (1...., .l.....M] M). and 3 E Rzo the inverse lemperature. Recall that. (.1,..., zs) and W, = (W,), Wyr).
For simplicity, we denote the output of dz 2) as de for given instantiations of Z, Z. We start by writing df as
M
d = Σ (8) - =1)2, 1=1
(27)
8, Tr(W, Zsoftmax Z ZWW,)), W,)), My = Tr(W," Zsoftmax 2T (WW,)).
(28)
(29)
We first investigate 8y:
22

---

Published as a conference paper at ICLR 2026
8j Tr Zsoftmax(3 Do W)))
(30)
S T
We
(31)
-1=1 exp (xp(3zFuuzr)
2(=1 exp( [2, w 2,,0111
(32)
Ee=1 21=1 exp( [2] wjr)
ΣT 1= k(Zz, Wj
(33)
=== Et= 1' (z Wji)
where we used the kernels k/ R xR + R with feature maps y :R? - H, 40: RD H'
and H. H'' denote Hilbert spaces. Given a, E R we have
k'(a,)) b) exp(Ba"b) (e'(a); 1 (b()))) k(a,b) k(aab)a"b exp(3a a b)a = (v((),
(34)
(35)
We define E H, wj E EH, E' E H', and w/j E Ht as follows.
8 ((z)) wi = T
T
s
T
2 = S 1=1
Σ
p(wsr) z' = s ==1
Σ
(2,,)
w =
(wijt)
T
(36)
8y can now be expressed as
Σ Σ k(z,, Wst)
83 E Σt 1e' (2,, Wp))
(37)
(2,ū5yHH (2', ,'')''''
(38)
We next investigate the term B4 == Tr(W?' 2softmax W))). Recall that 2 = (Z) Zx) where Z, = (x,) with Z, Z is the sequence representation of sequence Z;- Z, is sequence of stacked tokens (Zi1,.. ..... zis), and for simplicity, we assume uniform sequence length S for all Z; with i E (I,..., N). We define 2, E H, z E H', μ E H, and u' E H' as follows:
s
1
2:
(zia)
#
v Ziss
(39)
S
N
N
$
2
6 Ziaa
(40)
N
SN
1=1
i=
+=1
S
1
1
n'
2,
(zia)
(41)
N
SN
1=1
1=1
a=1
Thus, B2 can be expressed
23

---

Published as a conference paper at ICLR 2026
M = Tr(W3 Zsoftmax(8 ZT W) (M,, 12yHz
(42)
(43)
(pl, W/)P
ΣN 11 E Di=1 (Zis, Wge)
(44)
iil E. Etm1 (zis: Wge)
We can now express d² as follows.
d Σ Tr(W? Zsoftmax(B z"w,))
Tr(W"' Zsoftmax(B Z WW;)))²
(45)
2
(z, 1)
(M,, 20 (n', )2))
(46)
(z', w0 1')
Therefore, d² is a modified version of the "idcalized" distance di
M Σ
d2
((z, wyyH
(e, 20)
(47)
j=1
M
(2 - uT)
00007 Σ
(2 - 11).
(48)
In early experiments, we evaluated & using a range of kernel functions. It performed substantially worse than the d² with attention pooling. This might be due to gradients being either too large (e.g., with the exponential kernel) or too small (e.g., with the RBF kernel) for gradient-based learning.
B.4 AP-OOD REDUCES TO MAHALANOBIS DISTANCE WITH MEAN POOLING FOR 3 = 0
In this section, we show that as 3 = O and M= = D, d2, Z) reduces to the Mahalanobis
distance with mean pooling as used by Ren et al. (2023). To arrive at the result, we assume uniform sequence lengths.
softmax(0- -T0 = exp(0 x, w)
(49)
Σ exp(O) zfww)
5
1 s
= AttPoola (Z = Zsoftmaxx
=TTw ...0....m = Z
(50)
S
aml
N
N
1 AttPoolo () = 2softmax((- 2T) = SN
Zis
(51)
N
(=1
s=l
1=1
where we use the concatenated sequence Ž (Z ... ZNn) and the sequence representations
Z, = ((x;)) = (za..... Zis) E Dx8, The squared distance of AP-OOD reduces to
M
2(2,2)
Σ
W Zsoftmax(B ZTw,) - w/ 2 2softmax ZT TT,))
(52)
j=1
D
Σ
(w.. w.,u²² u22 (2.
(53)
j=1
24

---

Published as a conference paper at ICLR 2026
C ADDITIONAL ALGORITHMIC DETAILS
C.1 MINI-BATCH ATTENTION POOLING
In this section, we describe the process of performing attention pooling over long sequence representation Z Z that is too large to fit into memory by dividing Z into smaller mini-batches of size B E N. For this, we need the log-sum-cxponential (lsc) function. We follow the notation from Ramsauer et al. (2021):
()) = =3--log
Σ exp(BBa.)
(54)
The following algorithm computes μ Zsoftmax( Z"w) for B > 0, and denotes the logistic sigmoid function:
Algorithm 2 Atlention pooling over a long sequence
Require: 2 = (,,,..., Es) DxS, inverse temperature 8, weight vector w, batch size B
1: E -00
2: μ 0-0
3: for 5 to S step B do
4:
Load mini-batch B t- (ž ( ;,tt))
5:
Eg f- lse(3, BT w)
6:
p exp(3(B"
EB))
7:
PB - Bp
8:
PB + (BB(BBB F.))
9:
# t PBUB + (1- PB PB)H
É E+g log (exp(3EB) + exp(3B))
return
C.2 AP-OOD PYTORCH/ EINOPS-LIKE PSEUDOCODE
We detail the loss compulation for AP-OOD with multiple queries per head (Equation (14)) using Torchh Einops-style pseudocode. Assuming uniform sequence length S, Algorithm 3 demonstrates the computation via attention pooling over the sequences Z. Alternatively, Algorithm 4 presents a mathematically equivalent formulation that applies attention pooling the similarities WTZ.
C.3 ON THE DIFFERENCE BETWEEN HEADS AND QUERIES
We find that heads are learnt largely independently from one another while queries are not, which experimentally verify follows: We train AP-OOD using the SGD optimizer the summarization task using (i) 1 head and (ii) 2 heads, where the initialization of one of the heads in (ii) is identical to the initialization of the head of (i). We find that after training for 500 steps, the weight vectors associated with the heads with shared initialization between (i) and (ii) remain identical. In contrast, when repcating this experiment by varying the number of queries, the weight vectors associated with the queries differ after training. Intuitively, adding additional heads will help the model discover more local minima the parameter space (similar to Lakshminarayanan et al., 2017), while adding queries increases the capacity of each given head. The following observation supports this intuition: When testing different hyperparameter combinations for AP-OOD, of queries combined with a small number of heads leads to overfitting when training the model small ID data. sets (c.g., 10,000 sequences): In this case, the average distance of the ID training sequences substantially smaller than the average distance of the ID validation sequences.
found that large number
25

---

Published as a conference paper at ICLR 2026
Algorithm 3 AP-0OD loss in PyTorch/ Einops-like like style using attention pooling Z
def attention oonn porpatiioopoooinngggggoome (Zs, Ws): Zs IN DJ mini batch of N sequences with length
DJ weights of model with heads and queries
# pairwise similarities betueen tokens and weights similarities einsum (Zs, We, HSD, MTD -> N
softmax over query- and sequence dimensions probs similarities softmax (dim= (2, 3)) 4 T
pooling to form sequences
2_ bars einsum(Zs, probs, 'N S.D. NMST -M
return 2 bars
def
loss (Zs Ws);
D] mini- N sequences with length
# Ws [M T D] weights of model with heads and queries
# attention pooling over individual sequence bars attention.pooling (Zs, Ws) # DJ
# attention pooling over all sequences tilie 2s flattenn0,,, 1) unsqueeze (0) #[1, N*S, DJ pttention.poollngg (Z_tilde, Ws) #[1 T D]
# squared distance per head heads_2 einsum (2_bars, Ws, ' M T D. M
T D -> N M2)
heads_m = einsum (mus, Wa, ' M T D da_ squared (heads_Z heads_mu)
M T -> M²)
[N
M]
# regularized and loss regularizer torch. log + sum (dim= (1, 2)) # [M] losses (ds_squared regularizer) sum (dim=1) [NN] loss torch mean (losses)
return loss
D EXPERIMENTS
D.1 ADDITIONAL DETAILS FOR THE TOY EXPERIMENT
In the toy experiment in Figure 2, we modify the attention pooling process use the negative squared Euclidean distance instead of the dot product similarity because the Euclidean distance known lo work better low-dimensional spaces. Formally, the modified allention pooling process is:
exp( 1
wl13)
(Zt
(55)
L exp( E1122 w
D.2 HYPERPARAMETHR SELECTION
To find the values for 3, M,, and T in the unsupervised setting, we perform a grid search using the values { VD 0.25, 0.5, 1, and T' {1, 16, 64). We select M such that the total number parameters of AP-OOD equals the mumber of entries Σ the Mahalanobis method, i.e., such that MT D. We select the hyperparameter configuration by evaluating each resulting model on OOD detection using a validation split of the AUX data set (in the unsupervised setting, the AUX data set only for model selection, not for training the
26

---

Published as a conference paper at ICLR 2026
A Algorithm AP-0OD loss PyTorch, Einops-like style using attention pooling over W Z
def attention com_oooiig (Za, Wa)
D] batch N sequences with length
# Ns [M DJ weights of model with heads and queries
pairwise similarities between tokens and weights similarities einsum (Zs, We, D, M T NMSTT
# softmax over query- and sequence dimensions probs similarities softmar (dim= (2, 3)) # M T]
pooling over similarities pooled Asnummsimilarittes probs
return pooled
def loss (Zs, W
# Za [N S D] mini- of N sequences with length S # Ws [M T D] weights of model with M heads and queries
attention pooling over individual sequence heads_2 attention.pooling (Zs, Ws) #[N M]
# attention pooling over all sequences tiiie Zs f0, unsqueeze (0) #[1, M*S, D] heads_mus atention.pooling
Ws) #[1 M]
# squared distance per head
ds ds_squared (heads_2 heads_mul + [N M]
regularizer and loss regularizer torch. log torch.log((H) (Wa W sum (dim= (1, 2))) # #[M] losses (ds_squared regularizer) sum (dim=1) [N] loss torch mean (losses)
return loss
model), and we select the model with the highest AUROC. In the supervised setting, follow the same procedure, and we additionally select À E 10).
D.3 SUPERVISED EXPERIMENTS ON TEXT SUMMARIZATION
In the fully supervised setting, we train all methods on the embeddings of 100,000 ID examples and 10,000 AUX examples obtained from PEGASUSLARGE trained on text summarization using the XSUM data set. Table 4 shows that AP-OOD substantially improves fully supervised OOD detection results, improving the previously best mean FPR95 of 0.97% (binary logits) to 0.11% in the input OOD setting. Figure 5 shows the results for the semi-supervised setting when scaling the number of AUX examples on all OOD data sets for text summarization. We evaluate relative Mahalanobis only for N' > 1024, because Σ is not invertible when using fewer AUX examples. In contrast to Figure 3, Figure also shows the results for Reddit TIFU and Samsum. On these two data sets, all evaluated methods except relative Mahalanobis achiove near-perfect OOD detection results for N' 28.
D.4 VISUALIZING AP-OOD's ATTENTION MAPS ON THE SUMMARIZATION TASK
We analyze how the attention pooling process of AP-OOD allocates weight to individual tokens. We randomly select sample from each of the four OOD data sets in the summarization benchmark. We then investigate the attention weights a trained AP-OOD model over the generated output sequence. For each sample, we select the two heads with the
27

---

Published as a conference paper at ICLR 2026
AUROC
NO smiee Depor MuNindo
(a) CNN/DM
(b) Newsroom
AL8OC
AUROC
Dec
Mamber example
examples
Reddit TIFU
(d) Samsum
Figure 5: OOD detection performance on text summarization for all OOD data sets. We vary the number of AUX examples and compare results from AP-OOD, binary logits (Ren et al., 2023), relative Mahalanobis (Ren et al., 2023), and Deep SAD (Ruff et al., 2019).
Table 4: Supervised OOD detection performance on text summarization. We compare results from AP-OOD, binary logits (Ren et al., 2023), relative Mahalanobis (Ren et al., 2023), and Deep SAD (Ruff et al., 2019) on PEGASUSLARGE trained on XSUM as the ID data set. indicates "lower is better" and f "higher is better". All values %. We estimate standard deviations across five independent data set splits and training runs.
CNN/DM
Neweroom
Sumsum
Input DOD
Binary logits
FPR05
AUROC +
99 99.27100:
2.0g40.17
0.00
100.00300
99.3gi991
0.97
99.74
AUROC
734434072
91.7119.17
90,05
29..00
Rolutive Mahaalanabia
FPR05
28. 29.34
0.01 EX20
0.01
22.75
Deep SAD
FPR05
AUROC
99,34tm0
99 99.0gamuz
100.0g
1.00%..00
0 0.00.0.00
100 000000
0
98.70
AUROC
8930.0
00:0000000
00.9
(Ouss)
Ousput OOD
Binary logits
FPR05
AUROC
98 48.474
5.77
99. 99.25.40.0
1.8740.2
90.0g
0.00
D9 99.3540.00
0.0g
99.47
1.92
AUROC
823022
Mahalannbiia
FPR05
25 mm
0.96
8.72
Doep SAD
FPR05
AUROC
97 07.334852
99, 29.204
1.00e5222
0.01 0.01
90 09
0. 0.08
80,05 90.055...00
99.22
2.55
(Ouss)
AUROC
90.4620.ur
1.9g:0.1
73-073
30:0000
99.99=t0m
largest deviations in the positive and in the nogative directions before applying the square in the score function of AP-OOD. Figure 6 visualizes the token-wise attention weights of the selected heads. When manually examining the generated output sequences, we find hard to attribute the "OODness" of individual sequences to single token to small set
28

---

Published as a conference paper at ICLR 2026
0.6
2 Tokion Wesghn
Token Weightt
Positive
Negative
CNN DDM
year
again
you
0.3
year
again
when
you
0.4
0.3
stuck
stuck
Tokton
Token
security
Ine
0.1
Positive
Negative
(b) Newsroom
This
what
happens
when
This
what
happers
when
0.3
random
stranger
tries
Tolton Woight
random
stranger
trles
0.6
Toleen Weight
0.2
you
Positive
Negative
(c) Reddit
News
Iound
Tania
round
Michelle
0.6
Toxken Wesgit
Michelle
been
Tokenn Weightt
about
their
weekend
plans
about
their
weekend
plans
<>S>>
Positive
Negative
(d) Samsum
Figure 6: AP-OOD's attention weights randomly selected output sequences from OOD
data sets on text summarization. For each sequence, we visualize the heads with the highest deviation in the positive and negative direction of the dz(Z) before applying the square.
of tokens. Therefore, it difficult interpret the attention weights for the individual heads. However, the results indicate that the different heads exhibit distinct atlention patterns.
D.5 SUPERVISED EXPERIMENTS TRANSLATION
In the fully supervised setting, we train all methods on the embeddings of 100,000 ID embeddings and 100.000 AUX embeddings obtained from Transformer (base) trained on WMT15 En Fr translation. Table 5 shows that AP-OOD improves supervised OOD detection results w.r.t. the mean AUROC and mean FPR95 metrics.
29

---

Published as a conference paper at ICLR 2026
Table 5: Supervised OOD detection performance on Freeishhto-French translation. We compare results from AP-OOD, binary logits, relative mahalanobis (Ren ct al., 2023), and Deep SAD (Ruff et al., 2019) on Transformer (basc) trained on WMT15 En-Fr as the ID data set. + indicates "lower better" and f "higher is better". All values in %. We estimate standard devialions across five independent data set splits and training runs.
Lawe
Subelitee
ad12005
HL2014
Tuput 0on
Binory logits
PPR95 1
AUROC
85.
20.51
22.61 22.61-214
#2.14
90.56
51:23-5-4
73.4
50.14 80.14-140
41.74 41.7442.00
57.1842
55,69
80.50
BS.
54.74
55.29
Reaative Mahialanohis
FPRIG
AUROC
29 00.0099..0
88 21
22. 22-22
06.17-4
30.11-234
67 20
T9.45-140
00
26.71 26.71
88 65.62-954
07 07.87
07 07.84
57.63-904
82 42.7149914
59.10
79.21
AUROC
97 19:
06 06.7499-12
12:01:20:0
80 50.0000
BS 05:452444
01:11
88 273244
98.15
SAD
PTRO5
17 17
20:06-1:5
05.00-912
62
39 5770
57.40
07.00
70.96
$4.96
AUROC
-030
4.50
1.00
-8.49
32.5..4
-9.30
(Ours
PPRID
49.53
22000
Output OOD
Rinary logits
FPRO5
AUROU
28
96 06:12:9.7
83
72.91
37
50.00
ST.
20,80
81.78
56.12
Relative Mahalanobie
FTR95
ALBOC
87
37 27.33
05.86
24.51
06.72-0.4
3041-1
82
66.1739166
20 11.
92 422443
09.20 09.20-1.2
M6.35
87.70
68.5941446
$4.15 $4.154953
SL177
61.68
79.09
Dwup SAD
PPR05
AUROC
81 ag
51 aga1.3
76.54
01.23-148
68 38
-3-00-00
77 14th
72.54-887
84 72gorrz
80 50-40-400 40-4444
03 03.97-994
03.07 03:07
65
12. 12.0079905
05:35-4433
71.87
84.18
AF-OOD
AUROC
19.52-1.00
97.14-0.19
97.71-0-17
13.73-1.85
07.18
56.00-272
97
80.33
81.44-0.0
30.02-4.37
95 05:27:9.00
83.22
20-444
21:28:9944
89 99-006-9.30
1.77
87.50
Table 6: Unsupervised OOD detection performance on large-scale language modcling. We compare results from AP-OOD, Mahalanobis (Lee al., 2018; Ren al., 2023), KNN (Sun el al., 2022), DeepSVDD (Ruff et al., 2018), and model perplexity (Ren et al., 2023) Pythia-1600 trained on the Pile as the ID data set. indicates "lower better" and "higher is better". All values in %. Standard deviations estimated across five independent training runs.
4Chan
Reports
Covid
Clinical
Twitter
Mean
Input OOD
Perplexity
AUROC
65.05
18.32
89.51
85.88
99.22
77.59
FPR95
72.66
86.91
68.00
66.22
2.85
59.25
Mahalanobis
AUROCT
35.27
54.72
75.50
75.67
97. 56
67.80
FPR95
92.93
87.77
98.07
90.79
10.91
76.09
KNN
FPR95
AUROCT
98.85
39.31
93.26
90.03
70.62
95.85
61.12
89.62
Deep SVDD
64.06
72.54
69.34
FPR95
88.15
68.94
99.52
96.35
76.72
89.03
AUROCT
AP-OOD (Ours)
FPR95
88.34
91.47
40.34
57.38
1.52
$5.81
D.6 EXPERIMENTS ON DECODER-ONLY LANGUAGE MODELING
To verify the effectiveness of AP-OOD on the decoder-only language modeling paradigm used by LLMs, we conduct experiments on Pythia-160M (Biderman et al., 2023), a decoder-only language model trained on the Pile (Gao et al., 2020). We evaluate the discriminative power of AP-OOD trained in unsupervised fashion the 4Chan and Twitter subsets of Paloma (Magnusson et al., 2024), the EDGAR anmual reports corpus (annual reports of public companies between (Langnickel et al., 2022), and the MIMIC-III clinical corpus (Goldberger et al., 2000). In the decoder-only setting, we directly the encoded representations of the input sequences and do not generate output sequences. Table 6 shows that AP-OOD improves unsupervised OOD detection w.r.t. the mean AUROC and mean FPR95 metries.
Loukas et al., 2021), Long- MOn-CCOVDD related articles
D.7 OOD SCORE COMPARISON
We experimentally compare the min-based OOD score smin (Z) and its upper bound s (Z). For training, we use the loss from Equation (10) in both settings. The results in Table
30

---

Published as a conference paper at ICLR 2026
Table 7: Unsupervised OOD detection performance on text summarization. We compare results from AP-OOD when using s 2) and Smin Z), , on PEGASUSLARGE trained on XSUM as the ID data set. 4 indicates "lower is better" and "higher is better". All values in %. We estimate standard deviations across five independent dataset splits and training runs.
CNN/DM
Newsroom
Reddit
Samsum
Mean
Input OOD
AUROCT
96. 13:0.44
99. 99.10:0.00
99.91
99.80.10.0
a0::0.000
74
st 2
FPR95
19.51=2224
4.11#0.25
0.00
0.04**.05
AUROC
96.08=047
97.48+0028
99.71
97,6740.35
97.74
Smina
FPR95
18. 18.78=2.73
11.16#1.21
0.01
12.04
10.50
Outpot OOD
AUROC
93.372034
92.62-0.4
28.04
98.30-0.11
95.59
2
FPR95
23. 23.12=107
29.9 29.91=2.00
6.34*1.50
6,83
16.55
AUROCT
93.8 73.82=1.50
$8.30
95,91=2 25
90. 30
92.05
Sminn (Z)
PPR95
26.60=5.68
38.26*3.78
18.49=9001
36.71
30.02
Table 8: Unsupervised OOD detection performance text summarization. We compare results from AP-OOD trained on XSUM as the ID data set when varying B. + indicates "lower is better" and t "higher is better". All values in %. We estimate standard deviations across five independent dataset splits and training runs.
CNN/DM
Reddit
Mean
Input OOD
AUROC 1
G6 06-83-0.01
81.4240077
94.8140-3
93. 93.38
84.11
FPR95
97.17=0.10
76.31
11.12#3..
19. 90.96=0..0
AUROCT
97.76=0.11
98.75#0.07
81.7750000
98.46=0.08
98. 96
0.25
FPR95
11.07 01.07.0774
77550.41
00:00.00
0.0240.0 on
3. 3.96
AUROCT
96.13=0.44
99.10=0000
99. 99.91* +9.00
99 80
74
PPR95
19.51
4.11
0.00
0.04
5.91
AUROCT
91.36 -0.4
98.77 Lo.0s
90.75 20.75 L007
99. 1o.00
97.43
0
FPR95
38.78 =8.00
1.94
0. 0.0249.02
0.00=...00
10.91
81.29=0.01
97.58+00.0
700.7=00.00
B
FPR95
61.11=4.3
9.14
0.12#9.07
0.05
18.16
AUROCT
.0.85 G3.
40.5g
80.5g
99.871..0r
94.79
B 1/VD
FPR95
63.96=3.39
47.50
0.1740.1
0.0g
25.42
Ouiput OOD
AUROCT
77.67=1.37
85.10)=0.0
81.12
74
B 0
FPR95 82.07=1.0
69.32
57.30 57.30*1.79
29 29.3771175
50.52
AUROCT 91.37
15
94.79
96.56 10.27
94.10
8 = 0.25
FPR95 13.03=1-71
31. 30
38.3 38.38
18 18.61 61
33.68
AUROCT 93.37=0.84
92 02.6077
98 00..00
95.59
B 0.5
FPR95
23.1241.07
29 91
6. 6.34
6.83 6.83
16.55
A
AUROCT
24.04=1.95
93.06=0.07
91. 82.82.077
22.04422.07
27 97.66-00333
29
97. 97.91
8.82+1422
95.11
AUROCT
00.25 01.2520.45
91 tou7s
97.571000
97 97.97a
95.19
8 2
FPR95
23.69-1.m1
10.06
8.37
18.34
AUROCT
51.67:0.72
80.5g40.77
91.1240030
9199320
81.08
A 1/VD
FPR95
92.30=0.21
65.8
30.00=1.16
27. 2..20
53.87
show that s sZ) achieves better OOD discrimination w.r.t. the mean AUROC and FPR95. While Smin(2) roughly matches the OOD detection metrics of s(Z) on CNN/DN for both input and output, smiu (Z) lags behind s(Z) on the other OOD data sets.
D.8 ABLATIONS
Beta sensitivity analysis. We evaluate AP-OOD when varying the hyperparameter B on the summarization task. We select 3 from (0, 0.25, 0.5, 1,2], and we leave the settings for M and T unchanged (i.e., they are identical to the settings used in Table 1). Table 8 shows that AP-OOD on text summarization relatively insensitive to the selection of B inside the range [0.25,2] 2] in the input and output settings.
31

---

Published as a conference paper at ICLR 2026
Table 9: Unsupervised OOD detection performance on text summarization. We compare results from AP-OOD trained on XSUM as the ID data set when varying M and T. indicates "lower is better" and 1 "higher is better". All values in %. We estimate standard deviations across five independent dataset splits and training runs.
CNN DM
Newsroom
Reddit
Samarmn
Mesu
Inpat OOD
AUROCT
97.16 97.1640022
98.25486:11
98.8 09.824800
99 99.32
96, 61
M 1024 T
FPR35 14.72
7.54
0.00 0.00-0:00
0.64
5.72
M 612 7
FPR56 9.77-0040
AUROCT 97.96-0.1
4 4.6710.30
98.83=00.07
0.00 0.0040::00
39.875m0
99.
0.02
99.07
3.61
AUROCT
97.76
09.87:30.6
99.46
M
FPRO5
11,0720.74
4.71520441
0.00 0.000.000
0.02
1.96
AUROCT
97. 27.55
98. 4g
90 837307
98. 98.14
98. 75
M
12.48#1.54
29.45
00000
0.25
AUROCT 97.10 97.10
98.1 4tt..
98 #1
98. 47
M
61
T =
FPRO5 14.30
7.87
3..00
0.00
5.79
AUROCT 96.8429.35
97.78 15-15
90 555
98
M 82 T = 32
FPR96 14.9741.9c
10.18
0.01 A0.02
2.58
6.92
AUROCT 96.2329-45
35
73
12
977
16 7
FPRO5 16.65
12.5541-15
0.09
5 1.4.7
6.75
M
T
AUROCT
95.5649.36
18.16
16.34
90.17750.40
0.52
99.6759937
97. 97.44
97.
AUROCT 94.884959
94.75 94.7540.22
10 0022750000
96 22
B5. 916
M
T 256
FPR06 20.10
21.71
2.01 54400
24.58=1.81
17.10
M == T= 512
FPRS6 22.86+2.20
AUROCT 93.1739.76
27.09*1.4
98, 28.4
4.95+9.36
39.75
1888700.00
23. 66
883
AUROCT 41
88.1049.55
96 6.5.0.00
81.41-3
M
FPR95
27.14
32.6412.22
9.0g
52.73 52.73
30.39
Dutput ODD>
AUROCT
92.1730.45
91.1740.30
08.3g
07.77
95.
M 1024 T
FPRS6
39.11
39.69
3.11
12.3 m2
22 38
M
AUROCT
93.79 0.25
95. 85
10
99.02
0.95
96.9g
2.779
96.90
AUROCT 96.35
95. 95.48
98. 19
99. 03.055550 00000
96.77
M 256
T
FPR06 33.674977
88
2.7250
14.75
AUROCT 93.24
35.277
99 21:22110.4
96.68
M
128
T
FPR96 32.84
23.4
0.59 0.59
3 3.20
15.
AUROCT
02.96*9.45
11000 t0.00
88888
96.
M 61 T
FPR06
34.08
34.08-44.22
20.5841.47 41.87
1.4g
4. 4.10A.m
16.30
AUROCT
92.51=0.11
90.11
98, 67..77
98.63=8.41
95.
M 52
FPRUS
37.21 13:76
29. 13.71
14.00
6.11
19.
AUROCT
91,26*1
92.62 92.6g
97, 99
96, 98.58
95. 11
M 16 T
FPR36 41.9641.44
35.7845.7
8.75 11344
6, 6.191
17
= T= 128
FPR06 41.2438.00
AUROCT 90.91
36, 36.4247.55
97. 2.14
13. 13.13 13413
7.58
96.
94
24.59
90.35+2.64
05.91
97.79
M
T= 256
FPRO5
47.5249.04
41.77412.21
18.53
10.02
29.45
AUROCT
87 82
88 18.0691.29
01.00
96.91
91.70
M T
62.1849m
50.66
28, 937.00
13.08
36.31
AUROCT
86, 86.16
86 86.95+1779
93. 43
96. 16.100
90.3
==1 T= 1021
FPR06
49.61 16.70
29.61 18:37
14.824..02
36.24
Number of heads . and queries T. We ablate on the number of heads and the number of queries T of AP-OOD the summarization task. For this ablation, scleet T E 2,, 4,8, 16, 32, 64, 128, 512, 1024) and we then select M such that the total number of parameters of AP-OOD equals the number of entries in Σ of the Mahalanobis method, ie., such that MT = D. The results in Table 9 show that AP-OOD works best on the summarization fask for both input and output when M 512 and T = 2. Although the performance drops when decreasing M and increasing T, we find that AP-OOD is relatively insensitive the number of heads and queries.
Dot product and Euclidean distance. We compare using the dot product and the negative squared Euclidean distance for the attention pooling in AP-OOD. For a formal definition of attention pooling with the negative squared Euclidean distance, we refer to Appendix D.1. Table 10 shows that using the dot product works substantially better. This result aligns with the well-establiseed observation that measuring similarity using the dot product in high-dimensional spaces is more effective than using Euclidean distance.
32

---

Published as a conference paper at ICLR 2026
Table 10: Unsupervised OOD detection performance on text summarization. We compare results from AP-OOD trained on XSUM as the ID data set when using the dot product and the Euclidean similarity. & indicates "lower is better" and f "higher is better". All values in %. We estimate standard devialions across five independent dataset splits and training runs.
DN
Newsroom
Reddic
Sumssum
Mean
Input OOD
AUROC
76-0111
98. 08.75-00.07
99.87+00000
199446-0.0g 0.00
Dot product
FPR95
11.07=0.74
4.75#041
0.00 00
0.02
3.96
AUROCT
74.22=0.05
84.43*022
97.06-0041
08.30
Encliduan
FPR95
00.20
74.08
15.27=b.0
7.17
46.65
Outgut
Dot product
FPR55
AUROC
93. 37-07-054
23.12
29.90 29.93#22.8
92 62-20 as
6.3g*1100
98 08.0g429.20
98. 30
6, 6.83=0.04
95 58.58
16.56
Enclidean
FPR95
AUROC f 87,67
63.62=
68.0
88. 17
22.31=6.0
96 50
91, 91,28:1
53,8977.80
90.90
51.97
Methed
35000
Infermnce (ma)
25000
20000
10000
5000
Encoder +P-DD0
Encoderi Encoder-Mihhaeenn Mahalenchik
Figure 7: Comparing AP-OOD and the Mahalanobis method relative to the encoder inference time. The bars show the mcan over ten batches.
D.9 PERFORMANCE MEASUREMENTS
We analyze the inference time of AP-OOD in comparison to the transformer backbone and other OOD detection methods. To avoid bottlenecks during data loading, we measure inference times on single batches and report the mean and standard deviation across 10 batches. Our measurements only start after warm-up phase of 5 batches. If not stated otherwise, the measurements were performed with batch size of and context. length of 512 tokens. All measurements were performed a single NVIDIA A100-40GB GPU.
Figure compares the inference time of various OOD detection methods for different batch sizes. As expected AP-OOD has a strong linear relation to the batch size and is significantly slower than the reference models.
Although AP-OOD is slower than other methods like the Mahalanobis method, Figure 7 illustrates that it still takes less than 20% of the combined inference time the PEGASUS
Table 11: Inference times of OOD methods and PEGASUS transformer for a batch size of 32 samples, number of heads M 256, number of queries T = 4, and context length of S = 512 tokens. All values in milliseconds ms. We estimate the mean and standard deviation ten batches.
AP-0OD
Mahalanobie
PEGASUS Encoder
PEGASUS Geueration
6.58
0.5240.16
28.13=2.013
3:19 10.31-0..54
33

---

Published as a conference paper at ICLR 2026
Inference times es per batch
Mcthod
140
AP-D0D (supendsed)
120
BinaryLogits
Honalensble
100
80
20
Batch
Figure 8: Comparison of various OOD detection methods for increasing batch sizes. We estimate the mean and standard deviation over ten batches.
AP-DOD inference time at batch size=1
3.8
2.6
2.5
2.7
2.7
4.4
72
13
23
44
3788
2.6
2.6
2.5
2.7
2.6
2.8
6.7
12
22
64
2.5
2.7
3.9
12
Queries (P)
32
2.6
2.6
2.7
2.5
2.5
2.6
2.5
2.6
3.8
2.6
2.5
3.6
2.5
2.5
2.5
2.5
2.4
1.9
2.9
2.6
2.5
2.5
2.6
2.5
2.5
2.5
1.5
2.5
2
2.7
2.5
128
512
Heads
Figure 9: Inference times for AP-OOD over different numbers of heads and queries T. We estimate the mean over ten batches.
encoder and OOD detection method. We argue that the higher OOD detection rate mitigates the addition of overhead of AP-OOD since allows skipping the substantially longer generation time more often (see Table 11). The degree to which the overhead of AP-OOD is mitigated by skipping the decoder depends on the rate of detected OOD samples in future applications.
The heatmap in Figure 9 shows the inference time of AP-OOD for different selections of the hyperparameter number of heads M and number of queries T. While for small values of both parameters the inference time is constant, for larger parameters the inference time increases linearly with both parameters.
The inference time the decoder of the transformer depends on the length the longest output. sequence of batch. To obtain consistent measurements, to always produce the same sequence length. Figure 10 illustrates the inference times of the PEGASUS transformer model. The plots of the first row cover the performance of the PEGASUS encoder only, while the second row shows the combined inference time of the encoder and decoder. In the left column, the plots indicate that the model inference lime increases linearly with the batch size. Further, it is shown that the encoder takes about % of the overall inference time. The right column shows the inference limes for increasing number of context tokens. The inference time of the transformer encoder and decoder increases quadratically with the context length.
forced the decoder
34

---

Published as a conference paper at ICLR 2026
(batch_size:30
0.6
0.050
Interence timne
2.045
0.2
0.040
0.0
PECASUS (Tokons Sw512)
Decoder (batch. size 32)
Inference time (3)
Inference time
40
20
20
Batch size (M)
Tolkens (8)
Figure 10: Inference times for the PEGASUS model for various batch sizes (left) and various numbers of imput tokens (right). The top row illustrates the Encoder's 's performance, while the bottom row shows the combined performance of the Encoder and Decoder. We estimate the mean and standard deviation over ten batches.
35

---

Published as a conference paper at ICLR 2026
E DETAILS ON CONTINUOUS MODERN HOPFIELD NETWORKS
The following arguments are adopted from Hofmann et al. (2024); Fürst et al. (2022) and Ramsauer al. (2021). Associative memory networks have been designed to store and retrieve samples. Hopficld networks are energy-based, binary associative memorics, which were popularized as artificial neural network architectures in the 1980s (Hopfield, 1982; 1984). Their storage capacity can be considerably increased by polynomial terms in the energy function (Chen et al., 1986; Psaltis & Park, 1986; Baldi & Venkatesh, 1987; Gardner, 1987; Abbott & Arian, 1987; Horn & Usher, 1988; Caputo & Niemann, 2002; Krotov & Hopfield, 2016). Tn contrast to these binary memory networks, we use contimuous associative memory networks with far higher storage capacity. These networks are continuous and differentiable, retrieve with a single update, and have exponential storage capacity (and are therefore scalable, i.., able to tackle large problems; Ramsauer et al., 2021).
Formally, we denote a set of patterns ( NN ) C Rd that are stacked as columus the matrix X = (æI...., [NN) and state pattern (query) & E Rd that represents the current state. The largest norm of a stored pattern is M = max; || æ Then, the energy E of continuous Modern Hopfield Networks with state is defined as (Ramsauer et al., 2021)
N
E = ==-3" -B- log Σ
exp(Bxx E)
E"E C, + 2
(56)
where C= 88= log N that the update rule
M22 For energy E and state &, Ramsauer et al. (2021) proved
nnew X softmax(3x*
(57)
converges globally to stationary points of the energy E and coincides with the attention mechanisms of Transformers (Vaswani et al., 2017a; Ramsaucr et al., 2021).
The seporation A of pattern x; is its minimal dot product difference to any of the other patterns:
A min (of a; af m.)))
(58)
A pattern is well-separated from the data if A, is above given threshold (specified in Ramsauer et al., 2021). the patterns o; are well-separated, the update rule Equation (57) converges to fixed point close to stored pattern. If some patterns are similar to one another and, therefore, not well-soparated, the update rule converges to a fixed point close to the mean of the similar patterns.
The updale rule of a Hopfield network thus identifies sample-san sample relations between stored patterns. This enables similarity-based learning methods like nearest. neighbor search (see Schäfl et al., 2022).
Hopfield networks have recently been used for OOD detection (Zhang et al., 2023b; Hofmann et al., 2024). Hu et al. (2024) introduces Hopfield layers for outlier-efficient memory update.
F THE USE OF LARGE LANGUAGE MODELS
When creating this paper, we utilized large language models (LLMs) to refine our writing, to identify related work, and for rescarch idcation. When refining the writing using LLMs, we carefully review and verify LLM output to preserve sentence semantics. For related work, confirm the soundness of papers suggested by the LLM, and for research ideation, verify the factual accuracy of all statements.
36


Reviews:
[{"review_id": "DKnptU9WQO", "reviewer": "Reviewer_18Vp", "summary": "This paper points out the limitations of mean pooling for sequence representations and proposes Attention Pooling (AP-OOD) to compute OOD scores while preserving token-level information. The method uses attention (i) within each sequence and (ii) corpus-wide to estimate representative embeddings, then computes a distance/score between the input sequence’s token embeddings and the pooled corpus statistics.\n\nDuring training, the backbone LM is frozen and only the AP-OOD module is trained; the approach applies seamlessly from unsupervised (ID-only) to semi-supervised settings using AUX data. The evaluation covers summarization (PEGASUS-LARGE trained on XSUM), translation (Transformer-base trained on WMT15 En–Fr), and audio (MIMII-DG), and reports both input-OOD (encoder embeddings) and output-OOD (decoder embeddings).\n\nIn experiment results, the paper shows improved AUROC and reduced FPR@95TPR, with especially large gains over embedding-based alternatives on summarization and audio.", "questions": "**Please provide responses to the reviews mentioned in the Weaknesses section.**\n\n**Additional questions**\n\n(Q1) Application to modern LLMs & hallucination mitigation:\n- Why wasn’t AP-OOD validated on advanced open-source LLMs such as Llama, Qwen, or Phi?\n\n(Q2) Input-/Output-OOD for unsafe or malicious prompts:\n- (Q2-1) How should AUX be constructed to enable reliable detection of unsafe prompts (e.g., jailbreaks or prompts that solicit illegal activity)?\n- (Q2-2) Based on the method and the reported results, do you expect AP-OOD to be effective for blocking malicious inputs (and, by extension, for detecting unsafe outputs)? If so, under what conditions?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Replaces mean pooling with a learned attention-pooling formulation for OOD scoring; the paper clearly decomposes Mahalanobis into attention pooling and illustrates the failure mode of mean pooling with an intuitive toy example.\n\n- Broad empirical coverage across two NLG tasks and one audio task; AP-OOD improves AUROC/FPR95 in unsupervised, semi-supervised, and supervised regimes, with consistent AUX scaling curves. The audio experiment demonstrates modality generality (MIMII-DG).", "weaknesses": "- (W1) Scope limited to task specific backbone models: \nThe models used (PEGASUS-LARGE or Transformer-base) do not consider modern LLMs (e.g., Llama, Qwen, Phi, etc.). Even if one assumes task-specific language models, today’s practice allows fine-tuning LLMs or few-shot (in-context) learning. Defining ID vs. OOD simply as “training data vs. domain-shifted datasets” can be somewhat unrealistic for contemporary LLM usage.\n\n- (W2) Does not directly address hallucination:\nHallucination is a central issue in language generation. While OOD detection is related to hallucination mitigation, the paper offers no empirical evidence or discussion of how AP-OOD might help reduce hallucinations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NH0bQpgXJD", "reviewer": "Reviewer_WMzy", "summary": "The authors address the problem of out-of-distribution (OOD) detection in natural language processing, highlighting the limitation of mean pooling approaches that lose token-level information. To overcome this, the authors propose AP-OOD, a novel OOD detection method that uses attention pooling to assign higher weights to informative tokens within a sequence. The method builds upon the Mahalanobis distance and introduces attention pooling to better utilize token-level information during OOD scoring. AP-OOD can operate in unsupervised, semi-supervised, and supervised settings and demonstrates improvements over existing baselines across various tasks, including summarization, translation, and audio data.", "questions": "1. Attention weights indicate where the model is “looking,” but they do not necessarily reflect the tokens that causally contribute to the OOD decision. Could the authors empirically verify whether higher attention values actually correspond to greater influence on the final OOD score? Could the authors show if tokens with higher attention are indeed more important for the model’s OOD detection?\n\n2. In Section A.2, the paper states that the Mahalanobis distance and the proposed decomposition are “equivalent”.  However, this statement seems mathematically ambiguous. It is unclear whether this equivalence truly holds for arbitrary linearly independent w_j. As currently written, the derivation seems to assume equivalence without sufficient justification. Could the authors explain this step in more detail?\n\n3. The authors set the number of parameters in AP-OOD to match that of the Mahalanobis baseline for a fair comparison. However, it would be interesting to see how much further the performance could improve if the model were allowed to use a larger capacity.\n\n4. AP-OOD employs multiple heads and multiple queries per head to capture diverse token-level informations. However, it remains unclear whether different heads and queries actually learn distinct OOD-related patterns or if they are largely redundant. Could the authors provide qualitative or quantitative evidence showing the contribution of each head to OOD detection? Such analysis would help clarify whether the attention structure learns meaningful diversity rather than merely increasing model capacity.\n\n5. The authors mention that they ensure a fair comparison by matching the number of parameters between AP-OOD and the Mahalanobis baseline. However, this fairness applies only to model size, not necessarily to computational complexity or inference cost. It would be helpful to clarify how much additional computational overhead AP-OOD introduces in practice and whether this extra cost is justified by the observed performance improvements.\n\nPlease answer the questions in the rebuttal.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper moves beyond prior OOD detection approaches that focused on summarizing the entire sentence through mean pooling, introducing a new method that leverages token-level information via attention pooling.\n\n2. By using a toy experiment to intuitively illustrate why attention pooling outperforms mean pooling, the paper clearly and convincingly conveys the core idea of the proposed method.", "weaknesses": "1. The model requires extensive grid search over β, M, T, and λ, which significantly increases computational cost. Since AP-OOD relies on an attention pooling mechanism, each configuration must be trained separately, making the search process expensive. This can limit scalability and practicality when applied to large datasets.\n\n2. Although the paper claims that the unsupervised setting does not use AUX data for training, it still leverages AUX samples for hyperparameter selection. Moreover, the model’s performance appears highly sensitive to hyperparameter choices. This effectively makes the selection process semi-supervised and may introduce bias if the AUX distribution differs from the OOD distribution encountered at test time. As a result, the chosen model may not generalize reliably across unseen OOD scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NKtXgaqcSZ", "reviewer": "Reviewer_M9r8", "summary": "This paper introduces AP-OOD, a novel method for out-of-distribution (OOD) detection in natural language tasks that leverages token-level information instead of simple average-based embedding aggregation. Operating in a semi-supervised framework, AP-OOD flexibly combines unsupervised and supervised approaches by using limited auxiliary outlier data.", "questions": "Please refer to the weakness section.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "- The motivation for using standard Gaussian-based averaging is well explained.\n- The paper presents an interesting and original idea.\n- The authors thoughtfully explore multiple use cases to demonstrate the method’s applicability.", "weaknesses": "- The paper suffers from clarity and presentation issues. The introduction does not sufficiently articulate the limitations of existing methods or explain how the proposed approach addresses them. The methodology section is overly condensed and lacks precision, with several symbols insufficiently defined. Moreover, the authors do not reference the accompanying pseudocode. Overall, the manuscript appears hastily written and insufficiently polished.\n\n- Although the approach is presented as semi-supervised, the paper merely demonstrates a transition between supervised and unsupervised scenarios without providing a clear formulation or justification for the semi-supervised setting. Additionally, the proposed supervised out-of-distribution scenario appears limited in practical applicability, as it does not reflect realistic deployment conditions.\n\n- The experimental setup is restricted to a simplistic baseline where embeddings are averaged across sequences, effectively assuming a Gaussian distribution. This assumption is unnecessarily restrictive, as the embedding distribution could be more accurately modeled using mixture models or more expressive probabilistic frameworks such as normalizing flows.\n\n- The concatenation of Z-embeddings (Algorithm 1, line 8) is likely infeasible for larger datasets due to significant memory constraints, which raises concerns about the scalability of the proposed approach.\n\n- The proposed model assumes access to the training data during OOD detection, which severely limits its applicability in realistic or privacy-sensitive scenarios where such access is unavailable.\n\n- The experimental comparison is incomplete. The authors evaluate primarily against standard baselines while omitting several important domain-specific OOD detection methods, such as:\n\nDirected Sparsification – Yiyou Sun and Yixuan Li (2022), DICE: Leveraging Sparsification for Out-of-Distribution Detection.\n\nVirtual-logit Matching – Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang (2022), VIM: Out-of-Distribution Detection with Virtual-Logit Matching (NeurIPS).\n\nGradNorm – Rui Huang, Andrew Geng, and Yixuan Li (2021), On the Importance of Gradients for Detecting Distributional Shifts in the Wild (NeurIPS).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2LofC5OSJk", "reviewer": "Reviewer_QF7M", "summary": "This paper proposes AP-OOD, an iteration on OOD detection using Mahalanobis distance with token embeddings [1]. In particular, by replaces mean pooling of token embeddings with learnable attention pooling to improve OOD detection in text models. The authors provide rigorous theoretical background as well as empirical evaluations to showcase the validity of AP-OOD. Additionally, the paper presents the methodology in both its supervised formulation (leveraging auxiliary data) and unsupervised formulation.\n\n[1] Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mohammad Saleh, Balaji Lakshminarayanan, and Peter J Liu. Out-of-distribution detection and selective generation for conditional language models. In The Eleventh International Conference on Learning Representations, 2023.", "questions": "The reviewer would like to encourage the authors to revisit the presentation of the overall work. In particular, much of the background can be attributed to Ren et al's paper and could be instead used to visit some of the ablations experiments presented in the appendix [1]. Additionally, more focus on the novelty of the underlying work whether that be the theoretical or algorithmic contribution would help improve the overall quality of the paper.\n\n[1] Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mohammad Saleh, Balaji Lakshminarayanan, and Peter J Liu. Out-of-distribution detection and selective generation for conditional language models. In The Eleventh International Conference on Learning Representations, 2023.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The reviewer notes the following strengths:\n- The paper presents a very clear motivation, with clear justifications on why its nessacary to leverage token-level information beyond mean-pooling.\n- The paper is backed with sound theoretical background which helps frame the entire work and adds valuable contribution to the OOD detection field.\n- The empirical analysis showcases meaningful improvements upon the baseline, with indications that the methodology can be applicable beyond the base text modality (given audio classification evaluations).", "weaknesses": "The reviewer notes the following weakness:\n- Although the improvement is measurable, the incorporation of learnable attention pooling into Mahalanobis distance is not conceptually groundbreaking and may hinder the novelty of the overall work.\n- The paper presents few intuitive explanations for overall observations leading to difficulties in finding the novelty of the work.\n- The lack of experiments on larger, more realistic language models, potentially makes the work less applicable in real-world settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)