Mysteries of the Deep: Role of Intermediate Representations in Out of Distribution Detection
Authors: Imezadelajara
Cristian Rodriguez-Opazo
Damien Teney
Damith Ranasinghe
Ehsan Abbasnejad
Venue: NeurIPS 2025 poster ()
Keywords: out-of-distribution detection
intermediate representations
vision transformers
entropy-based fusion
pretrained models
representation analysis
contrastive learning
model robustness
Primary area: deep_learning
TL;DR: We show that intermediate layers enhance OOD detection and introduce a training-free method to fuse them effectively, improving performance without relying on OOD data.

Abstract: Out-of-distribution (OOD) detection is essential for reliably deploying machine learning models in the wild. Yet, most methods treat large pre-trained models as monolithic encoders and rely solely on their final-layer representations for detection. We challenge this wisdom. We reveal the intermediate layers of pre-trained models, shaped by residual connections that subtly transform input projections, can encode surprisingly rich and diverse signals for detecting distributional shifts. Importantly, to exploit latent representation diversity across layers, we introduce an entropy-based criterion to automatically identify layers offering the most complementary information in a training-free setting, without access to OOD data. We show that selectively incorporating these intermediate representations can increase the accuracy of OOD detection by up to $10\%$ in far-OOD and over $7\%$ in near-OOD benchmarks compared to state-of-the-art training-free methods across various model architectures and training objectives. Our findings reveal a new avenue for OOD detection research and uncover the impact of various training objectives and model architectures on confidence-based OOD detection methods.

Mysteries of the Deep: Role of Intermediate Representations in Out of Distribution Detection
Ignacio Meza De Jara 1.4 Cristian Rodriguez-Opazo" Damien Teney
Damith Ranasinghe
Ehsan Abbasnejad!..
Australian Institute for Machine Learning, University of Adelaide
Iddap Research Institute, Switzerland Monash University {f ifirstname.. last hastname) @adelaide..die au
"Naval Group
Abstract
Out-of-distrine iitoo-ddistributin (OOD) detection is essential for reliably deploying machine learning models in the wild. Yet, most methods treat large pre-trained models as monolithic encoders and rely solely on their final-layer representations for detection. We challenge this wisdom. We reveal the intermediate layers of pre-trained models, shaped by residual connections that subtly transform input projections, encode surprisingly rich and diverse signals for detecting distributional shifts. Importantly, exploit latent representation diversity across layers, introduce entropy- based criterion to automatically identify layers offering the most complementary information in training-frre free betting-withuut access to OOD data. We show that selectively incorporating these intermediate representations can increase the accuracy of OOD detection by up to 10% in far-OOD and over 7% in near-OOD benchmarks compared to state-of-the-art training-free methods across various model architectures and training objectives. Our findings reveal new avenue for OOD detection research and uncover the impact of various training objectives and model architectures on confidence-based OOD detection methods.
Introduction
fitoof-distribution (OOD) detection is essential for reliable machine learning, especially in open-
world settings with distribution shifts that risk unsafe predictions 21 (55). The problem is acute in safety-critical domains such autonomous driving, medical diagnostics, and cybersecurity [17 71).
Recent work has leveraged large vision-language models (VLMs) such CLIP [57] in attempts to address the problem. The methods enable zero-shot OOD detection by aligning image and text embeddings. However, the approaches implicitly treat these deep models as shallow because the infor- mation extracted for detection simply focus on the last layer. Indeed, deep neural networks typically rely on final-layer embeddings as compact, semantically rich representations of the input. But, sole reliance on the last layer overlooks the neural structure through which they are obtained. Therefore, we challenge this widespread wisdom and propose exploring intermediate-layer representations.
Interestingly, early work in convolutional models showed distinct functionality across layers 2 however pretrained vision transformers trained with diverse objectives--Supervissse contrastive, or masked modeling-hon hhve not been thoroughly examined in this context. Understanding how in- and df-ittiiiittni semanties are distributed across depth may offer means for improving detection robustness and generalization. Our study revisits the problem of zero-shot OOD detection to ask: Can intermediate representations be systematically leveraged improve detection performance?
To investigate, we conduct a comprehensive analysis across seven vision backbones, including CLIP, DINOv2 54) MAE 23 MoCo-v3 (8) SiGLiP-v2 [6], Perception Encoder (4), and supervised
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Effect Layer Combination Length OOD Detection Across Architectures
Best FPR vs Baseline
0.9
Architectrre
0.37
0.37
0.46
0.47
0.6
1.42
0.45
Average
29
0.42
0.27
5
9-
10
12
0.5
Size of the Selected Layer Set (N)
FPR @8 95%6
Buseline
CLIP
RN50
VIT-B/16
VIT-B/32
VIT B/pprSuee)vssed)
PEv1-B16-224
SiGLip2-B-16
Figure 1: Effect of layer combination length OOD detection across architectures. The line plot (left) shows average FPR@95 as a function of fused layer count N, where all combinations include the final layer (e.g., N may yield layers {1, 2, 11)). Shaded regions indicate the full range of FPR @95 values across combinations. The gray zone marks the baseline (single-layer) case. The bar plot (right) compares baseline FPR @95 (gray) with the best result from layer fusion (colored) for each model. See Appendix 6. for more details.
VITs [14] We examine how semantic diversity, entropy structure, and prediction stability vary across depth and training paradigms.
Our findings reveal strong contrasts between architectures. CLIP models exhibit high inter-layer diversity, broad top-1 agreement, and smooth entropy transitions across depth- Sypth--rropertiee that enable stable and effective multi-layer fusion. In contrast, supervised and self-supervised VITs often show abrupt shifts or redundancy across layers, limiting the utility intermediate feature fusion. Notably, more recent contrastive models such as SIGLIP and the Perception Encoder do not replicate CLIP's robustness, suggesting that architectural design plays central role in enabling successful fusion strategies. As shown in Figure I combining three to six well-selected layers yields the largest performance gains, while fusing more layers tends to introduce redundancy or noise, diminishing returns. Additional analysis appears in Section 3
Motivated by these insights, we introduce simple yet effective extension of Maximum Concept Matching (MCM) [47. Our method performs entropy-guided selection of informative intermediate fully training-free inference-
layers and aggregates their outputs to improve OOD detection. only, and does not require OOD data, fine-tuning, prompt engineering. The method improves performance, reducing FPR @950095-10 false positive rate when the true positive rate is at 95%-by over 12 percentage points far-OOD datasets and more than 7 points near-OOD benchmarks (see Section 5). These results are consistent with the ablations presented Section
Extensive ablations confirm that our method is robust to hyperparameters such as fusion length, histogram resolution, and temperature scaling. We also observe that the best-performing layer combinations are consistent within a dataset, but differ across domains, emphasizing the need for adaasiee strategies. These results challenge the assumption that the final layer is always optimal for OOD detection. By selectively leveraging signals from across the network, our approach offers practical and general enhancement to training-free OOD methods.
Our contributions are as follows:
We conduct systematic analysis of intermediate representations across seven vision backbones and three training paradigms. While early layers alone are weak, aggregating selected intermediate layers consistently improves OOD robustness (Section
We propose training-free extension of MCM that uses entropy-guide guide- selection to identify and fuse informative layers. The method remains training-free and architecture-agooste for CLIP (Section 4
We validate our approach across six OOD benchmarks and two ID datasets (ImageNet-11 K and PoScal-VOC), achieving consistent gains both single-label and multi-label settings. Our method outperforms prior inference-only basclines, particularly contrastive architectures (Section 5)

---

2 Preliminaries
OOD Detection. Out-of-distribution (OOD) detection aims identify test-time inputs drawn from distribution different from the training one. Classical methods define a scoring function G(x) : R using final-layer outputs of models pretrained on the in-distribution (ID) data. These approaches rely on post hoc confidence scores e.g. softmax, energy, or entropy- overlooking earlier representations that may encode complementary semantic cues.
Zero-shot OOD Detection with Pretrained Vision-Language Models. Recent advances in large- scale contrastive vision- vision-language models (e.g., CLIP (58) have enabled zero-shot classification by aligning images and text within shared embedding space. These models bypass conventional supervised training by leveraging natural language supervision for OOD detection. Given an ID label set y = {. ................................ yK² class-specific text prompts such as "a photo of yı" are embedded via text encoder 7 Ty::,, while images are encoded through visual encoder (xx) Classification is then performed via cosine similarity between modalities:
exp
/T
ply 2)) En exp cos(( [xx), T(y') /)
(I)
where cos(.,)) is the cosine similarity and T temperature parameter.
We use the Maximum Concept Matching (MCM 477) approach to OOD detection, which flags an example OOD if maxy p(v a:) < 0, with 0 controlling sensitivity.
3 Exploring the Value of Intermediate Layers for OOD Detection
We investigate whether leveraging intermediate-layer representations rather than relying solely on final outputs-cca improve OOD detection in the zero-shot setting. To assess the potential of these representations, we extract features from intermediate layers throughout the network and evaluate their predictive utility for OOD detection. Specifically, let d, denote the visual encoder of given model For each architecture, we obtain intermediate representations () from a predefined set of layers aveers]lllowiing us to analyze how OOD-relevant information emerges and evolves throughout the network. Our analysis is conducted on ImageNet-IK as the in-distribution dataset, along with
four diverse OOD datasets: iNaturalist 28 Places 75
SUN (69, and Textures [10]. Further details
can be found in Appendix D
For classification models, we apply the native classification head each intermediate repre-
0.5
AUROC
0.6
sentation oh For contrastive models lacking such heads, normalize d done at the final layer and apply the pretrained projection head p. To address laser-wise dimensionality esseecalll in ResNets- we insert fixed random linear projection before applying (see Appendix
0.4
0.3
0.54
0.6
0.4
Laper -VT (Supervised)
Importantly, this procedure is entirely inference- only: no new adapters or layers are intro- duced, no training phase is involved, and in-distribution labels are used. All components reused from the pretrained network. As result, for each layer f, we obtain a class proba- bility distribution computed h(ol (z)), where is the input image and (l. representation at layer E.
* CLIP RN50
- CLIP VIF
-SGGLip2 D PDv1
Figure 2: Layer OOD detection performance across architectures. Most architectures exhibit their best performance near the final layer, while carly layers generally under-perform.
(x)) is the intermediate
To compute OOD scores, we use confidence-based indicators. For classification models, we extract the maximum softmax probability per layer (MSP) 25 For contrastive models, we apply Maximum Concept Matching (MCM) [8). which computes softmax -orinaxizd cosine similarities between
'Lyeer definitions follow standard naming and indexing of the Hugging Face Transformers library 68

---

image and class embeddings. This yields a score tensor S E RNXLXX where N is the number of input samples. L the number of layers, and C the number of classes, Each entry Sa,te denotes the class-c probability at layer & for input n.
Impact of a Single Layer for OOD Detection We begin by assessing the performance of OOD detection using features of individual layers l in various architectures. As shown in Figure most models reach their best AUROC and lowest FPR @95 at the final layer. highlighting the discriminative power of deeper representations. For instance, CLIP ViT-B/16 and PEViT-B16 16 show consistent rise in AUROC with depth. Some intermediate layers offer isolated improvements, such as early gains in CLIP RN50 reduced 995 layer in DINOv2, but these cases infrequent and strongly depend on the architecture. Similar trends are observed in MAE and MoCo v3, where mid-level layers provide only minor advantages; see Appendix M.17 for details.
Thus, relying on single intermediate layer for OOD detection yields, best, marginal and incon- sistent improvements over the final layer, which remains the most reliable source of discriminative information.
Impact of Multi-Layer Fusion for OOD Detection Given the limited gains from single-layer prediction, we cxplore whether combining multiple layers can improve OOD detection. To this end, aggregate OOD scores from selected subsets of intermediate layers, always including the final layer due to its strong baseline performance. These subsets span all layers and include all possible combinations for given subset size.
Let E N denote the matrix of confidence-based OOD scores, where S,, the OOD scores for the instance at layer f. Given subset of layers & C (l...., LJ.. we computed fused score for each instance by averaging over the selected layers:
DEINX
1 Pr.
|21 ESu, Sm,
This procedure yields single fused OOD score p., per instance. [ the average false positive rate (FPR) at 95% function of the number of layers used in the fusion. Results are averaged over four OOD benchmarks SUN, Places, Textures, and iNaturalist- with shaded regions representing the full range of FPRs (minimum to maximum) observed across all layer combinations of equal size. The plot reveals substantial differences the FPR metric across architectures, with some models- as SiGLip2 [61] and DINOv2 [54 starting with notably higher FPRs, highlighting variability in baseline OOD performance.
Fusion improves performance in model-specific ways. CLIP variants show consistent FPR reduction as layers are added. MoCoV3 and MAE gain moderality, while supervised ViTs sees minimal change. SiGLIp2 and DINOv2 also improve, though less reliably. In contrast, larger models like CLIP ViT-L/14 and PEv1-B16-22 exhibit rising FPR with longer combinations, suggesting that aggregating too many layers may hurt performance when predictions conffict or lack discrimination.
While fusing multiple layers can enhance OOD detection, the specific number and choice of layers that contribute to this improvement vary across models and datasets. This raises the question of whether certain layers are consistently more informative. In Appendix we show that perfffrrrning layer combinations tend to be structurally consistent within given reference ID dataset, yet diverge significantly across datasets. These findings suggest that fusion strategies must be adapted to the domain, as no universal layer subset generalizes well across distributional shifts.
These results highlight the potential of multi- layer fusion for OOD detection while raising key questions: what makes certain combinations effective, and how do they help?
To better understand what makes certain layer combinations effective, we analyze the representational diversity of intermediate layers across models. We use SVCCA [59] to quantify structural similarity between layers by measuring the alignment of their activation patterns. As shown in Figure 3 contrastive models like CLIP exhibit a sharp decline in similarity as the layer distance increases, indicating high inter-layer representational diversity and more progressive transformations. In contrast, supervised models and hybrid architectures display
Uncovering the Diversity of Intermediate Layers

---

Layer-wise SVCCA Decay: CLIP vs. Supervised Methods
SVCCA Similarity (1)
0.8
0.6
0.4
0.2
2
Layer Distance A
Figure 3: Layer-wise SVCCA similarity as function of layer distance A for contrastive and classic models. CLIP models (blue) exhibit lower similarity across layers, indicating more progressive
transformations, while Supervised models (vellow) retain higher laver-wise redundancy.
Hocov
cui vitans C.P VITUU
CLP SIGLEED 2/15
20V1-816-22
Figure 4: Top-1 agreement similarity across transformer layers for various vision models. Each matrix shows pairwise agreement in predicted top-1 classes across layers.
slower SVCCA decay, suggesting higher redundancy and smoother transitions between layers. These models tend refine their representations gradually, concentrating most semantic abstraction near the final layers. By comparison, contrastive models distribute semantic information more evenly across depth, yielding more diverse intermediate features that may offer greater complementarity for fusion.
Layerwise Prediction Agreement and Entropy To complement the SVCCA analysis, mea- sure top-1 agreement across layers to assess prediction consistency. As shown in Figure 4 CLIP models especially ViT-B variants -ehiants-exhibit exhibit broad zones of agreement, reflecting stable prediction behavior across depth. This stability diminishes in larger models like ViT-L/14. Other contrastive models, such SiGLLip and PE, show lower agreement, suggesting abrupt shifts in prediction space. Supervised and self-supervised models (e.g., ViT, MoCo v3) display strong diagonals their agreement matrices, suggesting independent prediction evolution across layers and limited stability, which can undermine the effectiveness of intermediate fusion.
To further contextualize these observations, Figure M. shows the average class entropy across layers. Distinct calibration patterns emerge: supervised models and MAE exhibit sharp entropy drops near the final layers, indicating increasingly confident ofteen prrcoofidenttprrdcitiioss [2 46). Earlier layers, by contrast, retain high entropy with near-uniform class distributions, offering little semantic guidance This weak discriminative signal can limit their contribution in feature aggregation. In contrast, CLIP models display a more gradual change in entropy across depth, with moderate uncertainty in early layers and more confident predictions in deeper allows intermediate layers to contribute non-redundant yet coherent information. Conversely, models like SiGLip2 maintain uniformly high entropy, implying consistently flat softmax outputs that limit the effectiveness of multi-layer fusion. Perception Encoder shows a similar issue, with little differentiation between earlier and later layers.
This structured progression
These trends provide explanatory basis for the differing effectiveness of intermediate-lan fusion observed across architectures. Only architectures that balance inter-layer diversity and prediction consistency-men notably derive substantial benefit from multi-layer fusion. In contrast, models with unstable or flat prediction profiles across depth may require more selective weighted aggregation strategies avoid destructive interference.
Proposed Method to Exploit Intermediate-Layer Representations
Motivated by the patterns uncovered in our analysis of intermediate-layer fusion, presentts training-free OOD detection framework G(-), which leverages intermediate- representations
5

---

labela
Gemenal Method
Layer Selector
Thee asapelo
Ercoder
STEP
Fass such layer floer
MCCMM
Image
Visua Encoders
Layer
Riteal
AECIM
600
Layolr
6.0
Layer
loyes
Layers Gelocted Ae, 1i13
Lagar'
Calial
MCM
Vavilla
Figure 5: We propose general approach OOD detection that exploits features from intermediate layers of a visual encoder (left), extending the Maximum Concept Matching (MCM) method [47]. Section annalyzes the informativeness of intermediate features across architectures. Based these insights, Section introduces an mntropy-based layer selector (right) that identifies the most reliable combination of layers for training- ffre OOD detection.
from pretrained vision-language models. We extend the Maximum Concept Matching (MCM) method across multiple layers and introduce antropy-based layer selection mechanism that identifies the most discriminative combinations for OOD detection.
Text and Visual Encodings. Each in-distribution (ID) class label y/ E Jin is mapped to a text prompt (e.g.. "a photo of a y; and encoded into a semantic embedding T(4,) using pretrained text encoder. On the visual side, backbones such as ViT or ResNet yield intermediate feature representations (z),). (x)) Rach feature (f0) (1) is mapped into a shared embedding space via a projection head p. yielding [x(x) (x)))))) This enables alignment between visual and textual modalities for similarity-based matching.
Maximum Concept Matching (MCM). For each test input x', we compute cosine similarities between the visual embedding Ie(z') and all text embeddings (T(+)) The MCM score at layer is defined as:
s(0 McM (2))
exp
T(t)) /T)
) = max
E exp (cos (Ie(""), T(t)))/T)
where 7 is a temperature parameter that controls the sharpness the distribution. This normalized score approximates the model's confidence at each layer.
Entropy-Baad Layer Selection. To aggregate information across layers, we define a set of candidate combinations C = [CI,..., CM ), where each Cm S {1,..., .,. L.} always includes the final
layer. For each combination C., we compute the aggregated score:
SMCM(0'; Cm)
(4)
1Cml
EEC...
To evaluate each combination Cm. we compute the distribution {SMCM(4'; Cm); over set of unlabeled in-distribution samples {at ) and bin the scores into a normalized histogram {po )
We then compute the entropy:
B
H(Cm) = Σ Pa log(pt),
(5)
b=1
low-entropy distribution indicates confident and concentrated scores, which we use as a proxy for better ID-OOD separability. The optimal layer combination is then selected as:
C arg min H (Cm.).
(6)
C.

---

Table I: Training-free OOD detection results ImageNet-IK and Pascal-VOC ID datasets. We report the false positive rate at 95% TPR (FPR95, 4) and AUROC (1) across six OOD datasets: iNaturalist, SUN, Places, DTD, ImageNet-22K, and COCO. Best results per column are highlighted in bold.
Method
Backhone
iNaturalist
SUN
Places
Texture
ImageNar22K
COCO
Average
FPR. AUCT
FPR
AUCT
FPR. AUCT FPR.
AUCT
FPR.
AUCT
EPR AUCT
FPR AUCH
ImageNetIK
MCM
V/T-
33.85
93.62
40.99
91.56
46.71
89.25
60.90
85,03
45.61
89.87
MCM
VIT-R/16
30.67
94.63
37.41
92.57
43.67
89.96
57.34
86.18
42.27
90.83
SETAR MCM
VIT-B/16
26.92
94.67
35.57
92.79
42.64
90.16
55.83
86.58
40.24
91,05
GL-MCM
VIT-B/16
17.42
96.44
30.75
93.44
37.62
90.63
55.20
85.54
35,25
91.51
SeTAR GL-MCM
VIT-B/16
13.36
96.92
28.17
93.36
36.80
90.40
54.17
84.59
33.12
91.32
Ours
VIT-R/16
15.98
96.90
45.58
89.69
35.71
92.72
25.51
94.84
30.70
93.54
Ours
VIT-B/32
12.02
97.64
28.98
93.37
35.69
91 61
39.36
91.07
29.01
93.42
Pascal-VOC
MCM
VIT-B/32
34.80
95.35
30.60
93.74
37.70
91.99
51.60
91.68
55,00
91.16
59.10
89.23
44.80
92.19
MCM
VIT-B/16
10.51
97.93
30.45
94.25
36.11
91.86
53.21
91.77
53.82
91.12
57.10
89.02
40.20
92.66
VIT-B/16
98.70
26.24
94.95
28.67
93.28
50.32
92.32
44.61
92.63
49.80
89.68
34.00
93.59
GL-MCM
VIT-B/16
4.33
98.81
22.94
94.63
26.20
93.11
41.61
92.88
93.17
43.70
90.71
29.44
93.88
SETAR GL-MCM
V/T6
3.01
99.04
21.76
94.98
24.00
93.73
37.61
93.87
33.46
94.24
40.60
91.48
26.74
94.56
32.18
94.40
35.60
92.61
49.17
92.34
33.72
95.78
91.34
57.60
89.97
43.77
92.74
VIT-B/16
95.77
19.53
95.11
92.06
34.70
91.15
93.52
Inference. At test time, the selected combination C* is used to compute the final OOD score for any input x' using Eq. (4). This enables robust prediction by fusing complementary information across informative layers.
Experiments
5.1 Dataset and Experimental Settings.
We use two ID datasets: ImageNet-1K 12]. a standard large-scale classification benchmark, and VOC [5]. a multi-object detection dataset. Following prior work 30 51 Detailed dataset descriptions and statistics are provided in Appendix [ For ImageNet- KK. we use iNaturalist (28),
SUN [69, Places 775, and Textures
g as OOD datasets, following existing protocols exposed in 30 For Pascal-VOC, we use ImageNet-22K [60], iNaturalist, SUN, Textures and we add
MS-COCO as an auxiliary OOD source when Pascal-VOC is used as the ID) dataset.
Models and Setup. We use CLIP 58 with VIT-B/16 backbone our primary contrastive vision model. All evaluations are performed fully training setting without any additional fine-tuning. To ensure comparability, we apply the softmax function with fixed temperature = For consistency and reduce variance in the fusion process, we fix the number combined layers to L < 5 (see 6). Our experimental setup uses a single NVIDIA RTX 4090 GPU. The selection of the optimal layer combination takes approximately 50 seconds for both ImageNet-ID and Iascal-VOG
Evaluation Metrics and Baselines. We report (i) FPR@95: the false positive rate when the ID true positive rate is 95% (lower better), and (ii) AUROC: the area under the ROC curve for distinguishing ID and OOD samples (higher is better). We compare our method against recent training-free, zero-shot baselines, including MCM 48 GL-MCM [5], and SETAR
5.2 Results
Table 5.1 presents training-free OOD detection results across six benchmarks, using ImageNet-IK or Pascal-VOC as ID datasets. Our method achieves the best average performance both settings. With ViT-B/16 we obtain an FPR95 of 30.70% and AUROC of 93.54 on ImageNet- IK, and 26.97 % for the FPR95 and 93.52 for the AUROC on Pascal-VOC. Gains are consistent across iNaturalist, Texture, COCO, and ImageNet-22K, which span fine- mme-grained and complex domains. The largest improvements occur on COCO and Texture, where standard ID/OOD assumptions often break. ViT- B/32 yields smaller gains on Pascal-VOC, likely due to its lower spatial resolution, echoing prior findings from GL-MCM and SeTAR that emphasize the importance of localized features. We provide further insight into the semantic abstraction patterns driving these improvements in
7

---

Unlike MCM, which uses only the final-layer [CLS] token, our method aggregates selected interme- diate layers. On ImageNet- with ViT-B/16, this reduces FPR95 by 51.9% relative on iNaturalist, achieves 55.5% relative reduction on Texture, and 18.2% on Places. On Pascal-VOC with ViT-B/16 relative reductions of 79.2% on iNaturalist, 35.3% on SUN, 45.9% on Places, and 27.1 % on COCO are observed. These gains are most pronounced in high-variability settings, where final-layer predictions tend to be overconfident.
Near OOD is better with intermedi- ate layer We evaluate near-QOD per-
Table 2: Near-OOD results. Best results per column
are bolded.
formance the NINCO and SSB-Hard benchmarks, following the protocol intro- duced in 773. As shown in Table lever- aging intermediate-layer representations yields consistent improvements across both datasets. Our method outperforms the training-fret free baseline (MCM) by over 7.5% in AUROC and reduces the false pos- itive rate by more than 10 points. Com- pared to fine-tuned approaches such as SETAR+FT, our method achieves a 1.92% % lower FPR while maintaining competitive AUROC. These results indicate that the inductive biases encoded in intermediate-layye representations provide more robust foundation for handling subtle distribution shifts, especially in near-OOD regimes, than fine-tuning on in-distribution data alone.
Method
NINCO
SSB-Hard AUCT
Average
AUC?
FPR.
FPR.
AUC1
FPR.
MCM
73.53
79.69
62.71
90.47
68.12
85.08
GL-MCM
76.03
74.35
66.13
87.42
71.08
80.885
SCTAR
76.95
70.16
68.31
84.94
72,63
77.535
LoCoOp
73.90
80.04
65.73
89.48
69.81
84.76
ScTAR
77.56
70.86
69.68
83.99
73.62
77.425
Ours
80.98
66.74
70.44
82.80
75.71
74.77
Different architectures and prompt-based meth- ods To test portability across backbones and prompting strategies (Table 3. find intermediate- layer fusion remains effective beyond the default set- ting. Unlike single-layer baselines, our method ag- gregates selected intermediate layers. With MCM on RN50x4, we obtain an FPR95 of 41.61 % and AU- ROC of 90.80, while ViT-L/14 achieves 37.09% % for the FPR95 and 91.70 for the AUROC. Prompted ViT- B/16 also benefits substantially: NegLabel reaches 23.79% % FPR95 and 95.05 AUROC. while CSP achieves 15.52% for the FPR95 and 96.74 for the AUROC. These trends mirror our layer-agreement analysis: heterogeneous ResNet stages let fusion re- duce variance and suppress spurious peaks, whereas ViT-L/14's higher inter-layer redundancy and sharper softmax (with т = 0.01) leave less headroom. Dataset context matters]] the largest FPR drops appear on scene-centric SUN/Places where early layers retain layout/background cues; gains are smaller on DTD and iNaturalist, where final embeddings already align with object-centric semantics. Overall, fusion acts at the representation level and transfers robustly across architectures and prompts.
Table 3: Intermediate vs last layer perfor- mance improvement across scoring rules.
Method
Average
FPR AUC1
Extra backbones (MCM)
MCM (Last Layer) RN50x4
45.16
89.95
MCM (Intermediate)) RN5004
41.61
90.80
(Last Layer) VIT-L/14
91.66
(Intermediate) VIT-L/14
91.70
Prompt methods (composition with VT-B/166)
Negl abel
25.40
94.21
NegLabell + Tnt. Layers
23.79
95.05
CSP
17.51
95.76
CSP + Layers
15.52
96.74
Orthogonality to Scoring Rules To evaluate whether our contribution acts at the representation level rather than tai- loring specific decision rule, we report results across multiple OOD scores (Ta- ble 4 The considered scores capture compicmentary statistics: peak sensitiv- ity for MaxLogit (25) and MCM [47],
Table 4: Intermediate last layer performance im- provement across scoring rules. Results averaged across SUN, Places365, DTD, and iNaturalist datasets.
Method
RetNet
VIT-B/32
VIT-R/16
Average
AU
FPR
AUCt
FPR
AUC1
FPR
AUCt
FPR1
Distributional Methods
Entropy
+0.1
+0.0
+1.1
+2.5
+9.0
+16.0
+3.4
+6.2
and global aggregation for Energy 44
Energy
+4.2
+0.4
+2.4
Entropy, and Variance [47). Our method demonstrates consistent improvements across all scoring functions and architec- tures, with particularly strong gains on Vision Transformers where distributional methods benefit most from intermediate- layer fusion (More details in
Variance
+0.1
+1.0
42.3
+16.0
+3.3
+6.1
Max buved Methods
MaxLogit -1.2
+4.0
-0.9
+0.3
+3,2
+8.6
+1.2
+4.3
MCM
+0.8
+3.5
+3.6
+16.6
+2.7
+11.6
+24
+10.2
K
These consistent im-
8

---

provements span both peak-based and aggregation- based metrics, ruling out score-specific artifacts and ensuring compatibility with pipelines that already fix scoring rule. The consistency across this diverse family of scores confirms that intermediate-layer fusion provides broadly applicable representation improvements rather than optimizing for a particular decision boundary.
6 Discussion
Computational Cost Analysis of Inter- mediate Layer Usage We evaluate the computational implications of incorporat- ing intermediate layer features across three CLIP architectures: RN50x4, ViT-B/16, and ViT-B/32. The study varies batch size from 1 to 8 and reports peak GPU memory, per-image latency, total inference time, and throughput (images per second). As shown in Table 5 and Figures L.13] L.15] and [L.14 14 extracting intermediate features adds only modest overhead. Across all configurations, memory and latency remain within practical deployment limits, with minimal increases relative to the last-layer baseline. The effect is smallest for transformer architectures, which sustain competitive throughput. Full details appear in Appendix L
Table 5: Efficiency of intermediate-layer fusion across backbones and batch sizes.
Architecture
Memory
Latency (ms)
Last
All
Last
All
ResNet-5004
0.41+0.00
0.45+0.04
67.70460.43
73.80+63.52
VIT-B/16
0.57+0.00
0.63+0.05
218.47+154.88
224, 124.96+184.71
VIT-B/32
0.5740.01
0.5940.02
211.66175..88
219.46a16733
Ablation: Layer Selection Strategies. We evaluate the effect of different heuristics for selecting intermediate layers to fuse in training-free OOD detection with CLIP ViT-B/16. Since evaluating consider seven strategies based per-
all possible combinations computationally infeasible, layer output statistics: entropy, kurtosis, standard deviation, Gini coefficient, Jensen-Shannon (JSD) divergence, average scoring across all layers, and random selection. Each method selects up layers using poftmax-normallee
logits at fixed temperature T 1.
These metrics are chosen for their sensitivity to the shape of the class probability distribution. Our hypothesis is that layers with peaked, asym- metric distributions are more effective for OOD separation, while flatter distributions yield less discriminative scores. For example, the Gini coefficient -E p²) increases with concen- tration, while JSD divergence measures devia- tion from uniformity in a bounded, symmetric form. The average baseline aggregates scores from all layers without selection, serving as naive fusion strategy.
PPR
30
20
AUROC (8)
94
92
ImageNet-II IK
Pascal-Vol
Entropy Random IIl Kurtosis
STD m Average Gini ISD
Figure 6: Ablation of layer selection strategies.
Figure reports FPR@95 and AUROC ImageNet-IK and VOCC Entropy-based selection consistently outperforms all other methods, achieving the lowest false positive rates and highest AUROC. In contrast, statistical heuristics such as kurtosis and standard deviation yield less reliable results. Although random selection performs similarly on average, its high variance (omitted for clarity) highlights the advantage of informed scoring.
Comparison of selection heuristics.
Ablation: Sensitivity to Histogram Binning (B), Combination Length (T.), and Temperature (7) We evaluate the sensitivity of our method to three key hyperparameters: the number of histogram bins used for score normalization, the combination length (i.c., the number of layers fused), and the temperature applied to softmax outputs. Overall, the method exhibits stable performance across a broad range of configurations. For histogram binning, AUROC and FPR @95 remain consistent when using between 16 and 128 bins, with noticeable degradation outside this range due to under- or over- wer-discretization effects. In the case of combination length, fusing 3 to 6 layers yields optimal results, capturing sufficient semantic diversity without introducing excessive redundancy (see G more detailed analysis); shorter combinations lead to marginal improvements, while longer ones tend to incorporate noisy or less informative layers, reducing detection reliability, Finally, temperature scaling demonstrates that values in the range [0.25, 2.0) preserve stable behavior, whereas values

---

AUROC FIROO
Number
Histogram
Combination Length
Temperature
Figure 7: Ablation studies across different hyperparameter dimensions: number of histogram bins (left), combination length (center), and temperature (right). ImageNet-IK. Pascal-VOO
below 0.25 cause a sharp decline in performance, likely due overconfident predictions. These findings confirm that our method is robust to hyperparameter choices within the empirically validated ranges reported above.
7 Related Work
Out-of-distribution detection. OOD detection identifies inputs that deviate from the training distribution, a key challenge for safe deployment. Early methods used softmax-based confidence scores, such as MSP and ODIN [25] 41, or statistical distances like Mahalanobis [37]. Later work
introduced energy-based scores [44 and Outlier Exposure (OE) [77 better separate ID and OOD
data. Other advances explore residual features [66], hyperspherical embeddings [49]. and activation
shaping [13). Theoretical studies have analyzed learnability and scoring foundations K [6] [5]] while modality-specific taxonomies [ extend upplicability. In NLP. OOD work focuses on intent
classification (72). data augmentation 66, uncertainty [29] and unsupervised adaptation [70], often
using pretrained transformers [26 [6].
Vision-language models for OOD detection. VLMs like CLIP support zero-shot OOD detec-
tion by aligning image and text embeddings. Recent extensions explore concept matching 47 [1],
prompt tuning [50] 39], and synthetic outliers (5). Training-free variants such SeTAR 44) further expand this space. Broader efforts leverage contrastive learning 63 65], noisy supervision [33]
and models like ViLT, VERUa and Tip-Adapter [35 38 74 Studies on calibration and robust- ness [67] [9] highlight strengths and weaknesses under distribution shift. While prior work focuses on final-layer embeddings, we show that intermediate CLIPP layers offer improved detection, influenced by depth and patch size.
Intermediate-layer representations. Early distance-based detectors used intermediate fea- tures (37, but most modern methods rely on final-layer outputs. Recent work shows that intermediate layers carry transferable, shift-stable signals 662, 32], typically with added supervision or retraining. Lin et al. [43] target closed-set classifiers with intermediate exits and exit-wise energy plus input- conditioned early stopping, not VLM embeddings. Fayyad et al. (18) add supervised auxiliary heads; Guglielmo and Masana [20] mainly select single discriminative layer supervised backbones and small-scale benchmarks rather than fuse across depth. In contrast, our approach is training-fre free and VLM-native, select complementary layers using simple ID-only statistics.
8 Conclusion
We challenge the final-layer paradigm in OOD detection by showing that intermediate representations, especially in contrastive models like CLIP, encode diverse and complementary signals. Our training- free method selects and fuses informative layers via entropy-based scoring, yielding consistent improvements across near- and far-OOD benchmarks. These results highlight the architectural differences that shape layer utility and demonstrate the value of exploiting internal model structure for robust, inference-only OOD detection.
This work was supported by Naval Group. The authors are affiliated with the Centre National de la Recherche Scientifique (CNRS) and the CROSSING Lab.
Acknowledgment
10

---

References
[1] Udit Arora, William Huang, and He He. Types of out-of-distribution texts and how to detect them. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 10687-10701. Association for Computational Linguistics, 2021.
[2] David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissec- tion: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 654 6541-6549, 2017.
[3] Julian Bitterwolf, Alexander Meinke, Maximilian Augustin, and Matthias Hein. Breaking down out-of-distribution detection: Many methods based ood training data estimate a combination of the same core quantities. In Proceedings of the 39th International Conference on Machine Learning (ICML), volume 162, pages 2041-2074. PMLR, 2022.
[4] Daniel Bolya, Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Rasheed, Junke Wang, Marco Monteiro, Hu Xu, Shiyu Dong, Nikhila Ravi, Daniel Li, Piotr Dollár, and Christoph Feichtenhofer. Perception encoder: The best visual embeddings are not at the output of the network, 2025.
[5] Chentao Cao, Zhun Zhong, Zhanke Zhou, Yang Liu, Tongliang Liu, and Bo Han. Envisioning outlier exposure by large language models for out-of-distribution detection, 2024.
[6] Derek Chen and Zhou Yu. GOLD: Improving out-of-scope detection in dialogues using data augmentation. arXiv preprint vrXh:22109033099 2021.
[7] Mengyuan Chen, Junyu Gao, and Changsheng Xu. Conjugated semantic pool improves ood detection with pre-trained vision-language models, 2024.
[8] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers, 2021.
[9] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild, 2013.
[10] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3606-3613, 2014.
[11] Sanjoy Dasgupta. Experiments with random projection, 2013.
[12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proc. CVPR, pages 248-255. Ieee, 2009.
[13] Andrija Djurisie, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. Extremely simple activation shaping for out-of-distribution detection, 2023.
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations (ICLR), 2021.
[15] Mark Everingham, Lue Van Gool, Christopher K. I. Williams, John M. Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. Int. J. Comput. Vis., 88(2):303-338, 2010.
[16] Zhen Fang, Yixuan Li, Jie Lu, Jiahua Dong, Bo Han, and Feng Liu. Is out-of- detection learnable? In Advances Neural Information Processing Systems (NeurIPS), 2022.
[17] Zhen Fang, Yixuan Li, Jie Lu, Jiahua Dong, Bo Han, and Feng Liu. Is out-of-distribution detection learnable?, 2023.
[18] Jamil Fayyad, Kashish Gupta, Navid Mahdian, Dominique Gruyer, and Homayoun Najarran. Exploiting classifier lnter-level features for efficient out-r--distribution detection. Image Vision Comput., 142(C), February 2024.

---

[19] Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of detection. In Conference on Neural Information Processing Systems (NeurIPS), 2021.
[20] Guglielmo, Gianluca and Masana, Mare. Leveraging intermediate representations for better out-of-distributiin detection. In Proceedings of the 28th Computer Vision Winter Workshop (CVWW), pages 53-61 33-61. Verlag der Technischen Universität Graz, 2025.
[21] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks, 2017.
[22] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, pages 1321-1330. PMLR, 2017.
[23] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners, 2021.
[24] Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joe Kwon, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. Scaling out-of-distribution detection for real- world settings, 2022.
[25] Dan Hendrycks and Kevin Gimpel. baseline for detecting misclassified and out-of-distribution examples in neural networks, 2018.
[26] Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. Pretrained transformers improve out-of-distribution robustness. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL.) 2020.
[27] Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure, 2019.
[28] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alexander Shepard, Hartwig Adam, Pietro Perona, and Serge J. Belongie. The inaturalist species classification and detection dataset. In CVPR, 2018
[29] Yibo Hu and Lanfang Kuang. Uncertainty-awant reliable exit classification. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), 2021.
[30] Rui Huang and Yixuan Li. Mos: Towards scaling out-of-distribution detection for large semantic space, 2021.
[31] Paul Jaccard. Etude de la distribution florale dans une portion des alpes et du jura. Bulletin de la Societe Vaudoise des Sciences Naturelles, 37:547-579, 01 1901.
[32] Fran Jelenić, Josip Jukić, Martin Tutek, Mate Puljiz, and Jan Šnajder. Out-of-distribution detection by leveraging between-layer transformation smoothness, 2024.
[33] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hien Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Proceedings of the 38th International Conference Machine Learning (ICML) 2021.
[34] Xue Jiang, Feng Liu, Zhen Fang, Hong Chen, Tongliang Liu, Feng Zheng, and Bo Han. Negative label guided ood detection with pretrained vision-language models, 2024.
[35] Wonjae Kim, Bokyung Son, and Ildoo Kim. ViLT: Vision-and-language transformer without convolution or region supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML), 2021.
[36] Kasper Green Larsen and Jelani Nelson. The -ohnson-linneeestrauss lemma is optimal for linear dimensionality reduction, 2014.
[37] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. simple unified framework for detecting out-of-distrrbutiio samples and adversarial attacks, 2018.
12

---

[38] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai- Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.0355: 2019.
[39] Tianqi Li, Guansong Pang, Xiao Bai, Wenjun Miao, and Jin Zheng. Learning transferable negative prompts for dilto---iitrrbutionn detection, 2024.
[40] Yixia Li, Boya Xiong, Guanhua Chen, and Yun Chen. Setar: Out-of-distribution detection with selective low-rank approximation, 2024.
[41] Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing the reliability of out-of -istriibiint image detection in neural networks, 2020.
[42] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects context, 2015.
[43] Ziqian Lin, Sreya Dutta Roy, and Yixuan Li. Mood: Multi-level out-of-distribution detection, 2021.
[44] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Tao. Energy-based out-of nut-of-distribution detection. Advances in Neural Information Processing Systems, 2020.
[45] Mark D. McDomnell, Dong Gong, Amin Parveneh, Ehsan Abbasnejad, and Anton van den Hengel. Ranpac: Random projections and pre-trained models for continual learning, 2024.
[46] Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. In Advances in Neural Information Processing Systems, volume 34, pages (6682-16994, 2021.
[47] Yifei Ming. Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li. Delving into distr--disttibutton detection with vision-language representations, 2022.
[48] Yifei Ming, Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li. Delving into out-of-distribution detection with vision-language representations, 2022.
[49] Yifei Ming, Yiyou Sun, Ousmane Dia, and Yixuan Li. How to exploit hyperspherical embed- dings for
detection?, 2023.
[50] Atsuyuki Miyai, Qing Yu, Go Irie, and Kiyoharu Aizawa. Locoop: Few-shot out-of-distribution detection via prompt learning, 2023.
[51] Atsuyuki Miyai, Qing Yu, Go Irie, and Kiyoharu Aizawa. Gl-mem: Global and local maximum concept matching for zero-shot dutsr--distontttion detection, 2025.
[52] Peyman Morteza and Yixuan Li. Provable guarantees for understanding distriibuiiinoionnn detection. In Proceedings of the AAAI Conference on Artificial Intelligence (AAI), 2022.
[53] Tuomas Oikarinen and Tsui- Weng. Clip-dissect: Automatic description of neuron repre- sentations in deep vision networks, 2023.
[54] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza. Francisco Massa, Alaacldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po- Ya- Huang, Shang- Ween Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2024.
[55] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D Sculley, Sebastian Nowozin, Joshua V. Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift, 2019.
[56] Alexander Podolskiy, Dmitry Lipin, Andrey Bout, Ekaterina Artemova, and Irina Piontkovskaya. Revisiting mahalanobis distance for transformer-based out-of-domain detection. In Proceedings of the AAAI Conference Artificial Intelligence (AAAI), 2021.
13

---

[57] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal. Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
[58] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
[59] Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretabilityy, 2017.
[60] Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretrainin for the masses, 2021.
[61] Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alab- dulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. Siglip 2: Multilingual vision- language encoders with improved semantic understanding. localization, and dense features, 2025.
[62] Arnas Uselis and Seong Joon Oh. Intermediate layer classifiers for ood generalization, 2025.
[63] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 18x7. 00.44437 2018.
[64] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: a good closed-set classifier is all you need?, 2022.
[65] Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[66] Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of-distribttion with virtual-logit matching, 2022.
[67] Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R. Ledsam, Patricia MacWiliimms Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, Taylan Cemgil, S. M. Ali Eslami, and Olaf Ronneberger. Contrastive training for improved out-of- distribution detection. arXiv preprint arXfiv2007.055566 2020.
[68] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of- the-art natural language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45. Association for Computational Linguistics, 2020.
[69] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In Proc. CVPR, pages 3485-3492. IEEE, 2010.
[70] Keyang Xu, Tongzheng Ren, Shikun Zhang. Yihao Feng, and Caiming Xiong, Unsupervised out- of-domain detection via pre- transformers. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL), 2021.
[71] Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey, 2024.
[72] Li-Ming Zhan, Haowen Liang, Bo Liu, Lu Fan, Xiao-Ming Wu, and Albert Lam. Out-of-scope intent detection with self-supervision and discriminative training. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL), 2021.
[73] Jingyang Zhang, Jingkang Yang, Pengyun Wang, Haoqi Wang, Yueqian Lin, Haoran Zhang, Yiyou Sun, Xuefeng Du, Yixuan Li, Ziwei Liu, Yiran Chen, and Hai Li. Openood v1.5: Enhanced benchmark for
detection, 2024.
14

---

[74] Renrui Zhang, Rongyao Fang, Peng Gao. Wei Zhang, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-Adapter: Training-frr -fee CLIPeadapter for better vision-language modeling. arXiv preprini arXiv:21 11.09330 2021.
[75] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. TPAMI, 40(6); 1452-14524449- 2017.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope. The paper sets out investigate whether intermediate- layer representations in pretrained vision-language models can improve training-free OOD detection.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations discussed in Appendix A
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer: [NA]
The paper does not include formal theoretical results or proofs. While presents conceptual assumptions (such as the presence of progressive semantic abstraction across layers) and analyzes them empirically through metrics like SVCCA similarity and entropy, there are no formal theorems, lemmas, or mathematical proofs provided. The work is entirely empirical in nature, focused on evaluating and improving training-free OOD detection using intermediate-layer representations in pretrained vision- laiiunla models.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper provides sufficient detail to enable reproduction of its main experi-
mental results in and
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility.
17

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: The code is not yet ready for public release, but we plan share it upon publication to support future research.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
mublic/guidess httpe////oooeeuubbasssosoooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips. http/uullcggguucee http///ooeSuumiissseeeeeee asioonooicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper provides all necessary experimental details. Our method does not require training. but specify how features are extracted, how entropy-basee layer selection is performed, and how evaluation metrics are computed. Additional implementation details are provided in 5.1 and to ensure full reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Our method is fully deterministic and yields identical results across runs, so reporting standard deviation is unnecessary. For baselines involving stochastic processes, report the average and standard deviation over three runs.
18

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative етгог rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Our experimental setup uses a single NVIDIA RTX 4090 GPU (see Section 5.1 for details). All evaluations and ablations were performed on this hardware.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httph:tt//////tt httpa//:///u/is...eeee https//s.eeceepbbbtttttcttttttte
Answer: [Yes]
Justification: This work fully complies with the NeurIPS Code of Ethics. It does not involve human subjects, crowdsourcing, or the release of models or datasets that pose risks of harm, privacy violations, or misuse. A more detailed discussion B
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
19

---

Question:
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This work discusses this in Appendix C
Guidelines:
The answer NA means that there no societal impact of the work performed.
the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: This work does release any models, datasets, assets that pose risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper relies exclusively on publicly available pretrained models from Hugging Face (68] and benchmark datasets described in Appendix
Guidelines:
20

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwprswipeeaaaeeee thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of derived asset (if has changed) should be provided. this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our work does not introduce any new datasets, models, or code assets. Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification: This work does not involve any crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for rescarch with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
21

---

This paper does not involve human subjects research or any form of data collection from study participants, including crowdsoureing.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: This work does not involve large language models (LLMs) as part of core methodology, analysis, or contributions.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy auttee///eearigeccc///offrrrrrer for what should or should not be described.
22

---

A Limitations
Despite demonstrating consistent improvements, several limitations highlight promising directions for future research. First, while our method benefits from the structured and diverse intermediate representations found in architectures like CLIP, it remains less effective in models with flatter or redundant representations, such as MAE or Perception Encoder. Future work could explore adaptive or weighted fusion strategies to mitigate these limitations. Second, although our entropy-based layer selection is inference-only and label-free, the selected combinations vary significantly across ID datascts. This domain sensitivity suggests the need for more generalizable selection mechanisms, potentially incorporating Outlier Exposure or few-shot adaptation. Third, our study is restricted to visual inputs. Extending the method to other modalities- test the versatility of intermediate-layee fusion for OOD detection, especially in domains where semantic abstractions may emerge differently across depth.
audio or language-could further
B Ethical Considerations
This work proposes intermediate-layer fusion in pretrained vision-language models. The method does not involve the collection or of personal data, nor does it introduce new generative components learning from sensitive content. It operates entirely at inference time using publicly available models and datasets. As such, it poses minimal risk in terms of privacy, fairness, or misuse. Nonetheless, we acknowledge that robust OOD detection plays a vital role in the safe and ethical deployment of machine learning systems. Future research should continue examine the implications of model biases and domain shifts to support fair and transparent Al behavior in real-world applications.
training-free method for out-of-distribution (OOD) detection based on
C Societal Impact
Improving OOD detection is crucial for the reliability of machine learning systems deployed in open- world environments, including safety-criticn domains such as healthcare diagnostics, autonomous vehicles, and security applications. Our method contributes toward the development of AI systems that can identify and appropriately reject unfamiliar or anomalous inputs, thus preventing overconfident and potentially harmful predictions. Additionally, accurate OOD detection may support tasks such as active learning, human-ing numan-in-the-lle in-the-loop filtering, and data selection for continual learning. By enabling more trustworthy model behavior under distributional shift, this line of research contributes to the broader goal of responsible and safe Al deployment.
D Datasets
Following prior work 30 we adopt the standardized configuration for in-distribution (ID) and out-of-distribution (OOD) datasets that has been widely used in recent CLIP-based OOD detection benchmarks. Specifically, we use ImageNet- 1K (12) and Pascal-VOn VOC [15] as ID datasets. For OOD evaluation. we follow the curated protocol proposed in MoS [30], which provides de-duplicated subsets of four OOD datasets: iNaturalist (28). SUN [69, Places (75), and Texture (DTD) [10].
Additional OOD sets include ImageNet-22K
60 and MS-COCO 42
In-Distribution Datasets
ImageNetIK ImageNet-IK [2]
serves our primary in-distribution dataset. Consistent with prior evaluations 30/48, 48), we assess OOD detection using the full 50,000-image validation split, which
spans 1,000 object categories and is widely adopted for benchmarking zero-shot performance.
For object-centric evaluation, we utilize Pascal-VOC [5 as an dataset. Following the setup introduced by Miyai et al. [51), we select samples containing only a single labeled object per image. ensure comparability with SeTAR 40) we adopt the 906-image test set configuration used in their study.
Pascal-VOC
23

---

Out-of-Distribution Datasets
iNaturalist iNaturalist [2]] is a large-scale dataset of flora and fauna images. We employ the 10,000-image version curated by Huang et al. [30], where overlapping classes with ImageNet-Il I are removed ensure distributional shift.
Places The Places dataset 77 contains millions of labeled scenes across various environments. For evaluation, rely 10,000-image subset filtered by (00 eliminate semantic overlap with the ID categories.
SUN SUN [69] features broad collection of indoor and outdoor scene images. We use the filtered version compiled by [30], which retains 10,000 samples that are semantically distinct from ImageNet- IK.
Texture (DTD) The Deseribable Textures Dataset 10 includes fine fine-grained texture patterns organized into 47 visual descriptors. In line with (30), we employ the full set of 5,640 images, which are disjoint from the object categories in our ID datasets.
60 extends the standard ImageNet taxonomy to over 21,000 categories. We use the version filtered by Ming al. [48), which excludes all classes overlapping with ImageNet- K and Pascal-VOC, to serve as OOD input in both evaluation settings.
ImageNet-22K ImageNet22K
MS-COCO MS-COCO 42 is used in filtered setting (COCO-OOD) containing 1,000 samples that are class-wise disjoint from Pascal-VOS We use the version from [48] to test OOD detection performance when Pascal-VOO is the ID distribution.
Near-OOD Datasets
We follow the recommendations from OpenOOD 73 to evaluate near-OOD detection using two challenging benchmarks that emphasize subtle distributional shifts while maintaining high visual similarity to in-distribution data.
NINCO NINCO [24] is a curated benchmark for evaluating nuanced OOD detection in natural images. It includes near-OOD samples that are visually similar ImageNet-1K categories but originate from distinct distributions.
SSB-Hard SSB-Hard (64] introduced for open-set recognition on ImageNet-IK, It provides only OOD samples with minimal visual deviation but distinct semantics.
E
Diversity and Consistency of Oracle Layer Combinations
We assess the internal diversity of oracle-selected layer combinations in Figure E.8 using the Jaccard distance [31], computed among the top-10 performing combinations. A lower Jaccard distance indicates stronger agreement in the selected layers across different lop-performing combinations. As shown, both ImageNet and Pascal-VOL
exhibit low pairwise distances within their top- selections (mean Jaccard distances of 0.480 and 0.477, respectively), suggesting high internal consistency. This implies that, once an ID dataset fixed, best-performing combinations tend to share many common layers, indicating that certain representations are consistently beneficial for OOD detection within each domain.
However, this consistency does not extend across datasets. While individual ID datasets demonstrate stable oracle combinations, the actual sets of selected layers differ substantially between ImageNet and Pascal-VOC This observation is further supported by Fig which quantifies the overlap between top-k layer selections across datasets. Notably, the intersection between the top-50 combinations of ImageNet and Pascal comprises only four shared configurations. These shared combinations yield an average FPR of 0.300, which a worst fpr than the 0.288 average obtained by selecting layers independently for each ID dataset using an entropy-based heuristic. This discrepancy indicates that a single fixed set of layers is inadequate for generalizing across ID datasets, and that the optimal composition of layers for OOD detection is highly dataset-dependent.

---

These findings underscore two key insights: (1) within a given ID dataset, the top-performing layer combinations exhibit high structural consistency, and (2) across different ID datasets, the optimal configurations diverge significantly, reflecting distinct underlying representational demands. This analysis reinforces the conclusion that static fusion strategies are insufficient for general-punee purpose OOD detection.
(..4.6.0.33
76o-
6.7
05.6.9
111
3.50
0.22
0.25
3.13
0.00
1.52
5.5
9.5.4
2.20
2.50
0.7
2.50
04.6
70a-
0.50
2.
0.50
5.0
5.7
0.00
5.50
0.63
0.50
5.02
0.50
6A
Tua
0.67
0.50
0.67
0.00
5.00
4-40
13.3
2.40
0:30
0.50
6.22
0.00
0.25
2.40
0.00
2.57
0.42
Toa.6
0.20
0.
0.22
0.40
5.02
9.50
535
0.00
5.67
03.0
9.25
0:40
0.25
0.00
9:25
0.50
5.40
0.20
0.67
0.57
8.71
5.02
04.3.3 111
6.25
3.00
0.00
5.67
0.40
0.0
0.67
3.00
3.50
5.40
0.00
05.4 110
0.50
8.00
0.50
6.77
3.67
5.52
6.5
5.00
0.57
7a-30
8.17
030
0.43
0.35
5.
2.32
542
0.30
0.57
0.00
Tep
3:40
0.22
0.25
60
ImageNet (ID)
(b) Pacal-VOC (ID)
Figure E.8: Jaccard distance between the top-10 oracle combinations under two different ID datasets. Lower values indicate higher agreement between selected layer sets.
Top-k
Top-k 50
Top-k 100
96 0.312
0.296
0.226
0,303
0.300
0.231
0.309
0.237
ImageNet
Pascal-VOC
ImageNet
Pascal-VOC
ImageNet
Pascal-
Figure E.9; Overlap of top-k oracle layer combinations between ImageNet and Pascal-VOC. Inter- section regions are shaded in gray and color matthee to the scatterplot. The bold decimal values indicate the average FPR computed across all OOD datasets evaluated under each ID configuration (ImageNet or Pascal-VOG.
Semantic concept emergence across layers.
Figure F.10 offers interpretive bridge between theoretical foundations (Section H represen- tational analyses (Section [6). and empirical evaluation of intermediate- fusion strategies. visualizes the layer-wiss distribution of six high-level semantic categories, defined by [53], across eight datasets. Each curve denotes the average proportion of concept-relevant descriptors matched at each transformer layer, thereby revealing both architectural regularities and dataset-specific semantic emergence.
This figure directly substantiates Assumption A1 (Layered Semantics) from Section H Low-level concepts (c.g., colors, textures) consistently peak in early layers (typically layers 2 to 5), while high-level concepts (e.g., objects and machines, activities) peak in deeper layers (layers 9 to 12). This stratification reinforces the canonical coarse-to-fim information processing in vision transformers, but here it is quantified via explicit semantic concept activation. rather than inferred from feature abstraction or classification accuracy. These results confirm that semantic depth varies across layers, validating our use of intermediate-layer fusion to capture complementary features.
Moreover, Figure F.10 deepens the interpretation the SVCCA-based similarity trends in Figure 3 As discussed in Section 6 contrastive models exhibit lower similarity across layers, indicative of
25

---

greater representational diversity. The semantic traces in TigureFF.1 offer an explanation: layers specialize in distinct concept types. In particular, the delayed emergence of categories such as objects supports our hypothesis that last-layer-only approaches may overlook crucial intermediate semantics. This aligns with the performance gains we observe when fusing information across multiple layers.
Finally, we note that the concept distributions are remarkably consistent across datasets, suggesting that concept emergence not purely data-driven but also reflects model-internal inductive biases. This insight supports the transferability of our layer selection heuristics: strategies optimized on one dataset (e.g., ImageNet) remain effective across others (e.g., COCO, SUN), as demonstrated in our zero-shot evaluations for OOD detection.
0.8
Colorss
Textures and materials
Natural elements and organisms
0.F.
Proparion
Poporcon
Proputton
Layer
Layer Places and buildings
Layer
Objects and machines
Activitie
0.0
0.6
Propettio
Propurtinn
Propucton
0.4
0.2
0.0
Places
Net
Rascal-OC
Phaturatst
CoCO
ImageNet-224
Figure F.10: Layer-wise distribution of semantic concepts across six high-level categories proposed in [53] on CLIP ViT-B/16. Each curve shows the average proportion of concept words associated with given category that are matched in each transformer layer, evaluated independently for each dataset.
G Understanding the Role of Intermediate Layers
Empirical Validation of Entropy-based Selection. Our untropy-based selection criterion is empirically validated in Figure G.II which shows the relationship between en- tropy and the average false positive rate (FPR @95) across multiple challenging OOD datasets (iNaturalist, Textures, SUN, and Places) using CLIP ViT-B/32. Each point cor- responds to a specific combination of intermediate layers.
11.5
Anerage g90
0.40
0.35
0:30
Two distinct clusters emerge: low-entropy combinations achieving low FPR @95, and high-entropy combinations approaching baseline performance. A consistent positive correlation between entropy and F is observed across architectures, suggesting that entropy can be ex- ploited to unsupervisedly select promising layer combina- tions for OOD detection.
1:00
1.04
Entropy
Seloction
1od Selection
000
2500
Dare
Importantly, we find that this relationship strength- ens when restricting the maximum combination length. Shorter combinations yield more distinet clustering toward low entropy and low FPR @95, while longer combinations introduce redundancy and noise, weakening the discrim- inative signal.
Scan
Entropy vs. @95 for different layer combinations, highlight- ing good and bad selections. Bottom: ID and OOD score distributions for repre- sentative examples.
Figure G.11:
26

---

Effect of the size of the combinations in relation between average FPR and entropy. We observe that constraining the maximum combination length strengthens the relationship between entropy and detection performance. Specifically, shorter combinations yield more coherent clustering toward low entropy and low FPR @95, as they are less likely to include noisy or redundant layers. In contrast, longer combinations often aggregate conflicting or less informative signals, diluting discriminative power and degrading OOD separation.
This trend suggests that limiting combination length acts as a natural form of regularization, reducing fusion noise and producing more robust, confidently separated layer subsets. These findings reinforce entropy as principled selection criterion: when applied over constrained layer sets, it consistently identifies combinations that maximize zero-shot OOD detection performance. Further supporting evidence is provided in Section 6
(a) VTT- Entropy AVG FPR across varying combination lengths. Lower entropy correlates with improved OOD detection.
(b) ViT-B/16: Similar positive correlation between entropy and FPR, validating rohustness of entropy- hased selection.
Figure G.12: Empirical validation of entropy-base based selection for OOD detection. Across both ViT-B/32 and ViT-B/16, lower entropy values correspond to lower false positive rates (FPR). Shorter combinations reduce noise, yielding more confident and effective selections.
H
Why Intermediate Layers Help: A Theoretical View
Confidence--basee scoring across depth. Let E X' an input sample and let (1)(x) Rd denote the visual feature at layer & E L of the pretrained encoder. For pretrained classification models, we define Sn, max Softmax, (h())) (o!") (("((x)))) (x))) (MSP score). For contrastive vision-language models like CLIP, we follow the procedure in Section 2 applying the pretrained image-to-text projection p to layer and define S,, maxyey (oftmax,((ossppll (c))((((()(()))) (x)), (()))/T) (MCM score). These scores form the confidence tensor S E RRX where each element Sn,e is the scalar OOD score for instance n at layer E.
Motivation for fusion. For subset of layers L C (I...., L), define the fused score as:
5, = C Σ Sn,t,
REC
where S., represents the average confidence score across selected layers for instance (e.g the set of layers {1, 2, 12)). Fusion strategies aim to reduce noise and increase robustness by integrating complementary representations across depth.
Assumptions for effective fusion.
We propose the following sufficient conditions for the effective-
ness of intermediate- fusion:
A1 Semantic progression: Deeper layers encode more task-aligned representations; shallow layers remain task-agnostic.
A2 Representation diversity: Layer-level scores {Sm.eheec are not perfectly correlated, i.e., [ar[S,] 0.
27

---

A3 Prediction consistency: Scores fluctuate more for OOD than ID samples:
Varm(Sn,t) Varoon[Sn.e]-
Theoretical insight. Under mild cosuumptions-bbunddee scores, non-positive pairwise covariances (empirically supported in Fig. B), and conditions A1-A3-w can show that fusion reduces variance
in OOD scores and increases the separation between ID and OOD distributions in expectation.
Proposition 1 (Fusion reduces variance and amplifies ID-OOD gap). Let S(x) denote the average confidence score across subset of layers L S (1..... L}. Then:
Varood [x()]] <
Σ of.
Ex [8())] > max (Exm [S Ex (),,e). Ex [x,[S(())
[S(x)]
CC2
FEC
FEC
Thus, fusion reduces score variance on OOD data and increases the expected confidence gap between ID and OOD samples samples-yielding improved detection relative any individual layer.
Discussion
The theoretical conditions outlined above well-aligned with our empirical observations. CLIP models exhibit high inter-layer representational diversity (as indicated by low SVCCA similarity) and stable prediction behavior (as reflected by low top-1 disagreement), satisfying both A2 and A3. As shown in Figure 7 these properties translate into consistent gains from multi-layer fusion. In contrast, models such MAE and DINOv2-characterl entropy variation and unstable intermediate resulting diminished or even detrimental effects when layers are combined.
by limited violate these assumptions,
I Random Projection for Resnet Architecture
To harmonize feature map dimensions across architectures and layers-and to enable alignment with a shared text embedding space-we we apply a lightweight two-stage projection. Given feature tensor F R.x. HxW, we first apply adaptive average pooling resize the spatial dimensions fixed target shape (Hugrtt W warget):
F pooled = AdaptiveAvyPPol22DC. (Hurgets W Wirger))
Next, a 1 convolution projects the feature channels Curgets which matches the dimensionality of the text features:
Fproj Conv2D (F pooled: Craget)
This operation ensures that all visual representations, irrespective their originating layer or ar- chitecture, are transformed into common shape and embedding space compatible with the text modality. We interpret the x convolution as random linear projection over the channel dimension. Unlike spatial convolutions, this operation applies independent linear transformation each spatial location, akin to fully connected layer applied per pixel. When randomly initialized (e.g., using He or Xavier initialization), such projections approximately preserve the geometry of the input space with high probability, as guaranteed by the Johnson-Lindenstran lemma
[T] 36 45). This allows
the projected visual features remain discriminative while enabling seamless fusion or comparison with the corresponding text representations.
J Different architectures and prompt-based methods
To test portability across backbones and prompting strategies (Table]), we find intermediatc-layer fusion remains effective beyond the default setting: with MCM on RN50x4 it yields AUROC +0.85; FPR 3.55, while ViT-L/14 improves more modestly. Prompted ViT-B/16 also benefits, with NegLabel (AUROC +0.84; FPR - 1161) and CSP (AUROC +0.98; FPR - 1.99). These trends mirror our layer-agreement analysis: heterogeneous ResNet stages fusion reduce variance and suppress spurious peaks with MCM. Over this, this results highlight that the random projections I are able to improve this performance and specific in this network the intermediate layers able to improve the results in every ood dataset.
28

---

Whereas ViT-L/14's higher inter-layer redundancy and sharper softmax (with T = 0.01) leave less headroom. Dataset context matters: the largest FPR drops appear on scene-centric SUN/Places where early and mid layers retain layou/bacckrround cues; gains are smaller on DTD and iNaturalist, where final embeddings already align with object-centric semantics. This difficult can be generated by the deepness of the network that generate mordissimilar behavior acrees the extra layer that has in comparison with vit-B/16 ad vit=b/32 and the entropy in its output as we show in Sec 3
Prompt-based methods such as CSP 77) and NegLabel + typically rely on external resources (e.g., WordNet hierarchies) or handcrafted prompts and use method -specffic scoring. Our approach operates at the representation level by fusing intermediate layers, making it orthogonal to prompt design and broadly compatible with existing scoring rules. Applied to NegLabel, fusion yields consistent improvements across most datasets. CSP, although more memory- inteensvee often benefits even more, likely because its synthetic negatives enlarge the ID vs OOD margin and better exploit complementary mid-layer cues.
Table 6: Training-free OOD detection with additional backbones and prompt-based methods (ImageNet-IK as ID). Base: without intermediate fusion; +Fusion: our training-fret free intermediate fusion. Best per column in bold.
Method
Backbone
iNaturalist
SUN
Places
Texture
Averuge
FPR AUC
FPR|
AUC
FPR.
AUC
FPR
AUC
FPR|
AUCA
Extra backbones (MCM)
MCM (Last Layer)
RN50<4
44.00
91.59
35.13
92.83
44.30
89.38
57.22
85.99
45.16
89.95
MCM (Intermediate)
RN50<4
41.79
92.00
31.83
93.48
40.46
90.36
$2.38
87.37
41.61
90.80
MCM (Last Layer)
VIT-L/14
24.91
95.44
29.58
93.98
35,51
92.02
58.65
85.
19
37.16
91.66
MCM (Intermediate)
VIT-L/14
25.15
95.45
29.21
94.07
35.02
92.13
58.99
85.14
37.09
91.70
Prompt methods (composition with VIT-B/16)
NegLabel
VIT-B/16
1.91
99.49
20.53
95.49
35.59
91.64
43.56
90,22
25.40
94.21
NegLabel Int. Layers VIT-B/16
2.35
99.36
21.10
95.38
32.31
93.35
39.38
92,10
23.79
95.05
VIT-/116
1.54
99.60
13.66
96.66
29.32
92.90
25.52
93.86
17.51
95.76
CSP Layers
B/T
96.74
K
Orthogonality to Scoring Rules
Consider logits z E RK where K denotes the number of classes. Following our experimental setup (Section 3), we analyze OOD detection methods: MaxLogit (max, 24), MCM (max, Pt, with
exp(s;)
Energy (-log Σ exp(x;)) 44). Entropy (- (- Pt lo), and Variance
K
P: = Σ, exp(s;)
K E (p -))))
The differential improvements observed across methods reflect their computational characteristics. MaxLogit [25] and MCM 47 rely directly on peak logit or probability values, making them particularly sensitive to intermediate layer representations, where the separation between ID and OOD peaks, A peak = E E[max; 2/0 [D] E[max; (e")(OO)) ), is typically larger due to reduced saturation effects. The observed improvement for MCM specifically benefits from the softmax transformation applied over logits, which amplifies probability differences and enhances discriminative power between ID and OOD samples. This theoretical insight is empirically validated (Table K). showing substantial intermediate-layer improvements for MCM (average + +102%, notably + 16.6% for ViT- B/32) and consistent gains for Maxl (+4.3%).
In contrast, Energy 44 Entropy, and Variance methods aggregate global distributional information through log-sum-ex, operations and probability averaging, making them inherently less sensitive to peak-specific variations and thus displaying more modest improvements (+0.4%, +6.2%, and +6.1%, respectively). Architectural differences further modulate these improvements. Our analysis reveals that CLIP models demonstrate broad prediction agreement across layers (Figure promoting stable predictions conducive to effective fusion, whereas supervised models show independent prediction evolution across layers. These architectural patterns, combined with the representational diversity captured in our entropy calibration analysis (Figure M.)8), explain the variance in observed improvements, with VIT-B/32 achieving the highest gains due to optimal balance between inter-layer diversity and prediction consistency.
29

---

Table 7: Comprehensive OOD detection results using intermediate vs last layer features. We report the false positive rate at 95% TPR (FPR95, 4) and AUROC (f) across four OOD datasets: SUN, Places365, DTD, and iNaturalist. Each method is evaluated using ResNet, ViT-B/32, or ViT-B/16 backbones with both last and intermediate layer features. Best results per backbone are highlighted
in bold.
Method
Layer
Backbone
SUN
Places365
DTD
INaturalist FPR. AUC1
Average
FPR
AUC
FPR.
AUCT
FPR.
AUCT
FPR AUCT
ResNet Results
Eniropy
Last
ResNet
55.84
88.04
72.42
79.31
73.32
78.78
83.13
71.80
71.18
79.48
Entropy
Intermediate
ResNet
53.69
88.06
72.91
79.14
73.81
78.74
82.19
72.46
71.15
79.60
Energy
Last
ResNet
98.63
39.20
94.56
52.88
99.70
20.35
98.83
43.01
97.93
38.86
Energy
Intermediate
ResNet
98.49
39.65
94.12
53.43
99.68
19.79
98.57
44.53
97.72
39.35
Variance
Last
ResNet
55.09
88.19
72.03
79.51
73.14
78.84
82.90
72.15
70.79
79.67
Variance
Intermediate
ResNet
55.01
88.22
72.49
79.35
73.53
78.80
81.88
72.80
70.73
79.79
MaxL.ogit
89.11
52.05
71.19
65.44
88.23
MaxLogit Intermediate
ResNet
54.71
90.08
46.78
72.89
61.00
89.19
62.81
MCM
92.83
44.30
44.00
91.59
Intermediate
92.00
VTT-R/32 Results
Entrrpy
Last
VIT-B/32
59.13
84.38
67.38
79.81
76.72
72.08
73.34
79.29
69.14 14
78.89
Eniropy
Intermediate
ViT.B/32
65.09
80.03
84.96
68.69
66.86
80.54
49.72
90.68
66.66
79.98
Energy
Last
V/T.
97.77
42.25
95.79
53.02
98.92
26.50
99.79
37.98
98.07
39.94
Energy
Intermediate
VIT-B/32
97.64
45.85
96.07
57.90
97.29
40.18
99.88
32.71
97.72
44.16
Variance
Last
VII-B/32
58.78
84.60
67.10
80.06
76.63
72.18
73.11
79.58
68.91
79.11
Variance
Intermediate
VIT-B/32
64.92
80.21
84.84
68 94
66.91
80.64
49.67
90.78
66.59
80.14
MaxLogit
Last
VIT-B/32
64.98
86.78
56.36
88.02
87.00
70.45
65.22
87.58
68 39
83.21
MaxLogii Intermediate
VIT-B/32
69.30
83.70
67.89
83.95
80.25
72.03
54,82
89.56
68.06
82.31
MCM
Last
VIT-R/32
40.99
91.56
46.71
89.25
60.90
85.03
33.85
93.62
45.61
89.87
MCM
Intermediate
VIT-B/32
28.98
93.37
35.69
91.61
39.36
91.07
12.02
97.64
29.01
93.42
VYT-B/16 Results
Entropy
VIT.B/16
68.00
81.36
74.90
76.35
79.08
72.37
87.00
65.05
77.24
73.78
Entropy
Intermediate
VIT-R/16
66.86
78.25
81.86
72.64
36.33
92.12
59.78
88.25
61.21
82.81
Energy
VIT-B/16
98.68
37.66
97.21
45.39
99.22
24.68
99.77
36.95
98.72
36.17
Energy
Intermediate
VIT-B/16
98.49
41,02
99.48
33.19
94.52
38.66
99.87
41.52
98.09
38.60
Variance
Last
ViT-B/16
67.54
81.61
74.47
76.62
78.97
72.47
86.67
65.62
76.91
74.08
Variance
Intermediate
VIT-B/16
66.73
78.40
81.51
72.98
36.21
92.13
59.21
88.45
60.91
82.99
Maxl.ggi
Last
VIT-B/16
64.45
87.43
60.40
87.07
85.48
72.56
60.56
89.64
67.72
84.17
MaxLogit
Intermediate
ViT-B/16
70,32
85.04
69.05
84.15
50.51
88.18
46.61
91.97
59.12
87.33
MCM
Last
VIT.
37.41
92.57
43.67
89.96
57.34
86.18
30.67
94.63
42.27
90.83
MCM
Intermediate
VIT.//16
45.58
89.69
35.71
92.72
25.51
94,84
15.98
96.90
30.70
93.54
Formally, for any such score (s), increase in A, = E (2uee) ID] E (2 (usee)) OOD]
improves thresholded decisions for that (s), indicating robustness that is orthogonal to the choice of scoring rule.
Discussion
Fusing intermediate representations offers a training-free, plug-and-pla, mechanism that consistently strengthens OOD robustness across architectures, datasets, and prompt- ing regimes while adding only modest compute and memory cost. The effect arises from complementary early- and mid-layer signals that widen the separation between famil- iar and unfamiliar inputs and stabilize predictions. Because the mechanism acts at the representation level, integrates without changing existing pipelines. Gains can vary with architectural traits; models whose information is concentrated in late layers may show smaller improvements. Future directions include adaptive per-modes layer selection, dynamic fu- sion policies inference time, and exploring token- or region-level cues to further enhance reliability open-world settings.
30

---

L Computational Cost Analysis of Intermediate Layer Usage
We evaluate the computational implications of incorporating intermediate layer features across three CLIP architectures: ResNet-50x4, ViT-B/16, and ViT-B/32. The analysis spans batch sizes from 1 to 8 and considers peak GPU memory, per-image latency, total inference time, and throughput (images processed per second). As shown in Figures L.13] L.15] and [..44] the additional operations required for intermediate feature extraction introduce limited overhead. Across all configurations, memory and latency values remain within practical deployment limits, supporting the method's s suitability for real-world use.
ViT-B/32 exhibits the most efficient resource profile (Figure (oggureL.14. L.14 with memory overhead ranging from 1.7% to 7.1% (average: 3.4%) across batch sizes and average latency changes of just 3.3%. Notably, ViT-B/32 demonstrates latency reductions at batch sizes 4 and 8 (-1.1% and - -1.0%, respectively), indicating improved hardware utilization. ViT-B/16 shows higher memory overhead (average: 10.2%, up to 20.6% at batch size 8). yet achieves an overall latency "reduction* of 5.7% on average, with throughput impact of only -0.7% (Figure L.15). Both ViT architectures exhibit counterintuitive latency improvements at larger batch sizes, suggesting better parallelization and GPU kernel efficiency when processing intermediate features. ResNet-50x4 exhibits more conventional scaling behavior (Figure 13)). with stable memory overhead averaging 12.4% and latency increases ranging from 5.1% to 14.7% (mean 11). Throughput decreases by an average of 10.7%, primarily due to the use of external random projections align the dimensionality of intermediate-layee features.
Importantly, absolute performance values remain deploylent-friendly in all cases. Peak memory usage stays under 0.7 GB even at batch size 8, and per-image latency remains compatible with both real-time and thgh-throughput applications (maintaining 8-60 images/second depending on architec- ture and batch size). These results confirm that the use of intermediate features is computationally viable across architectures, offering substantial detection performance improvemens--up on far-OOD benchmarks (see Tables (5. [nd and consistent gains across diverse scoring functions (Table K) with minimal computational cost. Full breakdowns of the quantitative results are reported in Tables
and L
Real parfernoccs Ovethead dis
Avrago Referneecs misstt
Mimory uhage Scaling
14.7%
Luc3
Procssrig Themagrout
Pepingge Pressiii
140
10
Figure 13: Computational cost comparison for CLIP RNS0x4 using intermediate layers versus last-layer only inference.
31

---

Table 8: ResNet-500x Performance Analysis: Last Layer vs All Layers. We report memory usage (GB), latency per image (ms), and overhead percentages (%, 1 indicates increase) for CLIP ResNet-50x4 architecture, Results show mean + standard deviation across 5 measurement runs per
configuration.
Batch
Memory Usage (GB)
Latency per Image (ms)
Overhead (%)
Runs
Size
Last Layer
All I Layers
Tast Layer
All Layers
Memory!
Latencyf
(T
0,41#0.00
0,43=0.00
152.20=33,11 17 160.0*+11.58
+7.0
+5.1
5/5
0.41#0.00
0,4420.00
67.72=0.56
77.70=1.01
+8.5
+14.7
5/5
0.41=0.00
0.46+0.00
33.47=0.36
38.23+0.86
+11.9
+14.2
5/5
0.41+0.00
0.50=0.00
17.41=0.54
19.23+0.33
+22.3
+10.5
5/5
Average Overhead: Memory +12.4%, Latency +11.1%
Aad Feformance Oraffead by Daich Size
Assrago Nerarrrane maad
Nemory Uaage Scaling
Meney Aveiial
125
115
14C
0.59
E37
Pracesing Threushou!
Wnteraoo Lelong
Pelliag:: Pocessin Thee
16-
350
Figure 14: Computational cost comparison for CLIP ViT-B/32 using intermediate layers versus last-layer only inference.
Table 9: VIT-B/32 Performance Analysis: Last Layer vs All Layers. We report memory usage (GB), latency per image (ms), and overhead percentages (%, indicates increase, J. indicates decrease) for CLIP ViT-B/32 architecture. Results show mean = standard deviation across measurement runs per configuration.
Batch
Memory Usage (GB)
Latency per Image (ms)
Overhead (%)
Runs (L/A)
Size
Last Layer
All Layers
Last Layer
All Layers
Memory?
Latency!
0,5740.00
0.5820.00
454.64:25.7 75 452.42+14 36
+1.7
-0.5
5/5
2
0,5740.00
0.5820.00
227. 227.4919.11
263.49579.88
+1.8
+15.8
5/5
0,5740.00
0.59:0.00
109 2243.74 74
107.99t 16
+2.8
-1.1
5/5
0.5840.00
0.6240.00
54,47+1.30
53.94&1.11
+7.1
-1.0
5/5
Average Overhead: Memory +3.4%, Latency +3.3%
32

---

10.4%
6.07
Faary
funtral
655
Meler
Thoughpst
coreeciinn leens
prew
200
Figure L.15: Computational cost comparison for CLIP ViT-B/16 using intermediate layers versus last-layer only inference.
Table 10: VIT-B/16 Performance Analysis: Last I Layer vs All Layers. We report memory usage (GB), latency per image (ms), and overhead percentages (%, 1 indicates increase, + indicates decrease) for CLIP VIT-B/16 architecture. Results show mean + standard deviation across measurement runs per configuration.
Batch
Memory Usage (GB)
Latency per Image (ms)
Overhend (%)
Runs
Size
Last Layer
All Layers
Last Layer
All Layers
Memoryt
Larency"
(L/A)
0,5720.00
0.5840.00
151.84220.45
$20.31=162.03
+2.6
+15.2
515
2
0,57+0.00
0.61+0.00
221 86+5.01
213.37++.67
+7.2
-3.8
515
4
0.57=0.00
0.6340.00
133.13:39.90 96
110.89+2.09
+10.4
-16.7
S/5
8
0.57=0.00
0.69:0.00
67.04+20.14
55.2840.94
+20.6
-17.5
S15
Average Overhead: Memory +10.2%. Latency -5.7%
33

---

M Additional Figures
Effect of Layer Combination Length on OOD Detection Across Architectures
1.0
0.9
Average @ 95%
0.8
0.7
0.6
0.5
5
10
25
Number Layers Combination
Architecture
CLIP RN5D
CLIP VIT-L/14
(Supervised)
SiGLip2 8-16
CLIP VIT-B/32
MAE VIT-B
DINO v2 VIT-B
5/GLip2-8-32
CLIP VIT-B/16
MoCo v3 VIT-B
PEV1-816-224
Figure M.16: Impact of combination length on OOD detection performance across architectures. The plot reports average FPR @95% function of the number fused layers. Combining a moderate number of layers typically improves performance, while longer combinations may degrade it.
AUROC
0.5
0.0
1.0
FPR@95
0.8
0.6
0.4
1
2
3
4
6
8
9
10
11
Layer
CLPP PEv1-816-224
VIT-B-16
CLIP ViT-L-14
mae
-9-
CLP RN50x4
CLIP VIT-B-32
dinov2
moco-v3
Figure M.17: Layer-wise OOD detection performance across architectures. Most architectures exhibit their best performance near the final layer, while early layers generally under-perform.
34

---

Non-Contrastive Models
Contrastive Models
Average Entropy
Average
Layer
Layer CLIP VIT.144
DINOV2
McCo V3
PEV1 816 224
CLIP VIT 8/26
(Superissed
Figure M.18: Average entropy across transformer layers for various vision models. We compare softmax entropy across layers. Supervised models exhibit low entropy in later layers, reflecting overconfident predictions, while self-supervised and contrastive models (c.g.. MAE, CLIP) show greater entropy variation across depth. The inset highlights subtle differences the high-entropy regime across CLIP variants, SiGLip2, and PE.
MAE
MDCD-y3
VIT
Dino-V2
CUP VIHH222
CLIP WI-8/20
CUP VIt-U24
Figure M.19: Layer-wise Censen-Shanoor Divergence (JSD) between predicted class probabilities across different architectures. Each heatmap corresponds specific model and illustrates the pairwise JSD between layers. Color intensity reflects the degree of divergence, with brighter values indicating greater dissimilarity in output distributions. Note that each subplot uses its own color scale to emphasize internal variation within each architecture.
VALL windury DON UNIFGEN DICAA COWAARO
2000
Layer Citance AcTRECTton
Greek
Figure M.20: SVCCA similarity vs. layer distance across ViT architectures. Box plots show SVCCA distributions per A; overlaid lines denote mean similarity.
35


Reviews:
[{"review_id": "v7pTZb7b5M", "reviewer": "Reviewer_WaeU", "summary": "The authors propose a training-free strategy for OOD detection through using intermediate representations. They find that intermediate representations encode rich signals and propose entropy-based scoring for OOD detection which outperforms contemporary methods across various OOD detection benchmarks. The authors perform detailed experiments across ImageNet-1K and Pascal-VOC using CLIP with various backbones, and ablation studies over the hyperparameter and design choices of their method.", "questions": "How does your method perform across various architectures (& target modalities, such as perhaps text)?\n\nI don't think this is an easy question to answer nor expect the authors to resolve it within the response period - but it is one that naturally comes up and would improve your work, if answered.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The paper is very well written, with takeaways clearly highlighted and excellent visualizations.\n- The experimental results are convincing\n- The authors perform various ablation studies to analyse the robustness of their approach\n\nWeaknesses\n- The main weakness which if resolved, would take this paper a step beyond is evaluation on more models/architectures and possibly datasets", "was_revised": true, "final_justification": "The paper is well written and tackles a relevant problem. My main perceived weakness was the lack of evaluation on text processing tasks, which would show that the method works across modalities. This weakness remains as it is (somewhat understandably) out of scope of the authors work. I consider the paper well written and rounded and worthy of publication.", "rebuttal": "We thank the reviewer for their thoughtful and encouraging feedback. We are glad the strengths of our work, including its clarity, experimental depth, and visualizations, resonated with you. In response to your main suggestion, we will expanded our evaluation and provide a more detailed discussion on performance across architectures and modalities. Below, we address your questions and comments point by point.\n\n### **Q: Evaluation on more models/architectures**\n\nWe do demonstrate that the method performs well **across a diverse range of vision architectures**. Our systematic evaluation covers 7 vision architectures of diverse types: supervised (ViT), self-supervised (MAE, MoCo-v3, DINOv2), and contrastive (CLIP, SiGLip-v2, Perception Encoder). This allows discovering consistent trends (see next answer). We propose to **include additional results from larger models**, namely CLIP ViT-L/14 and PEv2-L/14 in the final version of the paper.\n\n\n### **Q: How does the method perform across architectures?**\n\nThe systematic evaluation across architectures shows that contrastive models (e.g. CLIP) consistently benefit from intermediate-layer fusion due to their high inter-layer diversity and stable prediction profiles. In contrast, supervised and self-supervised models often exhibit redundant or unstable representations across layers, which limits the benefits of fusion. These differences are driven both by the training objective and the dataset scale, which together influence cross-layer complementarity.\n\n### **Q: Other modalities**\n\nThe extension to other modalities such as text would be very interesting because similar layer-wise computations are likely at play. Indeed, prior work like Layer-by-Layer [1] and DoLa [2] shows that intermediate representations can improve performance and reduce hallucinations in LLMs. Some modality-specific adaptations would be required, for example replacing patch-based fusion with token-level strategies. We propose to add relevant discussion and suggestions in the \"Future Work\" section. \n\n[1] Skean et al. *Layer by Layer: Uncovering Hidden Representations in Language Models*\n\n[2] Chuang et al. *DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models*"}, {"review_id": "7UWPdQXj0L", "reviewer": "Reviewer_Zjg4", "summary": "This paper presents a study on the impact of intermediate-layer representations in CLIP to OoD detection and show that certain intermediate layers can help improve the performance. Following, an entropy-based criterion is proposed to automatically select informative layers, and then fuse based on Maximum Concept Matching (MCM). Experimental setups are comprehensive.", "questions": "1. In line 233-234, the authors claim the performance of “24.63% / 93.93 on Pascal-VOC.” However, I can not find these performance figures in the table. Please let me know if I overlooked any details.\n2. Can the entropy-based layer selection be extended to dynamically select layers per input sample, rather than per dataset?\n3. How does this method generalize to other CNN-based or deeper vision encoder architectures?\n4. What is the computation overhead of the proposed method?\n5. How does this method perform with a real-world applications/dataset (e.g., medical imaging, autonomous driving)\n6. Have you tried other statistical metrics for layer selection beyond entropy?", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* First study on the impact of intermediate-layer representations in CLIP to OoD detection\n* The method is training-free, requiring no fine-tuning, labels, or OoD data. \n* The evaluation is extensive including both far-OoD (ImageNet-1K, Pascal-VOC) and near-OoD (NINCO, SSB-Hard) benchmarks, multiple baselines such as MCM, GL-MCM, and SeTAR.\n* The design is strongly motivated by interesting empirical findings\n\nWeaknesses:\n* The findings of the impact of intermediate-layer to OoD Detection is limited to CLIP or training-free OoD Detection Methods such as MCM\n* While the improvement is significant compared to the baseline MCM, the advantage of the proposed method is minimal (or even lower performance) than SOTA training-free methods (See Tab. 1)\n* Since this study related to model’s representations, evaluation on ViT-B16/32 might be limited. Will this method generalize to other CNN-based or deeper vision encoder architectures?\n* I understand that the experimental setups are standard for training-free OoD detection. However, since the proposed method requires no training, it would be more interesting to evaluate on a critical real-world applications (e.g., medical imaging, autonomous driving) rather than ImageNet-1K or Pascal-VOC\n* Minor: Technically, the proposed method requires multiple MCM inferences at different layers, the computation overhead is not discussed in the paper.", "was_revised": true, "final_justification": "The rebuttal well addresses and clarifies my concerns.\n\nParticularly, the additional results on Per-Instance Dynamic Layer Selection are interesting.\n\nTherefore, I increase the final rating that this paper meets the acceptance standard.", "rebuttal": "We sincerely thank the reviewer for their thoughtful and encouraging feedback. We are pleased to hear that the writing, motivation, and experimental rigor of our work were appreciated. In response to the reviewer’s suggestions, including the evaluation on more architectures, per-instance selection, computational overhead, and comparisons to additional metrics, we conducted new experiments and analyses, which we detail below. Each point raised has been carefully addressed in the revised response.\n\n\n### **Q1: Clarification on Pascal-VOC Results**\n\nThank you for pointing this out. This was a typographical error in the text. The **correct performance values are those reported in Table 5.1**, which reflect the actual evaluation. We will update the manuscript to resolve this inconsistency in the final version.\n\n\n### **Q2: Per-Instance Dynamic Layer Selection**\n\nWe thank the reviewer for this valuable suggestion. While we did not implement per-instance dynamic selection in our current method, we conducted an **oracle-based analysis** to assess its potential. Specifically, for each architecture and dataset, we identified the **best-performing combination of layers per input sample**, allowing the selected layers to vary dynamically across instances.\n\nThis yields the following **lower-bound FPR\\@95** results:\n\n* **ViT-B/32:** iNat 0.0575, DTD 0.2472, SUN 0.2122, Places 0.2845 → **Avg FPR\\@95: 0.2004**\n* **ViT-B/16:** iNat 0.0693, DTD 0.1397, SUN 0.2305, Places 0.2325 → **Avg FPR\\@95: 0.1680**\n\nThese results indicate that **per-sample dynamic layer selection could lead to further improvements**, particularly on challenging datasets. While our current method performs entropy-based selection at the dataset level, this oracle highlights a promising direction for future work on **test-time adaptive strategies**.\n\n\n### **Q3: Generalization to CNNs and Deeper Vision Architectures**\n\n**Following the reviewer’s suggestion, we extended our evaluation to include both convolutional and deeper transformer-based vision architectures**. Specifically, we tested our method on **CLIP RN50x4** and **CLIP ViT-L/14**. The results below confirm that our approach generalizes well across different CLIP architectural types.\n\n**CLIP RN50x4 (ResNet)**\n| Method             | FPR Avg ↓ | AUROC Avg ↑ | FPR SUN | AUROC SUN | FPR Places | AUROC Places | FPR DTD | AUROC DTD | FPR iNat | AUROC iNat |\n| ------------------ | --------- | ----------- | ------- | --------- | ---------- | ------------ | ------- | --------- | -------- | ---------- |\n| MCM (Last Layer)   | 45.16     | 89.95       | 35.13   | 92.83     | 44.30      | 89.38        | 57.22   | 85.99     | 44.00    | 91.59      |\n| MCM (Intermediate) | **41.61** | **90.80**   | 31.83   | 93.48     | 40.46      | 90.36        | 52.38   | 87.37     | 41.79    | 92.00      |\n\n**CLIP ViT-L/14 (24-layer Transformer)**\n| Method             | FPR Avg ↓ | AUROC Avg ↑ | FPR SUN | AUROC SUN | FPR Places | AUROC Places | FPR DTD | AUROC DTD | FPR iNat | AUROC iNat |\n| ------------------ | --------- | ----------- | ------- | --------- | ---------- | ------------ | ------- | --------- | -------- | ---------- |\n| MCM (Last Layer)   | 37.16     | 91.66       | 29.58   | 93.98     | 35.51      | 92.02        | 58.65   | 85.19     | 24.91    | 95.44      |\n| MCM (Intermediate) | **37.09** | **91.70**   | 29.21   | 94.07     | 35.02      | 92.13        | 58.99   | 85.14     | 25.15    | 95.45      |\n\n#### **A: Model-Specific Behavior**\n\nWhile **ViT-L/14 shows smaller gains**, this aligns with our analysis in Section 3. Figures 2–4 and lines 142–146 show that ViT-L/14 exhibits **greater redundancy** and **flatter entropy profiles** across layers, limiting the benefits of fusion. This is further influenced by its **sharper softmax distribution**, which required a lower temperature setting (0.01). In contrast, **ResNet-based models like RN50x4 benefit more** from fusion due to **higher inter-layer diversity** and **smoother confidence dynamics**.\n\n### **Q4: Computational Overhead**\n\nAs requested, we performed additional experiments to quantify the computational overhead of our method. We evaluated inference time, memory usage, and throughput on ViT-B/32, comparing the use of the last layer versus our intermediate-layer fusion strategy. The results show that the use of intermediate layers introduces negligible overhead: memory usage increases by at most 0.04 GB, latency remains virtually unchanged, and throughput is preserved across all batch sizes.\n\n**ViT-B/32 | Last Layer vs Intermediate Layers**\n| Batch Size | Memory (GB) — Last | Memory (GB) — Ours | Latency (ms) — Last | Latency (ms) — Ours | Throughput (img/s) — Last | Throughput (img/s) — Ours |\n|------------|--------------------|--------------------|---------------------|---------------------|---------------------------|---------------------------|\n| 1          | 0.57               | 0.58               | 454.6 ± 25.8        | 452.4 ± 14.4        | 2.2 ± 0.1                 | 2.2 ± 0.1                 |\n| 2          | 0.57               | 0.58               | 227.5 ± 9.1         | 263.5 ± 79.8        | 4.4 ± 0.2                 | 4.1 ± 0.8                 |\n| 4          | 0.57               | 0.59               | 109.2 ± 3.7         | 108.0 ± 1.2         | 9.2 ± 0.3                 | 9.3 ± 0.1                 |\n| 8          | 0.58               | 0.62               | 54.5 ± 1.3          | 53.9 ± 1.1          | 18.4 ± 0.4                | 18.5 ± 0.4                |\n\n\n### **Q5: How does the proposed method perform when applied to real-world scenarios or domain-specific?**\n\nOur method could be well suited for real-world applications where inference efficiency, scalability, and reliability are essential, based on the computational results shown in the previous table. Unlike methods that require retraining or computationally expensive ensembles, our approach is lightweight and don't need a retraining.\n\nDespite this, we acknowledge that real-world scenarios often involve challenges such as long-tailed class distributions and subtle distribution shifts factors known to significantly impact OOD detection performance [1, 2, 3]. While our current study focuses on standard benchmarks, the proposed method is modular, lightweight, and inference-only, making it a promising candidate for extension to safety-critical domains. We consider this an important direction for future work.\n\n**References**\n[1] Miao et al. *Out-of-Distribution Detection in Long-Tailed Recognition with Calibrated Outlier Class Learning*\n[2] Wei et al. *EAT: Towards Long-Tailed Out-of-Distribution Detection*\n[3] Miao et al. *Long-Tailed Out-of-Distribution Detection via Normalized Outlier Distribution Adaptation*\n\n### **Q6: Have you tried other statistical metrics for layer selection beyond entropy?**\n\nYes, we explored several metrics for layer selection, as shown in Section 6 and Figure 6. These include entropy, kurtosis, Gini, JSD, and random selection. **Entropy consistently performs best**, selecting mid-to-deep layers (e.g., 6, 8, 11) that offer complementary information and result in the lowest FPR and highest AUROC. In contrast, other metrics often favor shallow or final layers, leading to weaker performance. We will include additional metrics and visualizations in the revised version to provide more detailed comparisons."}, {"review_id": "rWmIrxi4ZK", "reviewer": "Reviewer_rDhG", "summary": "This paper presents an empirical exploration into the value of leveraging intermediate layers when using zero-shot OOD detection methods like MCM. In particular, the authors finds that intermediate layers can contain rich signals that can be leveraged for improving OOD detection beyond the currently standard final-layer OOD detection methods. The proposed updated MCM method which fuses intermediate layers is OOD agnostic and can be directly leveraged during inference. Additionally, the authors provide extensive empirical evaluations across multiple common pre-trained architectures.", "questions": "Additional questions beyond weaknesses listed above:\n- The reviewer wonders if the authors have an intuitive understanding of why intermediate layers help particularly with near-OOD shifts.\n- Have the authors attempted to see if the selection of intermediate layers can be further improved with OOD data?\n- Any empirical evaluations on OOD detection tasks beyond image or on lower resolution CIFAR-based datasets?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The reviewer notes the following strengths and weaknesses.\n\nStrengths:\n- The paper presents a comprehensive empirical analysis on the value of intermediate layers when leveraging zero-shot OOD detection methods that is rarely explored.\n- The proposed methodology is simply to use and can be directly applicable given the zero-shot nature of MCM and entropy-based selection of intermediate layers.\n- The paper is well written and provides extensive empirical evaluations across multiple high resolution datasets and model architectures.\n\nWeaknesses:\n- It seems to the reviewer that OOD detection gains from intermediate layers is highly contingent on the underlying model, with CLIP-like models benefiting the most while others showing minimal benefits. This can limit the applicability of the methodology.\n- The proposed methodology is an simple extension of MCM, which limits the novelty of the underlying proposed method. \n- Much of the work, from the reviewer's perspective, is based on empirical evaluations and observations. Additional theoretical justifications may help improve the paper.", "was_revised": true, "final_justification": "This paper presents a thorough series of evaluations into the value of intermediate layers when performing OOD detection with MCM. The methodology is intuitive and readily applicable to real-world scenarios of OOD detections. Given the technic soundness and extensive evaluations, I will maintain my score to reflect my positive view on the paper's acceptance.", "rebuttal": "We thank Reviewer rDhG for the thoughtful and encouraging feedback. Below, we address each of the reviewer’s questions in detail. We elaborate on the intuition behind the effectiveness of intermediate layers for near-OOD shifts, discuss potential extensions involving OOD supervision, and clarify our evaluation choices and directions for future work.\n\n### **Q1: Why do intermediate layers help particularly with near-OOD shifts?**\n\n**Intermediate layers help in these cases because they capture diverse and complementary features that are often lost in the final layer**. As shown in Figure F.10, early layers focus on textures and materials, **middle layers on object parts and natural elements, while final layers emphasize high-level semantic concepts**. By fusing signals from intermediate layers, our method retains fine-grained information critical for distinguishing near-OOD samples, leading to improved performance on datasets such as NINCO, and SSB-Hard.\n\n[1] Oikarinen et al.: CLIP-Dissect: Automatic Description of Neuron Representations in Deep Vision Networks\n\n### **Q2: Can intermediate layer selection be further improved using OOD data?**\n\nThank you for the suggestion. While leveraging OOD data (e.g., Outlier Exposure \\[2]) can enhance robustness, it alters the problem setting by requiring access to OOD samples. In contrast, to explore the potential of OOD usage without supervision, we conducted an oracle-based analysis using per-sample dynamic layer selection. The results show clear room for improvement: average FPR\\@95 drops to **0.2004** for ViT-B/32 and **0.1680** for ViT-B/16. This highlights the potential of combining test-time adaptive strategies with OOD data as a promising future direction.\n\n[2] Hendrycks et al.: Deep Anomaly Detection with Outlier Exposure\n\n\n### **Q3: Any empirical evaluations on OOD detection tasks beyond image or on lower resolution CIFAR-based datasets?**\n\nYes. we also evaluate on Pascal VOC. Results are provided in Table 5.1. We do not use CIFAR-based benchmarks, as their low resolution images lack the semantic complexity needed for OOD detection with models like CLIP, and they are not widely used in recent CLIP-based OOD benchmarks."}, {"review_id": "mFZ6P3D7WG", "reviewer": "Reviewer_A4Qo", "summary": "This paper proposes to leverage the intermediate layers of pre-trained models can encode surprisingly rich and diverse signals for detecting distributional shifts. To exploit latent representation diversity across layers, this paper introduces an entropy\u0002based criterion to automatically identify layers offering the most complementary information in a training-free setting—without access to OOD data. Estensive experiments across seven vision backbones have been conducted to evaluate the propsoed method.", "questions": "See Weaknesses.", "limitations": "The idea of leveraging intermediate layers to enhance OOD detection has been proposed in several prior works and the reported results are not state-of-the-art.  More details can be seen in Weaknesses.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n1. The paper is well-structured and easy to follow.\n2. It has been experimentally validated on various pre-trained models and is supported by a comprehensive ablation study.\n\n**Weaknesses:**\n\n1. Limited Novelty: I have to say, the idea of leveraging intermediate layers to enhance OOD detection has been proposed in several prior works [1,2,3]. However, the authors have not discussed these related methods. Please elaborate on how the proposed approach differs from and improves upon these existing works.\n\n    [1] Lin, Ziqian, Sreya Dutta Roy, and Yixuan Li. \"Mood: Multi-level out-of-distribution detection.\" Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition. 2021.\n\n    [2] Guglielmo, Gianluca, and Marc Masana. \"Leveraging Intermediate Representations for Better Out-of-Distribution Detection.\" arXiv preprint arXiv:2502.12849 (2025).\n\n    [3] Fayyad, Jamil, et al. \"Exploiting classifier inter-level features for efficient out-of-distribution detection.\" Image and Vision Computing 142 (2024): 104897.\n\n2. Limited Scope of Scoring Methods: In Table 1 (based on CLIP ViT-B/16), why the discussion is limited to MCM scoring, do the conclusions generalize to other scoring strategies, such as energy score, MSP, and maxlogits? Please consider including results or discussion on these alternative scoring methods.\n\n3. In Line 102, the paper mentions that a fixed random linear projection is used to address layer-wise dimensionality mismatches. The authors further claim in Appendix I that “such projections approximately preserve the geometry of the input space.” Could the authors provide experimental evidence to support this claim?\n\n4. Does the use of intermediate layer features introduce additional GPU memory consumption? Please provide a comparison of the computational cost (e.g., memory usage, inference time) of the proposed method versus existing baselines.\n\n5. The reported results are not state-of-the-art. The previous method, such as CSP [4],NegLabel [5],  Clipscope[6], have achieved much better performance comapred to the your results in Table 1.\n\n    [4] Chen, Mengyuan, Junyu Gao, and Changsheng Xu. \"Conjugated semantic pool improves ood detection with pre-trained vision-language models.\" Advances in Neural Information Processing Systems 37 (2024): 82560-82593.\n\n    [5] Jiang, Xue, et al. \"Negative label guided ood detection with pretrained vision-language models.\" arXiv preprint arXiv:2403.20078 (2024).\n\n    [6] Fu, Hao, et al. \"Clipscope: Enhancing zero-shot ood detection with bayesian scoring.\" arXiv preprint arXiv:2405.14737 (2024).\n\n6. In Figure 6, why the layer selection strategies based on kurtosis, standard deviation, Gini coefficient, Jensen–Shannon (JSD) divergence peform even worse than random selection. Could the authors provide insights to explain this phenomenon?", "was_revised": true, "final_justification": "The authors’ response effectively addressed my concerns, so I have decided to raise my score to \"borderline accept\".", "rebuttal": "We thank the reviewer for their feedback. We have carefully considered each suggestion and incorporated new analyses, comparisons, and clarifications to address the concerns. We hope these additions better convey the motivation, novelty, and practical value of our work.\n\n### **Q1: Relation to Prior Work Using Intermediate Representations**\n\nWhile prior work [1–3] has explored intermediate layers for OOD detection, our method differs in both scope and design. Previous approaches focus on supervised models, require labeled ID data, auxiliar networks and select a single best layer using trained classifiers or regularization. In contrast, our method is fully training-free, uses no ID labels, and operates directly on pretrained CLIP models without fine-tuning. We fuse multiple intermediate layers instead of selecting just one, capturing complementary OOD signals and achieving consistent gains across datasets and architectures. Unlike prior work limited to CNNs, we conduct a systematic analysis across seven models with diverse training objectives, including contrastive and self-supervised vision-language models like CLIP, SiGLip-v2, and Perception Encoder. Our study reveals new insights into the role of intermediate representations for OOD detection and is orthogonal to scoring functions, making it complementary to existing methods (see Questions 2 and 5).\n\n\n### **Q2: Generalization to Other Scoring Methods**\n\nWe thank the reviewer for this suggestion. Based on it, **we conducted additional evaluations using alternative scoring functions**. While our main focus is on **MCM** due to its strong and stable performance, our method is **compatible with a wide range of scoring strategies**. We will include the detailed results in the revised version.\n\n\n**ResNet (RN50x4)**\n\n| Method | Last Layer (FPR / AUROC) | Intermediate Fusion (FPR / AUROC) |\n|---|---|---|\n| MCM | 45.2 / 89.9 | 41.6 / 90.8 |\n| MaxLogit | 66.8 / 84.4 | 62.8 / 85.6 |\n| Entropy | 71.2 / 79.5 | 71.1 / 79.6 |\n| Variance | 70.8 / 79.7 | 70.7 / 79.8 |\n| Energy | 97.9 / 38.9 | 97.7 / 39.4 |\n\n**ViT-B/32**\n\n| Method | Last Layer (FPR / AUROC) | Intermediate Fusion (FPR / AUROC) |\n|---|---|---|\n| MCM | 45.6 / 89.8 | 29.0 / 93.4 |\n| MaxLogit | 68.4 / 83.2 | 68.0 / 82.3 |\n| Entropy | 69.1 / 78.9 | 66.7 / 79.9 |\n| Variance | 68.9 / 79.1 | 66.6 / 80.1 |\n| Energy | 98.1 / 39.9 | 97.7 / 44.2 |\n\n\n**ViT-B/16**\n\n| Method | Last Layer (FPR / AUROC) | Intermediate Fusion (FPR / AUROC) |\n|---|---|---|\n| MCM | 42.3 / 90.83 | 30.7 / 93.5 |\n| MaxLogit | 67.7 / 84.1 | 59.1 / 87.3 |\n| Entropy | 77.2 / 73.7 | 61.21 / 82.8 |\n| Variance | 76.9 / 74.1 | 60.9 / 83.0 |\n| Energy | 98.7 / 36.2 | 98.1 / 38.6 |\n\n\n### **Q3: Validity of Random Projections for Geometry Preservation**\n\nAs detailed in Appendix I, our use of fixed random projections is supported both theoretically and empirically. The Johnson–Lindenstrauss lemma [10, 32, 40] guarantees that such projections approximately preserve pairwise distances. Empirically, RanPAC [40] shows that random projections maintain the discriminative structure of visual features across domains, validating their use for aligning intermediate features without retraining.\n\n### **Q4: Computational Overhead of Using Intermediate Layers**\n\nIn response to the reviewer’s suggestion, we evaluated the computational overhead introduced by our method on both **ResNet** and **ViT-B/16**. At batch size 8 (averaged over 5 runs), **ResNet** shows a memory increase from **0.41 GB to 0.50 GB**, latency from **17.1 ms to 19.5 ms**, and a **12% drop in throughput** (58.6 → 51.4 img/s). For **ViT-B/16**, memory increases from **0.57 GB to 0.69 GB**, while latency remains stable (**57.5 ms to 54.5 ms**) and throughput is unchanged (**54.1 to 54.4 img/s**). These results support that, as requested, our method incurs **minimal additional cost**, particularly for transformer-based architectures. Full details will be included in the appendix.\n\n\n### **Q5: Comparison to State-of-the-Art Methods**\n\nThis is an interesting point, and we thank the reviewer for highlighting it. While recent methods such as **CSP**[4], **NegLabel** [5], and **CLIPScope** [6] achieve strong results, they typically rely on external resources such as WordNet hierarchies or handcrafted prompts, and use scoring functions specifically designed for their frameworks. In contrast, our method is orthogonal to these approaches. It focuses on the internal structure of the model and leverages intermediate-layer fusion, making it broadly compatible with other scoring strategies. To illustrate this, we applied our method to **NegLabel** and observed consistent improvements:\n| Method | FPR Avg ↓ | AUROC Avg ↑ | FPR SUN | AUROC SUN | FPR Places | AUROC Places | FPR DTD | AUROC DTD | FPR iNat | AUROC iNat |\n|---|---|---|---|---|---|---|---|---|---|---|\n| NegLabel | 25.40 | 94.21 | 20.53 | 95.49 | 35.59 | 91.64 | 43.56 | 90.22 | 1.91 | 99.49 |\n| **NegLabel + Int. Layers** | **23.79** | **95.05** | 21.10 | 95.38 | 32.31 | 93.35 | 39.38 | 92.10 | 2.35 | 99.36 |\n\t\n### **Q6: Why Do Metrics Like Kurtosis, Gini, and JSD Perform Worse Than Random?**\n\nHeuristics like kurtosis, Gini, and JSD perform worse than random because they often select redundant or uninformative layers, typically shallow or final ones, which limits the diversity needed for effective OOD detection. In contrast, entropy consistently selects mid-to-deep layers (such as 6 to 8) that offer a better balance between abstraction and diversity, achieving lower FPRs (e.g., 0.307 compared to up to 0.435). This is supported by our SVCCA and agreement analyses (Figures 3 and 4), the entropy–FPR correlation in Figures G.11 and G.12, and the marginal utility study. Random selection, although unguided, sometimes includes strong layers by chance, which explains its surprisingly competitive average performance. For further details, we refer to **Response Q1 to Reviewer V3xf**, which compares selection strategies, and **Response Q2**, which analyzes the individual utility of each layer."}, {"review_id": "Le0eroUDcG", "reviewer": "Reviewer_V3xf", "summary": "The paper studies the how OOD detection changes when information from the intermediate layers of models is included, rather than focusing on the final layer representations. The paper evaluates the OOD performance across many architectures and benchmarks, and also presents a layer selection method which increases OOD detection performance.", "questions": "- Which layers get chosen from the different layer selection methods? For example, for the entropy-based method, do models typically get more confident as we progress through each layer, meaning that optimal combination could just be the last n layers? It would also be helpful to incorporate a few other baselines such as \"single best layer\" or \"penultimate layer\" in Figure 6 so I have a better sense of how each method compares. \n- We see in Figure 1 that increases the number of layers significantly hurts performance for PEv1-B16-224. Do you have any hypotheses for why this is happening? Do the layer selection methods choose N=1 for this model, or would they suggest larger Ns which would hurt performance?\n- In *Surgical Fine-Tuning Improves Adaption to Distribution Shifts* (Lee et al), the authors note the earlier layers seem to be helpful for input-level shifts, middle layers for feature-level shifts, etc for OOD generalization. Do you notice similar trends (or other trends) with your method for OOD detection, and do you have any intuitions for what layers are typically helpful for specific types of OOD detection problems?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n- The paper explores an important aspect of OOD detection, feature selection, where last-layer features are often used by default. Incorporating intermediate layers seems to generally have a positive impact on performance, and the authors also demonstrated results over a wide variety of settings and models. \n- I appreciate the number of ablations in the paper, and they really help illuminate the difference in behavior between model architectures. For example, I thought Figure 3 was really interesting and highlighted the differences in behavior between contrastive vs supervised models. The authors did a thorough job of benchmarking the impacts of many components of their method, such as the number of layers, layer selection method, and other hyperparameters. \n- The paper is well-written, and the motivation for each section of the paper is clear.  \n\nWeaknesses\n- There are quite a few papers which explore the effectiveness of intermediate layers for OOD detection. Although many of these works do not focus on vision-language models, many of them come to the same conclusions that we should consider more than just the penultimate layer. \n  1. Detecting Out-of-Distribution Inputs in Deep Neural Networks Using an Early-Layer Output (Abdelzad et al, 2019)\n  2. MOOD: Multi-level Out-of-distribution Detection (Lin et al, 2021)\n  3. Layer Adaptive Deep Neural Networks for Out-of-Distribution Detection (Wang et al, 2022)\n  4. Multi-layer Aggregation as a Key to Feature-Based OOD Detection (Lambert et al, 2023)\n- The proposed entropy minimization layer selection method seems a bit unintuitive to me, since it is unclear why choosing the subset of features that the model is the most confident on for ID data would translate to improved ID vs OOD separation. \n- The ablation of layer selection strategies could also be expanded on. It may be helpful to add a section in the appendix which clearly defines each of the methods and compare the differences in the layers which end up being selected.", "was_revised": true, "final_justification": "The authors explore an important aspect of OOD detection which often gets overlooked, and determine that features from middle to deep layers are often beneficial. This is an interesting finding, and their method is fairly straightforward and leads to positive performance. They also included more ablations during the rebuttal period which answered my questions about the types of features selected. However, there have been many previous works which explore the effectiveness of intermediate layers, so there is limited novelty. The entropy-based selection strategy could also have been better motivated.", "rebuttal": "We appreciate the reviewer’s positive remarks on the paper’s clarity, experimental rigor, and architectural insights. Motivated by the suggestions, we have expanded our analysis in the revision to better characterize selection behaviors across metrics and architectures, and to clarify the benefits and limitations of our entropy-based strategy.\n\n### **Q1: Which Layers Are Chosen by Each Selection Method?**\n\nWe thank the reviewer for the question. To clarify the behavior of each selection method, we will include a dedicated section in the appendix describing the layers typically selected by different metrics. Below, we provide representative examples:\n\n| **ENTROPY**                    | **KURTOSIS**             | **GINI**              | **JSD**                    |\n| ------------------------------ | ------------------------ | --------------------- | -------------------------- |\n| \\[3, 4, 6, 8, 11] — FPR: 0.307 | \\[0, 11] — FPR: 0.435    | \\[3, 11] — FPR: 0.408 | \\[0,1,2,9,11] — FPR: 0.367 |\n| \\[2, 3, 6, 8, 11] — FPR: 0.298 | \\[11] — FPR: 0.423       | \\[5, 11] — FPR: 0.411 | \\[0,1,3,9,11] — FPR: 0.381 |\n| \\[3, 4, 7, 8, 11] — FPR: 0.315 | \\[0, 1, 11] — FPR: 0.441 | \\[1, 11] — FPR: 0.429 | \\[0,2,3,9,11] — FPR: 0.364 |\n| \\[2, 3, 4, 8, 11] — FPR: 0.316 | \\[1, 11] — FPR: 0.429    | \\[0, 11] — FPR: 0.435 | \\[0,1,2,7,11] — FPR: 0.342 |\n\nThese results show that **entropy consistently selects a mix of mid-to-deep layers** (e.g., 3–8, 11), rather than simply favoring the final layers. In contrast, kurtosis, Gini, and JSD often focus on shallow–final pairs or layer 11 alone, leading to reduced performance. A qualitative explanation is provided in Figures G.11–G.12, where **only entropy exhibits a stable correlation with FPR\\@95**.\n\n### **Q2: Do Certain Layers Consistently Contribute More to Performance?**\n\nTo support the intuition that some layers are more valuable than others, we conducted a **marginal utility analysis** using ViT-B/16 on ImageNet. We ranked layers by their individual contribution to FPR reduction and AUROC improvement:\n\n| Rank | Layer | FPR Reduction (%) | AUC Improvement (%) | Overall Score |\n| ---- | ----- | ----------------- | ------------------- | ------------- |\n| 1    | 8     | **+3.42**         | **+0.54**           | **1.98**      |\n| 2    | 6     | +2.93             | +0.69               |  1.81         |\n| 3    | 7     | +2.53             | +0.54               | 1.54          |\n| 4    | 4     | +1.65             | +0.40               | 1.03          |\n| 5    | 2     | +0.59             | +0.12               | 0.36          |\n| 6    | 9     | –0.14             | –0.32               | –0.23         |\n| 7    | 3     | –0.71             | –0.05               | –0.38         |\n| 8    | 1     | –0.83             | –0.19               | –0.51         |\n| 9    | 0     | –0.85             | –0.21               | –0.53         |\n| 10   | 5     | –2.36             | –0.64               | –1.50         |\n| 11   | 10    | –2.15             | –1.15               | –1.65         |\n\nThis confirms that **layers 6–8 consistently provide strong utility** and complement the final layer, reinforcing the idea that **optimal combinations involve diverse mid-to-deep layers**, not just the last *n*.\n\nIn addition, our **consistency and diversity analysis** (Figures E.8–E.9) shows:\n\n* **Within each ID dataset**, top-10 oracle combinations have low pairwise Jaccard distance (≈ 0.48), indicating stable and convergent selections.\n* **Across datasets**, the overlap drops (≈ 30–50%), and fixed layer sets perform worse (e.g., FPR: 0.300 vs 0.288 with entropy-based selection), emphasizing the benefit of **dataset-specific layer selection**.\n\n### **Q3: Why Does Performance Degrade with More Layers in PEv1-B/16?**\n\nWe hypothesize that the performance degradation in **PEv1-B/16-224** when increasing the number of combined layers is due to its **distilled architecture**. Specifically, PE B/16 is distilled from the larger PE G model \\[1], which likely **compresses most of the useful information into the final layers**. This contrasts with CLIP models, which are trained end-to-end with contrastive objectives that promote **semantic diversity across all layers**.\n\nAs a result, intermediate representations in PEv1-B/16 tend to be less informative or redundant, and **combining multiple layers introduces noise rather than complementary signals**, ultimately harming OOD performance.\n\n### **Q4: Do Selection Methods Adapt to PEv1-B/16 Behavior?**\n\nYes. Our entropy-based selection method adapts accordingly and does **not favor large N** for PEv1-B/16. For example, it selects a small combination $[0, 1, 11]$, reflecting an attempt to minimize degradation. However, even this small combination performs **slightly worse** than using the final layer alone (FPR = 0.3816 vs. 0.3792), indicating that **intermediate layers are not reliably useful** in this model.\n\nThis outcome supports the idea that **layer fusion is less beneficial for distilled architectures**, where semantic abstraction is concentrated near the output and intermediate layers are poorly calibrated as shown in our analysis.\n\n[1] Bolya et al. *Perception Encoder: The best visual embeddings are not at the output of the network*"}]

Author rebuttal:

Meta-review: This paper investigates the role of intermediate-layer representations in pre-trained vision and vision-language models for out-of-distribution (OOD) detection. The authors propose an entropy-based layer selection method and show that fusing intermediate layers can improve zero-shot OOD detection performance across diverse architectures. The work is training-free, inference-only, and systematically studied across 7 backbones. The rebuttal period added new experiments, including integration with stronger baselines (CSP, NegLabel), evaluation on larger and convolutional architectures, per-instance oracle analysis, overhead profiling, and expanded comparisons across scoring strategies.

Overall, the reviews converge on the paper being technically sound, clearly written, and empirically comprehensive, with several reviewers explicitly raising their scores after the rebuttal. Concerns about limited novelty are valid, but the rebuttal effectively demonstrated that the method is orthogonal and complementary to existing state-of-the-art approaches. The authors are encouraged to take reviewers' suggestions to further improve the manuscript.
Decision: Accept (poster)