Title: X-Mahalanobis: Transformer Feature Mixing for Reliable OOD Detection
Venue: NeurIPS 2025
Accepted: yes
Keywords: out-of-distribution detection

long-tail learning
Abstract: Recognizing out-of-distribution (OOD) samples is essential for deploying robust machine learning systems in open-world environments. While conventional OOD detection approaches rely on feature representations from the penultimate layer of neural networks, they often overlook informative signals embedded in intermediate layers. In this paper, we present a straightforward feature mixing approach for pre-trained Transformers, which combines multi-layer representations via calculated importance weights, and identifies OOD samples using Mahalanobis distance in the blended feature space. When in-distribution samples are accessible, we show that parameter-efficient fine-tuning strategies effectively balance classification accuracy and OOD detection performance. We conduct extensive empirical analyses to validate the superiority of our proposed method under zero-shot, and fine-tuning settings using both class-balanced and long-tailed datasets. The source code is available at https://github.com/SEUML/X-Maha.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes X-Mahalanobis, an OOD detection method that fuses shallow and deep layer features in an adaptive way to form the feature space in which the Mahalanobis distance is computed. The work also identifies that parameter-efficient fine-tuning is favorable over full fine-tuning for pre-trained transformers for OOD detection. Experiments on CIFAR-100 and ImageNet (with both class-balanced and long-tailed setting) are conducted to demonstrate improved performance.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. In general leveraging both shallow and deep features has great potential for OOD detection, yet the key lies in how to fuse/aggregate the information from different features. Having another method on this, which is what this paper is doing, helps push the progress in this direction.\n2. It's good to consider both class-balanced and long-tailed situations for OOD detection.\n\n### Weaknesses\n1. I understand that this is for motivating the paper and that later in line 173 - 181 the authors make distinction with prior works. However, the claim that existing works \"neglect the rich information in the features learned by the layers of shallow neural networks\" is false (or at least an over-claim). The authors have already mentioned Mahalanobis and Trusted as two examples, and there are many more that have already tried to leverage shallow layer features for enhanced detection (e.g., [1, 2]). I would suggest rephrasing some of the sentences to make them more accurate.\n2. In ablation study \"Ways to fuse shallow features\", could the authors try simply averaging the layer-wise scores for Mahalanobis when comparing with Score Aggregation, instead of using the weights derived from your method?\n3. OpenOOD shows that Mahalanobis with only the final layer feature consistently outperforms Mahalanobis with multiple layers' features (see Table 2 MDS v.s. MDSEns). Although there could be many extra nuances here (e.g., they are mostly evaluating on CNNs while this work is on transformers), I'd like to see some discussion on this seemingly contradicting observation. For example, does model architecture have any effect here? Or is it purely because that X-Mahalanobis uses a more appropriate fusing strategy such that it outperforms MDS (Mahalanobis with the final layer feature), while MDSEns (Mahalanobis with multiple layers' features) underperforms MDS?\n\n[1] Detecting Out-of-Distribution Inputs in Deep Neural Networks Using an Early-Layer Output\n\n[2] MOOD: Multi-level Out-of-distribution Detection\n\nQUESTIONS:\nPlease see Weaknesses 2 and 3.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method for computing a score used for out-of-distribution (OOD) detection. The authors use the well known Mahalanobis distance as the score. Their approach differs from prior works in the used feature space. They compute weighted sum of representations from different layers of the transformer feature extractor, and compute the Mahalanobis distance in this new space. \n\nThe authors report state-of-the-art (SotA) results on the OOD detection benchmark OpenOOD.\n\nSTRENGTHS AND WEAKNESSES:\n## Notes:\n- line 40 and lines 318-319: \"... **attention-based** fusion module ...\"\n        - Could you explain what is meant by attention-based? I am unable to see the connection between the computation of your coefficients $\\alpha^l$ and the defacto standard \"query, key, value attention\".\n- lines 126-128:\n        - Perhaps a lower index $i$ is missing in the layer feature representations.\n- Eq. 2:\n        - The input $x$ is an image. It is split into patches, linearly projected, positional embeddings are added, and a class token is concatenated. This results in the embedding of an image being a sequence of length \"number of patches + 1\", no? Does the $x^l$ correspond to the embedding of the class token? Or the entire sequence over patches? I could not find this in the paper. Please correct me if my understanding of ViT architectures is incorrect.\n        - It should be noted that this aggregation assumes that the features from different layers are compatible. I believe that this should be the case due to the residual connections, however, it might not be the case for a general architecture. \n- line 149:\n        - Why is the covariance matrix computed using the feature representation from the final layer ($f(x_i)$), but the mean vector is computed in the aggregated feature space from $\\phi(x_i)$?\n- line 155 and lines 162-164:\n        - What is the argument for high variability in the features => high discriminative capacity? Low intra-class variability and high-inter class variability do indeed suggest discriminative ability of the features. How does feature variability (independent of classes) play a role here?\n- lines 170-172:\n        - Wasn't \"feature mixing\" already demonstrated to be effective by Trusted [7] and Mahalanobis [27]? The novelty of your method lies precisely in how the mixing weights are computed, no? I believe that your argument is stronger in lines 179-181, as your method may choose to ignore representations from some layers.\n- Fig. 3:\n        - Could you also plot the model accuracy before any fine tuning?\n- line 209: \"... our approach is model-agnostic ...\"\n        - Can it be adapted to convolutional networks, or models without residual connections?\n        - Could you evaluate it with the model checkpoints provided in the OpenOOD benchmark?\n- line 217:\n        - This is the first time that the scoring function is defined in the paper. I assume that for vision only models, you use the scoring function $G(x)=\\max_{c\\in[C]}M_{X-Maha}(x; \\tilde\\mu_c, \\tilde\\Sigma)$, is that correct?\n- line 245:\n        - Typo \".. our methoMSP, MSP ...\"\n- Table 1:\n        - On Texture, LSUN, and Places365 the method Trusted ties with X-Maha. It should also be highlighted in bold. Same applies e.g. to Appendix Table 4.\n- Figure 4\n        - SFM is not defined, what does it stand for? From the context of the paper I believe that SFM should be X-Maha, no?\n- Table 2:\n        - Trusted achieves AUROC below 50%, is this correct? This would mean it performs worse than a random OOD detector.\n- lines 296-298:\n        - On what data are the coefficients $\\alpha^l$ computed? Are they computed on the training / validation ID data and fixed? Or are they computed on the unlabeled deployment set which is a mixture of ID and OOD?\n- Figure 5:\n        - Is the method equivalent to Mahalanobis [27] for number of layers = 1? This would coincide with the values in the subfigures for CIFAR-100 and ImageNet, with the values 98.92 and 93.03 (of Mahalanobis from Table 2), which seem to fit in the figures. If this is the case, why does the value for ImageNet CLIP (85.12) not fit into the third subplot?\n        - For number of layers = 12 (all layers of ViT-B) the method is exactly the presented X-Maha, correct? Why does the value for number of layers = 12 not coincide with the reported X-Maha result in Table 2 of 90.15?\n- Table 3:\n        - ImageNet CLIP PM AUROC should be highlighted in bold instead of X-Maha. How is the AUROC so high when PM achieves the worst FPR95 here?\n- Appendix A.3, line 510: \"... X-Maha consistently achieves remarkable performance ...\"\n        - From the ablation Tables 7,8,9,10 it seems that the method performs well (sometimes better than X-Maha) even with other layer weights, e.g., $W_{0.5}$. Do you believe the proposed method for computing $\\alpha^l$ to be crucial for the method to work?\n        - From the layer effect ablation, you concluded that only the final ~6 layer should be used. Wouldn't e.g., uniform weights over these layers perform the same as X-Maha?\n\n## Strengths:\n- Strong empirical results; Reported SotA results on OpenOOD benchmark\n- Good literature review with proper citations of methods related to X-Mahalanobis\n- Ablation studies\n\n## Weaknesses:\n- Novelty - Though the method is new, it does not present a significant difference from prior works\n- Weak motivation for Eq. 4 which is the central concept of the method\n- Inconsistencies in the manuscript, e.g., 1) use of $f(x_i)$ on line 149. Either the method is not precisely described or the explanation for why $f(x_i)$ is the correct term is missing from the paper. 2) The feature representation from layer $i$ is a sequence of feature vectors. How is $x^l$ precisely defined is not clear.\n- Results are not evaluated with the checkpoints standard to the OpenOOD benchmark, making comparison to other methods difficult (Please correct me if I am wrong)\n\nQUESTIONS:\n1) Does the $x^l$ correspond to the embedding of the class token? Or the entire sequence over patches? Why is the covariance matrix computed using the feature representation from the final layer ($f(x_i)$), but the mean vector is computed in the aggregated feature space from $\\phi(x_i)$?\n2) What is the argument for high variability in the features => high discriminative capacity? How does feature variability (independent of classes) play a role here?\n3) Wasn't \"feature mixing\" already demonstrated to be effective by Trusted [7] and Mahalanobis [27]? The novelty of your method lies precisely in how the mixing weights are computed, no?\n4) On what data are the coefficients $\\alpha^l$ computed? Are they computed on the training / validation ID data and fixed? Or are they computed on the unlabeled deployment set which is a mixture of ID and OOD?\n5) Is the method equivalent to Mahalanobis [27] for number of layers = 1? If this is the case, why does the value for ImageNet CLIP (85.12) not fit into the third subplot in Figure 5? For number of layers = 12 (all layers of ViT-B) in Figure 6 the method is exactly the presented X-Maha, correct? Why does the value for number of layers = 12 not coincide with the reported X-Maha result in Table 2 of 90.15?\n6) Do you believe the proposed method for computing $\\alpha^l$ to be crucial for the method to work? Wouldn't e.g., uniform weights over these layers perform the same as X-Maha?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an adaptive fusion module that dynamically assigns importance weights to the representations learned by each Transformer layer and detects OOD samples using the Mahalanobis distance. Compared to prior works such as Mahalanobis and Trusted, the proposed approach computes importance weights directly from the training data and does not require a validation set. By dynamically assigning importance weights to different layers, it effectively mitigates the degradation of overall OOD detection performance, even when features from certain layers are less effective. Extensive experiments demonstrate the effectiveness of the proposed method. \n\nHowever, the underlying principle of the method is not clearly explained, and the experimental results do not achieve state-of-the-art performance compared to recent CLIP-based OOD detection methods.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength:**\n\n1.The paper is well-structured and easy to follow.\n2. It has been experimentally validated on various benchmark datasets and is supported by a comprehensive ablation study.\n\n**Weaknesses:**\n\n1. In lines 72-73, the authors calim that \"...fuse the shallow layer features with the importance weights by measuring the covariance of features in each layer\". However, the underlying relationship between feature covariance and discriminative capacity is not clearly explained. Please provide theoretical insights or experimental evidence to support this proposal.\n\n2. In Figure 3, the comparison between FFT and PEFT using a learning rate of 1e-4 is insufficient to justify the claim that “PEFT outperforms FFT,” as FFT is commonly trained with smaller learning rates (e.g., 1e-5 or 1e-6). A more comprehensive comparison across different learning rates would strengthen this conclusion.\n\n3. Regarding the experimental comparisons, numerous recent zero-shot and few-shot CLIP-based methods have been proposed for improving OOD detection. Prior works such as GL-MCM, NegLabel, and CLIPScope have achieved significantly better performance than the results reported in Table 2. These should be included as baselines for a fair comparison.\n\n   Miyai, A., Yu, Q., Irie, G., & Aizawa, K. (2025). GL-MCM: Global and Local Maximum Concept Matching for Zero-Shot Out-of-Distribution Detection. International Journal of Computer Vision, 1-11.\n\n   Jiang, Xue, et al. \"Negative label guided ood detection with pretrained vision-language models.\" arXiv preprint arXiv:2403.20078 (2024).\n\n   Fu, Hao, et al. \"Clipscope: Enhancing zero-shot ood detection with bayesian scoring.\" arXiv preprint arXiv:2405.14737 (2024).\n\n4. Does the use of shallow layer features result in additional GPU memory consumption?  Please provide a comparison of the computational cost (e.g., memory usage, inference time) of the proposed method versus existing baselines.\n\n5. Do the conclusions of this work generalize to other architectures, such as CLIP ResNet-50? Clarifying the generalizability across different backbone models would enhance the impact and applicability of the method.\n\nQUESTIONS:\nSee Weaknesses\n\nLIMITATIONS:\nThe underlying principle of the method is not clearly explained, and the experimental results do not achieve state-of-the-art performance compared to recent CLIP-based OOD detection methods.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce an adaptive fusion module that dynamically assigns importance weights to the learned feature representations at each layer, including both intermediate and final layers, to enhance (OOD) out-of-distribution sample detection using the Mahalanobis distance metric. This contrasts with conventional OOD detection approaches, which typically rely solely on the final layer's feature representations. The authors further demonstrate and benchmark the effectiveness of their proposed approach across a range of diverse OOD datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. The authors proposed a really simple concept of fusing lower/shallow layer features with importance weight metric derived from a pre-trained Transformer model to improve OOD separation.  \n2. The manuscript is well-written, well-organized, and easy to follow.\n3. The authors present an extensive experimental setup and validate their approach with satisfactory evaluation results, benchmarking it against prior SOTA methods across nine diverse OOD datasets.\n\nWeakness:\n1. The authors mentioned as a distinction from prior work from Mahalanobis [27], as it calculates \"OOD score using the representation of each layer individually and weights them together by training a logistic regression model using a validation set\" . While the proposed approach does not require a separate validation set and computes from same training data. This point need to be discussed elaboratively, specifically what practical and methodological challenges/implications to encounter related to the dependence on a validation set rather a training-set only usage. Is there any risk of overfitting on the validation dataset?\n2. Please ensure consistent formatting for result representation (bold: best, underline: second best etc.) across all tables including \"Additional experiments\" section. It will improve the readability of the manuscript and improve clarity in data presentation.\n3. The statement in the Limitations section \"A promising avenue for future research would be to extensively study the impact of mixing weights in X-Maha and theoretically show that mixing shallow features can improve performance.\" --> appears somewhat self-contradictory when compared to the empirical results presented in the manuscript. If there are specific scenarios where the generalizability of the proposed approach is limited, the authors are encouraged to briefly discuss them and provide illustrative examples to clarify these limitations.\n\nQUESTIONS:\nAlready listed as weakness points to be discussed:\n\n1. The authors mentioned as a distinction from prior work from Mahalanobis [27], as it calculates \"OOD score using the representation of each layer individually and weights them together by training a logistic regression model using a validation set\" . While the proposed approach does not require a separate validation set and computes from same training data. This point need to be discussed elaboratively, specifically what practical and methodological challenges/implications to encounter related to the dependence on a validation set rather a training-set only usage. Is there any risk of overfitting on the validation dataset?\n\n2. The statement in the Limitations section \"A promising avenue for future research would be to extensively study the impact of mixing weights in X-Maha and theoretically show that mixing shallow features can improve performance.\" --> appears somewhat self-contradictory when compared to the empirical results presented in the manuscript. If there are specific scenarios where the generalizability of the proposed approach is limited, the authors are encouraged to briefly discuss them and provide illustrative examples to clarify these limitations.\n\nLIMITATIONS:\nThe statement in the Limitations section \"A promising avenue for future research would be to extensively study the impact of mixing weights in X-Maha and theoretically show that mixing shallow features can improve performance.\" --> appears somewhat self-contradictory when compared to the empirical results presented in the manuscript. If there are specific scenarios where the generalizability of the proposed approach is limited, the authors are encouraged to briefly discuss them and provide illustrative examples to clarify these limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-14"}