Prototype-based Optimal Transport for Out-of-Distribution Detection

Authors: Ao Ke, Wenlong Chen, Chuanwen Feng, Yukun Cao, Xike Xie, S Kevin Zhou, Lei Feng
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Out-Of-Distribution Detection, Optimal Transport

Abstract:
Detecting Out-of-Distribution (OOD) inputs is crucial for improving the reliability of deep neural networks in the real-world deployment. In this paper, inspired by the inherent distribution shift between ID and OOD data, we propose a novel method that leverages optimal transport to measure the distribution discrepancy between test inputs and ID prototypes. The resulting transport costs are used to quantify the individual contribution of each test input to the overall discrepancy, serving as a desirable measure for OOD detection. To address the issue that solely relying on the transport costs to ID prototypes is inadequate for identifying OOD inputs closer to ID data, we generate virtual outliers to approximate the OOD region via linear extrapolation. By combining the transport costs to ID prototypes with the costs to virtual outliers, the detection of OOD data near ID data is emphasized, thereby enhancing the distinction between ID and OOD inputs. Experiments demonstrate the superiority of our method over state-of-the-art methods.

Decision: reject
Reviewer scores: 3, 8, 6, 3, 3

Reviews and discussion:
<thread>
creation date:
1729978904990

thread_type:
official_review

<thread_object>
creator: reviewer_zwxm

<content>
summary:
This paper introduces an approach to out-of-distribution (OOD) detection leveraging optimal transport (OT). The core idea is based on the observation that OOD inputs typically incur higher transport costs when mapped to the in-distribution (ID) in latent space. From this, they develop an OOD detection method by first compute the optimal transport plan from the test distribution to the ID distribution. OOD detection is then achieved by calculating the transport cost for each input and applying a threshold to determine whether the input is OOD. The author then benchmark their method with varied number of datasets.

strengths:
- The comparison with other baseline methods appears comprehensive and thorough.

weaknesses:
One of my concern is that  the paper lacks a deeper understanding for why OT would be better than other OOD detection methods. Standard approaches have increasingly focused on understanding geometric representations in neural networks, either through uncertainty estimation [1] or, more rigorously, likelihood estimation [2], sometimes aided by reconstruction autoencoders [3]. Given the connection between geometric representation and OT through the transport map, I would expect interpretative insights along this line rather than a direct application of OT for OOD detection. Without this, the current proposal, in my opinion, is incremental compared to the existing distance-based approaches.

The method presentation is not clear. I have some confusion in terms of how the OT plan is constructed using both in-distribution (ID) inputs and out-of-distribution (OOD) datasets. See question (1).

Finally, similar ideas of using optimal transport to characterize OOD error have been explored with more concrete theoretically properties, as in [4].

[1] Cao, Senqi, and Zhongfei Zhang. "Deep hybrid models for out-of-distribution detection." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[2] Kamkari, Hamidreza, et al. "A Geometric Explanation of the Likelihood OOD Detection Paradox." Forty-first International Conference on Machine Learning.

[3] Zhou, Yibo. "Rethinking reconstruction autoencoder-based out-of-distribution detection." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[4] Lu, Yuzhe, et al. 'Characterizing out-of-distribution error via optimal transport.' Advances in Neural Information Processing Systems 36 (2023): 17602-17622."

questions:
(1) Can the author specify how the validation set is built in the OT formulation, especially the test batch size in Section 4.3? Specifically, my impression is that we only have 1 unique transport map from the validation set and we will use that for detecting OOD samples. As such, I'm not sure how varying the test batch size affect the result. Are we computing a distinct transport map for every test batch size? if so how do we decide the threshold for OOD samples for these test batch size? What if their statistic (portion between ID and OOD data) changes?

(2) How to apply this idea when the OOD set is small? How the validation set size affect the performance? 

(3) What is the fundamental idea that OT is better than other methods such as likelihood estimation?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730472119949

thread_type:
official_review

<thread_object>
creator: reviewer_7p4j

<content>
summary:
This paper addresses the inherent distributional differences between ID and OOD samples by proposing the use of optimal transport (OT) to measure the discrepancy between test inputs and ID prototypes, a method termed Prototype-based Optimal Transport (POT). The authors assess distances by calculating the transport cost from each test input to the ID prototypes. Additionally, they apply linear extrapolation to generate virtual outliers, effectively regularizing the decision boundary. By evaluating distributional differences between ID and OOD samples through optimal transport, this work introduces a novel perspective. Furthermore, the use of linear extrapolation enables optimal transport to function effectively within this framework.

strengths:
1.	Explore a new OOD score by measuring the discrepancy between test images and ID prototypes from the perspective of optimal transport, which is well-supported theoretically.
2.	Use linear extrapolation to regularize the decision boundary, offering a successful perspective on applying optimal transport in OOD detection.

weaknesses:
1.	In the use of OT, calculating ν for test samples in Equation (3) requires all test samples, which is impractical in real-world applications.
2.	When generating outliers via linear extrapolation, the method appears to rely on information from the test set, which is also impractical.

questions:
Why do you claim that your method measures differences directly in the latent space? It is clear that you are also using features obtained from the feature extractor. When comparing with DDE, you state that DDE measures differences in the representation space, which is influenced by the transformations of the DNN features, while POT measures differences in the latent space.

I will adjust the score based on your response.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730514398409

thread_type:
official_review

<thread_object>
creator: reviewer_jbpb

<content>
summary:
The work considered the problem of OOD (Out-Of-Distribution) detection, and proposed a method based on optimal transport from test inputs to ID (in-distribution) prototypes. It first constructed the prototype of each class as the average representation for that class extracted from the feature encoder, then computed the optimal transport between the encoded feature of the test inputs and the class prototypes and used the sum of the transport costs from a test input to prototypes as an indication of the input's outlier-ness. It then noted that only using this is insufficient for detecting OOD data closer to ID, and proposed to generate virtual outliers by linear extrapolation between the prototypes and the mean of the test inputs, and finally used the difference between the transport cost to the ID prototypes and the cost to the virtual outliers as the final OOD score.

The work then performed extensive experiments. The results showed that the proposed method often outperforms the competitors and gets close-to-best performance otherwise. It can be used with different pretraining schemes without graceful performance degradation, and can use the network weights as the ID prototypes when training data are not available. Ablation study shows that the virtual outliers are important for the performance, and that the method is robust to hyperparameters: the extrapolation coefficient, the regularization coefficient, and the test batch size.

strengths:
- The work addressed an important topic of OOD detection, and designed a pratical method.
- The proposed method used two key techniques: optimal transport and virtual outliers via linear extrapolation. The work illustrate the insights behind these techniques quite clearly.
- The experiments are quite extensive and the results are quite convincing, showing the strong performance and investigating important aspects of the method like robustness to hyperparameters.

weaknesses:
- See the questions below.

questions:
- The performance will depend on the feature encoder used. Is the method robust to the choice of the feature encoder? How large is the impact of the encoder? 
- What if we use more than one class prototypes for each class? 
- What are the \mathcal{P} and \mathcal{M} in Line 241? Better to formally specify them.
- Eqn (9): the second "+" should be ":"?
- Table 1, Textures FPR95, NAC-UE result: should be underlined rather than boldfaced.
- Maybe also underline the second best results in Table 2, so as to show the performance margin and be consistent with Table 1.
- Figure 3 shows the ablation result on virtual outliers on Far-OOD datasets. What about ablation study on Near-OOD datasets?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730518240541

thread_type:
official_review

<thread_object>
creator: reviewer_q2aw

<content>
summary:
This paper focuses on detecting Out-of-Distribution (OOD) inputs, which is a crucial task for improving the reliability of deep neural networks. Since there exists an inherent distribution shift between ID and OOD data, the authors propose a novel method Prototype-based Optimal Transport (POT) which utilizes the prototype-based optimal transport to assess the discrepancy between test inputs and ID prototypes and use the obtaining transport costs for OOD detection. Since relying solely on the cost of ID prototypes is insufficient for discerning OOD data with smaller distribution shifts from ID data, they propose to generate virtual outliers to approximate the OOD region using representation linear representation extrapolation. Finally, experiments on several benchmark datasets are provided.

strengths:
1. This paper addresses the OOD detection problem by measuring distribution discrepancy and proposes a novel approach using prototype-based optimal transport.
2. To identify OOD data with smaller distribution shifts from ID data, this paper proposes to generate virtual outliers to approximate the OOD region using representation linear representation extrapolation.
3. Comprehensive experimental results on various benchmark datasets demonstrate the effectiveness of the proposed detection method.

weaknesses:
1. The experimental results and ablation studies require more explanation and discussion, rather than just describing the phenomena.
2. Although this paper reduces computational complexity by introducing the entropy regularization term and the Sinkhorn-Knopp algorithm, the optimal transport problem is still computationally intensive on large-scale datasets. This may limit the application of the proposed method POT in resource-constrained environments.
3. Is the constraint w<0 in Eq. (8) necessary? Because there is no corresponding description in this paper, and this case was not considered in the parameter analysis.

questions:
1.	Can the authors provide more discussions on the experimental results and ablation studies?
2.	Can the authors show the execute time of the proposed method compared with other methods?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730622464339

thread_type:
official_review

<thread_object>
creator: reviewer_49jf

<content>
summary:
Summary:This paper proposes a new prototype-based optimal transfer (POT) method to detect out-of-distribution (OOD) data by measuring the optimal total transfer cost between the test sample and the ID prototype and the virtual outlier. In addition, this paper also proposes to obtain the virtual outlier of OOD by simple linear extrapolation, and demonstrates its effectiveness by comparing with the existing technology. The paper has a clear structure and rigorous logic, covering relevant background, methods, experimental evaluation and results analysis.

strengths:
- This paper provides a new perspective on the OOD detection problem by combining the concepts of prototype and optimal transmission.
- The author conducted exhaustive experiments on multiple standard datasets, comparing the performance of POT with 21 existing technologies. The experiments are adequately designed and the results are clearly interpreted.

weaknesses:
- Formulas 2 and 3 in the paper mention that the calculation of the transmission plan and cost matrix comes from the ID prototype and test samples. The simple linear extrapolation to obtain the OOD area is also based on the test samples. This solution is not realistic under the task setting of OOD detection.
- The paper mentions that the OOD area is obtained by simple linear extrapolation, but based on the author's description and existing research, the difficulty of OOD samples lies in the approximate ID features in the feature space. The OOD area obtained by simple linear extrapolation may not be representative, and there is a lack of discussion on the interpretability of the process.
- The results of the paper method should also be compared with the results of some of the latest papers.

questions:
- Are you skeptical about the effectiveness of obtaining OOD data areas through simple linear extrapolation?
- Is it realistic for OOD detection to use test data for linear extrapolation?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732294054486

thread_type:
official_comment

<thread_object>
creator: area_chair_jcvq

<content>
title:
Interactive Discussions

comment:
Dear Reviewers,

Thank you for your efforts in reviewing this paper. We highly encourage you to participate in interactive discussions with the authors before November 26, fostering a more dynamic exchange of ideas rather than a one-sided rebuttal.

Please feel free to share your thoughts and engage with the authors at your earliest convenience.

Thank you for your collaboration.

Best regards,
ICLR 2025 Area Chair
</content>
</thread_object>

</thread>


Metareview:
metareview:
This submission focuses on out-of-distribution (OOD) detection and proposes a novel method based on prototypes and optimal transport. Specifically, the method first calculates the optimal transport distance from each test batch to the in-distribution (ID) prototypes. To improve performance, the distance to virtual outliers, generated through linear extrapolation, is then incorporated into the OOD detection score.

One key weakness highlighted by reviewer #7P4j concerns the problem setting, where the proposed method relies on a minibatch of test samples to obtain the OOD detection scores. This approach is impractical for real-world applications and differs from existing OOD detection methods. Additionally, other reviewers, such as #zWXM, expressed similar concerns. The Area Chair concurs with these points and suggests that the authors address this issue in a future submission.