SIREN: Shaping Representations for Detecting Out-of-Distribution Objects
Authors: Xuefeng Du
Gabriel Gozum
Yifei Ming
Yixuan Li
Venue: NeurIPS 2022 Accept (2022)

Abstract: Detecting out-of-distribution (OOD) objects is indispensable for safely deploying object detectors in the wild. Although distance-based OOD detection methods have demonstrated promise in image classification, they remain largely unexplored in object-level OOD detection. This paper bridges the gap by proposing a distance-based framework for detecting OOD objects, which relies on the model-agnostic representation space and provides strong generality across different neural architectures. Our proposed framework SIREN contributes two novel components: (1) a representation learning component that uses a trainable loss function to shape the representations into a mixture of von Mises-Fisher (vMF) distributions on the unit hypersphere, and (2) a test-time OOD detection score leveraging the learned vMF distributions in a parametric or non-parametric way. SIREN achieves competitive performance on both the recent detection transformers and CNN-based models, improving the AUROC by a large margin compared to the previous best method. Code is publicly available at https://github.com/deeplearning-wisc/siren.

SIREN: Shaping Representations for Detecting Out-of-Distribution Objects
Xuefeng Du, Gabriel Gozum, Yifei Ming, Yixuan Li Department of Computer Sciences University of Wisconsin Maddison
{xfdu, gozum, alvi viimiing, sharonli (Ccs wisc.edi
Abstract
Detecting dittrf--iitribution (OOD) objects is indispensable for safely deploying object detectors in the wild. Although distance-based OOD detection methods have demonstrated promise in image classification, they remain largely unexplored in object-level OOD detection. This paper bridges the gap by proposing a distance- based framework for detecting OOD objects, which relies on the model-agnostic representation space and provides strong generality across different neural architec- tures. Our proposed framework SIREN contributes two novel components; (1) a representation learning component that uses trainable loss function shape the representations into mixture of von Mises-Fisher (vMF) distributions on the unit hypersphere, and (2) a test-time OOD detection score leveraging the learned vMF distributions in a parametric or pan-aaraetticc way. SIREN achieves competitive performance on both the recent detection transformers and CNN-based models, im- proving the AUROC by a large margin compared to the previous best method. Code is publicly available at https: //////ggttt///// com coom/deppeerrnngg wisc/sire siren...
1 Introduction
Teaching object detectors to be aware of out-of-diittrbuttin (OOD) data is indispensable for building reliable Al systems. Today, the mainstream object detection models have been operating in the closed-world setting. That is, a model will match an object to one of the given class labels, even it is irrelevant. Instead, the open-world setting emphasizes that objects from the unknown classes can naturally emerge, which should not be blindly predicted into a known class. In safety-critical applications, such autonomous driving, failing detect OOD objects the road directly lead to disastrous accidents [50]. The situation can be better avoided if the object detector recognizes the object unfamiliar and appropriately cautions the human driver take over.
In this paper, we pioneer a distance-based framework for detecting OOD objects. Currently, the distance-based method remains largely unexplored in object-level OOD detection. In particular, by operating in the representation space, distance-based methods are model-agnostic and provide strong generality across neural architectures. In contrast, existing approaches derive highly specialized OOD detection scores based on the outputs the object detectors, which may not be seamlessly applicable across architectures. For example, the classification output of the Faster R-CNN [52] is optimized by the multi-class softmax loss, whereas the recent transformer-base based object detection networks such [75] uses multi-label focal loss [36]. Thereby, while output-based OOD scoring functions may be limited to specific architectures, distance-basee methods are not.
DEFORMABLE...
Although distance-based OOD scoring functions have been studied in image classification, they do not trivially transfer to object detection models. For example, [33] modeled the feature embedding space as mixture of multivariate Gaussian distributions and used the maximum Mahalanobis distance [41] to all class centroids for OOD detection. However, we observe that the modern object detection
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(a) Vanilla DETR
(b) SIREN (Ours)
Figure 1: (a) Feature embeddings from the penultimate layer of vanilla DEFORMABLE-DDTE [75] trained on the PASCAL-VOC dataset [14], which display irregular distributions. (h) Feature embeddings shaped by the proposed SIREN, which form compact clusters on the unit hypersphere.
models such as DEFORMABLE-DETR [75] produce highly irregular embeddings (Figure (a)), which do not fit the Gaussian distributional assumption. As result, the OOD detection score relying such suboptimal embeddings can misbehave.
We propose novel framework called SIREN, tackling two highly dependent problems--representttio learning and OOD detection- in one synergistic framework. Concretely, SIREN contributes two novel components: (1) We introduce an end-to-end trainable loss that enables ShapIng the RepresENtations into desired parametric form (Section 3.1). In particular, we model the representations by the von Mises-Fisher (vMF) distribution, classic probability distribution in directional statistics for hyperspherical data with the unit norm. Our loss function encourages the normalized embedding to be aligned with its class prototype and shapes the overall representations into compact clusters for each class. Compared to the Gaussian distribution, using the vMF distribution avoids estimating large covariance matrices for nigh-dimensional data that is shown to be costly and unstable [7, 65]. (2) We explore test-time OOD detection by leveraging the optimized embeddings in a parametric or non-parametric way (Section 3.2). We propose new test-time OOD score based on the learned class- conditional vMF distributions. The parameterization of the vMF distribution is directly obtainable after training, without requiring separate estimation. Different from Mahalanobis distance [33], the proposed parametric score in principle suits the learned vMF distributions on the hypersphere. Additionally, we explore non- nan-aaraettric nearest neighbor distance for OOD detection [60], which is agnostic to the type of distribution of the feature space.
Empirically, SIREN establishes superior performance on both transformer-based and CNN-based models. On PASCAL-VOC, SIREN outperforms the latest baseline OW-DETR [18] by a significant margin (122.53% in AUROC). Moreover, our framework is model-agnostic and does not incur changes to the existing network architecture. The proposed loss can be flexibly added as a plug-in module top of modern architectures,
show in Section 4.
Our key contributions are summarized as follows:
1. To the best of our knowledge, SIREN pioneers a distance-based approach for object-level OOD detection. Different from previous works, SIREN does not rely specialized output- based OOD scores, and can generalize across different architectures in model-agnostic fashion.
2. SIREN establishes competitive results on challenging object-level OOD detection task. Compared to the latest method [18], SIREN improves the OOD detection performance by a considerable margin while preserving the mAP on the ID task. We show that SIREN is effective for both recent transformer-ba based and classic CNN-based models.
3. We shape representations via novel vMsed formulation for object-level OOD detection. We conduct in -e-tepth ablations to understand how different factors impact the performance of SIREN (Section 5).

---

Prediction Head
Shaping Representations
SIREN
clss
LaIREN
Classification Head
OOD Detection
Regression Head
Llae
Input X
Scon
Modeling Latent Distributions
4xx, b)
bject Detection Network
Projection Head ( R
palri x) Zalk) exp (xz (r)
Figure 2: Overview of the proposed learning framework SIREN. We introduce a new loss Lsren which shapes the representations on the unit hypersphere into compact class classcconiitiona vMF distributions. The embedding E R has unit norm ||rll² = In testing, we can employ either parametric non- oa--aaaaettrc distance functions for OOD detection. See Section for details.
2 Preliminaries: Object-level OOD Detection
We start by introducing the OOD detection problem for object detection in the open-world setting, which has received increasing research attention lately [13, 18]. Our goal is train object detection networks that can simultancously: (1) localize and classify objects belonging to known categories accurately, and (2) identify unfamiliar objects outside the training categories. Compared to image- level OOD detection, object-level OOD detection is more suitable for real- machine learning systems, yet also more challenging as it requires reasoning OOD uncertainty at the fine-grained object level. Since natural images are composed of multiple objects, knowing which regions of an image are anomalous allows for safe handling of unfamiliar objects.
Notations. We denote the input and label space by X' R and y (1. respectively. Let x E X' be the input image, b E R be the bounding box coordinates associated with objects in the image, and y E y be the semantic label of the object. An object detection model is trained on ID dataset Din = (xx, b;, })));= drawn from an unknown joint distribution P. We use neural networks with parameters 0 to model the bounding box regression po x) and the classification po pe(y)x,b) b).
Object-level OOD detection. The OOD detection can be formulated as a binary classification problem, distinguishing between the in- vs. -uttt--distiiuution objects. Let Px denote the marginal probability distribution on X. Given test input x' ~ Px, as well as an object b' predicted by the object detector, the goal to predict a binary outcome g(x', b'). We use g = 1 to indicate detected object being ID), and 0 being OOD, with semantics outside the support of y
Proposed Method
Overview. Our framework SIREN is illustrated in Figure 2, which trains object detector in tandem with a representationn-saapin branch. The object detector backbone f X + R" maps an object to its feature embedding h(x, b) E R (often referred to as the penultimate layer). In addition, we introduce a new MLP projection head ø R" Ed that maps the hxx, b) to lower-dimensional embedding E IRd (d < m) with unit norm ||||| = 1. The normalized embeddings are also referred to as hyperspherical embeddings, since they are on unit hypersphere. In designing SIREN, we address two key challenges: (1) How to shape the hyperspherical representations into desirable probability distributions during training time (Section 3.1 (2) How to perform test-time OOD detection by leveraging the learned distributions (Section 3.2)? Our method does not incur any change to the object detection network backbone. The proposed regularization can be flexibly used plug-in in module top of the modern architectures, will show in Section 4.
3

---

3.1 SIREN: Shaping Representations
Modeling the latent distributions. We propose to model the latent representations by the von Mises-Fisher (vMF) distribution [43], probability distribution in directional statistics for spherical data with unit norm ||r|]² The probability density function for unit vector in Id is given as follows:
pa(r; 14 x) = Za(n:) exp (repa T r).
where x 2 0, |||22 1, and the normalization factor Za(k) is defined as:
1:4/2-1 Za(w) (2mrd//
(2)
where I is the modified Bessel function of the first kind with order v. Za(n) can be calculated in closed form based on K and the dimensionality d. Importantly, the vMF distribution is characterized by two parameters: the mean vector μl and concentration parameter k. Samples that are more aligned with the center z have higher probability density, and vice versa. Here к. indicates the tightness of the distribution around the mean direction μ.. The larger value of i, the stronger the distribution concentrated in the mean direction. In the extreme case of K = 0, the sample points are distributed uniformly on the hypersphere.
When considering multiple classes, we can model the embedding space as a mixture of class- conditional vMF distributions, one for each class e 21,2..... C}:
pa(r; Mer No) Za(Ke)) exp (NoHt r)),
(3)
where Kg and ut are class-conditional parameters. Under this probability model, an embedding vector is assigned class with the following normalized probability:
Za (Ne) exp (Not4
ply = c|r; (Ky, (3 = Σ5 Za (1x) exp (k511 r)
(4)
Shaping representations. Our key idea is to design an end-to-end trainable loss function that enables ShapIng the RepresENtations into mixture of vMF distributions, which facilitates test-time OOD detection in the representation space (Section 3.2). We therefore name our method SIREN. The learned mapping funetion projects an input to point in the embedding space, where higher probability is assigned to the correct class in comparison to incorrect classes. To achieve this, we can perform maximum likelihood estimation (MLE) on the training data:
M argmax I Pludne (1
(5)
where i is the index of the object embedding and M is the size of the training set. By taking the negative log-likel,hood, the objective function is equivalent to minimizing the following loss:
1 M
Za (Rig.) exp (H..HTT
LSIREN
log
(6)
M
Σラ Zu (K₂) exp (NgH4]
where yi is the ground truth label for the embedding r, In effect, LSIREN encourages the object embeddings to be aligned with its class prototype, which shapes the representations such that objects in each class form a compact cluster on the hypersphere; see Figure I (b).
Prototype estimation and update. During training, SIREN estimates the class-conditional object prototypes e,,e c E 21,2,..., C). The conventional approach for estimating the prototypes to calculate the mean vector of all training samples (or subset of them) for each class, and update it periodically during training [74]. Despite its simplicity, this method requires alternating training and prototype estimation, which incurs heavy computational toll and causes undesirable latency. Instead, we update the class-conditional prototypes in an
(EMA) manner [34, 63]:
He = Normalize(ap,, + -a)r), (1,2...,CC, C),

---

Algorithm SIREN: Shaping Representations for object-level OOD detection
Input: ID training data Dir {xx,, b1, M
randomly initialized object detector and MLP
projection head with parameter 0, loss weight 3 for CSIRN,, and learnable {NoJE=)
Output: Object detector with parameter 0 and OOD detector G. while train do
1. Update class-conditional prototypes M4 with the hyperspherical embeddings r by Equation (7).
2. Calculate the vMF- based representation shaping loss LSIREN by Equation (6).
Update the learnable fKe ) and the network parameters 8 using Equation (8).
end
while eval do
1. Calculate the OOD score by Equation (9) or Equation (11).
2. Perform OOD detection by Equation (10).
end
where a is the prototype update factor, and denotes the normalized object embeddings from class c. The update can be done efficiently with negligible cost, enabling end-to-end training.
Overall training objective. The overall training objective combines the standard object detection loss, along with our new representation shaping loss LSIREN:
min b., A.*
[Lels Lioel + +B- LSIREN,
where /3 is the weight of our representation shaping loss. Lelss and Lioe are losses for classification and bounding box regression, respectively. We provide extensive empirical evidence in Section demonstrating the efficacy of our loss function.
To the best of our knowledge, our work makes the first attempt to explore vMFFbased learning and inference for object-level OOD detection. To highlight the novelty of the loss itself: we introduce a novel learnable )=1 for each ID class in Equation (6), instead of using fixed values. Our loss allows concentration parameter K to adaptively and flexibly capture the class-conditional feature statistics during training. This is desirable when each ID class may have its concentration in the hypersphere. We will later show that such Icarnable {N,] enables better OOD detection performance (Section 5).
3.2 Test-time OOD Detection
During inference, explore and contrast types of uncertainty scores for detecting OOD objects.
Parametric vMF score. We propose a new test-time OOD score based on the learned class- conditional vMF distributions, parameterized by {Re, MC -- Here Re denotes the Icarned concen- tration parameter for class c, which captures the concentration of representations for class c. For a test-time object (x', b'), we use the largest estimated dass-conditional likelihood the OOD score:
('x max (R(Re) exp (R.P.T))
where r'= ===<<(((x b')) is the normalized embedding from the MLP projection head. Our OOD detection score thus in principle suits our learned embeddings and vMF distributions. For OOD) detection,
the level set to distinguish between IDD and OOD objects:
if b') 2
G G b') 0
(10)
S b') <>
The threshold can be chosen so that high fraction of ID data (e.g.. 95%) is correctly classified. For objects classified as ID, one can obtain the bounding box and class prediction using the prediction head as usual.
Non-paramettr KNN score. To relax the distributional assumption on the learned embeddings, we additionally employ non-parametric KNN distance for OOD detection, which performs well compact and normalized feature space. Following Sun et al. [60], the KNN distance defined as:
S (x',b', b),k)
||r rwmll2>
(11)
where T(k) denotes the normalized embedding of the k-th nearest neighbor (in the training data), for the test embedding r'. Our algorithm is summarized in Algorithm 1.
5

---

Remark 1. Different from Mahalanobis distance [33], our parametric vMF-based OOD detection score operates under the same distributional model as the training process, and hence enjoys mathematical compatibility. Computationally, we can di- rectly use the parameters of vMF distri- butions (such as k) learned from training. In contrast, the Mahalanobis distance quires a separate test-time estimation of involves expen-
(4) WMF S00R0
(b) Mahalanobia distance
The uncertainty surface is calculated using vMF score (a) and the Mahalanobis distance (b). We showcase one class for visual clarity.
Figure 3:
feature statistics- sive and numerically unstable step of cal- culating the covariance matrix. The distinction of the uncertainty surface calculated by both our vMF score and the Mahalanobis distance is qualitatively demonstrated in Figure 3. The data points sampled from mixture of three elass-conditional vMF distributions (see details in Appendix F).
4 Experiments
In this section, we validate the effectiveness of SIREN on object detection models, including the latest transformer-base based (Section 4.1) and flagship CNN-based models (Section 4.2).
Datasets. Following [13], we use PASCAL-VOC' [14] and Berkeley DeepDrive (BDD 100K)² [73] datasets as the ID training data. For both tasks, we evaluate on two OOD datasets that contain a subset of images from: MS-COCO [37] and OPENIMAGES (validation set) [30]. Extensive details on the datasets are in Appendix A.
Metrics. For evaluating the OOD detection performance, report: (1) the false positive rate (FPR95) of OOD objects when the true positive rate of ID samples is at 95%; (2) the area under the receiver operating characteristic curve (AUROC). For evaluating the object detection performance on the ID task, we report the common metric mAP.
4.1 Evaluation on Transformer-based Model
Experimental details. We adopt very recent DEFORMABLE-DETT (DDETR) architecture [75]. DDETR introduces multi-scale deformable attention modules in the transformer encoder and decoder layers of DETR [4], and provides better convergence and lower complexity. The multi-scale feature maps in DDETR are extracted from ResNet-50 [21] pre-trained on ImageNet in a self-supervised fashion, i.e., DINO [5]. We use the embeddings the penultimate layer of the decoder in DDETR for projection. For the projection head, we use a two-layer MLP with ReLU nonlinearity, with dimensionality 256 - d + d. The dimension 64 for BDD BDD100K. The default weight / for the SIREN is 1.5 and the prototype update factor is 0.95. We initialize the learnable x to be 10 for all classes The in the KNN distance is set to 10. Ablations on the hyperparameters are provided in Section 5 and Appendix C. Other hyperparameters are the same as the default ones in DDETR [75].
the unit hypersphere is 16 for PASCAL- VOC and
SIREN achieves superior performance. In Table 1, we compare SIREN with competitive OOD detection methods in literature. For a fair comparison, all the methods only use ID data for training. SIREN outperforms competitive baselines, including Mahalanobis distance [33], KNN distance [60], CSI [61], Gram matrices [53] and Dismax [39]. These baselines operate on feature embedding space, allowing fair comparison. Note that other common output-based methods (such MSP [23], ODIN [35], and energy [38]) are not directly applicable for multi-label classification networks in DDETR. For these methods relying comparisons on the Faster R-CNN model in Section 4.2. Implementation details and the training time for all the baseline methods are reported in Appendix D and E. We highlight few observations:
multi-class classification model, we will later provide
1) The comparison between SIREN vs. Mahalanobis highlights precisely the benefits of our embedding shaping loss LSIREN. As shown in Figure/ 1, the vanilla DETR model produces ill-
"PASCAL-VOC consists of the following ID labels: Person, Car, Bicycle. Boat, Bus, Motorbike, Train, Airplane, Chair, Bottle. Dining Table, Potted Plant, TV, Sofa. Bird, Cat, Cow, Dog, Horse, Sheep. BDDD10OK consists of ID labels: Pedestrian, Rider, Car, Truck, Bus, Train, Motorcycle, Bicycle, Traffic light, Traffic sign.

---

In-distribution dataset
Method
FPR95
AUROC 1
mAP (ID)1
OOD: MS-COCO/ OpenImages
Mahalanobis [33]
97.39 /9.88 97.88
50.28 / 49.08
60.6
Gram matrices [53]
94,16/ 95.29
43,97 / 38.81
60.6
KNN [60]
91.80/ 91.36
62.15 59.64
60.6
CSI [61]
84.00 / 79.16
55.07 / 51.37
59.5
PASCAL-VOC
VOS [13]
97.46 97.07
54,40 / 52.77
60.3
OW-DETR [18]
93.09 / 93.82
55.70/ 57.80
58.3
Dismax [39]
82.05 / 76.37
75.21 / 70.66
60.1
SIREN-VMF (ours)
75.49:10.8 78.3668/7787111111 78.3611
76.10=10.1 100000.117110 71.05500
60.8:00,1
SIREN- KNN (ours)
64.77.10.2 65.99
78.23 74.933...
60.8:10.11
Mahalanobis [33]
70.86 71.43
76,83 / 77.98
31.3
Gram matrices [53]
73.81 71.56
60.13 / 57.14
31.3
KNN [60]
64.75 / 61.13
80,90 / 79.64
31.3
CSI [61]
70.27 71.30
77.93 / 76.42
29.9
BDD100K
VOS [13]
76.44 / 72.58
T7.33 / 76.62
31.0
OW-DETR [18]
80.78 / 77.37
70.29 / 73.78
28.1
Dismax [39]
77.62 / 81,23
72.14 / 67.18
31.2
SIREN-vMF (ours)
67.54111.3 1..9
80.06 79.77
31.31:0.0
SIREN-KNN KNN (ours)
53.97 47.28
86.561..1 / 0.4
31.31:0.0
Table I: Main results. Comparison with competitive out-of-distribute
detection methods. All baseline methods are based on the same model backbone DDETR. t indicates larger values are better and 4. indicates smaller values are better. All values are percentages. Bold numbers are superior results. We report standard deviations estimated across runs. SIREN-VMP/KNN denotes using vMF score and KNN distance during inference.
conditioned embeddings that do not conform multivariate Gaussian distributions, rendering the Mahalanobis approach ineffective (with AUROC around 50%-which is random guessing). In contrast, SIREN-vMF improves the OOD detection performance (AUROC) by 25.82% PASCAL-VOC (MS- COCO as OOD). Different from the Mahalanobis distance, our parametric vMF scoring function naturally suits the learned hyperspherical embeddings with vMF distributions. The advantage of the vMF loss can be further verified by observing that SIREN-KNN outperforms directly applying KNN distance the vanilla DDETR (16.08% AUROC improvement VOC with COCO OOD).
2) SIREN outperforms the latest methods VOS [13] and OW-DETR [18], which are designed for object detection models and serve strong baselines for us. Compared with OW-DETR, SIREN-KNX substantially improves the AUROC by 22.53% on PASCAL-VOC (COCO as OOD). OW-DETR uses the unmatched object queries with high confidence the unknowns, and trains binary classifier to separate ID and unknown objects. However, the unmatched object queries might be distributionally too close the ID classes and thus displays limited improvement for OOD detection. In addition, VOS synthesizes virtual outliers from the class-conditional Gaussian distributions of the penultimate layer but fails to perform well due the ill-conditioned embedding distribution in DDETR Gaus-Gaas)i
4.2 Evaluation on CNN-based Model
Going beyond detection transformers, we show that SIREN also suitable and effective on CNN-based object-level OOD detection mod- els, e.g., Faster R-CNN [52]. Table show- cases the OOD detection performance with SIREN trained on PASCAL-VOC dataset and evaluated on both MS-COCO and OPENIM- AGES datasets.
Method
FPR95 +
AUROC1 1 Time
COCO/ OpenImages
MSP ODIN
70.99/73.13
83.45 81.99
2.15 h
59.82/63.14
82.20/82.50
Mahalanobis
96.46/96.27
2.1 h
Energy score
56.89/58.69
83.69 82.98
Gram matrices
62.75/67.42
79.88
77.62
2.1 h
KNN
52.67/53.67
87.14/84.
84.54
CSI
59.91 /57.41
81.83
82.95
4.9 h
addition to basclines con- sidered in Table 1, we include common output- based methods relying on multi-class classifi- cation, such as MSP, ODIN, and energy score.
GyN-shnthesis
60.93/59.99
83.67
/ 82.67
3.7 h
VOS
47.53/51.3
88.70
85.23
4.3 h
Dismax
84.38 86.93
74.56
71.53
2.2 h
-TREN-vMF (ours)
64.68 68.53
85.36
82.78
SIREN KNN (ours)
47.45 50.38
89.67
88.80
2.1 h
In Table 2, we additionally report training time comparison. STREN consistently improves OOD detection performance on both OOD datascts. Notably, SIREN performs better than
Table 2: OOD detection results of SIREN and com- parison with competitive baselines on two OOD datasets: COCO and OpenImages.
7

---

7521
060
72D
200
213
16.44
114
11.0
71.5
15.03
26:66
Fosd aggga
pyparpaes diendes
1MM 1211
Figure 4: (a) Ablation study on using different fixed values of concentration parameter & in Equa- tion (12). (b) Ablation on different weights 3 for LSIREN in Equation (8). (c) Ablation on the
dimension of the hyperspherical embeddings r. (d) Ablation study on the parameter & in the KNN- based OOD detection score for SIREN. Numbers are AUROC. The ID training dataset is PASCAL-VOC and OOD dataset MS-COCO.
the previous best OOD detection approach VOS on Faster R-CNN while preserving the same training time vanilla Faster R-CNN.
Ablations and Discussions
In this section, we provide ablation results on how different factors impact the performance of SIREN. For consistency, we present the analyses below based on the DDETR model. Unless otherwise pointed, we use the KNN distance by default.
Ablations on learnable vs. fixed K. In this ablation, we show that our approach using learnable concentration parameters {Ni))== is better than using the fixed ones. Our method with learnable is desirable, since each ID class may have its own concentration in the hypersphere. With fixed k, the loss function can be simplified as follows:
1 M
exp [1ipe r2)
L -
log
(12)
Σラ= exp (Niaa] r:)
Empirically, we indeed observe that employing learnable fKo [n=)c- achieves better OOD detection performance, with AUROC 78.23% on PASCAL-VOC model (MS-COCO as OOD). In Figure 4 (a), we show SIREN's performance when trained under different fixed & values. The best model under the fixed 10 achieves an AUROC of 77.67%. Too small of value (e.g., 1) leads to almost uniform distributions and is therefore not desirable.
Ablations on the SIREN loss weight 3. Figure 4 (b) reports the OOD detection results as we vary the weight 3 for the representation shaping loss LSiREn- The model is evaluated on the MS-COCO dataset as OOD, Overall a mild weight works well. Across all the 3 values considered, SIREN consistently outperforms the baseline OOD detection methods in Table 1. One can use our STREN loss as an easy plug-in module, with minimal hyperparameter tuning.
Ablations on the dimension d of hy hypersphere. SIREN projects the object feature embeddings into a shows the effect the embedding dimension on the OOD detection performance. We find that a lower dimension between 16 and 64 achieves favorable and stable performance. In the extreme case with dimension d 8, the model suffers from considerable information loss and degraded performance. On the other hand, too large of d causes training instability, which is not desirable either.
hypersphere in IR,, which allows tractable vMF estimation. Figure 4 (c)
Ablations on the uncertainty score. We perform ablation on two variants of the vMF-based OOD detection score (c.f. Equation (9)): using the learned [Ka]C= vs. approximately estimate the parameters directly from the converged embeddings. In literature, there are several established methods for approximating [43, 55]. Denote F. the average of object embeddings for class the simplest approximate solution is given as follows:
F. (d
c
(13)
8

---

MP
Figure 5: Visualization of detected objects on the OOD images (from MS-COCO and OPENIMAGES) by vanilla DDETR (lop) and SIREN (bottom). The ID is BDD100K dataset. Blue: OOD objects classified as the ID classes. Green: OOD objects detected by DDETR and SIREN, which reduce false positives.
The proof is given in Appendix B. We show in Table 3 that using learned {Ke 1 avoids the imprecision in approximating k, directly from embeddings. This affirms the importance of employing learnable concentration parameters in SIREN loss. Moreover, the non-parametric KNN density estimation provides stronger flexibility and generality, and leads to better performance.
FPR95
AUROCT
COCO/ Openlnuges OOD
vMF w/ from [55] vMF w/ learned & (ours)
78.60 78.42
73.03 70.27
75.49 / 78.36
76.10 / 71.05
Non-parametric KNN distance
64.77 65.99
78.23 74.93
Table 3: Ablation different OOD detection scores. The ID dataset is PASCAL-VOC.
Ablations on the projection head. We ablate on the nonlincarity in the projection head by comparing with SIREN trained with a linear layer in Table The result shows using nonlinear mapping for projection helps obtain more expressive hypersphere, which improves OOD detection by 4.37% in terms of AUROC (MS-COCo as OOD, vMF the OOD score).
FPR95
AUROCT
COCO/ Openlmages as OOD
vMF w/o nonlinearity
82.85
83.69
71.73/ 66.82
vMF w/ nonlinearity
75.49 78.36
76.10 71.05
KNN w/o nonlinearity
67.03
72.11
78.02 72.42
KNN w/ nonlinearity
64.77
65.99
78.23 74.93
Table 4: Ablation on the projection head. The ID dataset PASCAL-VOC.
Qualitative analysis
In Figure 5, we visualize the predictions on several OOD images, using object detection models trained without SIREN (top) and with SIREN (bottom), respectively. The in-distribution data is BDD L00K. SIREN better identifies OOD objects (in green) compared to a vanilla object detector DDETR, reducing false positives. Moreover, the confidence score of the false-positive objects of SIREN lower than that of the vanilla model (see the train/bicycle in the 3rd/Sth column).
7
Related work
OOD detection for classification can be broadly categorized into post hoc and regularization-basse approaches [72]. [2), the OpenMax score is proposed for OOD detection based on the extreme value theory (EVT). Subsequent work [23] proposed simple baseline using maximum softmax confidence

---

(MSP), Several improvements have been proposed, such as deep ensemble [31], ODIN [35]. distance- based score [33, 51, 53, 60]. energy-based score [38. 49, 64]. DICE [59] and ReAct score [58]. While post hoc methods often considered OOD scoring function alone, our framework considers both representation learning (at training) and OOD detection (at testing).
Along the line of regularization- based approaches, most prior works focused on regularizing the model's output [3, 16, 22, 26, 28, 42, 44, 47, 48, 62, 66, 71J. For example, the model is regularized to produce lower confidence [24, 32] or higher energy [13, 38] on the outlier data. In contrast, SIREN regularizes the model by shaping the representations during training. [39, 40, 61] employed self- supervised learning and learnable prototypes to learn desirable representations for novelty detection, respectively, but they used complex output-based score ensembles during inference. which are not generalizable and perform sub-optimally (see Tables and 2). [57] shaped the latent representations as multivariate Gaussians but they trained a variational autoencoder and utilized the reconstruction error for open-set recognition. In contrast, STREN directly shapes the embeddings of discriminative-bun bassed model, and does not require training generative model.
OOD detection for object detection is a rising topic with very few existing works. For Faster R-CNN, Du al. [13] proposed to synthesize virtual outliers in the feature space for model regularization, which we compare in Section 4. [12] explored unknown-aware object detection by leveraging videos in the wild, whereas we focus on settings with still images only. For DETR, [18] adopted unmatched object queries that are with high confidence as unknowns, which did not focus on regularizing the model for desirable representations. Several works [9, 11, 19, 45, 46] used approximate Bayesian methods, such as MC-Dropout [15] for OOD detection. They require multiple inference passes to generate the uncertainty score, which are computationally expensive on larger datasets and models.
vMF distribution in ML has been adopted for supervised classification [4], face verification [20], generative modeling [8], segmentation [25] and clustering [17], etc. Some works exploited vMF distribution for anomaly detection, they employed generative models and used the vMF distribution as the prior for zero-shot learning [6] and document analysis [76]. Both the problem settings and approaches are orthogonal to SIREN. To the best of our knowledge, our work makes the first attempt to employ vMF-based learning and inference with learnable parameters for object-level OOD detection. Metric learning has been explored for both classification [10, 67, 70, 74] and object detection [27, 56, 68, 69]. Unlike ours, they did not focus on the OOD detection problem.
Conclusion
In this paper, we propose a novel framework SIREN, which tackles object-level OOD detection with a distance-based approach. SIREN mitigates the key shortcoming of the previous output-based OOD detection approach, and explores new vMF loss to shape representations for OOD detection. To the best of our knowledge, SIREN makes the first attempt to employ vMseb learning and inference for OOD detection. SIREN establishes competitive performance on challenging object-level OOD detection tasks, evaluated broadly under both the recent detection transformers and CNN- based models. Our in-depth ablations provide further insights the efficacy of SIREN. We hope our work inspires future research on OOD detection with representation shaping.
Broader Impacts
Our project aims to improve the reliability and safety of modern machine learning models. Our study can lead to direct benefits and societal impacts, particularly for safety-critical applications such as autonomous driving. Our study does not involve any human subjects or violation of legal compliance. We do not anticipate any potentially harmful consequences to our work. Through our study and releasing our code, we hope to raise stronger research and societal awareness towards the problem of out-of-distribution detection in real-world settings.
Acknowledgement
We thank Yiyou Sun and Ziyang (Jack) Cai for their valuable suggestions. The authors would also like to thank NeurIPS anonymous reviewers for their helpful feedback. The work supported by the AFOSR Young Investigator Program Award.
10

---

References
[1] Arindam Banerjee, Inderjit Dhillon, Joydeep Ghosh, and Suvrit Sra. Clustering on the unit hypersphere using von mises-fisher distributions. J. Mach. Learn. Res., 6:1345-1388 2005.
[2] Abhijit Bendale and Terrance E. Boult. Towards open set deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1563-1572, 2016.
[3] Petra Bevandić, Ivan Krešo, Marin Oršić, and Siniša Segvić. Discriminative out-of-distribution detection for semantic segmentation. arXiv preprint
2018.
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Proceedings of European Conference Computer Vision, pages 213-229, 2020.
[5] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in supe-viue vision transformers. In Proceedings of IEEE/CVF International Conference Computer Vision, pages 9630-96400 2021.
[6] Xingyu Chen, Xuguang Lan, Fuchun Sun, and Nanning Zheng. A boundary based out-of- distribution classifier for generalized zero-shot learning. In Proceedings of European Conference on Computer Vision, pages 572-588, 2020.
[7] Xixian Chen, Michael R. Lyu, and Irwin King. Toward efficient and accurate covariance matrix estimation on compressed data. In Proceedings of the International Conference on Machine Learning, pages 767-776, 2017.
[8] Tim R. Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M. Tomczak. Hy- perspherical variational auto-encoders. In Proceedings of the Thirty-Fourdh Fourth Conference Uncertainty in Artificial Intelligence, pages 856-865, 2018
[9] Kumari Deepshikha, Sai Harsha Yelleni, P. K. Srijith, and C. Krishna Mohan. Monte carlo dropblock for modelling uncertainty in object detection. CoRR, abs/2108.0 08.03614, 2021.
[10] Jiankang Deng. Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 4690-4699, 2019.
[11] Akshay Raj Dhamija, Manuel Günther, Jonathan Ventura, and Terrance E. Boult. The over- looked elephant of object detection: Open set. In Proceedings of IEEE Winter Conference Applications of Computer Vision, pages 1010-1019, 2020.
[12] Xuefeng Du, Xin Wang, Gabriel Gozum, and Yixuan Li. Unknown-aware object detection: Learning what you don't 't know from videos in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13678-13688, 2022.
[13] Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. Vos: Learning what you don't It know by virtual outlier synthesis. In Proceedings of the International Conference on Learning Representations, 2022.
[14] Mark Everingham, Lue Van Gool, Christopher K. I. Williams, John M. Winn, and Andrew Zisserman. The pascal visual object classes (VOC) challenge. International Journal of Computer Vision, 88(2)3303-338, 2010.
[15] Yarin Gal and Zoubin Ghahramani. Dropout bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of International Conference on Machine Learning, pages 1050-1059, 2016.
[16] Yonatan Geifman and Ran El- El-Yaniv. Selectivenet: A deep neural network with an integrated reject option. In Proceedings of International Conference on Machine Learning, pages 2151- 2159. PMLR, 2019.
[17] Siddharth Gopal and Yiming Yang. Von mises-fisher clustering models. In Proceedings of International Conference Machine Learning, pages 154-162, 2014.

---

[18] Akshita Gupta, Sanath Narayan, KJ Joseph, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. Ow-detr: Open-world detection transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9235-9244 2022.
[19] David Hall, Feras Dayoub, John Skinner, Haoyang Zhang, Dimity Miller, Peter Corke, Gustavo Carneiro, Anelia Angelova, and Niko Sünderhauf. Probabilistic object detection: Definition and evaluation. In Proceedings of IEEE Winter Conference Applications of Computer Vision, pages (020-1029, 2020.
[20] Md. Abul Hasnat, Julien Bohné, Jonathan Milgram, Stéphane Gentrie, and Liming Chen. von mises-fisher mixture model-based deep learning: Application to face verification. CoRR, (bss17006.04644) 2017.
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[22] Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 41-50, 2019.
[23] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and iit-of-distribution examples in neural networks. Proceedings of International Conference on Learning Represen- tations, 2017.
[24] Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In Proceedings of International Conference on Learning Representations, 2019.
[25] Jyh-Jing Hwang. Stella X. Yu, Jianbo Shi, Maxwell D. Collins, Tien-Ju Yang, Xiao Zhang, and diang-Chieh Chen. Segsort: Segmentation by discriminative sorting of segments. In Proceedings of IEEE/CVF International Conference on Computer Vision, pages 7333-7343, 2019.
[26] Taewon Jeong and Heeyoung Kim. Ood-maml: Meta-learning for few-shot ofi-t--diittiuutionn detection and classification. In Proceedings of Advances in Neural Information Processing Systems, pages 3907-3916, 2020.
[27] KJ Joseph, Salman Khan, Fahad Shahbaz Khan, and Vineeth N Balasubramanian. Towards open world object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5830-5840, 2021.
[28] Julian Katz-Samuels, Julia Nakhlch. Robert Nowak, and Yixuan Li. Training ood detectors in their natural habitats. In Proceedings of International Conference on Machine Learning, 2022.
[29] Takumi Kobayashi. T-vmf similarity for regularizing intra-class feature distribution. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages (616-6625,200 2021.
[30] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4. International Journal of Computer Vision, pages 1-26, 2020.
[31] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Proceedings of Advances Neural Information Processing Systems, pages 6402-644 6413, 2017.
[32] Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibrate classifiers for detecting outtt--disttibution samples. Proceedings of International Conference on Learning Representations, 2018
[33] Kimin Lee, Kibok Lee, Honglak Lee, and Jínwoo Shin. A simple unified framework for detecting distr--disttitttio samples and adversarial attacks. In Proceedings of Advances in Neural Information Processing Systems, pages 7167-7177, 2018.
12

---

[34] Junnan Li, Caiming Xiong, and Steven C. H. Hoi. Mopro: Webly supervised learning with mo- mentum prototypes. Proceedingd of International Conference on Learning Representations. 2021.
[35] Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of- distribution image detection in neural networks. In Proceedings of International Conference on Learning Representations, 2018.
[36] Tsung- Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. IEEE Trans. Puttern Anal. Mach. Intell., 42(2)318-327,, 2020.
[37] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proceedings of European Conference on Computer Vision, pages 740-755, 2014.
[38] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy- out-of-distribttion detection. Proceedings of Advances in Neural Information Processing Systems, pages 21464- 21475, 2020.
[39] David Macêdo, Cleber Zanchettin, and Teresa Bernarda Ludermir. Distinction maximization loss: Efficiently improving -ot----diitiiuttinn detection and uncertainty estimation simply replacing the loss and calibrating. CoRR, abs/2205.05877 2022.
[40] David Macêdo, Tsang Ing Ren, Cleber Zanchettin, Adriano L. I. Oliveira, and Teresa Luder- mir. Entropic out-of distribbutiii detection: Seamless detection of unknown examples. IEEE Transactions on Neural Networks and Learning Systems, 33(6):2350-2366 2022.
[41] Prasanta Chandra Mahalanobis. On the generalized distance in statistics. National Institute of Science of India, 1936.
[42] Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. In Proceedings of Advances in Neural Information Processing Systems, pages 7047-7058, 2018.
[43] Kanti V Mardia, Peter E Jupp, and KV Mardia. Directional statistics, volume 2. Wiley Online Library, 2000.
[44] Alexander Meinke and Matthias Hein. Towards neural networks that provably know when they don't know. In Proceedings of International Conference on Learning Representations, 2020.
[45] Dimity Miller, Feras Dayoub, Michael Milford, and Niko Sünderhauf. Evaluating merging strategies for sampling- uncertainty techniques in object detection. In Proceedings of International Conference on Robotics and Automation, pages 2348-2354, 2019.
[46] Dimity Miller, Lachlan Nicholson, Feras Dayoub, and Niko Sünderhauf. Dropout sampling for robust object detection in open-set conditions. In Proceedings of IEEE International Conference on Robotics and Automation, pages 1-7, 2018.
[47] Yifei Ming. Ying Fan, and Yixuan Li. Poem: Outtof-distribution detection with posterior sampling. In Proceedings of International Conference on Machine Learning, pages 15650- 15665, 2022.
[48] Sina Mohseni, Mandar Pitale, J. B. S. Yadawa, and Zhangyang Wang. Self-supervised learning for generalizable distr--iistiitutinn detection. Proceedings of AAAI Conference on Artificial Intelligence, pages 5216-5223, 2020.
[49] Peyman Morteza and Yixuan Li. Provable guarantees for understanding out-of-distribution detection. Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
[50] Julia Nitsch, Masha Itkina, Ransalu Senanayake, Juan Nieto, Max Schmidt, Roland Siegwart, Mykel J. Kochenderfer, and Cesar Cadena. Out-of-distribution detection for automotive per- ception. In Proceedings of IEEE International Intelligent Transportation Systems Conference, pages 2021.
13

---

[51] Jie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Lakshmi- narayanan. simple fix to mahalanobis distance for improving near-ood detection. CoRR, abs/2106.09022,, 2021.
[52] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-enn: Towards real-time object detection with region proposal networks. In Proceedings of Advances in Neural Information Processing Systems, pages 91-99, 2015.
[53] Chandramouli Shama Sastry and Sageev Oore. Detecting out-of-distribution examples with gram matrices. In Proceedings of International Conference Machine Learning, pages 8491-8501, 2020.
[54] Tyler R. Scott, Andrew C. Gallagher, and Michael C. Mozer. von mises-fisher loss: An exploration of embedding geometries for supervised learning. In Proceedings of IEEE/CVF International Conference on Computer Vision, pages 10592-10602, 2021.
[55] Suvrit Sra. A short note on parameter approximation for von mises-fisher distributions: and fast implementation of I s (x). Comput. Stat., 27(1):177-190, 2012.
[56] Bo Sun, Banghuai Li, Shengcai Cai, Ye Yuan, and Chi Zhang, FSCE: few-shot object detection via contrastive proposal encoding. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 7352-7362, 2021.
[57] Xin Sun, Zhenning Yang, Chi Zhang. Keek Voon Ling, and Guohao Peng. Conditional gaussian distribution learning for open recognition. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13477-13486, 2020.
[58] Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activations. In Proceedings of Advances in Neural Information Processing Systems, pages 157, 2021.
[59] Yiyou Sun and Yixuan Li. Dice: Leveraging sparsification for outto--distri detection. In Proceedings of European Conference on Computer Vision, 2022.
[60] Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. nearest neighbors. In Proceedings of International Conference on Machine Learning, pages 20827-20840, 2022.
detection with deep
[61] Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. In Proceedings of Advances in neural information processing systems, volume 33, pages 11899-1 1 852, 2020.
[62] Joost van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using single deep deterministic neural network. In Proceedings of the International Conference Machine Learning, pages 1590-9700, 2020.
[63] Haobo Wang, Ruixuan Xiao, Yixuan Li, Lei Feng, Gang Niu, Gang Chen, and Junbo Zhao. Pico: Contrastive label disambiguation for partial label learning. In Proceedings of International Conference on Learning Representations, 2022.
[64] Haoran Wang, Weitang Liu, Alex Bocchieri, and Yixuan Li. Can multi-label classification networks know what they don't 't know? In Advances in Neural Information Processing Systems, 2021.
[65] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In Proceedings of the International Conference on Machine Learning, pages 9929-9939, 2020.
[66] Hongxin Wei, Renchunzi Xie, Hao Cheng. Lei Feng, Bo An, and Yixuan Li. Mitigating neural network overconfidence with logit normalization. In Proceedings of the International Conference Machine Learning, pages
2022.
[67] Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A discriminative feature learning approach for deep face recognition. In Proceedings of European Conference on Computer Vision, pages 499-515, 2016.
14

---

[68] Aming Wu, Yahong Han, Linchao Zhu, and Yi Yang. Universal-protofes enhancing for few-shot object detection. Proceedings of IEEE/CVF International Conference on Computer Vision, pages 9547-9556, 2021.
[69] Yongyu Xiong, Peipei Yang, and Cheng-Lin Liu. One-stage open set object detection with pro- totype learning. In Proceedings of International Conference on Neural Information Processing, pages 779-291, 2021.
[70] Hong-Ming Yang, Xu- Yao Zhang, Fei Yin, and Cheng-Lin Liu. Robust classification with convolutional prototype learning. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 3474-3482, 2018.
[71] Jingkang Yang, Haoqi Wang, Litong Feng, Xiaopeng Yan, Huabin Zheng, Wayne Zhang, and Ziwei Liu. Semantically coherent out- cittof--isttiibutin detection. In Proceedings of the International Conference on Computer Vision, pages 6281-8289, 2021.
[72] Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey. arXiv preprint arXiv:21 10.14 2021.
[73] Fisher Yu, Haofeng Chen, Xin Wang. Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. BDD100K: A diverse driving dataset for heterogeneous multitask learning. In Proceedings /EEE/CVF Conference Computer Vision and Pattern Recognition, pages
2020.
[74] Xuefei Zhe, Shifeng Chen, and Hong Yan. Directional statistics-based deep metric learning for image classification and retrieval. Pattern Recognit. 93:113-123, 2019.
[75] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable DETR: deformable transformers for end-to-end object detection. In Proceedings of International Conference on Learning Representations, 2021.
[76] Honglei Zhuang, Chi Wang, Fangbo Tao, Lance M. Kaplan, and Jiawei Han. Identifying semantically deviating outlier documents. In Proceedings of Conference Empirical Methods in Natural Language Processing, pages 2748-2757, 2757, 2017.
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Have you read the ethics review guidelines and ensured that your paper conforms to
them? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 8.
(d) Did you describe the limitations of your work? [N/A]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments...
(a) Did you include the code, data, and instructions needed reproduce the main perimental results (either in the supplemental material or as a URL)? [Yes] Code is released at https: / //g thub. com/ caappllarninn---ee wisgg/wwssssiiree
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Section 4 and Appendix A and C.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] See main results Table 1.
(d) Did you include the amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix G.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section 4.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "nPbmCQVnuQO", "reviewer": "Reviewer_aJVX", "summary": "This paper proposes a novel OOD detection framework SIREN relied on the vMF distributions, which unifies representation learning and OOD detection. Detailed experiments and solid ablation studies demonstrate the effectiveness of the proposed method.", "questions": "1. This article select the vMF distribution to shape the image embedding, but does not clarify why the vMF distribution is the chosen one, and whether any Gaussian-like distribution is feasible?", "limitations": "Limitations have been adequately addressed. No serious concerns.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n1. This paper is organized well and the oveall representation is sophisticated.\n2. The method is plausible, and the experimental results perform well.\n3. Extensive experiments and detailed ablation analysis.\n4. The theoretical proof of the pivotal distribution deviation vector K has been explained in the appendix.\n\n**Weaknesses**:\n1. The originality is a bit limited, and the impact on the field is also incremental.\n\n       Let's briefly summarize this paper, the authors introduce a pretext task of shaping the final image map, then exploit the corresponding maximum class-conditional likelihood to classify whether the current image is OOD or not. Thus, this apporach can still be coverd as an OOD method based on uncertainty and fixed thresholds.\n\n2. Tets-time OOD dectection can hardly be highligthed as a new contribution, casue it is more like a test solution specific to this method.\n\n3. Updating the mean of a distribution by using prototype is not fresh for me, in fact, it has appeared in the following paper:\n\n    > Wang, Haobo, et al. \"PiCO: Contrastive Label Disambiguation for Partial Label Learning.\" arXiv preprint arXiv:2201.08984 (2022).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "YYvWJ_t8j3", "reviewer": "Reviewer_d8dM", "summary": "The paper's main assumption is that we need to construct representations during training that are compatible with the detection distributions we observe during inference.\n\nUnlike the Mahalanobis approach, which is an inference-only solution because it does not affect the network's training, the paper proposes constructing a loss to train the network to produce consistent distribution in training and test sets.\n\nIt proposed to train the network to impose the representations to follow a mixture of von Mises-Fisher (vMF) distributions and to make both train and test distributions compatibles to improve OOD detection performance.\n", "questions": "We would like the authors to update the paper considering the comments presented on the above topics.", "limitations": "The paper does not properly comment on the approach's limitations. Unlike many other cited approaches, it does not allow pure end-to-end backpropagation training. It also should better explain the hyperparameter validation procedures required for the alpha, beta, and the dimension of the last layer.", "rating": 2, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Weaknesses:**\n\n_\"Shaping representations\" and \"leverage from the learned representations\" is what loss-based approaches (IsoMax, Scaled Cosine, SNGP, DUQ, and IsoMax+) have been doing since 2019. The fact that these loss-based approaches for OOD detection were not even cited may explain why the paper understand \"shaping representations\" and \"leverage from the learned representations\" as a significant novelty. Unfortunately, it is clearly not the case._ \n\n_Unfortunately, the paper entirely ignores the last three years of relevant advances in loss-based approaches that are completely reshaping the area after the Mahalanobis inference-based approach faded out. The paper insistently compares against Mahalanabos, a four years old approach that has been outperformed very easily in the last three years._\n\n**1. Making representation compatible with the OOD Detection procedure is at least three years old.** \n\nThe Mahalanobis paper is from 2018. At that time, inference-based metric learning approaches for OOD were trending. Since 2019, we have observed that loss-based methods are now mainstream. Therefore, training the model using a loss to make training representation and test/inference distribution compatible, which is the central claim of the paper's novelty, is not novel at all.\n\nUnfortunately, the paper lacks to realize that training the model to produce train and inference representations compatible with performing OOD has been the mainstream approach since 2019 and was proposed by IsoMax, Scaled Cosine, DUQ, SNGP, and more recently by DisMax. Kapa appears similar to essentially the entropic scale of IsoMax.\n\n**2. Forcing representation to the unit hypersphere is not novel.**\n\nForcing representation to the unit hypersphere during training to improve OOD detection was done in Scaled Cosine and IsoMax+, and more recently by DisMax. \n\n**3. Very limited novelty.**\n\nConsidering that point #1 above showed that leveraging representations/distributions learned during training to perform OOD detection has already been proposed. Moreover, noticing that point #2 above showed that unit hypersphere representations have also already been proposed, we conclude that the novelty of the proposed approach is very limited.\n\n**4. Neither cited nor compared with similar previous approaches.**\n\nConsidering that the paper essentially proposes a loss to train the network to improve OOD detection, we believe that the paper should have mentioned and compared with (at least) IsoMax, Scaled Cosine, DUQ, IsoMax+, and SNGP. Considering that this did not happen, we do not know if using vMF helps in any way.\n\n**5. Results for CIFAR10 do not look impressive.**  Apparently, the results for CIFAR10 are not remarkable compared to the approaches mentioned above (a direct comparison is absolutely mandatory). We would also like to see results for CIFAR100. \n\n**6. Results for CIFAR10 need to show classification accuracy.** We also would like to see classification accuracy results to evaluate a possible classification accuracy drop, which is very usual in loss-based approaches.\n\n**7. We need much more experiment results using CIFAR10. Furthermore, we need CIFAR100 results as well.** CIFAR10 and CIFAR100 still are the _de facto_ standard datasets on which major OOD detection approaches have published results. The paper requires much more experimentation using CIFAR10 and CIFAR100 datasets. Classification accuracy need also to be shown and analyzed, as classification accuracy drop is a significant issue when using loss-based approaches for OOD detection.\n\n**8. Unlike recent approaches, the proposed solutions do not allow regular pure end-to-end backpropagation-based training.**\n\n**9. The proposed approach presents much more hyperparameters than current state-of-the-art approaches.**\n\n**10. Results should show mean and standard deviations.**\n\nIsoMax: https://arxiv.org/abs/1908.05569\n\nIsoMax (journal): https://arxiv.org/abs/2006.04005\n\nScaled Cosine: https://arxiv.org/abs/1905.10628\n\nDUQ: https://arxiv.org/abs/2003.02037\n\nSNGP: https://arxiv.org/abs/2006.10108\n\nIsoMax+: https://arxiv.org/abs/2105.14399\n\nDisMax: https://arxiv.org/abs/2205.05874\n\nAfter rebuttal: I will keep my score because the differences between the proposed model and the cited training-based approaches are not significant. No direct comparison against the methods we presented was shown. Finally, the majority of the concerns we presented were not covered in the rebuttal. Neither citing nor comparing against the previous related work for the last four years is hard to accept.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XiUPC8GQW79", "reviewer": "Reviewer_azqk", "summary": "This paper shows that the representation embeddings in the transformer-based models  do not fit the Gaussian distributional assumption. As a result, previous OOD detection methods relying on such assumptions can not work. \n\nThis paper proposed the method (SIREN) to shape the representations into compact class-conditional vMF distributions, which unifies the distributional model between training and testing. And the proposed method is effective for both transformer-based and CNN-based models.\n", "questions": "Please see the \"strengths and weakness\".", "limitations": "This paper didnot describe the limitations of this work. Every paper has limitations and corner cases. I recommend the authors to supplement the limitations of this work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths：\n\n- SIREN shapes the representations for OOD detection under a coherent distributional model, which does not depend on the distribution of model representation embedding.\n- SIREN unifies the distributional model between training and testing for OOD detection and shows effectiveness for both transformer-based and CNN-based models.\n\nWeaknesses and questions：\n\n- Why choose the vMF distributions as the coherent distributional model, rather than other distributions.\n\n- The performance on CNN-based object detection is insufficient (table 2), and the author needs to provide more results. What's more, the VOS performs better than SIREN in CNN-based object detection, which needs some explanation.\n\n  | Method | FPR95       | AUROC       |\n  | ------ | ----------- | ----------- |\n  | SIREN  | 65.45/67.33 | 84.60/81.51 |\n  | VOS    | 49.02/52.11 | 88.66/85.36 |\n\n- From figure 6, the SIREN improves the embedding quality, does this influence the performance in ID data？ And it's better to show the performance under the setting about: fixed detector and only training the projection head.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept