Contrastive Class Anchor Learning for Open Set Object Recognition in Driving Scenes
Authors: Zizhao Li
Kourosh Khoshelham
Joseph West
Venue: Accepted by TMLR ()

Abstract: Conventional object recognition models operate under closed-set assumptions presuming that the training dataset is sufficiently comprehensive that any object detected during inference can be assigned to some known prior class. This assumption is flawed and potentially dangerous for real-world applications such as driving scene perception where diverse objects and unexpected behaviours should be expected.
In order to progress towards trusted autonomous platforms object recognition models need Open Set Recognition (OSR) methods
capable of identifying unknown classes while maintaining good performance on known classes. 
Existing OSR methods are mostly designed for image data and utilize generative models which are hard to train. In this paper, we propose S2CA, a Supervised Contrastive Class Anchor learning method which leverages contrastive learning principles to effectively reject unknown classes by increasing intra-class compactness and inter-class sparsity of known classes in feature space. We train a feature encoder through contrastive learning while ensuring that features of known classes form compact clusters, and then transfer the trained encoder to the OSR task. During inference, the model rejects unknown classes based on class-agnostic information in feature space and class-related information in logit space. The proposed OSR method is simple yet powerful. It is not only suitable for image-based object recognition models, but can also be used for a variety of lidar-based object recognition models. We demonstrate superior performance of S2CA when compared with state of the art methods on two widely used driving scene recognition datasets, i.e., KITTI and nuScenes.

Published in Transactions on Machine Learning Research (09/2024)
Contrastive Class Anchor Learning for Open Set Object Recognition in Driving Scenes
Zizhao Li
zishao.li lil iistudent.unimelte edu..ou
The University of Melbourne
Kourosh Khoshelham
The University of Melbourne
Joseph West The University of Melbourne
joseph. weapghh.wwes@@nimmmlbbeh
Reviewed on OpenReview: https: // openreview. net/ forum? id= 100um9SJgM
Abstract
Conventional object recognition models operate under closed-set assumptions presuming that the training dataset is sufficiently comprehensive that any object detected during infer- ence can be assigned to some known prior class. This assumption is flawed and potentially dangerous for real-world applications such as driving scene perception where díverse ob- jects and unexpected behaviours should be expected. In order to progress towards trusted autonomous platforms object recognition models need Open Set Recognition (OSR) meth- ods capable of identifying unknown classes while maintaining good performance on known classes. Existing OSR methods are mostly designed for image data and utilize generative models which are hard to train. In this paper, we propose S2CA, a Supervised Contrastive Class Anchor learning method which leverages contrastive learning principles to effectively reject unknown classes by increasing intra-class compactness and inter-class sparsity of known classes in feature space. We train feature encoder through contrastive learning while ensuring that features of known classes form compact clusters, and then transfer the trained encoder to the OSR task. During inference, the model rejects unknown classes based on class-agnostic information in feature space and class-related information in logit space. The proposed OSR method is simple yct powerful. It is not only suitable for imago-based object recognition models, but can also be used for variety of lidar-based object recog- nition models. We demonstrate superior performance of S2CA when compared with state of the art methods on two widely used driving scene recognition datasets, Le., KITTI and muScenes. The source code is available at /attps://gitiin com com/344gl1ysppp//22ccc
1 Introduction
Autonomous driving has received a great deal of attention in recent years due potential for eliminating human error, which is the major cause of most road accidents. To ensure safe and efficient navigation, autonomous vehicles must be able to perceive their surroundings using onboard sensors and understand the driving scene. This involves localizing and recognizing surrounding objects.
Conventional object recognition models typically make the assumption of a closed set, requiring that all object categories which the model will encounter are available in the training set (Scheirer el. al. 2013). When the recognition model encounters an unknown object, will misclassify as one the known classes. In the real world, autonomous vehicles encounter wide varioty of objects. It is not feasible to include all object categories in the training set. For example, autonomous driving datasets designed for the northern hemisphere do not typically contain kangaroos, yet in Australia kangaroos frequently appear the road.

---

Published in Transactions on Machine Learning Research (09/2024)
Open-set Recognition
Contrastive Class Anchor Learning
Known Class
Unknown Class
Aftraction
Repulsion
Figure 1. Our proposed approach to open-set recognition aims to effectively recognise unknown objects while correctly classifying known classes by increasing intra-class compactness and inter-class spansity in the feature space, thereby reducing the chance of mixing known and unknown classes. This is achieved by supervised contrastive learning (SCL) which encourages intra-class attraction and inter-class repulsion. To overcome the limitation of the original SCL, which contrasts the instances only in the same mini-batch, introduce Contrastive Class Anchor Learning, which applics global constraint contrastive learning by placing class anchors in the logil space. The proposed method is efficient and suitable for both image-based based and fidarbbasd object recognition.
Therefore, the closed-set assumption will inevitably have an adverse impact on the autonomous driving system.
An ideal driving scene perception system should be able to detect all object instances the road surface, then reject unknown objects and classify the known ones. Unknown objects can then be stored for later labelling, e.g. by a human supervisor. [Scheirrr et al. (2013) define the problem of rejecting unknown classes (U) and simultancously classifying known classes (K) as open-sct recognition (OSR). Unknown classes are sometimes referred to as "unknown unknowns"., because they are not only unseen but also have no side information (e.g., semantic attributes or known properties) during training (Oza & Patel 2019 Geng al. 2021).
An intuitive OSR method is to set threshold for the classification confidence score (e.g. softmax probability)
of the most probable class produced by the model (Hendrycks & Gimpel 2017). However, previous research
has shown that samples in U can also produce equally high softmax scores [Bendale & Boult 2015). State-
of-the-art OSR methods often involve generative models (Ge et al., 2017 Neal et al.) 2018 Oza & Patel
2019 Perera al. 2020 Kong & Ramanan 2021), which help separate known and unknown classes
the feature space. While improved feature learning leads to better OSR performance, the complexity and challenges of training generative models limit their application in open-set recognition in diverse scenarios. Miller al. [2021) proposed Class Anchor Clustering (CAC), which forces known classes to form compact clusters around class anchors in logit space and rejects unknown objects based on their distance to known clusters. However, the cluster compact ness achieved by minimizing the CAC loss can limit, the network's 's ability to learn discriminative features (Miller et al.) 2021), which is critical for achieving acceptable OSR performance (Perera et al. (2020). A further limitation of existing OSR methods relates their application single data modality, i.c., RGB images, whereas OSR performance with other data modalities, such lidar, has not. been evaluated in previous work.
2

---

Published in Transactions on Machine Learning Research (09/2024)
To overcome the limitations of existing OSR methods, we propose new contrastive learning framework which can effectively distinguish unknown objects from known classes in different. data modalities. We achieve intra-class compactness and inter-class sparsity in the feature space by training a feature encoder through contrastive learning while minimizing the class anchor clustering loss (Miller et al. 2021). During inference, the model rejects unknown classes by thresholding the virtual logit. score (Wang et al. 2022
Fig. 1 illustrates the proposed Supervised Contrastive Class Anchor (S2CA) learning approach to the OSR task. We evaluate the proposed S2CA approach with a wide range of image- based and lidar-based encoder networks, and compare its performance with state-of-the-a open-set recognition and out-of-distribunt (OOD) detection methods on two widely used datasets of KITTI (Geiger et al. 2013) and nuScenes (Caesar et al. 2019).
Our paper makes the following contributions:
We propose novel OSR method based on contrastive class anchor learning which enables effective rejection of unknown classes by increasing intra-class compactness and inter-class sparsity of known samples.
We demonstrate that the proposed method can be applied to different data modalities and encoder networks including image-basod and lidar-basod networks.
We present thorough performance evaluation and comparison with a wide range of OSR methods, which shows the superior performance of the proposed S2CA method.
2
Related Work
2.1
Open-set Recognition
Open-set recognition has long been an attractive topic in the computer vision community. Early studies Scheirer et al. 2013: 2014 JJain al.) 2014; Scherreik & Rigling, 2016) often involve traditional machine learning models, such SVM, and use the probabilistic methods threshold the prediction score.
OpenMax (Bendale & Boult 2015) is one of the first OSR methods based on deep neural networks, which determines the center of each known class in the logit space and then creates a statistical model of the distances between correctly classified samples.
uses extreme value theory (EVT) to detect outliers by fitting weibull function to the tail of the distance distribution. Many subsequent works involve genera- tive models. G- G-OpenMax (Ge al.) 2017) trains a Generative Adversarial Network (GAN) to synthesize open-set samples and uses these to estimate open-set class activations. Ncal et al. (2018) propose data augmentation technique called counterfacutal image generation. This approach uses GANs to generate im- ages that are visually similar to known ones, but are actually from unknown classes. The network is trained to have low confidence scores for these "known unknowns". Kong & Ramanan (2021) propose OpenGAN, which is suitable for both image classification and somantic segmentation. Generative models can do more than synthesize known unknown classes. In (Oza Patel 2019), the aulo-encoder only learns leature repre- sentations from known classes, which leads to higher reconstruction errors for unknown classes. Yoshihashi et al. 2019 integrate classification and input reconstruction in model training, which enhances the learned feature representation. Perera et al. (2020) combine the auto-encoder with multi-task classifier trained by minimizing a self-supervision loss and a classification loss. The self-supervision learning branch incorporates random transformations to improve the descriptiveness of feature representations. Miller et al. (2021) pro- pose a simple OSR method, which forces known classes to form tight clusters around class anchors in logit space and reject unknown objects based on their distance to known classes. While this model is easier train than the previous generative models, anchoring class centers equal distances can restrict the lcarning of semantically meaningful features (Miller al.) 2021).
2.2 Out-of lott-of-dise distribution Detection
Detection of out-of distribbbti (OOD) samples is concept closely related to OSR, and the two terms are sometimes used interchangeably. However, OOD detection is slightly different from OSR in that OOD
3

---

Published in Transactions on Machine Learning Research (09/2024)
samples can be completed unrelated to the training set, whereas OSR aims to reject valid but unknown classes GGillert & von Lukas 2021
OOD detection methods usually map the imput x onto a single scalar that indicates OOD-ness. Assuming that the network is more confident about known objects, the OOD-ness leads to softmax scores that are higher for known classes than the unknown ones, making it a baseline OOD detection method (Hendrycks) & Gimpel| 2017). Unknown objects tend to have a uniform softmax distribution, leading to higher entropy (Chan et al. )021). Liang et al. (2018) propose ODIN, which adds small perturbations to the input thereby increasing the softmax score gap between known and unknown objects. Hendrycks et al. (2022) use the maximum logit as the OOD score, and show that it outperms the softmax baseline (Hendrycks & Gimpel. 2017). Kullback Leibler divergence between the softmax probabilities and class templates has also been used as the OOD score (Hendrycks et al.] 2022). Lit et al. (2020) measure OOD-ness of objects based on the Helmholtz free energy of the network output. Lee et al. (2018) use the Mahalanobis distance (MD) between the feature vector and the closest class-condition conditional Gaussian distribution to mcasure OOD-ness. [aang et al.) (022)) propose Virtual-logit Matching (ViM), which combines class-agnostic information in the fealure space and the class-dependent information in the logit space for OOD detection.
2.3 Contrastive Learning
Contrastive learning aims to learn a representation of data such that similar instances are close together in the representation space, while dissimilar instances are far apart. He et al. (2020) propose the Momentum Contrast (MoCo) for large-scale unsupervisod learning, which converts the image-based contrastive learning into dictionary look-up problem. Chen et al. (2020) propose SimCLR, which maximizes the similarity between augmented versions of the same image (posilive pairs) and minimizes the similarity between dilfer- ent images (negative pairs). SimCLR employs an MLP projection head after the encoder network, which introduces non-lincar transformations to the feature space. It also incorporates a normalization stop after the projection head, which ensures that the representations have unit length, facilitating the use of cosine similarity to measure similarity between pairs. Khosla et al. (2020) propose Supervised Contrastive Learning (SCL), which extends SimCLR (Chen et al., (020) to fully supervised scenarios. SCL contrasts all samples from the same class positives against the negatives from the remainder of the batch, thereby forming tight class clusters in the feature space. Contrastive learning methods (He et al. 2020; Chen et al., 2020; Khosla et al. 2020) usually train an encoder with contrastive loss first, then freeze the encoder and train a linear classifier. [Wang et al. (2021) argue that this two-stage learning may not be the most effective ap- proach in a fully supervised setting, as it can reduce the compatibility between the feature encoder and the classifier. To address this issue, Wang et al. (2021) propose a hybrid framework that allows simultancous learning of features and classifiers for long-tail image classification. Ming et al. (2023) show that contrastive learning reduces intra-class sparity and improves inter-class repulsion in the feature space, which benefits distance- OOD detection.
3 Proposed Method
Given the various data modalities and the different model architectures used for object recognition in driving scenes, we propose perform OSR in feature space. This makes our method elficient and applicable to different data modalities and encoder networks.
Fig. 2 shows an overview of the proposed S2CA framework. We train a fcature encoder with Contrastive Class Anchor learning, which combines Supervised Contrastive Learning (SCL) (Khosla et al. 2020 and Class Anchor Clustering (CAC) (Miller et al.) 2021). Contrastive Class Anchor learning aims to train the encoder so that the known classes show intra-class compactncss and inter-class sparsity in the feature space. In this way, feature representations of known and unknown classes are separated, allowing us to model the distribution of known classes and reject unknown classes.
4

---

Published in Transactions on Machine Learning Research (09/2024)
Model Training
Contrastive Leaning Branch
SCL Loss
Angmentatiin
fe
Embeuldine
Classifier Learning Branch
Known classes
fc
CAC Loss
Embedding
Logit S
2. Virtual Logit Computation
3. Model Inference
OOD Score
K
fe
Unknown
fe
fo
Closed-set Posteriors
Known classes
Figure 2. Overview of the proposed S2CA framework. We use shared encoder fe to extract feature embeddings from the closed set and feed these supervised contrastive learning branch (SCL) and a CAC loss based classifter learning branch (step 1). The contrastive learning branch contains non-linear MLP combined with Ig-normalization, which translate the feature embeddings for contrastive loss. The classifier learning branch a single linear layer fe; trained by the CAC loss. SCL is batch contrastive approach. For each instance, SCL contrasts all instances (including the augmented instance) from the same class as positives against negatives from the remainder of the batch. Therefore, create augmented pairs for batch of training samples of size B from the known classes, and the subsequent embeddings are also in pairs. A curriculum is designed to control the weight of these two branches, i.e., 5 and - 8, during model training. Once training complete, we compute the distribution of virtual logits lu (step 2) based on the projection embeddings ||h*1] which indicate O0D-ness. During model inference (step 3), we replace fa with N +1 classifier fe, which produces closed-set posteriors and virtual logit 4u- We threshold the virtual logit-based OOD score to reject unknown samples and classify the known ones using the normal logits.
3.1 Model Training
S2CA consists of contrastive learning branch and CAC loss based classifier learning branch. They share the same encoder fe but optimize different objective functions. The two branches together constitute Contrastive Class Anchor Learning.
3.1.1 Contrastive Learning Branch
Supervised Contrastive Learning (SCL) (Koolla et al., 2020 encourages objects the same class to be tightly clustered in the feature space, while clusters belonging to different. classes are far apart. These charactoristics are beneficial to the OSR task. The intra-class compactness of known classes helps maintain robust recognition performance for known objects, while the inter-class sparsity makes it easier to reject unknown objects based on their distance to the known classes.
Let x and denote the input instance and the corresponding label, with E N known classes. For cach instance, S2CA creales Iwo augmented views, ã; and ăj, of x from a predefined family T. For image input, augmentation may include random cropping, random flipping and color jittering (for range images, we only use random flipping). For lidar point clouds, we apply geometric transformations such as random translation, random rotation and random scaling. For mini-batch of size B, the corresponding augmentations are denoted { and (7,)
5

---

Published in Transactions on Machine Learning Research (09/2024)
When is passed to the encoder network fe(-), it produces the feature embedding h = fe()). Then the MTP projector s(-) maps h to a 128-dimensional vector. Applying l2 normalization to this vector gives a normalized embedding 2. The projector g(-) itself is invariant to data transformation, while it helps the encoder f-)-) preserve richer information for downstream tasks (Chen et al.) 2020). The SCL loss (Khosla jet al. 2020) used maximize the agreement between { and {2 for instances of the samne class. For each vector 2i, the SCL. loss
exp (z; + 23/T)
Coup
log
(1)
2)E{2!} Ekss exp (2s zk TT
where {x+²) represents all instances the batch that have the same label ūt K is the number of instances in {2+t) }. and I is a temperature scaling parameter.
Accordingly, the loss for each mini-batch is defined as:
2B
LSCL
sup
(2)
3.1.2 Classifier Learning Branch
In the classifier learning branch, the feature embedding h is passed to linear classification layer fo produce the N-class logit vector The CAC loss (Miller et al. 2021 defined based on the distance between the logit vectors and a set of anchored class centers C' (c.. ,n) with predefined magnitude a set equally for all classes. Let d denote a vector of Fuclidean distances between and each anchored class center in C, and f(-) denote the classification function, the CAC loss is written as:
B
LCAC = log(1 + Σ -)) ds) +1)l(=)) Gyll2
(3)
jøy
where À is a hyperparameter to balance the two loss terms.
The first term of CCCC is modified from the tuplet loss (Sohn 2016 It forces s to maximize the difference between the distance the correct class center and the distances to all other class centers. The sccond term minimizes the distance between and the correct class center.
3.1.3 Transition from Contrastive Learning to Classifier Learning
To ensure compatibility between the feature encoder and the classifier, we follow a curriculum strategy (Zhou ct al. 2020) adjust the weighting of the two branches. The model will first focus on supervised contrastive learning (Khosla al. (2020) and then smoothly transitions classifier learning as the training progresses.
The final objective function is the combination of Fq. (2) and Eq. 33
CS2CA = 8LSCL + (1- 8)C8C
(4)
Note that LCaC in Eq. 4 is the sum of each individual loss in the mini-batch (like the batch form of the SCL loss shown in Eq. 2 8 is a parameter that decreases as the training progresses. For each epoch 4, 8 = 1 kmag optimizing the SCI loss to learn semantically meaningful features. As the training progresses, 8 is reduced and the model smoothly shifts its focus to the CAC loss, which fine-tunes the model from global perspective.
fi where tmax is the maximum epoch number. Initially, is large, and the model focuses
The combination of SCL and CAC has several advantages. While using the CAC loss alone can restrict the discriminative power of the learned features (Miller et al.) 2021), the SCL helps the encoder learn a richer feature space. The SCL also benefits from the CAC. As a batch contrastive algorithm, SCL only contrasts samples in the same mini-batch. To achieve better performance, SCL requires larger batch sizes more

---

Published in Transactions on Machine Learning Research (09/2024)
training epochs, which increases the computational complexity of the training (Khosla et al. 2020 The CAC mitigates this by comparing each individual sample with its corresponding class anchor. In each epoch, all training instances are pulled to the corresponding class anchors to form tight clusters. This results in faster convergence of contrastive learning with smaller batch sizes.
3.2 Model Inference
For model inference, we use virtual logit (Wang et al. 2022: to extract information in the feature space and combine it with ordinary logit to form the OOD score and reject unknown objects. Unlike the distance- based rejection score used by Miller et al. (2021), which ignores the class-agnostic information the feature space, the virtual logit matching (ViM) combines the class-agnostic information the feature space with the class-dependent information reject unknown objects and classify the known ones.
3.2.1 Virtual Logit
The virtual logit is the scaled norm of the projection of the feature vector to the orthogonal complement of the principal subspace of the feature space. To simplify the subsequent logit computation, we offset h with a new origin -(WT)+b, where W is the weight fe; b the bias and (+)+ is the Moore-Penrose inverse. This transforms the feature vectors the previous features the feature vectors in the new coordinate system denoted h. We further denote the feature matrix of the training set H.
new bias-free coordinate system. To distinguish these from
The principal subspace P of the feature space is defined by the eigen decomposition of H
='H=QQQ-
(5)
where A contains the eigenvalues in descending order, and the first D columns of Q define the D-dimensional principal subspace P. Let T1 be the orthogonal complement of P, and h the projection of h to pt, After this projection, known data tend to have a smaller norm, whereas unknown data exhibit a larger norm. This facilitates the differentiation between known and unknown samples. The virtual logit is then obtained
(Wng et al. 2022
L = -
(6)
where / is the scaling constant:
8 Ein max -= I...,N {6)
(7)
Σ 1
where N is the number of known classos, K is the number of training samples, and 13 is the j-th logit of h
Following Wang et al. 2022), we obtain the ViM score by appending the energy score
Liu et al. 2020 to
the virtual logit:
N
= 3
(8)
which has better performance than each individual score.
3.2.2 Inference Pipeline
During inference, the final classification layer fc replaced by a new module fv. which produces a logit vector of length N 1, that is N normal logits and one virtual logit. f, is able invoke both the encoder fa and the classification layer fe. It uses the encoder fe to extract the feature matrix H from the training set X, project H into bias-free coordinate system and calculate the D dimensional principal subspace P. We
7

---

Published in Transactions on Machine Learning Research (09/2024)
Table 1. Data Modalities and Encoder Networks
Input Type
Network
Architecture
RGB Image
WRN-10-2
Wide ResNet Zagoruyko & Komodakis
2016
Range Image
WRN-4002
Wide ResNet ZZgoruyko Komodakis
2016
3D) Voxel Grid
VoxNet
Volumetric CNN (Maturana & Scherer
2015)
3D Voxel Grid
LightNet
Volumetric CNN 7hi et al. 20188)
Point Cloud
DGCNN
Dynamic Graph CNN (Wang et al.
2019)
Point Cloud
PCT
Point Transformer (Guo et al. 2021
calculate the scaling variable B based on Eq. to fe.
For normal logits, fu directly passes the feature embodding
Now, we have all the parameters to compute the OOD score (Eq. S the object is more likely to be unknown. As such, unknown objects can be rejected by setting threshold O. The decision rule is as follows:
A larger value for indicates that
class i = urgmar(()), unknown,
v S 0 v > 0
prediction
(9)
The threshold be determined without requiring auxiliary unknown data by analyzing the distribution of OOD scores of the known samples in the training set.
3.3 Implementation Details
For training, we use the Casine Anncaling Learning Rate [Loshchloov & Hutter 2017). with a maximum learning rate of 0.1 and a minimum of 10-4, In the initial epochs, the learning rate decreases slowly to speed up the contrastive learning process. The learning rate begins to drop rapidly midway through the training to facilitate classifier learning. Towards the end the training, the learning rate becomes very small, so that the classifier can be fine-tuned. The training takes up 200 epochs, and is optimized by stochastic gradient descent. (SGD) with momentum of 0.9. For the SCL loss, the temperature set to 0.1 and the batch size is limited to 256, For the CAC loss, the anchor magnitude is set to 10 and the weight variable X is set to 0.3. For the virtual logit, the dimensionality of the principal subspace is set to 64.
4 Experiments
4.1 Comparison with State-of-the- Methods
We compare the performance of S2CA with existing OSR and OOD detection methods, for which source
code is available from PyTorch- OOD library KKrchheim et al. 2022 or GitHub. Specifically, we select
SoftMax (Hendrycks & Gimpel 2017), ODIN (Liang et al. 20088 MD Lee et al. 2018), Energy (Liu et al. 2020), Entropy (Chan et al.) (021), MaxLogit [Hendrycks al. 2022). KL-M (Hendrycks al. 2022). ViM (Wang al. 2022), ; and CAC (Miller al. 2021). The selected methods compatible with different data modalities and encoder networks.
4.2 Encoder Networks
To evaluate the open-set recognition performance with both image and lidar data several encoder networks were used as shown in Tab. [. For RGB image data, we use a 40-layer wide residual network with a widening factor 2 (WRN-40-2) (Zagouyyko
Komodakis 2016) for feature extraction. For lidar data, we use encoder networks operating on raw point clouds, range images, and 3D voxel grids. For range images, we use the same Wide ResNet for RGB image data. For lidar data represented as 3D voxel grids, we use VoxNet (Maturana) & Scherer 2015) and NightNet (7hi et. al. 2018 the encoder networks. For lidar point clouds, use
8

---

Published in Transactions on Machine Learning Research (09/2024)
DGCNN (Wang et al. 2019 and Point Cloud Transformer (PCT) point clouds as input.
2021 which directly take Guo et al.
4.3 Datasets
We use two public autonomous driving datasets for evaluation: nuScenes (Caesar et, al. 2019 and KITTI (Geiger et al. 2013). For the object detection task, nuScenes has 23 classes, while KITTI Geiger et al. 2013) has classes. Each dataset was divided into three parts: Etrain: Ptest and TOOD- Strain and Piest contained samples of known classes for training and testing. FooD refers to unknown classes (out of distributtion). We selected the classes with the fewest annotations personal mobility vehicle, animal, and ambulance. In KITTI, unknown classes were truck, mise (miscellaneous objects, such as trailers and segways), tram, and sitting person. We first located each object using the given bounding box and then extracted these from the image and lidar point cloud data. Heavily occluded or completely invisible objects were filtered out. Following Neal et al. (2018). we calculate the openness of our datasets according to: openness = 1- K+U where K is the number of known classes and U is the number of unknown classes. Accordingly, the openness of our nuScenes dataset was found to 9.1%, while the openness of KITTI was 29.3%.
unknown. In nuScenes, unknown classes were stroller,
K
4.4 Evaluation Metrics
The effectiveness of an open set recoguition model can evaluated by its overall classification accuracy or F-score when detecting unknown classes in mixed data. However, these combined metrics are affected not only by the quality of the OSR model, but also by the calibration parameters chosen (Neal et al. 2018). As result, evaluate the performance of OSR models using two metrics: Area Under the ROC Curve (AUROC) and Closed Set Accuracy.
Area Under the ROC Curve (AUROC) AUROC uses all possible thresholds to plot an ROC curve of the True Postive Rate (TPR) against False Positive Rate (FPR). Ideally, known objects should have a lower OOD-ness score, whereas unknown objects should have higher score. all unknown classes have higher scores than the known ones, the TPR will be 1 and the FPR will be 0 after trying all thresholds, rosulting in perfect AUROC value of 1. Lower AUROC values indicate that unknown objects are mixed with the known ones.
Closed-set Accuracy An open-set classifier must maintain its capacity for recognizing closed-set objects as well. As a result, we will further evaluate the classification accuracy of known classes.
5 Results and Discussion
5.1 Open-set Recognition Performance
Tab. 2 shows the performance of the OSR methods with different, encoder networks measured by AUROC. It can be seen that overall S2CA achieves leading performance on both datasets across different data modality and encoder networks. Interestingly, the OSR methods perform well with lidar data when paired with suitable encoder networks. Point- based encoder networks generally outperform projection based and voxel- basod models. The range image representation of lidar data does not seem to benefit the open-sot recognition performance, especially when the lidar point cloud is sparse. This can be seen from the performance of the range image-based models on nuScenes, which is noticeably worse than on KITTI. This is because muScenes lidar data are captured by 32-channel lidar, the point cloud is relatively sparse and the range images have a lower resolution as compared to KITTI which uses 64-channel lidar. Notably, ODIN (Liang et al. 2018) and MD (Lee et al. (2018) perform quite poorly on nuScenes range images. This can be attributed to the addition of noise during preprocessing step in these methods, which has a negative impact on the recognition performance. The comparable performance of the OSR methods with different imago-based and encoder networks implics that the power of open-set recognition lies in the feature learning. The modality of the data is crucial the information it carries.

---

Published in Transactions on Machine Learning Research (09/2024)
Table Performance of S2CA compared with state of the art OSR methods. Open-set recognition performance is measured using AUROC (%). For each encoder network, the best and second-best OSR results are emphasized.
Encoder Network
Dalaset
OSR Method
Trainable
WRN
WRN
VoxNel
LightNet
DGCNN PCT
(RGB)
(Range)
(Voxel)
(Voxel)
(Point)
(Point)
MoftMax [Hendrycks & Gimpel 2017)
79.26
60.52
71.78
69.17
80.50
80.92
ODIN Liang et al. 2018
73.44
34.30
71.70
69.30
81.16
80.20
MD Lee et al. 2018
78.07
23.42
74.43
72.02
78.91
80.51
Energy LLuu al. 20200)
79.72
62.53
73.93
71.11
81.73
80.82
Entropy (Chan et al. 2021
79.56
61.80
73.05
70.27
81.14
81.09
nuScenes
MaxLogit (Hendrycks et al.
2022
79.65
62.04
73.20
70.36
81.45
81.25
KL-M (Hendrycks et al. 2022
78.42
52.58
56.87
52.93
61.69
67.52
VIM (Waug et 2022
80.73
70.59
75.53
74.59
80.41
82.28
CAC Miller et al. 2021
85.32
69.63
72.40
74.04
77.65
82.80
S2CA (ours)
82.88
70.99
76.05
77.25
83.12
83.21
SoftMax (Hendrycks & Gimpel
2017
74.07
75.68
76.88
77.92
72.50
75.40
ODIN (Liang et al. 2018
62.09
61.72
76.98
77.90
70.85
75.31
MD [ee al. [0188)
79.78
61.23
74.01
76.77
86.90
78.19
Energy [Luu et al.] 20200
74.91
72.10
76.03
77.49
72.89
74.88
Eutropy (Chan et al. 2021
74.08
75.66
77.01
78.29
72.60
75.27
KITTI
MaxLogit (Hendrycks et
2022
74.90
71.99
76.07
77.48
72.82
74.93
KT-M (Hendrycks et al. 2022
62.55
69.15
61.70
60.47
72.88
75.33
VIM Wang et al. 2022
80.38
76.56
76.68
77.41
85.27
77.39
CAC Miller et al.
2021
73.71
72.12
74.44
77.58
71.83
74.39
S2CA (ours)
81.23
79.64
77.39
79.28
87.48
81.94
0.5
Category
0.10
Category
Probability Density
0.4
Known
Probability Density
Known
Unknown
0.08
Unknown
0.3
0.06
0.2
0.04
0.1
0.02
0.0
0.00
0.0
0.2
0.4
0.6
0.8
1.0
0.00
0.25
0.50
0.75
1.00
Normalized Score
Normalized Score
(a) SoftMax
(b) S2CA
Figure Distribution of OOD scores for known and unknown samples obtained by the Softmax (inverted) baseline (a), and the proposed S2CA (b). For fair comparison, all scores are normalized to [0.1].
The superior performance of the proposed S2CA be attributed to its ability to learn discriminative features, which results in OOD scores that higher for unknown classes. Fig, 3 shows the probability density distributions of the normalized OOD scores for both known and unknown classes in the KITTI dataset. For the SoftMax baseline (Hendrycks & Gimpel 2017) in Fig. 3a both distributions exhibit a long-tailed pattern and there is a high degree of overlap between them, whereas S2CA ( (Fgg. 3b) presents a noticcable difference between the two distributions, making it casier determine an appropriate threshold distinguish the known and unknown classes effectively. The significant difference between the figures explains the superior performance of S2CA in the OSR task.
10

---

Published in Transactions on Machine Learning Research (09/2024)
100
100
CE
CE
S2CA
S2CA
Accuracy (%)
Accuracy (%)
95
90
90
80
85
70
80
WRN (RGB)
WRN (Range)
VoxNet
LightNet
DGCNN
PCT
WRN (RGB)
WRN (Range)
VoxNet
LightNet
DGCNN
PCT
(a) nuScenes
(b) KITTI
Figure 4. Closed-set accuracy of S2CA with different encoder networks compared with the cross-entropy baseline
muScenes (a) and KITTI (b).
Fig. shows the closed-set accuracy of S2CA compared with the conventional cross-entropy baseline. It can be seen that for all encoder networks the S2CA framework maintains a comparable closed-set performance the conventional cross-entropy baseline. This again indicates that the ability of S2CA to extract rich features from the data not only useful for rejecting unknown objects, but also helps maintain a high classification accuracy on known classes.
Van
Unknown
(a) Van
(b) Truck
Figure 5, Examples of failure cases. Van (a) is known class, while Truck (b) is unknown.
A close examination of the errors reveals that most misclassified samples are objects that are similar in appearence and are too far from the sensor. Fig. shows an example of a known (van) and an unknown object (truck) which were misclassified by S2CA. In such cases, where the unknown object looks very similar to known class, rejecting the unknown objects and correctly classifying the known ones is challenging.
5.2 Ablation Study
The S2CA framework consists of three important components: contrastive learning, class anchor clustering, and virtual logit matching. To evaluate the contribution of each individual component to the final perfor-
11

---

Published in Transactions on Machine Learning Research (09/2024)
mance, systematically remove these components from the S2CA framework to create set of bascline models:
CLS - A standard N class classifier, denoted f, trained with cross-entropy loss. Unknown objects are
identified using the SoftMax score.
CLS + VIM (CLS V.) This bascline uses the same classifier f, but rejects unknown objects based on the ViM (Wang al. 2022
score.
CLS + CAC VIM (C.C.V) The classifier f is trained with CAC (Miller et al. unknown objects based on VIM (Wang et al. 2022) score.
2021 loss, but rejects
Contrastive CLS (Cont. CLS) We further add the supervised contrastive learning (SCL) Khosla et al.
2020 branch to f. The model rejects unknown objects based on the ViM (Wang et al. 2022 score.
S2CA The complete S2CA framework.
Tab. 3 illustrates the results of the ablation study, The complete S2CA framework with different encoder networks consistently outperforms all competing baseline models in terms of AUROC. It is evident that ViM is key contributor to open-set performance. We can see substantial improvement in AUROC between CLS and CLS V. The SoftMax score in CLS depends solely on the information in the logit space, and OOD samples tend to have smaller SoftMax probabilities than in-distribution samples. In contrast, ViM incorporates class-agnostic information in the feature space and class-dependent information in the logit space, which better reflects OOD-ness (Wang et al. 2022). However, ViM neglects the variance of maximum logits in the calculation of the scaling parameter per model, weakening the importance of information in the feature space in the final decision-making process. This limitation is effectively addressed by CAC loss, which forces known objects to form tight clusters in logit space. Our results show that CAC alone provides limited improvements open-set performance, with AUROC scores of CLS V. and C.C.V showing small differences. We can also observe that contrastive learning leads to overall improvements in AUROC. By incorporating supervised contrastive learning, Cont. CLS surpasses the performance of both CLS V. and C.C.V. Furthermore, the complete S2CA framework also uses CAC loss mitigate overconfident classification and address the limitations associated with the ViM score. The cumulative effect of these components makes the S2CA framework superior to all baseline methods, underscoring its potential as a comprehensive and effective approach to open-set recognition.
Table 3. Performance of the complete S2CA framework in terms of AUROC compared to other baselines
Encoder Network
Dataset
Baseline
WRN
WRN
VoxNet
Light.et (Voxel)
DGCNN PCT
(RGB)
(Range)
(Voxel)
(Point.
(Point)
CLS
79.26
60.52
71.78
69.17
80.5
80.92
CLS V.
80.73
70.59
75.53
74.55
80.41
82.28
muScenes
C.C.V.
80.32
70.63
75.4
74.04
80.43
82.8
Cont. CLS
81.59
70.89
75.8
76.12
82.53
82.87
S2CA
82.88
70.99
76.05
77.25
83.12
83.21
CLS
74.07
75.68
76.88
77.92
72.5
75.4
CLS V.
80.38
76.56
76.68
77.41
85.27
77.39
KITTI
C.C.V.
80.71
76.12
76.44
77.58
85.83
77.56
Cont. CLS
80.33
77.58
77.23
78.56
86.21
80.04
S2CA
81.23
79.64
77.39
79.28
87.48
81.94
Following Sun et al. (2021) we experiment with different projector dimensions and temperatures for con- trastive learning. The results for the RGB image classifier trained with S2CA are shown in Tab. 4 As it can be seen, the choice of hyperparameters has a minor impact on the performance of S2CA.
Tab. shows the performance of post-training OOD scores applied to the RGB image classifier trained with S2CA. ViM WWagg al. 2022) has the best performance since it combines class-agnostic information in the feature space and the class-dependent. information in the logit space for OOD detection.
12

---

Published in Transactions on Machine Learning Research (09/2024)
Contrast Head
Temperature (r)
Dimension
0.07
0.2
0.5
De 128
81.23
80.83
80.70
De 256
80.98
79.12
79.92
Table 4. Effect of contrastive learning hyperparameters on OSR performance as measured by AUROC KITTI.
Post-training OOD score
nuScenes
KITTI
SoftMax Henddyykk & Gimpel 2017
79.95
74.77
MaxLogit
Hendrycks et al.
2022
80.50
75.17
ODIN (Liang et al. 2018
72.23
65.49
MD
Lee et al., (2018)
78.98 82.88
80.59
ViM Wang et al.
2022
81.23
Table 5. Comparison of OOD scores applied with S2CA after training.
-1
20-
20
L:EE 2
L-SNE
0-
-20
-20
400
-40
-60
-60
-60
-40
-20
0
20
40
60
80
-60
-40
-0
0
20
40
60
t-SNE
t-SNE
(a) Cross Entropy
(b) CAC
60
40
SSNE 2
t-5NE 2
0-
-20
-40
-60
-40
-20
0
60
-60
-40
-20
20
40
1-5NE
(c) SCL
(d) S2CA
Figure 6. T-SNE visualization of feature space trained different objectives. Numerical labels correspond to object categories: -1 denotes unknown objects, represents Car, represents Cyclist, 2 represents Pedestrian and 3 represents Van.
13

---

Published in Transactions on Machine Learning Research (09/2024)
Fig. 0 shows the feature space of the image classifier trained by different objectives. Without contrastive learning (Fig. 6a and Fig. 6b there is significant overlap between the features of known classes and OOD samples. Contrastive learning promotes the intra-class compactness, with S2CA (Fig. 6d) showing more compact clusters as compared to the original SCL (Fig. Gc/. Due to the semantic similarity between OOD samples and known classes, they are still close to each other in the feature space. But contrastive learning
makes OOD samples gather at the edges of the known clusters, making it easier to reject unknown samples.
Table 6. OOD detection performance of S2CA with ResNet-34 backbone trained on CIFAR-100 (TD) compared with state-of-the-ar methods.
OOD Dataset
Average
Method
SVHN
Places365
LSUN
ISUN
Texture
FPR.
AUROCT
FPRI
AUROCT
FPR.
AUROCT
FPR,
AUROCT
FPR]
AUROCT
FPR.
AUROCT
Without
Contrastive
Learning
MSP
78.80
79.80
84.61
72.53
83.12
75.27
ODIN
70.16
$4.88
82.16
75.19
76.36
80.10
79.54
79.16
85,28
73.23
78.70
79.11
Mahalanobis
87.09
80.82
$4.63
73.80
84.15
79.43
83.18
78.53
61.72
84.87
80.15
79.53
Energy
66.91
GODIN
71.61
81.03
89.13
68.96
93.33
67:22
91.25
65.20
86,52
09.39
87. 57
70.97
Norm
50.60
90.74
80.25
78.58
81.07
82.90
84.19
80.77
86.64
75.00
78.35
81.74
Contrastive Learning
Arocyynchaor
87.21
82.44
70.10
79.84
37.19
91.68
70.00
$1,96
65.64
84,98
66.03
$4.78
CE SinCLR
24.82
94.45
$6.63
71.48
56.40
89.00
68.52
83.52
63.74
82.01
59:62
84.15
CSI
$1.83
SSD+
31.19
94.19
77.74
79.90
79.39
85.18
80.85
$4.08
66.63
86,18
67.16
85, 90
KNN
39.23
92.78
90.74
77.58
48.90
80.30
74.99
82.80
57.15
68.35
60.22
86.14
CIDER
23.09
79:03
73.43
96.33
71.08
82.98
43.87
46.89
87.67
S2CA
27.08
94.56
89.90
50.87
40.91
92.41
67.68
84.34
49.04
88.90
50.92
88.23
5.3
Evaluation on DistribDiittiittiion Benchmarks
Our results generally support the hypothesis that the combination of contrastive learning and class anchor clustering provides convincing solution to the problem of open set recognition in driving scenes. Similar findings have been reported for general OOD benchmarks by another recent work, CIDER (Ming et al. 2023 which also adopts the concept of supervised contrastive learning and class anchor (or prototype).
To evaluate the performance S2CA on OOD benchmarks, we follow the experimental setup of CIDER Ming et al.) 2023), and use CIFAR100 (Krizhevsky et all 2009) as the in-distribution dataset to train S2CA with a ResNet-34 backbone (He et al. 2016). OOD Lest datasets include SVHN (Netzer el al. 2011 Places365 [Zou al., 2017p, Textures (Cimpoi al., 2014), LSUN (Yu et al. 2015), and iSUN (Xu 2015). As shown in Tab. 6 S2CA achieves similar results to CIDER, with a slightly higher overall AUROC
and slightly worse FPR. When compared to other contrastive learning- based methods, such as Proxy Anchor
[Km et al. 2020), CST (Tack et al., 2020). SSD+ (Sehwag et al. 2021 and KNN+ (Sun et al. demonstrates the best overall performance on both metrics.
2022 S2CA
While both CIDER and S2CA use contrastive learning and class anchors, they have subtle differences. S2CA places fixed class anchors in the logit space, as the ground truth label of known classes, whereas CIDER estimates the prototypes (anchors) directly from sample embeddings using EMA in the feature space. Furthermore, S2CA jointly optimizes SCL and CAC loss. In contrast, CIDER uses maximum likelihood estimation (MLF) loss and dispersion loss to promote intra-class compactness and inter-class sparsity in the feature space. Also S2CA uses the virtual logit as OOD score, taking advantage of information from both feature space and logit space, whereas CIDER adopts k-NN cosine similarity as the OOD score, which only uses information in the feature space.
5.4 Theoretical Analysis
Softmax probability as a baseline OOD detection method achieves modest success, because the neural net- work less confident with unknown input. This observation supports previous results, such as (Hendrycks & Gimpel 2017) and (Kang et al. 2024), which also show that the predictions of the neural network converge to constant. value when inputs become more OOD. However, as shown by Pearce et al. (202)), since neural
14

---

Published in Transactions on Machine Learning Research (09/2024)
networks not bijective, the features of in-distribution input and OOD imput in the final-layer can be the same, resulting in the failure of softmax probability in rejecting unknown samples. The network can also produce lower softmax confidence for corrupted known samples. Since OOD data are absent in the training process, neural networks need to learn robust features of known classes to reduce overlap with OOD features. Contrastive learning promoles the aligmment of positive pairs and uniformity of the feat space (Wang & Isola 2020). In a fully supervised setting, where positive pairs are defined by class label (Khosla et al. 2020). the aligument of positive pairs encourages known classes to form tight clusters, thus improving invariance to noise, while the uniformity helps preserve as much information from the training samples as possible, but also encourages inter-class sparsity. As OOD samples lie between ID clusters, optimizing large distance among ID clusters benefits OOD detection (Ming et al. 2023). Traditional contrastive learning methods such as SCL (Khosla et al., 2020) usually train an encoder with contrastive loss first, then freeze the encoder and train a linear classifier. As shown by Wang et al. (2021). this two-stage learning may not be the most effective approach in a fully supervisod sctting, as can reduce the compatibility between the feature encoder and the classifier. S2CA jointly optimizes the model with feature-based SCL loss and logit-based CAC loss (Miller et al. 2021p. Contrastive learning helps the encoder learn richer feature space invariant to noise, while anchor loss will penalize the network when it makes low-confidence wrong predictions. As driving scenes are characterized by varying weather and lighting condition (Michaclis et al., 2019), the S2CA framework enables the model to learn robust feature embeddings from noisy data, improving the classification of known classes and rejection of unknown ones.
6 Conclusion
We proposed an effective and adaptive OSR method based contrastive class anchor learning. Our ex- perimental results show that the proposed method achieves state-of- shate-of-the-art performance with different data modalities and encoder networks. Future work will investigate failure detection, which focuses on detecting misclassification, covariate shifts (corruption shift and domain shift), and new class shifts (Jaeger et al. 2023). In the future work, we will work on a unified failure detection method, which detects both misclas- sified known samples and out-of-distribution samples under various domains. For real-world application, S2CA classifier can be integrated into a two-step object detector to make complete open-set object detec- tion pipeline, for different data modalities. For image input, S2CA can be applied to methods such as VOS (Du et al. 2022). which appends open-set classifier after Region Proposal Network (RPN) (Ren et al. 2015). For LiDAR input, object proposals can be obtained via ground segmentation and clustering (Nunes et al. 2022
Acknowledgement
This research was supported by The University of Melbourne's Research Computing Services and the Petas- cale Campus Initiative. The first author acknowledges the financial support from the University of Melbourne through the Melbourne Research Scholarship.
References
Abhijit Bendale and Terrance Boult. Towards open world recognition. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1893-1902, 2015. doi: 10.1 D0.1199/CVP.20010 CCVP.20015.72987990
Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. muscenes: A multimodal dataset for autonomous driving. arXiv preprint 1rXii:19033.1000 2019.
Robin Chan, Matthias Rottmann, and Hanno Gottschalk. Entropy maximization and meta classification for out-of-distributtion detection in semantic segmentation. In ICCV, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In ICML, 2020.
15

---

Published in Transactions on Machine Learning Research (09/2024)
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEFF Conference on Computer Vision and Pattern Recognation, pp. 3606 3613, 2014.
Xuefeng Du, Zhaoning Wang, Mu Cai, and Sharon Li. Towards unknown-aware learning with virtual outlier synthesis. In International Conference on Learning Representations, 2022. URL https://openrevies: het/fforummdd--T//dbbYYaa
ZongYuan Ge, Sergey Demyanov, Zetao Chen, and Rahil Garnavi. Generative openmax for multi-class open set classification. In Proceedings of the British Machine Vision Conference, pp. 42.1-42.12, 2017.
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets roboties: The kitti dataset. International Journal of Robotics Research (IJRR), 2013.
Chuanxing Geng, Sheng-Jun Huang, and Songcan Chen. Recent advances in open set recognition: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(10)33614 3631, 2021. doi: 10.1109/ TPAAM.202002290 2020.2981004.
Alexander Gillert and Uwe Freiherr von Lukas. Towards combined open set recognition and out-of- distribution detection for fine- ane-ggained classification. In 16th International Joint Conference on Computer Vision, 2021.
Meng-Hao Guo, Jun-Xiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R. Martin, and Shi-Min Hu. Pct: Point cloud transformer. Compulational Visual Media, 7(2):187- 199, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsuper- vised visual representation learning. In 2020 IEFE/ /VVF//F Conference on Computer Vision and Pattern /ttp///CCR220000000
Recognition (CVPR), pp. 9726 9735, 2020. doi:
Dan Hendrycks and Kevin Gimpel. A bascline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations (ICLR), pp. 1010-1019, 2017.
Dan Hendrycks, Steven Basart, Mantas Mazeika, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. Scaling out-of-distributio detection for real-world settings. In International Conference on Machine Learning (ICML), 2022.
Paul Jacger, Carsten Tim Lüth, Lukas Klein, and Till J. Bungert. A call to reflect on evaluation prac- lices for failure detection in image classification. In The Eleventh Internalional Conference on Learning Representations, 2023. URL https://ppeneeriiieee///ooorrrre www..eeetoouuriidd-niinnn..C.
Lalit P Jain, Walter J Scheirer, and Terrance F. Boult. Multiclass open set recognition using probability of inclusion. In European Conference on Computer Vision, 2014.
Katie Kang, Amrith Setlur, Claire Tomlin, and Sergey Levine. Deep neural networks tend to extrapolate predictably. In The Twelfth International Conference on Learning Representations, 2024. URL https: //openreview.cot wtciiiewwwwee/eeeemm
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Tn Adu. Neural Inform. Process. Syst., 2020.
Sungyeon Kim, Dongwon Kim, Minsu Cho, and Suha Kwak. Proxy anchor loss for deep metric learning. Tn IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Konstantin Kirchheim, Marco Filax, and Frank Ortmeier. Pytorch-ood: A library for out-of-disttibutiin detection based on pytorch. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern June 2022.
Recognition (CVPR) Workshops, pp.
16

---

Published in Transactions on Machine Learning Research (09/2024)
S. Kong and D. Ramanan. Opengan: Open-set recognition via open data generation. In 2021 IEEE IEEE/CVE /CFF In- lernalional Conference Computer Vision (ICCV), pp. 793 802, Los Alamitos, CA, USA, oct 2021. TREE Computer Society, doi: 10.1109/ N0.1109/ICCV499200
org/ 2CCC44892220220000000000 URL httph::/tooo weeeccompteessocitty....... o
10. N0.1109//CCV4892..oo
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images, 2009.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of- distribution samples and adversarial attacks. In Adv. Neural Inform. Process. Sysl., 2018.
Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing the reliability of out-of-distribinn image detection in neural networks. In International Conference Learning Representations, 2018. URL. https:// openreview wwarrrvvewwne//forrrrie
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distributtio detection. Adv. Neural Inform. Process. Syst., 2020.
Πya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In ICLR, 2017.
Daniel Maturana and Sebastian Scherer. Voxnet: A 3d convolutional neural network for real-time object recognition. In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 922-928, 2015. doi: N0.1109/TROS.200 7353481.
Claudio Michaelis, Benjamin Mitzkus, Robert Geirhos, Evgenia Rusak, Oliver Bringmann, Alexander S. Ecker, Matthias Bethge, and Wieland Brendel. Benchmarking robustness in object detection: Autonomous driving when winter is coming. arXiv preprint arXiv:1907.07488 2019.
Dimity Miller, Niko Sünderhaaff Michael Milford, and Feras Dayoub. Class anchor clustering: A loss for distance based open set recognition. In 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 3569-3577 2021. doi: 10.1109/ A..CVV8663.202.00000
Yifei Ming, Yiyou Sun, Ousmane Dia, and Yixuan Li. How to exploit hyperspherical embeddings for out- of-distribution detoction? In The Eleventh International Conference on Learning Representations, 2023. URL httts::/ www.:eeoooomdddaaaaaaaaoo..
Lawrence Neal, Matthew Olson, Xiaoli Fern, Weng- Keen Wong, and Fuxin Li. Open set learning with coun- terfactual images. Open set learning with counterfactual images. In ECCV, pp. 613-628 2018.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised fcature learning, 2011.
Lucas Nunes, Xieyuanli Chen, Rodrigo Marcuzzi, Aljosa Osep, Laura Leal-Taixé, Cyrill Stachniss, and Jens Behley. Unsupervised class-agnostic instance segmentation of 3d lidar data for autonomous vehicles. IEEE Robotics and Automation Letters, 7(4):8713 (14):8713-8720, 2022. doi: 10. 10.1100 100.119//..0 wT9....A..022..11111
Poojan Oza and Vishal M. Patel. C2ae: Class conditioned auto-encoder for open-set recognition. In CVPR, pp. 2302-2311, 2311, 2019. doi: 10.1109
Tim Pearce, Alexandra Brintrup, and Jun Zhu. Understanding softmax confidence and uncertainty, 2021.
Pramuditha Perera, Vlad I. Morariu, Rajiv Jain, Varun Manjunatha, Curtis Wigington, Vicente Ordonez, and Vishal M. Patel. Generativ--iisciinnnaave feature representations for open-set recognition. In CVPR, pp. 11811-11820, 2020. doi: 20.109//C
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time ob- ject detection with region proposal networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (ods.), Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL https://proceednnss www.nnuuriiccmm 7252660112277//bbb00222222222233 httpp//cwwa.ppppeeccc//ppp/aae/p http:ppppp..eeee/pppr///////////
17

---

Published in Transactions on Machine Learning Research (09/2024)
Walter J. Scheirer, Anderson de Rezende Rocha, Archana Sapkota, and Terrance E. Boult. Toward open set recognition. TEEE Transactions on Pollern Analysis and Machine Intelligence, 35(7):1767-1772 2013. 2013. doi:
Walter J. Schcirer, Lalit P. Jain, and Terrance E. Boult. Probability models for open set recognition. IEEE Transactions on Pallern Analysis and Machine Intelligence, 36(11):2317 2324, 2014. doi: 10.1109/
2P142.2214..3223922
Matthew D. Scherreik and Brian D. Rigling. Open set recognition for automatic target classification with rejection. IEEE Transactions on Aerospace and Electronic Systems, 52(2))632 642, 2016. doi: 10.1109/ TAES. 150..
Vikash Schwag, Mung Chiang, and Prateck Mittal. Ssd: A unified framework for self-supervised outlier detection. In International Conference on Learning Representations, 2021.
Kihyuk Sohn. Improved deep metric learning with multiclass n-pair loss objective. In Adv. Neural Inform. Process. Syst., 2016.
Bo Sun, Banghuai Li, Shengeai Cai, Ye Yuan, and Chi Zhang. Fsce: Few-shot object detection via contrastive proposal encoding, In 2021 IEEE/CVE Conference Computer Vision and Pattern Recognition (CVPR), pp. 7348 -388,, 2021. doi: 10.1109 D0.1099/CVPR44433 CVPRR4437.2202 20113.02220007721
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distributiio detection with deep nearest neigh- bors. 1CML, 2022.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty dctection via contrastive learning on distributionally shifted instances. In Advances in Neural Information Processing Systems, 2020.
Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of-distribution with virtual-logit matching. In 2022 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4911 4920, 2022. doi: Wttp:/CCVRR26...220
Peng Wang, Kai Han, Xiu-Shen Wei, Lei Zhang, and Lei Wang. Contrastive learning based hybrid networks for long-tailed image classification. In 2021 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 943 952, 2021. doi: 10.1109 0tBp///CCRR643..22211111 2021.0000...22000000.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity the hypersphere. In Proceedings of the 37th International Conference Machine Learning, ICML/20. JMLR.org, 2020.
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon. Dynamic graph cnn for learning on point clouds. ACM Trons. Graph., 38(5), oct 2019. ISSN 0730-0301. doi:
URL https://doi. htt:///oo.org///// wt..o///////.................... 1145/3326662
Pingmei Xu, Krista A Ehinger, Yinda Zhang, Adam Finkelstein, Sanjeev R Kulkarni, and Jianxiong Xiao. Turkergaze: Crowdsourcing saliency with webcam based eye tracking. arXiv preprint orXiv:1504.0677 2015.
Ryota Yoshihashi, Wen Shao, Rei Kawakami, Shaodi You, Makoto Tida, and Takeshi Nacmura. Classification- reconstruction learning for open-set recognition. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4011 4011-4020, 2019. doi: 10.1008/CVPPR2219.001.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Con- struction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03333 03365, 2015.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. Tn BMVC. BMVA Press, 2016.
18

---

Published in Transactions on Machine Learning Research (09/2024)
Shuaifeng Zhi, Yongxiang Liu, Xiang Li, and Yulan Guo. Toward real-time 3d object recognition: A
light volumetric cmn framework using multitask learning. Computers & Graphics, 71:199-200 2018.
ISSN 0097-8493. doi: http:::///oii.org/10.1110... htt.://....././......... 2017.10.00.. .00.0//.......1..00..000.
URL bttrs://ww..sceencciiree
com/ cttp://caaaooaaacre//////////110
Bolci Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transaclions pallera analysis and machine intelligence, 40(6): 1452-14 1464, 2017.
Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: Bilateral-branch network with eumulative learning for long-tailed visual recognition. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 9716 9725, 2020.
doi: 10. 101108//CPR22880.22000099
19


Reviews:
[{"review_id": "UoqjCjWmMC", "reviewer": "Reviewer_Aes7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ This is a principled way to construct the problem through a combination of contrastive, supervised and virtual logits. \n+ The results are definitely convincing, outperforming existing methods. The evaluations test against different modelling backbones and show consistency in performance. \n- Novelty appears to be slightly lacking, as it looks like a mix and match approach from different sources. \n- I think the analysis could use more ablations. I am looking at the impact of the different pieces used in the modelling (e.g. supervised, contrastive). \n- More insight would be useful as to how each contributes to the performance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the Open Set Recognition problem is examined, to reject unknown objects. The paper tackles this problem by using a combination of supervised contrastive learning (class anchoring) and classifier learning (minimizing distance to correct class, maximizing distance to incorrect class) - together with a weighting between the two branches, together with the virtual logit methodology in [1]. The model is then run on autonomous driving datasets (nuScenes and KITTI) and performance is recorded for open-set recognition.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "On the lines indicated above, can the authors run more ablation analysis showing the effect of the various components used in the modelling?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eqLOgevRkN", "reviewer": "Reviewer_hhVc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Apologies as I am quite pressed for time, so I may have misunderstood/missed information in the paper. I welcome the authors to correct me if that is the case.\n\n**Strengths**\n- The approach is relatively simple whilst being seemingly effective. The authors present a clear ablation of their approach, demonstrating how each component contributes to the overall performance.\n- The paper benchmarks lidar data, which is typically overlooked in the field of OSR/OOD detection.\n\n**Weaknesses**\n1. The core idea of increasing intra-class compactness and inter-class sparsity has been proposed before in [1], which is not referenced at all in the paper.\n2. The choice to extract objects using ground truth bounding boxes is a bit odd. This effectively reduces the problem of detection down to classification. The paper does not perform any actual object detection experiments on 2-stage object detectors, for which the proposed method should be readily applicable [2].\n3. Missing discussion on Failure Detection [3,4]. Recent work has proposed an alternative perspective where other misclassified samples of known classes and samples of unknown samples should be detected. This is not mentioned at all in the paper, even though, in scenarios such as autonomous driving, the misclassification of a known class could be just as problematic as the treatment of an unknown class as a know one. \n4. It's unclear in Tab. 2 which approaches are training-based and which are post-training. It's also unclear whether choosing a different post-training score rather than ViM (e.g. Mahalanobis distance) would benefit S2CA.\n\n\n[1] Ming et al. *How to Exploit Hyperspherical Embeddings for Out-of-Distribution Detection?* ICLR 2023\n\n[2] Du et al. *VOS: Learning What You Don't Know by Virtual Outlier Synthesis* ICLR 2022\n\n[3] Jaeger et al. *A Call to Reflect on Evaluation Practices for Failure Detection in Image Classification* ICLR 2023\n\n[4] Xia et al. *Augmenting Softmax Information for Selective Classification with Out-of-Distribution Data* ACCV 2022", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an approach for Open Set Recognition/OOD detection. Concretely, a number of existing methods (from OOD detection/Contrastive Learning etc.) are combined synergistically resulting in improved detection of unknown classes. The basic idea is to increase intra-class compactness and inter-class sparsity in the hope that OOD samples will more likely fall into the larger volume between known class centres in the feature space leading to better detection.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. (Critical) Add proper comparison to CIDER [1], both in terms of discussing the similarities and differences to S2CA, as well empirical results.\n2. (Critical) Add discussion with regards to real-world object detection and how S2CA may be applied. (Non-critical) Compare with [2] on their 2-stage object detector benchmark (which is fully open source)\n3. (Critical) Add discussion on Failure Detection including references to [3,4], e.g. as related/future work. (Non-critical) Evaluate S2CA on failure detection. \n4. (Non-critical) Make clear in the tables which approaches require altering the training procedure. Also ablated difference choices of post-training OOD score for S2CA.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qUZtEboqCH", "reviewer": "Reviewer_gKuR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed method is comprehensively explained through detailed formulas and illustrative figures.\n2. The effectiveness of the method is demonstrated using a variety of encoder networks.\n\nWeaknesses:\n1. The novelty of the framework is limited, as it seems to only use contrastive loss based on the previous method.\n2. Lack of theoretical derivation. Though contrastive loss improves model closed-set accuracy, it would be better to provide corresponding theoretial analysis to prove why this module works. Some related works[1][2] may help.\n3. Missing some ablation study. For example, the ablation study of projector g(.). It could be better to provide corresponding ablation study like Table 6 in FSCE[3].\n\n[1] Wang, Tongzhou, and Phillip Isola. \"Understanding contrastive representation learning through alignment and uniformity on the hypersphere.\" In International conference on machine learning, pp. 9929-9939. PMLR, 2020.\n\n[2] Khosla, Prannay, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. \"Supervised Contrastive Learning-Supplementary Material.\"\n\n[3] Sun, Bo, Banghuai Li, Shengcai Cai, Ye Yuan, and Chi Zhang. \"FSCE: Few-shot object detection via contrastive proposal encoding.\" In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7352-7362. 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "1. Introduce supervised contrastive loss into open set object recognition task to improve model's ability to reject unknown sample and known sample class distribution.\n2. Use different modalities data, different encoder networks, and different datasets to prove framework effectiveness.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. It would be better to provide a comprehensive illustration (theoretical derivation or experimental proof) of why contrastive learning could improve open set object detection tasks to enrich paper content.\n2. The author should provide detailed analysis of the experiment, such as an ablation study on contrastive learning and t-sne visualization on features to illustrate different module effects.\n3. Some visualization images are unsightly. For example, the author could reference Figure 6 in yolo[1] to improve the quality of Figure 5.\n\n[1] Redmon, Joseph, Santosh Divvala, Ross Girshick, and Ali Farhadi. \"You only look once: Unified, real-time object detection.\" In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779-788. 2016.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The proposal of Supervised Contrastive Class Anchor overcomes some of the limitation of Open Set Recognition (OSR) methods. The experimental results on two widely used driving scene recognition datasets are convincing. The reviewers, while not extremely enthusiastic about the paper, recommend that it be accepted as some researchers in the community would benefit from the methods introduced herein.
Decision: Accept (poster)