# Detecting Out-of-Distribution Samples in Complex IoT Traffic Based on Distance Loss
###### Abstract  
Out-of-distribution (OOD) detection is critical for securing Internet of Things (IoT) systems, particularly in applications such as intrusion detection and device identification. However, conventional classification-based approaches struggle in IoT environments due to challenges like large class numbers and data imbalance. To address these limitations, we propose a novel framework that combines class mean clustering and a group-level feature distance loss to optimize both intra-group compactness and inter-group separability. Our framework utilizes Mahalanobis distance for robust OOD scoring and Kernel density estimation (KDE) for adaptive threshold selection, enabling precise boundary estimation under varying data distributions. Experimental results on real-world IoT datasets show that our framework outperforms baseline techniques, achieving at least a 10% improvement in AUROC and a 33% reduction in FPR95, demonstrating its scalability and effectiveness in complex, imbalanced IoT scenarios. IEEEexample:BSTcontrol  
Recent research has primarily focused on post hoc modifications of scoring functions to enhance the distinction between in-distribution (ID) and OOD samples in OOD detection. For example, the ODIN method increases the difference in OOD scores between ID and OOD samples by applying temperature scaling and adding small perturbations [4]. React further mitigates overconfidence in classification-based OOD detection by modifying activation patterns [5]. UM and logit normalization alleviate model overconfidence by adjusting the loss function, thereby improving classification-based OOD detection performance [6; 7]. However, the complexity of IoT traffic is increasing, and the types of traffic training samples that can be collected are becoming more diverse and imbalanced in distribution [8; 9]. The aforementioned supervised learning-based OOD detection techniques struggle to handle such complex real-world conditions. To verify this limitation, we replicated classification-based OOD detection methods using both open-source and real IoT traffic datasets [4; 10; 11; 12; 13]. As shown in Figure 1, as the number of labeled samples increases and distribution imbalance grows, the performance of these methods deteriorates. These methods often overfit to ID samples, leading to misclassification of ambiguous samples as ID, reducing OOD detection reliability [6; 7]. Additionally, the imbalanced nature of IoT data causes issues, as minority class samples are poorly learned and their features overlap with OOD samples. This results in the misclassification of OOD samples as minority class samples. As the number of ID classes grows, decision boundaries become more complex, further hindering confidence-based OOD methods [14]. These challenges emphasize the need for improved OOD detection techniques that can handle the multi-class and imbalanced nature of IoT traffic [15]. Moreover, most existing research evaluates OOD detection performance using metrics like AUROC, AUPR, and FPR95, but often overlooks the importance of selecting the right threshold for real-world applications. While methods like Otsu's [16] exist, they struggle with skewed data distributions. Figure 1: Variation in OOD detection performance (AUROC/FPR95) under different class numbers and imbalance levels on the CIC IoT 2022 dataset. (**a,b**) show the changes in AUROC and FPR95 as the number of in-distribution classes increases. (**c,d**) present the changes in AUROC and FPR95 as the in-distribution class imbalance increases. To address the above limitations of existing techniques, we propose a model combining feature clustering, distance loss optimization, and an adaptive threshold selection strategy. First, feature clustering is performed based on class means, grouping similar classes together to simplify decision boundaries while preserving class information.