An Efficient One-Class SVM for Novelty Detection in IoT
Authors: Kun Yang
Samory Kpotufe
Nick Feamster
Venue: Accepted by TMLR ()

Abstract: One-Class Support Vector Machines (OCSVM) are a common approach for novelty detection, due to their flexibility in fitting complex nonlinear boundaries between {normal} and {novel} data.  Novelty detection is important in the Internet of Things (``IoT'') due to the threats these devices can present, and OCSVM often performs well in these environments due to the variety of devices, traffic patterns, and anomalies that IoT devices present. Unfortunately, conventional OCSVMs can introduce prohibitive memory and computational overhead at detection time. This work designs, implements and evaluates an efficient OCSVM for such practical settings.  We extend Nystr\"om and (Gaussian) Sketching approaches to OCSVM, combining these methods with clustering and Gaussian mixture models to achieve 15-30x speedup in prediction time and 30-40x reduction in memory requirements without sacrificing detection accuracy. Here, the very nature of IoT devices is crucial: they tend to admit few modes of \emph{normal} operation, allowing for efficient pattern compression. 

Published in Transactions on Machine Learning Research (11/2022)
An Efficient One-Class SVM for Novelty Detection in loT
Kun Yang Cohumbia University
ky24440@cokumbicacc edu
Samory Kpotufe
skle175@columiic.co edu
Columbia University
Nick Feamster University of Chicago
feamster&uucicagg.... edu
Reviewed on OpenReview: https: // openreview. net/ forum? id= LFRRUCaFFF
Abstract
One-Class Support Vector Machines (OCSVMs) are a set of common approaches for novelty detection due to their flexibility in fitting complex nonlinear boundaries between normal and novel data. Novelty detection is important in the Internet of Things ("IoT") due to the potential threats that ToT devices present, and OCSVMs often perform well in these environments due to the varicty of devices, traffic patterns, and anomalics that IoT devices present. Unfortunately, conventional OCSVMs can introduce prohibitive memory and computational overhead in detection. This work designs, implements, and evaluates an efficient OCSVM for such practical settings. We extend Nyström and (Ganssian) Sketching approaches to OCSVM, combining these methods with clustering and Gaussian mixture models achieve 15-30x speedup prediction time and 30-40x reduction in memory requirements without sacrificing detection accuracy. Here, the very nature of loT devices is crucial: they tend to admit few modes of normal operation, allowing for efficient pattern compression.
1 Introduction
As devices ranging from consumer electronics to building control systems become connected to the Internet part of the "Internet of Things" (IoT), both these devices and the network itself are subject to new threats. Novelty detection, which aims to detect unusual activity based on observable properties of network traffic, is a common defense. Onc-Class Support Vector Machines (OCSVMs) are one of the common approaches for novelty d due to their ability to identify a wide range of nonlinear classification boundaries. Such flexibility appropriate for IoT devices and applications, which exhibit complexity due to the vast heterogeneity of devices and the wide range of traffic patterns under different operating modalities.
The heterogeneity of IoT devices and operating regimes introduces a broad class of activities (and corresponding network traffic patterns) that could be classified as normal novel. In contrast to general-purpose computing devices where the main novel behavior of interest is typically security event such as infection loT devices raise a more diverse set of anomalies, including physical device failures, the introduction of rogue devices on the network, physical security incidents, and abnormal interactions with control systems. The devices themselves are also heterogeneous, with the normal operating regime for each device type or manufacturer exhibiting distinct normal baseline patterns. Given such diversity of anomaly patterns, desired anomaly
'The work was done at Columbis University.
In the context of security, novelty detection is often referred to as anomaly defection; the problem is equivalent. We prefer the of novelty detection in this paper bocause
the term novelty detection to
refer the same class of algorithms, the classes of events that aim detect include conventional anomalics (in the security sensc), well broader class of novel events, e-g., activities, and devices that might he simply "new", though these new events may necessarily have negative connotation.
1

---

Published in Transactions on Machine Learning Research (11/2022)
detection methods ought to be flexible in the generality of anomalies they can fit, but easy to use. OCSVM is clear candidate meeting these criteria and has thus been frequently applied to novelty detection problems in IoT, with demonstrable efficacy in detecting novel traffic patterns corresponding to either unseen modalities
or malicious activities (Shilton et al., 2015 Lee et al.) 2016 Mahdavinejad et al.) 2018: [A Shorman et al.)
2020: Razaak et al. 2020). We rllustrate OCSVM's detection performance against some other flexible methods
on typical ToT devices in Figure ] with further discussion in (Related Work) Section 2
However, many IoT deployments require fast novelty detection in field deployments (e.g., embedded devices such as home network routers embedded sensors), where both computational and memory requirements may be limited. In operational deployments, there may be the need to quickly detect an attack, a rogue device, or a novel activity (perhaps non-malicious). Unfortunately, OCSVMs can be computationally expensive at detection time. Given a new observation x to classify as normal or novel, detection consists of evaluating a scoring function f(x) of the form Σi= a, (X,, æ)) defined with respect to training data (X.) of size n and a so-called kernel function K; such evaluation of f(x) takes time and space Q(n)) for typically large training data size n in the thousands. In the context of ToT, each training data point X; represents vectorized representation of normal traffic data over short time periods. Given an Internet-conneeted device that is continuously generating network traffic, detection using OCSVM is currently prohibitive in practico.
Goal and Method. The goal of this work is to speed up detection time and reduce memory requirements of OCSVM, while maintaining detection performance; we demonstrate these gains in the context of loT, which imposes time and memory constraints in practice. Our focus is on detection time and space, rather than training time and space, and we particularly emphasize that we seek to not tradeoff detection performance, in contrast to usual expectations for computational speedup.
Although novelty detection is unsupervised problem i.e, we only have access to normal dala as opposed to both normal and novel data points- we draw initial inspiration from the related supervised learning methods of Support-Vector-Mant Vector-Machines (SVMs), which, similarly to OCSVMs, uncovers lincar relationships between classes of data. Namely, various speedup approaches such as so-called Nyström and Sketching (Drineas et al.) 2005 Yang et al.] 2017) have recently been developed for SVMs, which we aim to build on. However, we will argue that such speedup approaches cannot be applied as usual in the unsupervised case considered here: in particular, while it is usually expected that a lincar decision boundary is fit after the Nyström or Sketching (compression) step, we argue that we need to instead fit nonlinear clustering boundaries in the unsupervised regimes considered here, if we are to maintain detection accuracy.
To better understand relevant distinctions between unsupervised and supervised OCSVMs in the context of speedup methods, we need to get into a bit more detail. Most significantly, these methods all operate on a so- called grom matrir IC E R=*x, encoding relations between data points, i.c., inner-products Ki, = corresponding an implicit data transformation + (xx). Operations on K. are often the bottleneck in training and prediction time, and approaches such as Nyström and Sketching approximate KC with a lower-rank matrix 1C' that allows faster operations, while nearly preserving the original relations between transformed data points ((X,)ss s. In particular, in the case of SVM, the Nyström or Sketching matrix AC' results in a new transformation x + ((x) + ø' (x) where for any two data points X,, from separate classes, ø' d((),) and d((,,) remain linearly separated if o(X.) and Φ(X,) were linearly separable. In other words, one can simply proceed similarly with K' in place of IC and train a lincar classifier. Unfortunately, as we will see (Section 3.2, in the unsupervised case of OCSVM, such &' (X.) and o'(x,) might no longer be linearly separable an issue particularly true in ToT requiingg us to develop a different approach on top of Nyström or Sketching.
(X())
To address this issue of nonlincarity in the case of OCSVM after applying Nyström or Sketching, we rely on recent interpretations of these speedup approaches (Yang et al. 2012 Rudi et al., 2015 Kpotufe & Sriperumbudur 2020) which show that the resulting transformation o' preserves distances between transformed points ((X)), ((x,) (i.e., distances in the transformed kernel space {o(x))), even if linear separability between classes is not preserved. As such d-distances are preserved, one might then expect that cluster structures that may be apparent under d-transformation remain apparent under d'. Building on this intuition, detection will therefore just consist of flagging any future query point x as abnormal if d'(x) falls far from clusters in the remapped training data. (x.)),) To implement this idea, model clusters in (o((X.) as components of a Gaussian Mixture Model (GMM), which has the benefit of allowing for
2

---

Published in Transactions on Machine Learning Research (11/2022)
simple detection rule based on density levels (sec Section -. Finally, as the GMM model introduces new hyperparameter on top of vanilla OCSVM, namely, the number k of Gaussian components (or number of clusters), we further propose a basic approach to automatically set the parameter k by estimating high density regions of fØ (xx)))== via existing methods such as QuickShift+++ (Jiang et al.) 2018
Results Overview. We implement the above described approach based on mapping the normal training data as {d' (xx))) using either Nyström or a simple form of Sketching termed Kernel Johnson-Linderstruuee (KJL) shown recently to preserve cluster structures w.r.t. the original mapping e induced by kernel methods such as OCSVM (Kpotufe & Sriperumbudur 2020). For simplicity, we will henceforth refer to these approaches respectively as OC-Nyström and OC-KJL, where OC stands for One Class (as in OCSVM) to emphasize the unsupervised nature of these methods. We evaluate OC-Nyström and OC-KJL, both with and without automatic GMM parameter selection, on multiple loT datasets encoding a variety of detection use cases of interest from detecting benign new devices to malicious activity from infected devicos.
To evaluate the effectiveness of our techniques in the context of IoT anomaly detection, we lest against both public network datasets that apply to loT environments and datasets that we have generated in the lab based common interactions with consumer loT devices. In addition to lo'T-specific datasets, we also evaluate our algorithms on several public datasets involving traffic generated by general-purpos computing devices that are pertinent to ToT settings, including distributed denial of service (DDoS) attack detection and novel device activity. The very nature of 1oTT device's 's network behavior facilitates faster detection time and space: typical IoT devices, e.g,, smart appliances and traffic monitors, have few operational modalities, inducing few clusters of normal traffic; as a result, we can expect a small number of clusters, ie., GMM components needed to faithfully model normal operational traffic, leading to smaller memory footprint and detection time complexity, Our results are as follows:
Significant reduction in detection time and space. We observe typical detection time speedups (w.r.t. the baselin OCSVM) between 14 to 20 times faster using either OC-Nyström OC-KJL, and 40+ times for some datasets. Typical space complexities decrease by factor of 20 or more w.r.t. OCSVM. In particular, for cnmputattionall-cootttaannnn
deployment platforms, o.g., a home router, OCSVM detection time is typically in the order of 100 ms per data point (see Table 3), which is now reduced to about ms per data point: combined with the reduction in space, this opens up processing multiple data points from multiple 1oT devices simultaneously on the same router, while maintaining detection performance, as explained in the next bullet point.
Equivalent or improved detection performance. Given that detection performance of any machine learning method depends on hyperparameter choices, we consider two situations: (1) a situation where hyperpa- rameters are adequately calibrated using side data (i.e., a small validation set independent of future test data), and (2) a situation where such side data might be missing and basic rules-of-thumb are employed to select hyperparameters. Such a situation might arise in IoT settings where some activities and devices might be labeled, but the vast majority remain unlabeled due to the dataset scale and heterogeneity.
Upon proper calibration of all three procedures, both OC-Nyström OC-KJL achieve detection per- formance on par with the baseline OCSVM as measured by area under the curve (AUC). Both slightly outperform OCSVM in some cases, which is likely due to the fact that the new mapping d' acts as a lower-dinensional projection which at limes recovers intrinsic structure not present in abnormal traffic.
In the second sit sation i.e., under rule-of-thumb choices of the main hyperparameter shared by all three procedures (i.e., a so-called kernel bandwidth parameter), OC-Nyström and OC-KJL (with automatic choices of number of GMM components k) attain at Icast. 0.85% of OCSVM's AUC on most datasots and manage improvements in AUC over that of OCSVM on many datasets. Given the lack of proper calibration, we observe some rare satuations where AUC degrades more considerably compared to OCSVM. We include these results to give fair and broad sense of the range of performance that one could potentially observe in practice.
The rest of the paper is organized as follows. In Scction 2 we go over related work on novelty detection in computer networking and for IoT in particular, and further discuss the appeal of OCSVM
Paper Outline.
2Such single-board computers have similar specifications as Raspberry PT or Nano devices used in our experiments.
3

---

Published in Transactions on Machine Learning Research (11/2022)
in this domain. Section gives more detailed background on OCSVM, and relevant intuition Nyström and KJI,; we then build on this intuition in Section I to derive OC-Nyström and OC-KJI approaches in detail. In Section 5 we describe our experimental setup, including preprocessing choices for encoding network traffic as vectors, and evaluation metrics in detail. This is followed by experimental results in Scction
1.00
0.98
0.95
OCSVM
0.90
GMM
KDE
0.85
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
Train size
SFRIG
DWSHR
MAWI
Figure 1: Delection performance OCSVM that GMM and KDE two ToT dalasels (a and and
ceneral-purpose PC (c). OCSVM remains competitive against these other classical and flemible methods (GMM and KDE). Shown on all datasets, are the best performances obtained across hyperparometer choices (such choices are described in Section 5.1
2
Related Work
We survey related work in anomaly detection in networking, both generally and in the context of IoT.
Network Anomaly Detection. Anomaly detection in networks widely studied. Ahmed et al. provide a more complete survey of these techniques (Ahmed et al. 2016); we briefly overview some of the general classes of techniques. Various supervised learning techniques have been applied to the problem of network anomaly detection, including Support Vector Machines (Eskin et al. 2002). Bayesian networks (Kruegel & Vigna 2003), scquential hypothesis testing (Jung et al. 2004),, and neural networks (Hawkins et al. 2002 Wang et al. 2017). In many of these cases, supervised learning has been applied in very specific context when labeled datasets were available, such as detecting port scans (Jung et al. 2004) or web-based attacks (Kruegel & Vigna 2003). OCSVM remains a common technique for performing anomaly detection in loT, having been used in a variety of contexts, including sensor networks Rajascgarar al. (2010), intrusion detection of
system calls Heller al. (2003), network intrusion detection Zhhang et. al. (0015 5 and anomaly detection in
wireless sensor networks Zhang et al. (2099).
Common unsupervised approaches have involved principal component analysis (Shyu et al. 2003 Lakhina et al. 2004) and generalized likelihood ratio Thottan & Ji 2003). Principal component analysis in particular has proved problematic in the context of network anomaly detection due to the fact that transforming network traffic into a matrix representing a multidimensionall time series involves quantization and discretization that render the resulting underlying models brittle (Ringberg et al.) 2007). Ringberg et al. found that when applying PCA to network traffic anomaly detection, the false positive rate is sensitive to the selection of the number of principal components in the normal subspace and the level of traffic aggregation [Ringberg et al. 2007..
Anomaly Detection in IoT. Unsupervised learning approaches popular approach for IoT, where obtaining detailed labels for large, heterogenous set of devices is impractical. Over the past several years, unsupervised lcarning techniques have been developed for novelty detection specifically for IoT devices and activities; One-class SVMs (OCSVMs) have been particularly effective for delecting anomalies in IoT settings (Shilton et al.) 2015 Lee et al. 2016 Mahdavinejad et al. 2018 Al Shorman et al. 2020 Razzak ct al. 2020 OCSVMs are appropriate for novelty detection IoT due to their ability to learn complex, nonlinear decision boundaries, which can be important in IoT environments where activities are diverse and heterogeneous. Note that OCSVMs do not always have superior performance, alternative methods
4

---

Published in Transactions on Machine Learning Research (11/2022)
can indeed be better on given IoT device's datasets; however, they've received much attention in the IoT domain as they are simple to operate and compete well across a variety of ToT problems as argued in the aforementioned works. Such flexibility and competitiveness is illustrated in Figure vs. other classical (flexible) novelty detection approaches.
Unfortunately, despite its efficacy in these settings, OCSVM can be particularly costly terms of both time and memory requirements, rendering the previous work impractical for many deployment settings where novelty detection algorithms would be deployed in practice. Specifically, IoT deployments involve the deployment of resource-constrained devices; in the case of consumer 1oT deployments, for example, anomaly detection systems may need to operate on home routers, where processing and memory capacity is limited. The algorithms we develop in this paper achieve a speedup of up to 40 times as compared to the best-known implementations of OCSVM, thus making it possible to deploy these anomaly detection algorithms in practice IoT settings. To demonstrate this feasibility, we evaluate the real-time performance and memory requirements of our algorithms on embedded single-board computers that are often deployed in home network settings.
3 Background on Methods
3.1 (Gaussian Kernel) OCSVM
Basic Background. OCSVM first maps data x E RD as <(x) into an infinite dimensional space H (a so-called reproducing kernel Hilbert space (RKHS)). As Hilbert space, H admits basic vector operations as in Euelidean RD, in that it has a well-defined inner-produce (ø(z), ((z') inducing a norm |((x))] (6(x), o(x)) and hence notion of distance between points and space geometry (clusters, lincar projections, hyperplanes, spheres, etc). All that is therefore needed for geometric operations is access to the inner- product operation (,,), which is readily provided by RKHS theory: for any data points 2, E RD, there exists a so-called kernel fumction K satisfying K ((x,²') = (ő(x), Φ(x')) Therefore, given access to K, the mapping é need not be explicitly computed, as all geometric operations are implicit through K alone, and in particular, all geometric operations involved in learning hyperplane separating classes of points are thus determined by K alone. The most common kernel function in machine lcarning, and especially in OCSVM, is the Gaussion kernel K(,,z') 2²) C' exp - || x' /2h²) (for bandwidth hyperparameter h to be chosen in practice, and a normalizing constant C² = C(h)).
C'expp
Key Intuition and Operations. A main intuition behind the mapping 6, implicit in both supervised SVM and unsupervised OCSVM, that it manages to seporale classes of data, i.., pull corresponding data points far apart in H, even when they are not eas- ily separable in their original representation in RD. This is illustrated in Figure 2 It follows that, after the mapping o, the data might become linearly separable in H, i..,, the two classes of data, normal and abnormal, fall on different sides of a hyperplane in H. Therefore, in supervised learning (c.g., with SVM) where we have access to both classes of data at training time, simply would learn hyperplane that most faithfully separates the training data into the two class labels. However, in the case of OCSVM, only one class is available during training, namely, normal data. It therefore unclear how sep- arate it from unseen anomaly data. The main insight is that, if the kernel K satisfies K(xx C' for some constant C, as with the Gaussian kernel, then all points z E R are mapped in to the surface of sphere of radius vc, since K(x, = [| C. It follows that if the two classes are linearly separable, then they can be separated by hyperplane that puts
o (z) Rd
d(x)
GMM
R
T. E IR"
OCSVM maps data points E B.. as 6(x) onto a ball in R inducing linear separation between normal points (gray) and yet unseen novel points (in red). We consider mappings ø', which then remap down to IR, d & D, while maintaining separation (únto clusters), but not necessorily linear separabilily. Given the few modalities of 1oT devices, we can then learn GMM with few components to model the remapped normal data.
Figure 2:
5

---

Published in Transactions on Machine Learning Research (11/2022)
maximal margin between the normal class and the center of the sphere since unseen anomaly data is also constrained to map to the surface of the sphere. This is illustrated in Figure 2
OCSVM, thus, using normal data {X:}-- alone, returns a hyperplane that isolates normal data from future anomalous observations. Such a hyperplane can be estimated without actually computing ((x;) E H. simply through geometrical operations encoded by all pairwise inner-products (((X;), ((x,)) given by ( X)). These inner products are encoded for convenience in so-called gram matrir K R=** Ki K (X,, X3) the training phase just operates on Æ to return implicit representation of the separating hyperplane in the form of coefficients {as)== and threshold ag used as follows:
A future test point E RD, is deemed anomalous if it maps as o(x) to the wrong side of the hyperplane, that is, if f(z) = Σi= (XK,X,,r) < ag.
In other words, as in Euclidean spaces, f(z can be viewed as the projection of o(z) onto a vector normal to the separating hyperplane, and the a; 's are coefficients determining this vector.
Detection Time and Space. It should be clear by now that computational complexity is determined by the number S of nonzero a; 's.. The corresponding data points X/'s are called the support vectors and have to be kept in memory to estimate f(x). Thus the OCSVM detector takes space ñ (D + 1), while the computation time for f is li x D). Unfortumately, it is often the case that ñ or is of the same order, while the larger n, the more accurate the detector is.
3.2 Nyström and KJL Sketching
A main approach adopted recently to speedup training time, e.g., in the context of SVMs, is to reduce operations on the gram matrix K. E "==x" by approximating it with rank d & n matrix K' E "R" that might induce faster operations, while preserving much of the geometry induced by the kernel K on the implicit mapping {o(X,))²= E H. These come in different forms under the name of Nyström and Sketching. In particular, some implementattioss we can view K'' as inducing a new mapping x + o (xx) for ('()) E Rdd i.e,, a dow-dimensional mapping that preserves some geometry in H.
Critically, explained in the introduction, such o' often no longer allows for linear separability from 0- i..., using just one class in the training data- as in the case of the original OCSVM map σ, since the remapped data {Ø' ())( no longer lies on the surface of a sphere (sce Figure 2). However, cluster structures uncovered by the original é are preserved since ø' preserves interpoint distances (see e.g.. Calandriello & Rosasco (018);; Kpotufe & Sriperumbudur (2020)), ) which we build on in Section below.
The Embedding &'. Crucially, in order to leverage cluster structures towards efficient outlier detection, we make the embedding ø' explicit as opposed to operating on K' -nd and work directly in Rdd This is based recent reinterpretations of forms of Nyström and Sketching as dinndimensional projections (Yang et al. 2012: Kpotufe& & Sriperunbudur 2020). In both cases, let Sm denote a random subsample of size m < n of the training data Sn = [X, (w..o.g., we can let Sm. = (x)))))) Furthermore, for any subset of indices J C [1,...,n}, let KG,J denote the submatrix of K. corresponding to rows in I, and columns in J. Then, for I 1= [1: ::::i] m} and {1 n), we will consider the submatrices, Kr, E gmxm i.e., the gram matrix on Sm, and K2,J E Rmxx the gram submatrix of inner-products between Sm and Sn,.
Nyström. Let K7.} denote rank d pseudo-inverse of K then setting K' = KJ,, KK,, K1,,, the problem is to come up with o' E IRd such that (ø'(X,), (' o'(X,)) is exactly K' Recalling a bit of linear algebra, we can see that suitable ø' can be defined as follows YYang al. 2012). Let A Rddd denote the diagonal matrix containing the top d eigenvalues AI,..., Àa of KT,J, and V (u,,..., va] E mxd contains the corresponding (column) eigenvectors Ug's. Now, for any x E RD, let K(x) denote [K (x, XI).... K(I, XT)] we then have
(i.....,all
d((r) = P. K(x), where we let P 11/ VT.
(1)
We can verify that setting K V A-1, VT indeed recovers KC' defined above.

---

Published in Transactions on Machine Learning Research (11/2022)
KJL Sketching. In general, Sketching consists of multiplying gram matrix K. (or K1,1) by matrix Z with random entries. It was recently shown (Kpotufe & Sriperumbudur 2020) that when Z has i.i.d. (0, 1) Gaussian entries, sketching can be understood as a random projection operation in H, leading to the following mapping o' E R For any x E RR,, let (x) again denote the vector [K, K(x, Xm)] and let Ze E Rdxm with random (0, 1) entries. We then have:
o' (x) = P. Px) where we let P Z KI,I-
(2)
Embedding Time and Space. Notice that in both cases of Nyström and KJT,, we only have to retain P E _dx at testing time, along with the m data points in 5m. Namely, the model d' requires space complexity exactly m (d (d+Dl D). Similarly, the time complexity for evaluating &' (xx is of order m (d D), so it does not depend on n.
As turns out m, d can be kept considerably smaller than n, while achieving the benefits of both methods. This is illustrated in Figure 3 on simulated data of size n = 10000, with two classes that are not easily clustered in RD, but which are clusterable not only in H. but also after Nyström of KJLL In that simulation, used d d= 2 and m 100. Similar small values are used for our experiments on real-world IoT data (sce the experimental setup in Section 5.1
Data
Projected data by KJL
Projected data by Nystrom
8
100
1.0
50
0.5
0
0.0
-4
-50
-0.5
-8 -8
-100
-1.0
0
8
-150
-100
-50
0
50
-1.0
-0.75
-0.5
-0.25 0.0
Normal
Novelty
Figure 3: Clusters after mapping ø': the simulation dala Cluster in Cluster has 5000 points, shown before and after KJL/Nystron mapping. The KJL/Nyströnn mapping &', shown on the right, retains the clusters uncovered by the initial kernel mapping 6.
4 Efficient Detection Procedures
Once the data mapped to Rdd as (x(())) through Nyström or KJI., our next step is to learn an efficient model of the normal class embedded in Rd. Recall that cluster struc- tures are preserved, but not necessarily linear separability from 0 (see c.g., simulation of Fig- ure where the normal class is not necessarily linearly separable from the origin 0 R²).
Looking somewhat ahead, this intuition is validated with the results of Figure 4 where we compare fitting a linear separator after KJLL projection (denoted OC-KJL-SVM) to our proposed method (OC-KJII. soon to be described. The detection performance metric is the AUC, which is consistently higher for OC-KJL across datasets. A natural
1.0
0.75
AUC
0.5
0.25
idea therefore is to flag future points novelty if they fall far from clusters in (d' (X,)) This may be implemented in a number of ways via existing clustering procedures such as K-MEANS; however, is then unclear how to efficiently and soundly evaluate what we might be how far a new point is from estimated clusters. We opt for simple implementation consisting of fitting a Gaussian
0
UNB
CTU
MAWI
MACCDC SFRIG AECHO DWSHR
OC-KJL
OC-KJL-SVM
Figure 4: OC-KJL (using a GMM) performs better
than OC-KJL-SVM (using a linear separtor).
7

---

Published in Transactions on Machine Learning Research (11/2022)
Minture Model (GMM) to the normal remapped data (d)( with & components that encode clusters; such a GMM is defined as probability density of the form
h f(z) =
Σ m N (2: Ex), for any E Bd
where M (z; Mrs 21) denotes Gaussian density with mean μr and covariance Mn evaluated at 2, and my's denote the probability mass of each cluster 1 E {l,...,k) and sum up to 1. The GMM I would have modes μt, ie., local maxima a.k.a. high-density cores, centered on clusters, as illustrated in Figure 2 An immediate advantage of employing GMM's as a clustering approach is that, once f is learned, detection simply consists of flagging x as a novelty if f(ø' (x)) smaller than a threshold 6, ie., if the remapped o' (æ) falls in a low-density region far from cluster means. In practice, such a threshold can be picked depending on the amount of tolerable false positive; for instance, 95th quantile of values (in decrcasing f order) on the negative data, i.c., the embodded normal data (d'(X)) In our experiments below, report the performance of detectors across all such thresholds choices, as captured by AUC (Sections 5.1 and
get at most 5% false positives, we as the
Meta Procedurcs. The resulting OC-Nyström and OC-KJL approaches are summarized below. Given a Gaussian kernel K with bandwidth h, embedding choices m, d < training size nc:
Training: Given normal data (X.)
"R do:
Embed X,'s as o' (X..) Rd via Nyström equation or KJL equation 2
Parameter k is passed in or is chosen via Quickshift (see paragraph below) on embedded data (0'(X.)) ER4;
Estimate GMM with & components on [d'((.))?=
Return GMM ff along with projection ø' (i.e., matrix P and subsample Sm)
D
Detection: Given new E B and model (ø', do:
Embed as (xx into R4;
Flag novelty iff f( (x)) threshold
Choice of Number of Components k. As discussed earlier, we may automatically choose the number of components & by first identifying the number of high-density regions in the mapped data (d)(X.)) This might be done a number of ways, and we propose to use available density-mode estimators such as from the Meanshift family Comaniciu & Meer (1999); these are procedures that automatically identify the modes, i.e., local maxima, of the underlying data density, which in simple terms are just the regions of highest density in the data. In particular, in this work employ recent fast version of these mode estimators denoted QuickShifl++ Jiang et al. (2018). which automatically returns points in locally high-density regions of the data, with no a priorí knowledge of the number of such regions, which we will identify with clusters. However, if labeled side data is available to cross-validate for the OCSVM or Nyström and KJL bandwidth parameter h, the same data can be used to choose (see Section 5.1p
Detection Time and Space. As described in Scction 2.2., saving the model o' takes space m (d D); while f now takes additional space & (d+ + d²) for GMM parameters. As m, d can be chosen small, detection time mostly depends on k; fortunately, discussed in the introduction, can be chosen small (between and 20 in our experiments) clusters naturally correspond to the typically few modes of normal operation of loT devices.
OC-Nyström vs. OC-KJL, As we will see in the results Section 5.2 both procedures achieve our intended goal of efficiency while maintaining detection performance on par with that of OCSVM; while advantages vary across datasets, OC-Nyström tends to trade a bit of efficiency for better detection, as its embedding require larger k values.
8

---

Published in Transactions on Machine Learning Research (11/2022)
5 Experimental Setup and Evaluation
5.1 Experimental Setup
Data Sources. We consider both publicly available traffic traces and traces collected private consumer IoT devices. We aim to evaluate representative set of devices, from multi-purpose devices, such as laptop PCs and Google Home, to less complex electronics and appliances with few modes of operations, such as smart cameras or smart fridges. Furthermore, we aim at a representative set novelties, from benign novelties (new activily or new device lype) to novelties due to malicious activities (DDoS atlack). Table describes the datasets we used in the main paper, along with the associated types of novelty being detected. There are seven datasets in total, in which three of them are IoT datasets collected from three IoT devices deployed at the University of Chicago, and the remaining four are public datasets (i.e., CTU IoT, UNB IDS, MAWI, and MACCDC). Furthermore, the types of novelty vary from benign novelties (new activity or new device type) to novelties due to malicious activities (DDoS attack).
Table 1: Datasets' details.
Type of Novelty
Datasets
Description
Devices
Data traces are generated by Samsung Fridge (SCam) with IP 118 282.202 a private lab enviroument. has two types of traffic traces labeled as normal when there (such as opening the fridge).
Lab IoTT SPRIG
Novel activity
One fridge
human interaction and, novel when being operated by human
Data traces are generated by an Amazon ECHO (AECHO) with IP 1192.168. 168.228. 202.74 in private environment. labeled normal when there is human interaction, and novel when being operated by a human (such as buying fond by the BCHO).
Lah ToT AECHO
two types of traffic traces
One Amazon
ECHO
activity
Data traces are generated by a dishwasher (DWSHIt) with IP 1192. 192.168.202.77 202.76' a private environment. mal when there is no human interaction, and novel when being operated by a human (such as opening the dishwasher). We also add another novel traffic (such as buying food by ABCHO) collected from Amazon ECHO (with IP '192. 168.202.1747 into novel to get a bigger testing set.
types traffic: traces labeled nor-
One dishwasher
Lab ToT DWSHR
Novel activity
and
Amazon ECHO
CTU loT
Bitcoin- Bitcoin-Mining and Botnet traffic traces generated by two Raspberries; we use traffic (with IP '192.168. 1...1.196) as normal Bitcoin Mincinggiining traffic (with IP 192. 168.1.69911111 novel.
Two infected Raspberry Pis
Novel (infected) device
Garcia
2019
UNB
Normal traces are generated by one personsl computer (PC) with IP address '192.168 16w.166..00 10.9' Attack traces generated by three PCs with IP addresse:: 168.168.11. 10..1 1192.168 112.166..1.14: and '192.168 10..1.
Sharafaldia
Four PCs
DDoS attack
ct al.
2018
Normal traffic are collected on July 01, 2020; we choose one kind of traffic gen- crated by PC with IP '203.78.7.166 as normal, and another kind traffic generated by PC with TP address 11868.54.240 as novel.
MAWI
Naga &
Kaisaki
2020
PCs
(normal) device
MACCDC O''Brien et al 2012)
Data traces are collected in 2012; we choose one kind of traffic generated by PC IP '192. 168. 202.70..7 79' normal and kind of traffic generated PC with IP 11922166 112.122..2277 from another peap as novel.
Novel (normal) device
Two PCs
Although some the devices that we test are multipurpose (as such might display more modalities than a special-purpost IoT device), including them allows us to test how well our approach scales, In particular, we will see that efficient detection is possible even in such cases, even then k 20 clusters suffice to maintain detection performance over these datasets, keeping d 5 and 100.
Representation: Flows. Our unit of measurement consists of traffic flows, described below, i.e., as we aim to flag flows normal or novel. We parse bidirectional flows from datasets in Table I using Scapy [Biondi 2021) and extract interarrival times and packet sizes. Because certain devices can have arbitrarily long flows, we truncate each flow from a given dataset to have duration at most that of the 90th upper-percentile of flow durations in the dataset. Henceforth, a flow refers to these choices of flows involving truncation. We randomly split the obtained data into training, validation, and test sets of sizes dctailed in Table B. in Appendix.
Representation: Features. Every flow represented as a vector of the interarrival times between packets, iee, in microseconds elapsed between consecutive packets, along with the size in bytes of each packet in the
9

---

Published in Transactions on Machine Learning Research (11/2022)
flow (IAT+ We select these features as it results in competitive detection accuracy for OCSVM. This is illustrated, e.g., against 2 common alternative feature choices, namely, STATS and SAMP SIZE, shown in A different choice, STADS+HEADER, corresponds to common statistics on flows, e.g., flow duration, mean, standard deviation and quantiles of packet sizes, in addition to packet header information (Yang et al.
Table 2
2020 as explained in detail in Appendix E.2
Results for the alternative features set STATS+HEADER are given in Appendix E.3 further demonstrating that our speedups generally hold over choices of data representation, as significant savings in time and space over baseline OCSVM remain. Such generality is expected because the main source of savings in both time and space stands, namely, the succinct finite-dimensonnal modeling of the infinite-dimensional representation inherent in OCSVM, made possible by the few modalities displayed by typical loT devices.
Implementation and Hyperparameters. All detection procedures are implemented in Python, calling on the scikit-learn package for existing pro- cedures such as OCSVM and GMM. While OCSVM training uses the standard libsvm package, we re- implemented its detection routines (as described in Section
Table Average AUCs for allernative features.
Dataset
MAWI
SPRIG
DWSHR
IAT+SIZE
0.00
0.93
0.93
STATS
1.00 0.00
0.86 0.00
0.63 0.00
SAMP_SI2
0.99 + 0.00
0.93 +: 0.00
0.92 +: 0.00
3.1 using Numpy to ensure fair, apples-to-apples execution time comparison with OC-Nyström and OC-KJL, which are implemented in Numpy, a Python library that calls on fast algebraic operations and parallel processing on multicore machines (Harris et al., 2020). The Nyström and KJL projections are implemented as described above; All the source codes can be seen at KJL
Training Scenarios. We consider two practical scenarios: one where some small amount of labeled novelty data is available validate hyperparamcter choice in a controlled lab environment, and one with no such labeled validation data, where we have to result. in default choices of hyperparameters. Although each detection procedure may have many internal parameters, this distinction in scenarios only applies to two key choices of hyperparamcters:
Kernel Bandwidth h. For all methods, i.c., OCSVM, OC-Nyström, and OC-KJL, we use Gaussian kernel of the form K(x,x') x exp(-llæ- z'1l²/h²) where the bandwidth h is to be picked as quantile of (*) distances wetw the n training data points. In all our results, we consider 10 quantiles 0.2, .......)))) 0.9] U (0.95) of increasing interpoint distances.
Number of GMM components k. As explained above, OC-Nyström and OC-KJL also require a choice of the number of GMM components to fit. We consider choices in the range [1, 4, 6, 8, 10, 12, 14, 16, 18, 20). Thus the number of components or clusters k is capped at 20, as IoT devices are expected to display relatively few modes of operations, i.c., clusters of normal network activity.
For the results presented in the main tezt, the choice of k is made by QuickShift++, and the resulting procedures are denoted OC-Nyström-QS and OC-KJI-QS: these versions of our fast methods therefore only leave the choice of bandwidth h and will be our main focus.
Next, we discuss how the above parameters are picked in each of the use cases or scenarios discussed above.
Minimal Tuning. To simulate the first training scenario where some small amount of labeled novelty data is available, we subsample a small amount of the novelty data, which paired with equal amount of normal data is used to form validation set to be used in hyperparameter choice. We then proceed to choose h or l (when Quickshift++ is not used) to minimize AUC over the validation data, so that these choices are independent of the random test set on which final results are reported.
No Tuning. In this case, we choose the bandwidth à by a common rule-of thumb as the 0.3 quantile of increasing interpoint distances on the training data. The choice of the number of components & is then always made by Quickshift We observed the same speedups under these settings, although these results are related to Appendix D for space.
10

---

Published in Transactions on Machine Learning Research (11/2022)
All Other Algorithmic Parameter Choices are Fixed. We now describe all other choices inherent in our procedures, OC-Nyström and OC-KJL, and their variants OC-Nyström-QS and OC-KJI-QS.
Projection Parumeters. As discussed in Section | subsamples size m and projection dimension d are fixed to m = 100 and d' = 5, choices which remarkably preserve detection performance across datasets and types of novelty, despite the considerable amount of compression they entail.
Quickshift++ Parameters. We use the implementation of (Jiang et al. 2018 Jiang al.), which requires internal parameters set to 0.9 (this performs density smoothing) and the number of neighbors set to n2/3 (to build a dense neighborhood graph whose connectivity encodes high-density regions), two choices that work well across device datasets and types of novelty.
Here, due to variability in the data, Quickshift+++ can often return too many outlier clusters (despite the conservative setting of its internal parameters). To remove those, we only retain large clusters, namely, the smallest number of clusters that account for at least 95% of the data if this number is less than 20; otherwise, we retain the 20 largest clusters discovered by Quickshift++.
Gaussian Misture Models Parameters. We have the choice of using either full Gaussian covariances in fitting GMM model to the projected data after KJL or Nyström or using only diagonal covariances for faster fitting especially when operating in high dimensional settings - but at the usual cost of some loss in accuracy. Since GMMs are fit after projection to low dimension d = 5, it turns out that full Gaussian covariances are in fact efficient to fit in our case, so we only report results for full covariances.
When using Quickshift+++; we initialize GMM with the clusters returned, i.e. local means and covariances of these clusters, and train till convergence.
Computing Platforms. We perform our experiments on two computing platforms: (1) a well-provisioned server, for the use case where all training and detection might occur offline; and (2) resource-constraing devices, specifically a Raspberry Pi and an NVidia Jetson Nano, corresponding to the use case where detection is to be real-time, local to the loT device. Table A.] in Appendix provides details.
5.2 Evaluation Metrics
Detection Performance. In novelty detection, there is a well-known tension between false delection a.k.a. folse positive rates (FDR, i.e., the proportion of normal data wrongly flagged novel) and true detection a.k.a. true positive rates (TDR, i..., the percentage of abnormal data rightly flagged as novel). Such tradcoffs are well captured by a Receiver Operating Characteristic (ROC) curve, which plots the delection rale TDR against the false alarm rate FDR as the detection threshold t is varied from small to large; thus, the area under the ROC curve (Area Under the Curve (AUC)) when is large, i.e., close to 1, indicates that good tradeoffs are achieved by the given detection approach. In contrast, AUC below 0.5 signals poor tradeoffs. AUC is therefore commonly adopted under the complete range of detection choices.
sensible measure of detection performance, it captures tradeoffs
In practice, a single threshold is chosen, driven by application-specifc constraints, as one might prefer high TDR over low FDR, or vice versa (think of an infected medical device, e.g,, a pacemaker, where high TDR would be preferred, vs. an infected smart home appliance, c.g., a toaster, where low FDR might be preferred). Large AUC, thus, indicates that the detector allows for good choices in any of these situations. For our proposed fast detectors, we will be interested in the fraction of AUC retained over OCSVM, i... the AUC of our detector divided by that of OCSVM.
Training and Detection (or Testing) Time. We will measure time as the wall-clock lime taken by any of the methods for training (not-including data preprocessing into feature vectors, but inclusive of all actual training, i.c., modeling fitting) and testing, i.c., actual detection computations, on given machine environments (see Section 5.1 below) after a model is obtained. We report the speedup, the ratio of wall-clock time for OCSVM over that of our detector, separately for training and testing.
11

---

Published in Transactions on Machine Learning Research (11/2022)
Detection (or Testing) Space. We report the space taken by the model returned by the detection procedure in kiloBytes. Namely, we report the minimal amount of information on the learned model to be saved towards future detection. That is, (1) support vectors and coefficients for OCSVM, and (2) projection parameters and GMM components for OC-Nyström and OC-KJL (with or without Quickshift++) all as described in Section 3.1 and While memory usage depends on the programming language, under Python 3.7.3, memory usage is machine-independent, as Python enables porting across 64 or 32-bit machine architectures via its pickling process. All of our models are first trained on a 64-bit machine (Section 5.1
Averaging and Data Splitting. To reduce uncertainty in reported results, we introduce repetitions in various stages of our experiments and report averages and standard deviations on performance metrics. For each dataset, first. all flows (normal and abnormal) are preprocessed into the IAT+SIZE features, which creates at set of normal and novel data, from which we draw random subsamples. Experiments on each datasct follow the steps outlined below.
(i) Draw subsample of size 600 to 2500 from normal data and subsample of size 600 to 2500 from novelty data to form a test dataset of size 1200 to 5000. Exact sizes are given in Appendix Table B.1
(ii) Repeat 5 times for accurate AUC: 10K from normal data to form the training data, except for MAWT (n 5.7K).
Draw a subsample of size
If tuning: draw a validation sample (1/4 test set size).
Choose parameters h, le as described in Section 5.1
Train with the choice of h, and save model on disk.
Load and test model Test data: repeat 100 times for accurate timing on machine (retain aggregate time).
For the baseline OCSVM, we report the average and standard deviation of performance metrics over the 5 repetitions. When reporting speedups for OC-Nyström and OC-KJL over OCSVM, we use the corresponding average performance of OCSVM, say μ.. In other words, if we observe AUCs ......, a5 for OC-KJL, we report the mean of ai/p,... as/u a + the std of these ratios. We proceed similarly for time ratios.
6 Results
We focus in the main body on the first situation where some validation is available to tune model hyperparam- eters as described in Section 5.1 Results for the case of no tuning (as described in Section 5.1) reveal similar speedups in time and space as shown here and are given in detail in Appendix D We further focus here the OC-Nyströn-QS and OC-KJI.QQS for fair comparison as OCSVM comes with single parameter h-while results for OC-Nyström and OC-KJL (which require h, k to be tuned) given in Appendix C As previously discussed in Section 5.1 all testing procedures are implemented in Numpy with parallelism turned on to take advantage of multicores.
OCSVM Baseline Performance. Table shows OCSVM baseline performance. All training is performed on the server; testing is performed on all platforms. The table reports (1) AUC, same for all machines, since the same models and test data are used; (2) training time on the server; (3) test time for all three machines; and (4) test space, which is the same across all machines.
Table 3: OCSVM baseline performance. Time is in milliscconds 100 dala points and space is in kiloByles.
Dataset
UNB
CTU
MAWI
MACCDC
SFRIG 0.93 + 0.00
AECHO 0.90 + 0.00
DWSHR
AUC
0.67 + 0.01
0.66 + 0.02
0.99 + 0.00
0.85 +: 0.00
0.93 + 0.00
Server Train Time (ms)
93.93 4.75
77.87 1.72
82.59 1: 1.53
84.19 1.93
80.83 1: 7.72
111.26 1 6.59
110.53 1: 5.77
RSP1
125.86 +: 0.08
118.19 +: 4.54
74.53
4: 0.05
124.58 4: 0.22
120.12
0.22
125.43 4 0.22
124.54
0.28
Test Time (ms)
NANO
83.87 1 0.18
86.71
0.82
58.57 1: 0.05
$6.89 1: 0.80
69.62 1: 0.09
88.80 4: 0.08
84.07 1: 0.04
Server
19.84 +: 0.16
19.97 4: 1.36
11.22 1: 0.14
19.99 1: 0.23
16.15 1: 0.20
19.98 4 0.03
19.81 1: 0.01
Space (kB)
1763.53 + 0.45
961.52 + 0.75
2792.
16 + 0.48
1012.92
+ 0.45
641158
0.10
1441.75 + 0.14
1202.39 + 0.30
AUC Retained. As stated earlier, we now verify that our proposed methods manage to retain the accuracy of the baseline OCSVM and also do not sacrifice training efficiency. These are the results of Table 4 We
12

---

Published in Transactions on Machine Learning Research (11/2022)
Table Retained AUC (method over OCSVM) and server train time speedup (OCSVM over method).
Dataset
UNB
CTU
MAWT
MACCDC
SERIG
ARCHO
DWSHR,
Method
OC-KJT.QQ::
AUC Retained
1.33 + 0.02
1.05 = 0.03
0.95 +: 0.03
1.03 + 0.01
0.97 + 0.02
1.07 # 0.00
0.99 + 0.00
Train Speedup
1.36 +: 0.07
1.10 0.02
1.65 d: 0.03
1.14 0.03
0.97 1: 0.09
1.18 0.07
1.19 +: 0.06
OC-Nystršm-QS:
AUC Retained
1.37 + 0.02
1.13 = 0.04
0.90 +: 0.03
1.06 + 0.02
0.96 :: 0.01
1.09 0.01
0.98 + 0.01
Train Speedup
1.32 # 0.07
1.12 0.02
1.74 + 0.03
1.16 + 0.03
0.93 + 0.09
1.26 + 0.07
1.17 = 0.06
see that our detection methods largely retain the detection performance of OCSVM, often within a ratio of 1 or more, except in the case of MAWI, SFRFIG, and DWSHR which are still high AUCs considering OCSVM's very good performance on these datasets. Moreover, for some datasets, such as UNB and CTU, all procedures manage to actually outperform OCSVM. It is likely that such higher performance is due to the additional regularization inherent in the dimension reduction performed by our methods, In particular, it is well known that dimension reduction approaches such as PCA have the added benefit of making clusters more salient by keeping cluster centers apart while reducing cluster diameters (see e.g., Saajeev & Kannan (2001); Vempala & Wang (2004)). Here, we also re-emphasize that all methods presented are tuned fairly against the baseline OCSVM M over the same bandwidth parameter h as detailed in Section [5.] Versions with Quickshift+++, namely, OC-Nyström-QSS and OC-KJL-QS, tend to achieve slightly smaller AUC compared to the + counterparts where the number of components k is tuned by validation (see Table C.1 yet they also manage to maintain or sometimes outperform the baseline AUC of OCSVM.
Training Time. Our main goal is to maintain the training efficiency of OCSVM, and we see in Table 4 that our methods using Quickshift++ achieve this goal and even manage some minor speedup over OCSVM. We will see Appendix C.T that our methods without Quikshift+++ achieve significantly faster training time, although with the added complexity of an additional tuning parameter k. OC-Nyström-O0 QS's performance is similar to its non Quickshift+ counterpart, while at the same time it is more applicable in all scenarios, including when no validation data is available for tuning (results of Section Dj.
Significant Savings in Detection Time and Space. We report significant savings on detection time and space for all proposed variants of our approaches, which is the main motivation of this work. Table 5 and Table G present results for OC-KJL-QS and OC-Nyström-OR QS, while similar time and space savings under OC-KJL and OC-Nyström are presented Appendix Tables C.2 and C.3
Table OC-KJL-QS: Test time speedup (OCSVM over method) and space reduction (OCSVM over method).
Datasct
UNB
CTU
MAWI
MACCDC
SFRIG
AECHO
DWSHR
RSPI
24.68 0.02
24.22 0.98
15.61
27.56 +
26.38 + 0.05
Test Time Speedup
NANO
27.18 = 0.06
31.67 0.30
21.29
0.02
33.95 + 0.31
27.20 0.04
30.44 + 0.03
28.78 0.01
Server
26.33 = 0.21
28.10 1.92
17.17 0.21
29.72 + 0.34
24.15 0.30
32.32 + 0.05
30.94 4 0.01
Space Reduction
11.36 = 0.01
36.30 + 0.03
27.18 + 0.00
38.21 + 0.02
32.83 + 0.00
40.73 + 0.00
37.42 + 0.01
Table 6: OC-Nyström-QQ2 Test time speedup (OCSVM over method) and space reduction (OCSVM over method).
Dataset
UNB
CTU
MAWT
MACCDC
SFRIG
AECHO
DWSHR
RSPI
22.86 1 0.02
22.63
0.87
15.54
0.01
27.08 1: 0.05
26.48
0.05
26.24 1. 0.05
25.56 1. 0.06
Test Time Speodup
NANO
24.73 = 0.05
29.63
0.28
21.53
0.02
33.56 + 0.31
27.39
0.04
30.07 + 0.03
29.14
0.01
Server
24.16 à: 0.20
25.90
1.77
17.31
0.21
29.44 4: 0.33
24.36
0.30
31.73 d: 0.05
31.23
0.01
Space Reduction
39.99 = 0.01
35.09
0.08
27.21
0.00
37.98 4: 0.02
32.97 d: 0.00
40.66 d: 0.00
37.63 + 0.01
Our approaches are at least. 15 times faster than OCSVM on every machine we considered: Nvidia Nano, Raspberry Pi, and the server. Speedups on Raspberry Pi and the server are most considerable, up to 20+ times faster than OCSVM on many datasets. The smaller amount of speedup that we observe on the Nano can be attributed to the relatively smaller amount of memory that this device has compared to the Raspberry Pi, which likely forces more memory swap operations all test data is loaded at once. We also note that.
13

---

Published in Transactions on Machine Learning Research (11/2022)
unlike the Nano and Raspberry Pi, the server may have had more competing processes, yet even the server, the trend of large speedups is observed across datasets. We see small distinction between OC-KJI-QS and OC-Nyström-QSS whereby the former tends to achieve higher speedups on all machines for most datasets. As such both approaches seem to offer a tradeoff where, as per Table 4 the Nyström based approaches tend to achieve slightly higher AUC on most datasets.
In all cases, we observe significant space reductions, our models can be stored upwards of 27 times less space and up 41 times less than the baseline OCSVM model. This smaller memory footprint implies the possibility the
for wider deployment than conventional OCSVM, especially on memory-restricted devices such embedded devices we evaluated. Although we focused much of our evaluation on memory-constrained devices, which is a common deployment scenario for loT, the space efficiency of these models is important even in server settings, where a server might host large numbers detection tools each dedicated to monitoring a given machine on client networks.
7 Conclusion
Because many IoT devices have a few well-defined operating regimes, it is possible to model their network flows in lerms of relatively few clusters of activity under appropriate representations the data. In this paper, we have extended the OCSVM approach, which has received much attention in loT, to more efficient represcntations using projection and clustering. We have demonstrated that these procedures result in 15-40x improvements in both time and space without sacrificing detection accuracy across a wide range of novelty detection problems in IoT. These approaches, OC-Nyström-QS and OC-KJL-QS, are more widely applicable under practical use cases of novelty detection in IoT and in particular deployable not only on powerful servers but also on single-board computing devices with more limited memory and computing resources. Our evaluation of these techniques have also exposed some clear tradeoffs: when minimally tuned with a few labeled data, OC-Nyströn-QQS tends achieve higher detection performance than OC-KJL-QS at the cost of some increase in computation time. As explained carlier in Section [6] additional results for other variants and use cases with no tuning shown in Appendix show similar significant speedups.
14

---

Published in Transactions on Machine Learning Research (11/2022)
References
Mohiuddin Ahmed, Abdun Naser Mahmood, and Jiankun Hu. A survey of network anomaly detection techniques. Journal of Network and Computer Applications, 60:19-31, 2016.
Amaal Al Shorman, Hossam Faris, and Ibrahim Aljarah. Unsupervised intelligent system based on one class support vector machine and grey wolf optimization for iot botnet detection. Jowrnal of Ambient Intelligence and Humanized Compuling, 11 11(7):2809-288 2825, 2020.
ARM. ARM Cortex-A72 BCM2711 processor.
2016. Accessed: 2021-10-11.
Philippe Biondi. Scapy. btttps://scay..eete index 2021.
Daniele Calandriello and Lorenzo Rosasco. Statistical and computational trade-offs in kernel k-means. In NeurIPS, pp. 9379-9379999 9389, 2018.
Dorin Comaniciu and Peter Meer. Mean shift analysis and applications. In Proceedings of the Seventh IEEE International Conference on Computer Vision, volume 2, pp. 1197-1203. TEEE, 1999.
Petros Drineas, Michael W Mahoney, and Nello Cristianini. On the nyström method for approximating a gram matrix for improved kernel-basod learning. journal of machine learning research, 6(12), 2005.
Eleazar Eskin, Andrew Arnold, Michael Prerau, Leonid Portnoy, and Sal Stolfo. A geometric framework for unsupervised anomaly detection. In Applications of data mining in computer security, pp. 77-101. Springer,
2002.
Sebastian García. Malware on IoT Dataset (CTU 2019. Accessed: 2019-12-13.
Charles R. Harris, K. Jarrod Millman, St'efan J. van der Walt, Ralf Gommers, and Pauli Virtanen al. Array programming with NumPy. Nature, 585(7 :35 362, September 2020. doi: 100-0038//1158-0002226666
URL https://dol. htttp:://do..org/b55
Simon Hawkins, Hongxing He, Graham Williams, and Rohan Baxter. Outlier detection using replicator neural networks. In International Conference on Data Warehousing and Knowledge Discovery, pp. 170 180. Springer, 2002.
Katherine Heller, Krysta Svore, Angelos D Keromytis, and Salvatore Stolfo. One class support vector machines for detecting anomalous windows registry accesses. 2003.
Heinrich Jiang, Jennifer Jang, and Samory Kpotufe. Quickshift++ https://github.ccm www.tau...om/ooooo.//. g quickshift Accessed: 2020-06-13.
Heinrich Jiang, Jennifer Jang, and Samory Kpotufe. Quickshift++:: Provably good initializations for sample-based mean shift. arXio preprint 1rXii:1805.001 07909, 2018.
Jaeyeon Jung, Vern Paxson, Arthur W Berger, and Hari Balakrishnan. Fast portscan detection using soquential hypothesis testing. In IEEE Symposium Security and Privacy, 2004. Proceedings. 2004, pp. 211 115. IEEE, 2004.
KJI. An efficient ono-class svm for novelty detection source codc.
Samory Kpotufe and Bharath Sriperumbudur. Gaussian sketching yields a jl lemma in rkhs. In International Conference on Artificial Intelligence and Statistics, pp.
2020.
Christopher Kruegel and Giovanni Vigna. Anomaly detection of web- based attacks. In Proceedings of the 10th ACM conference Compuler and communicattons securily, pp. 251-261, 2003.
Anukool Lakhina, Mark Crovella, and Christophe Diot. Diagnosing network-wide traffic anomalies. ACM SIGCOMM computer communication review, 34(4)221 230, 2004.
15

---

Published in Transactions on Machine Learning Research (11/2022)
Scokjun Lee, Hyunguk Yoo, Jungtack Seo, and Taeshik Shon. Packet diversity- based anomaly detection system with ocsvm and representative model. In 2016 IEEE International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (Green Com) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData), pp. 498-503 IEEE, 2016.
Mohammad Saeid Mahdavinejad, Mohammadreza Rezvan, Mohammadamin Barekatain, Peyman Adibi, Payam Barnaghi, and Amit P Sheth. Machine learning for internet of things data analysis: A survey. Digital Commanications and Networks, 4(3):161-175, 2018.
Andrew Moore, Denis Zuev, and Michael Crogan. Discriminators for use in flow-based classification. Technical report, 2013.
Kenjiro Naga and Ryo Kaizaki. MAWI WIDE Dataset (MAW1). Accessed: 2020-09-13.
wwweii/iiieeade wwd.:.....dddda.aaaaaa.aa..... 2020.
Casey W. O'Brien, Lewis Lightner Costis Toregas, and Davina Pruitt-Mentle. National CyberWatch Mid- Atlantic Collegiate Cyber Defense Competition (MACCDC) Data. MACCDC 2012. Accessed: 2020-05-13.
com/?pag==
Travis Oliphant. Numpy configuration. htttg///waayyooo///////////eeeer htw.:/hupyyy..oo...... config. html 2005. Accessed: 2019-12-13.
Sutharshan Rajasegarar, Christopher Leckie, James C Bezdek, and Marimuthu Palaniswami. Centered hyperspherical and hyperellipsoidal one-class support vector machines for anomaly detection in sensor networks. IEEE Transactions Information Forensics and Security, 5(3):518- 000885533--11 2010.
Imran Razzak, Khurram Zafar, Muhammad Imran, and Guandong Xu. Randomized nonlinear one-class support vector machines with bounded loss function to detect of outliers for large scale iot data. Future Generation Computer Systems, 112:715-723, 2020.
Haakon Ringberg, Augustin Soule, Jennifer Rexford, and Christophe Diot. Sensitivity of pca for traffic anomaly detection. In Proceedings of the 2007 ACM SIGMETRICS international conference on Measurement and modeling of computer systems, pp. 109-120, 2007.
Alessandro Rudi, Raffaello Camoriano, and Lorenzo Rosasco. Less is more: Nyström computational regularization. In NIPS, pp. 1657-1665, 2015.
Arora Sanjeev and Ravi Kannan. Learning mixtures of arbitrary gaussians. In Proceedings of the thirty-third annual ACM symposium on Theory of computing, pp. 247-257, 2001.
Iman Sharafaldin, Arash Habibi Lashkari, and Ali A Ghorbani. Toward generating new intrusion detection dataset and intrusion traffic characterization. ICISSp, 1:108-116 2018.
Alistair Shilton, Sutharshan Rajasegarar, Christopher Leckie, and Marimuthu Palaniswami. Dplsvm: A dynamic planar one-class support vector machine for internet of things environment. In 2015 International Conference on Recend Advances in Internet of Things (RIoT), pp. 6. IEEE, 2015.
Mei-Ling Shyu, Shu-Ching Chen, Kanoksri Sarinnapakorn, and LiWu Chang, A novel anomaly detection scheme based on principal component classifier. Technical report, 2003.
Marina Thottan and Chuanyi Ji. Anomaly detection in ip networks. IEEE Transactions on signal processing, 51(8):2191-22 -2244, 2003.
Santosh Vempala and Grant Wang. A spectral algorithm for learning mixture models. Journal of Computer and System Sciences, 68(4):841-860, 2004.
Wei Wang, Yiqiang Sheng, Jinlin Wang, Xuewen Zeng, Xiaozhou Ye, Yongzhong Huang, and Ming Zhu. Hast-ids: Learning hierarchical spatial-temporal features using deep neural networks to improve intrusion detection. Jeee Access, 6:1 6:17 1806, 2017.
16

---

Published in Transactions on Machine Learning Research (11/2022)
Kun Yang, Samory Kpotufe, and Nick Feamster. A comparative study of network traffic representations for novelty detection. Xix preprint arXim:2006. 16993, 2020.
Tianbao Yang, Yu-Feng Li, Mehrdad Mahdavi, Rong Jin, and Hua Zhou. Nyström method random fourier features: A theoretical and empirical comparison. Advances in neural information processing systems, 25:476-484, 2012.
Yun Yang, Mert Pilanci, Martin J Wainwright, et al. Randomized sketches for kernels: Fast and optimal nonparametric regression. The Annals of Statistics, 4593):99,-1123, 2017.
Ming Zhang, Boyi Xu, and Jie Gong, An anomaly detection model based on one-class svm to detect network intrusions. In 2015 11th International conference on mobile ad-hoe and sensor nelworks (MSN) pp. 102-10 102-107. JEEE, 2015.
Yang Zhang, Nirvana Meratnia, and Paul Havinga. Adaptive and online one-class support vector machine- based outlier detection techniques for wireless sensor networks. In 2009 international conference on advanced information networking and applications workshops, pp. 990 995. IEEE, 2009.
17

---

Published in Transactions on Machine Learning Research (11/2022)
A Computing Platforms
Table. A.1 shows all three machines' information (i.e., Large server, Raspberry Pi, and Nvidia Nano), which includes operating system, CPU, memory, storage, programming language, and a scientific computing package (Numpyy) Note that 'Isepu' command is used to get CPU and CPU cache information. In addition, for Raspberry Pi, we get the cache information from the ARM official document of the Cortex-Te processor
ARM 2016).
Table A.1: We frain on server and test on all 3 machines.
Machine
Description
Large Server
61-bit, rumning Debian GNU/Limux 9 (stretch) with Intel(R) Xeon(I) processor (32 CPU Cores, 1200-3100 MHz each), 100GB memoryy and disk.
Caches: 3 layers of CPU caches (i.e. 32KiB Lid and L1i, 256KIB L2, and 20MB L3).
Programming language: Python
Numpy 1.19.2 (install from numpy wheel that includes an OpenBLAS implementation of the BLAS and PACK linear slgebra APIs (Oliphant) 2005)).
Raspherry
32-bit, running Raspbian GNU/Linux 10 (buster) with Cortex-A72 processor (4 CPU cores, 600-1500 MHz each), 8GB memory, and 27GB disk.
Caches: 2 layers of CPU caches (t.e., 32KIB Lid, 48KBB L1i, and 1MBB (RM 2016
Programming language: Python 3.7.3 with l-emable-ottimizatinn option.
Numpy 1.18.2 (install from the numpy source codes that searches for BLAS and LAPACK dynamic link libraries at build time influenced by the aystem environment variables (Oliphant 2005
Nvidia Nano
61-bit, running Ubuntu 18.04.5 LTS (Bionic Beaver) with Cortex-A57 processor CPU cores, 102-1479 MHz each), 4GB memory, and 30GB disk.
Caches:
2 layers of CPU caches (i.e., 32KIB Ltd, 48KiB Lii, and 2MiB 122)
Progrsmming language: Python 3.7.3 with -ensble-optimization option.
Numpy 1.18.2 (install from the source codes Oliphant 2005)).
B Dataset Sizes and Dimensions
Table B.1 shows Train set, Validation (Val.) set, Test set, and feature dimensions. The size of validation
sets is always 1/4 of the corresponding test set sizes. In Validation set and Test set, the number of normal and abnormal data is equal. Moreover, the IAT+SIZE dimension each dataset is less than 50, except for MAWI (its dimension 121).
Table B.1: Dataset sizes (# of data points) and dimensions.
Dataset
UNB
CTU
MAWI
MACCDC
SFRIG
AECHO
DWSHR
Train Set
10,000
10,000
5,720
10,000
10,000
10,000
10,000
Val. Set
462
1,250
1,040
1,250
1,250
280
508
Test Set
1,854
5,000
4,160
5,000
5,000
1,120
2,032
Dimensions
43
121
25
29
C OC-Nyström and OC-KJL Results (no Quickshift+++)
In the main paper body, we left out some of the detection time and space savings results for the OC-KJL and OC-Nyström variants (which do not use Quickshifl++ for automatic cluster-nunber identification). Here consider the effect of additionally tuning the number of GMM components rather than selecting them automatically via +ui The main message here is that not much is lost in AUC by automatically choosing & via Quickshift++.
C.1 Retained AUC and Training Efficiency
Table C.1 compares AUC and training times of OC-KJL and OC-Nyström to that of the baseline OCSVM.
18

---

Published in Transactions on Machine Learning Research (11/2022)
Table C.1: Retained AUC (method over MCSVM) and server train time speedup (OCSVM over method).
Dataset
MAWT
MACCDC
SPRIG
AECHO
DWSHR
Method
OC.KJT.
1.36 + 0.04
1.13 + 0.04
0.99 + 0.01
1.08 = 0.01
0.99 = 0.01
1.08 + 0.00
0.99 + 0.00
AUC Retained
Train Speedup
2.75 + 0.14
2.44 0.05
4.13 d: 0.08
2.47 0.08
2.22 +: 0.21
2.84 0.17
2.75 1: 0.14
OC-Nyström:
AUC Retained
1.41 + 0.02
1.19 + 0.03
0.99 +: 0.01
1.07 0.08
0.97 # 0.01
1.09 0.00
0.99 + 0.01
Train Speedup
3.14 + 0.16
2.59 + 0.06
3.68 + 0.07
2,60 0.06
2.17 + 0.21
2.94 + 0.17
2.72 + 0.14
AUC Retained. We that detection methods, OC-Nyström and OC-KJLL retain the detection performance of OCSVM, all within ratio of 1 or more, except in the case of MAWI, SFRFIG, and DWSHR - which are still high AUCs considering OCSVM's high accuracy on MAWI, SFRIG, and DWSHR (Table 3 Moreover, for some datasels, such as UNB and CTU, all procedures manage to actually outperform OCSVM. It likely that such higher performance is due to the additional regularization inherent the dimension reduction performed by our methods.
Although our original goal was just to maintain the training efficiency of OCSVM, especially considering the various additional steps inherent in our methods, our methods without Quickshift+++ fact achieve speedup factors of 2-4 in some cases over OCSVM training time which involves more expensive model fitting steps.
Training Time.
C.2
Significant Savings in Detection Time and Space
Tables
and
C.3
presents results delection time and space savings for both OC-KJL and OC-Nyström.
Table C.2: OC-KJL: Test time speedup (OCSVM over method) and space reduction (OCSVM over method).
Dataset
UNB
CTU
MAWI
MACCDC
SFRIG
AECHO
DWSHR
RSPI
26.66 0.02
22.81
13.25 0.01
25.91 + 0.05
26.20
0.05
24.52 # 0.04
25.07 0.06
Spendup
NANO
29.06 + 0.06
18.24 0.02
31.45 + 0.29
27.13
0.04
27.36 + 0.02
28.38
Server
28.80 + 0.23
26.41 1.80
14.73 0.18
28.19 + 0.32
23.63
0.29
29.11 # 0.05
30.48 0.01
Space Reduction
42.55 + 0.01
35.12 + 0.03
26.52 = 0.00
36.74 +: 0.02
32.58 + 0.00
39.20 - 0.00
37.13 + 0.01
Table C.3: OC-Nyström: Test time speedup (OCSVM over method) and space reduction (OCSVM over method).
Dataset
UNB
CTU
MAWI
MACCDC
SFRIG
AECHO
DWSHR
RSPI
27.17 + 0.02
24.30 + 0.93
12.59 = 0.01
27.02 + 0.05
21.87 + 0.05
25.33 4 0.04
21.49 0.06
Test Time Speedup
NANO
29.98 + 0.07
31.92 # 0.30
17.14 + 0.02
33.62 + 0.31
25.88 + 0.03
28.24 # 0.02
27.94 + 0.01
Server
28.31 + 0.23
27.60 # 1.88
13.95 0.17
29.71 + 0.34
23.22
0.29
30.24 # 0.05
29.91 # 0.01
Space Reduction
42.79 0.01
36.55
26.25 0.00
37.98 + 0.02
31.60 0.00
39.82 # 0.00
36.89 0.01
Testing Time Speedup. We observe speedups of at least 13 times over the baseline OCSVM detection times across all machines and datasets.
Space Reduction. As before, space reductions significant w.r.t. the baseline OCSVM from 26 to 42+ times less space than required by the bascline.
Results under No Tuning
We now consider the scenario where no validation data available to tune any the procedures, i.e., in choosing the bandwidth parameter h. While in general preferable perform some minimal tuning before deployment, in practice, may be difficult to obtain labeled data for the types of novel activities of interest. that commonly arise in actual deployment environments.
19

---

Published in Transactions on Machine Learning Research (11/2022)
In the practice of novelty detection with OCSVM, when no labeled data is available, various rule-of-thumbs are used, a popular one being to pick h as quantile of inter-point distances. For uniformity, explained in Section 5.1] here we pick h for all methods, as the 30th percentile of increasing inter-point distances in the training data.
Naturally, detection performance suffers w.r.t. that of a tuned procedure for any of the methods. Furthermore, since the choice of bandwidth affects the learned model, it is to be expected that time and space comparisons would also differ from that under minimal tuning in the previous Section B
D.1 Baseline OCSVM Performance
Table D.1 shows the performance of the baseline OCSVM. We observe decrease in AUC for most datasets, most considerably for UNB and CTU, which already were hard datasets even under tuning (Table 3 Interostingly, MAWI, MACCDC, SFRIG, and DWSHR still admit high AUCs even without tuning, attesting to the general appeal of OCSVM as an adaptable and robust novelty detection approach. Although AECHO and MACCDC dropped in accuracy (from 0.85+), they still retain reasonable accuracies with AUC's at 0.78.
Table D.1: OCSVM baseline performance, no tuning. Time in ms per 100 datapoints and space in kB.
Dataset
UNB
CTU
MAWI
MACCDC
SFRIG
AECHO
DWSHR
ALC
0.60 0.01
0.57 0.01
0.98 + 0.00
0.78 0.00
0.85 d: 0.00
0.78 0.00
0.87
0.01
Server Train Time (ms)
94.23 + 1.66
78.48 1: 0.93
84.72
85.15
2.99
117.65 1: 6.46
117.29 8.81
RSPI
124.54 + 0.12
124.12 + 0.35
74.48
+ 0.64
124.17 + U.39
123.70 + 0.36
126.31 + 0.95
125.46 + 0.72
Tost Time (ms)
NANO
89.30
0.11
86.32 + 0.17
50.71 + 0.05
86.70 + 0.21
83.50 = 0.17
88.85 = 0.02
88.00 + 0.09
Server
19.20
0.13
19.42 4: 0.22
9.76 + 0.13
19.10 + 0.27
18.62 + 0.15
19.95 + 0.02
19.72
0.09
Space (kB)
1761.21
961. + 0.23
2798.32 + 0.78
1041.31 0.13
611.07
1441.75 0.14
1201.38
0.32
D.2 Retained AUC and Training Efficiency
Table D.2 compares AUC and training times of OC-KJL-QS and OC-Nyströn-QS to that of the bascline OCSVM, using the exact same default choice of bandwidth h as OCSVM.
Table D.2: No tuning. Retained AUC (method over OCSVM) and train time speedup (OCSVM over method).
Dataset
UNB
CTU
MAWI
MACCDC
SFRIG
AECHO
DWSHR
Method
OC-KJL-QS:
1.49 1 0.02
1.20 1 0.05
0.14 1: 0.00
0.90 0.14
0.98 1 0.03
1.20 0.03
0.95 1 0.02
AUC Retained
Train Speedup
1.32 : 0.02
1.14 + 0.03
1.37 + 0.02
1.16 0.01
0.98 = 0.03
1.29 + 0.07
1.26 + 0.09
OC-Nyström-QSS
AUC Retained
1.54 + 0.03
1.19 + 0.10
0.17 + 0.00
0.88 0.17
0.77 = 0.11
1.18 0.00
0.90 + 0.05
Train Speedup
1.32 + 0.02
1.14 + 0.03
1.47 + 0.02
1.14 0.01
0.97 + 0.03
1.26 + 0.07
1.22 + 0.09
AUC Retained, OC-KJL-QS and OC-Nyström-QS manage to retain the AUC of OCSVM on most datascts. However, MAWI, neither OC-Nyström-Oe nor OC-KJI-QS does well, arriving at just fraction of the baseline AUC. MACCDC and SFRIG appear to cause problems for OC-Nyström-QS under the default h setting.
Training Time. As before, training time remains competitive with that of OCSVM with some significant. reduction time for instance in the case of UNB, AECHO, and DWSHR.
D.3 Significant Savings in Detection Time and Space
Tables D.3 and D.4 present results on detection time and space savings for both OC-KJI-QS and OC- Nyström-QQ. again with the same default choice of bandwidth h as OCSVM. The trends on savings are similar, but in fact even better than those under minimal tuning of these 3 methods.
20

---

Published in Transactions on Machine Learning Research (11/2022)
Table D.3: OC-KJL-QS, no tuning. Test time speedup (OCSVM over method) and space reduction (OCSVM
over method).
Datasct
UNB
CTU
MAWT
MACCDC
SFRIG
AFCHO
DWSHR
RSPI
24.30 1: 0.02
25.81 + 0.07
12.39 1 0.11
27.07 4: 0.13
27.37 1: 0.08
25.66 4: 0.19
25.40 1: 0.15
Test Time Speedup
NANO
28.94 + 0.01
32.18 # 0.07
15.99 = 0.02
31.43 + 0.08
31.21 + 0.07
29.97 # 0.01
30.88 + 0.03
Server
25.80 1 0.18
28.28
0.33
13.38
0.18
28.85 1: 0.41
28.30
0.23
32.34 1 0.03
31.49
0.15
Space Reduction
41.30 0.00
26.28
0.01
0.00
33.57
0.00
40.67 #
Table D.4: OC-Nyström- QS, no tuning. Test time speedup (OCSVM over method) and space reduction (OCSVM
over method).
Dataset
UNB
CTU
MAWI
MACCDC
SFRIG
AECHO
DWSHR
RSPI
22.70 + 0.02
24.50 0.07
12.38 0.11
26.31 +: 0.13
26.79 0.08
23.75 # 0.18
24.79 0.14
Test Time Speedup
NANO
26.36 + 0.03
30.28 = 0.06
15.97 = 0.02
33.22 + 0.08
33.29 + 0.07
27.37 = 0.01
29.92 = 0.03
Server
23.06 + 0.16
26.89 + 0.31
13.38 + 0.18
28.06 + 0.40
28.37 +: 0.23
29.33 4 0.03
30.60 + 0.15
Space Reduction
39.94 + 0.00
35.34 # 0.01
26.28 0.01
37.78 + 0.00
33.02 0.00
39.28 # 0.00
37.31 0.01
Testing Time Speedup. We observe speedups of at least 12 times over the baseline OCSVM detection times across all machines and datasets.
Finally, we again observe the trend where OC-KJL-QS manages faster times than OC-Nyström-OS in most cases, especially on Raspberry Pi and the server.
Space Reduction. As before, space reductions are significant w.r.t. the baseline OCSVM from 26 to 41+ times less space than required by the baseline.
E
Alternative Features
E.1 SAMP-SIZE Features Description
SAMP-SIZE: flow is partitioned into small time intervals of equal length, and the total packet size (i.e., byte count) in each interval is recorded; thus, flow is represented as time series of byte counts in small time intervals. Here, we obtain time intervals according to diffcrent quantiles (i.c., [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.95]) of flow durations. To ensure that each sample has the same dimension D, we select D for all flows as the 90th percentile of all flow lengths in the dataset (here, flow length stands for the number of packets a flow not its duration).
Now for any given flow, if the number of fixed time intervals in the flow is less than D, append 0's to arrive at vector of dimension D. Tf instead the number of fixed time intervals is greater than D, we truncate the resulting vector representation down to dimension D.
E.2 STATS+HEADER Features Description
a set of statistical quantitics compiled from a flow. In particular, we choose 10 of the most common such statistics in the literature (see e.g., Moore el al. (2013)), namely, flow duration, number of packets sent per second, number of bytes per second, and the following statistics on packet sizes (in bytes) in flow: mcan, standard deviation, the first third quantiles, the minimum, and maximum. Also, We incorporate packet header information (i.c., Time to Live (TTL) and TCP flags (FIN, SYN, RST, PSH, ACK, URG, ECE, and CWR) into the STATS to form the STATS+ DEEE feature.
HEADER:
21

---

Published in Transactions on Machine Learning Research (11/2022)
Table E.1: OCSVM performance with STATS+HEADER. Time is ms per 100 datapoints and space is KB.
Datasct
UNB
CTU
MAWT
MACCDC
SPRIG
AFCHO
DWSHR
AUC
0.00
0.00
J.
::
Server Train Time (ms))
77.20 = 1.75
94.34
2.73
64.63 + 0.82
92.23
2.25
88.69 # 0.90
100.38 +: 2.78
92.39 + 2.93
RSPI
123.90 + 0.39
119.29 + 3.31
72.39
= 0.08
114.20 + 0.33
121.33 + 0.57
123. + 0.31
125.70 70 + 0.22
Test Time (ms)
NANO
76.86 0.26
87.13
0.31
54.33 + 0.01
87.22 + 0.06
82.89 + 0.05
78.33 + 0.04
83.53 + 0.06
Server
17.60
11.17
0.16
19.09 0.23
19.88 19.883.22 0.02
Space (kB)
1655.41 2.72
1240.96 0.16
1831.75 + 0.00
1280.82
0.13
1084.04 + 0.22
1482.07 0.30
1303.32 = 0.28
E.3
Results under STATS+HEADER
E.3.1 Results Under Minimal Tuning
Table.
shows the baseline results obtained by OCSVM under minimal turning. Accuracies of the baseline OCSVM are similar to those using the features of LAT+ in the main paper and reported for completion.
F.1
Similar to the case of TTT+ features, both OC-KJL and OC-Nyström with or without Quickshift+ retain the AUC of the baseline OCSVM (on UNB and DWSHR, our procedures even get higher AUCs than OCSVM). Moreover, these procedures have times train time speedup. More details are shown Table
B2
Table E.2: Retained AUC (method over OCSVM) and server troin time speedup (OCSVM over method) with HTATS*HEADER
Dataset
UNB
CTU
MAWI
MACCDC
SFRIG
AECHO
DWSHR
Method
OC-KJL:
AUC Retained
1.33
1:
+
0.03
Trriin Speedup
2.63 t 0.06
3.11 +: 0.09
3.02 + 0.04
2.98 0.07
2.63 + 0.03
3.33 + 0.09
2.52 + 0.08
OC-KJI-QS::
AUC Retained
1.13 + 0.05
0.87 + 0.04
0.98 + 0.01
0.89 0.02
0.98 = 0.01
0.98 + 0.01
1.05 + 0.08
Truin Speedup
0.04
+: 0.01
1L 0.04
OC-Nyström:
1.45 1 0.02
1.02 1: 0.05
0.99 1: 0.00
0.94 de 0.05
0.96 1 0.02
0.99 d: 0.01
1.18 1: 0.04
AUC Retained
Train Speedup
2.45
0.09
3.24 + 0.04
3.15
0.08
3.15 = 0.03
3.17
2.60 + 0.08
OC-Nyström-QSS
AUC Retained
0.04
0.96
+ 0.03
0.04
0.94 + 0.01
1.06 + 0.03
Train Speedup
1.03 +: 0.02
1.29 1: 0.04
1.46 +: 0.02
1.48 +: 0.04
1.31 + 0.01
1.39 4: 0.04
1.19 4: 0.04
We also see that these methods under the alternative features attain significant detection time speedups over OCSVSM, which are shown in Tables E.3
and E.4
Testing Time Speedup. We observe specdups of at least 16 times over the bascline OCSVM detection times across all machines and dalasets.
Space Reduction. As before, space reductions are significant w.r.t. the baseline OCSVM from 26 to 41+ times less space than required by the baseline.
E.3.2 Results Under No Tuning
OCSVM results under no tuning for STATS- features are presented in Table E.5 As with the case of our preferred features of IAT+SIZE, we observe a significant decrease in AUC w.r.t. the tuned OCSVM case.
We also get similar significant test time speedup and space reduction results for both methods as shown Tables E.7 and E.g This goes show that the reductions inherent in our approach are likely not tied feature representations of the networking data.
22

---

Published in Transactions on Machine Learning Research (11/2022)
Table E.3: OC-KJL-QS: Test time speedup (OCSVM over method) and space reduction (OCSVM over method) with S HEADER.
Datasct
UNB
CTU
MAWT
MACCDC
SFRIG
AFCHO
DWSHR
RSPI
23.81 4: 0.07
28.17 + 0.78
16.45 1 0.02
27.00 4: 0.08
29.66 1: 0.14
24.92 1: 0.07
26.83
0.05
Test Time Spendup
NANO
25.74 + 0.09
36.96 # 0.13
21.17 0.01
35.58 + 0.02
35.52 + 0.02
25.93 # 0.01
33.73 + 0.02
Server
27.23 1 0.59
30.73
0.58
17.37 0.25
29.35 1: 0.27
30.53
0.37
29.77 4 0.03
36.73 0.03
Space Reduction
0.07
0.00
Table E.4: OC-Nyström- QS: Test time speedup (OCSVM over method) and space reduction (OCSVM over method) with STATS HEADER.
Dataset
UNB
CTU
MAWI
MACCDC
SPRIG
AECHO
DWSHR
+:
25.11 #
Test Time Speedup
NANO
1. 0.09
25.24 1:
Server
27.08 + 0.39
29.11 0.35
17.34 0.25
29.16 + 0.27
30.67 0.37
28.76 # 0.03
37.39 0.03
Space Reduction
39.70 + 0.07
10.33 = 0.01
26.48 = 0.00
40.64 + 0.00
40.12 + 0.01
39.94 = 0.01
41.48 = 0.01
Table E.5: OCSVM performance with STATS HEADER, tuning. Time in ms per 100 datapoints and
space is in kB.
Dataset
UNB
CTU
MAWI
MACCDC
SPRIG
AECHO
DWSHR
AUC
0.15 + 0.00
0.41 = 0.00
0.99 = 0.00
0.50 + 0.00
0.79 + 0.00
0.96 0.00
0.38 = 0.01
Server Train Time (ms)
92.28 = 2.55
89.55 + 2.28
62.00 + 2.11
96.15 + 1.96
89.70 # 1.70
95.50 + 2.22
97.06 + 2.16
d:
70.74 1: 0.36
4:
Test Time (ms)
NANO
89.86 = 0.90
87.65
0.31
46.35 d: 0.04
86.49
0.14
84.04 4: 0.19
88.14 4: 0.07
85,96
0.08
Server
19.91 - 0.65
19.32 4: 0.36
10.00 +: 0.05
19.61 1: 0.20
18.91 +: 0.23
20.82 1: 0.01
19.91 +: 0.02
Space (k:B)
1641.24 + 0.26
1241.11
0.12
1832.14 +: 0.31
1282.76 1: 0.54
1081.45 1: 0.22
1481.54
0.12
1361.80 : 0.24
Testing Time Speedup. We observe speedups of at least 12 times over the baseline OCSVM detection times across all machines and datasets.
Space Reduction. As before, space reductions are significant w.r.t. the bascline OCSVM from 25 43+ times less space than required by the baseline.
Table E.6 shows that OC-KJL-QQS and OC-Nyström-QS retain the AUC and train time of the baseline OCSVM using the STATS+ HEADER foatures.
Table E.6: no luning. Retained AUC (method over OCSVM) and train time speedup (OCSVM over method) with STATS+HEADEN
Dataset
UNB
CTU
MAWI
MACCDC
SFRIG
AECHO
DWSHR
Method
OC-KJL-QS:
2.29 + 0.50
1.02 0.06
0.22 + 0.23
0.93 + 0.09
1.00 + 0.02
0.52 + 0.28
1.60 + 0.09
AUC Retained
Train Speedup
1.41 + 0.04
0.03
1.19 # 0.04
1.38 0.03
1.22 + 0.02
1.28 0.03
1.27 0.03
OCC-yystrmm-Q)
AUC Retained
2.32 1: 0.05
1.20 0.11
0.13 4: 0.01
1.11
0.06
0.57 1: 0.20
0.49 d: 0.20
1.70 1: 0.10
Train Speedup
1.40 1. 0.04
1.22 0.03
1.31 4. 0.04
1.43 0.03
1.24 4: 0.02
1.31
0.03
1.24 d. 0.03
23

---

Published in Transactions on Machine Learning Research (11/2022)
Table E.7: OC-KJL-QS with STADS+HEADER, no tuning: Test time speedup (OCSVM over method) and space reduction (OCSVM over method).
Datasct
UNB
CTU
MAWT
MACCDC
SPRIG
AFCHO
DWSHR
RSPI
28.50 1: 0.22
28.91 1 0.19
12.99 1 0.07
28.50 1: 0.02
28.83 1: 0.05
25.42 1: 0.21
28.04 1 0.11
Test Time Speedup
NANO
38.31 + 0.39
36.42 # 0.13
15.53 0.01
35.37 + 0.06
36.16 + 0.08
29.71 # 0.03
36.86 + 0.03
Server
38.39 1 1.25
30.22
0.56
14.20
0.07
29.82 1: 0.30
30.26 0.37
32.25 1 0.01
39.59
0.04
Space Reduction
43.53
0.01
40.65
Table E.8: OC-Nyström- QS with HEADER, no tuning: Test time speedup (OCSVM over method) and space reduction (OCSVM method).
Dataset
UNB
CTU
MAWI
MACCDC
SFRIG
AECHO
DWSHR
RSPI
28.16 0.22
0.07
27.15
28.74
24.38 +
Test Time Speedup
NANO
27.86
Server
37.60 + 1.23
30.18 0.35
14.01 0.07
28.35 + 0.28
30.39 0.37
29.88 + 0.01
37.92 0.04
Space Reduction
43.22 + 0.01
40.87 = 0.00
25.13 = 0.00
39.94 + 0.02
39.88 + 0.01
39.61 + 0.00
41.37 + 0.01
24


Reviews:
[{"review_id": "8XAVHXtliS", "reviewer": "Reviewer_MEPD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe target problem (i.e., reducing the test time and memory) is interesting and well-motivated\n\nThe writing itself is clear, but maybe reorganizing the paper will improve the overall presentation.\n\n------------\n\n**Weakness**\n\nHard to agree that the OCSVM is the state-of-the-art method.\n- There exists various (non-deep learning) methods such as KDE, IF, Mahalanobis distance, and Minimum Covariance Determinant estimator (MCD).\n- The detection time of Mahalanobis and MCD does not depend on the training dataset size.\n\nWhile the author claim that GMM is a natural selection for unsupervised clustering, it is slightly hard to agree (since there is no theoretical explanation). Since the main algorithm is highly dependent on the GMM, proving some evidence will strengthen the paper’s claim, e.g., empirical support by comparing more clustering algorithms.\n\nTable 2 is quite confusing. Can the author provide the actual AUC value rather than the Retained AUC..? (Does Retained AUC > 1 mean that the proposed method is better than the original OCSVM?)\n\nAccording to Figure 2, reducing the training data size did not reduce the AUC much. Hence, one naive way is to randomly subsample the training data to reduce the time and memory complexity of OCSVM (a better way is to select the prototypical sample, e.g., using the k-medoids clustering). Does the proposed method perform better than this naive way?\n\nThe overall presentation can be improved.\n- For instance, a more abstract version of the Introduction section will be more readable (especially from the results overview). \n- Also, making Sections 5,6,7 into one section and divide with subsections will be much better.\n- The Table captions are hard to understand.\n\n[1] Parzen et al., On Estimation of a Probability Density Function and Mode, 1962\\\n[2] Liu et al., Isolation Forest, ICDM 2008\\\n[3] Mahalanobis et al., \"On the generalised distance in statistics, 1936\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper targets an important problem to make a test-time efficient anomaly detection method in low-resource hardware (e.g., IoT). To this end, the paper target to make OCSVM (which shows an effective performance in this area) efficient. The main contribution of this paper is to extend the Nystrom and (Gaussian) Sketching approaches (used in efficient SVM) into OCSVM by combing with clustering and GMM. The experimental results show that the proposed method indeed improves the memory and detection time highly efficient.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1) Some experiments to support that the OCSVM is the current state-of-the-art\n\n2) Justification of using GMM (mentioned in the weakness part)\n\n3) More baselines (e.g., KDE, IF, subsample training set for OCSVM) by comparing both efficiency (i.e., wall-clock-time) and effectiveness (i.e., AUROC)\n- 2D plot will be much more readable by putting efficiency on the x-axis and effectiveness on the y-axis\n\n4) Clearer presentation\n- Detailed explanations in the Table caption, especially Table 2 (mentioned in the weakness part)\n- The introduction and Section 5,6,7 (mentioned in the weakness part)\n\n5) (minor suggestion for improvement) Considering non-IoT datasets will strengthen the paper. Since the proposed method is can be applied to any scenario, showing non-IoT datasets will be worth it, e.g., MNIST.\n", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nkY33Xh5QA", "reviewer": "Reviewer_tP4r", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n\n- The authors tackled an important and practical problem: anomaly detection with the constraints of memory and latency.\n- The proposed method makes sense and the experimental results are significant.\n\n**Weakness**\n\n- Some of the claims do not have enough supports. For instance, the authors should provide more quantitative analyses on why OC-SVM is SOTA in this area.\n- The authors did not properly consider alternatives such as directly applying GMM or KDE for anomaly detection method.\n- The problem settings are unclear. Unsupervised usually means unlabeled data. But the authors used the term unsupervised as only with normal samples. As I understand, if we want to gather only with normal samples, we should exclude all the anomalies from the unlabeled data. It should be clarified in the rebuttals.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- The authors tried to solve the novelty detection problems in IoT domains with the constraints of low memory and low latency (at inference).\n- The authors claimed that the current OC-SVM's prediction performances are satisfactory; however, its memory and latency constraints are not practical to utilize in IoT domains.\n- The authors proposed an approximation of original OC-SVM and combined with GMM to significantly reduce the necessary memory size and the latency at inference time only with marginal performance loss.\n- The experimental results show that the latency and memory reduction are significant (x20) which makes OC-SVM practical in IoT settings.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "**1. Is OC-SVM state-of-the-art?**\n- In order to claim that the OC-SVM is SOTA, the authors should provide some quantitative evidence.\n- At least for the datasets that the authors used in this paper, the authors should provide some alternative anomaly detection methods (instead of OC-SVM) and discuss the performances of those alternatives. - Something like the accuracy and the inference time.\n- Currently, it is hard to defend why we need to focus on OC-SVM instead of many other anomaly detection algorithms.\n\n**2. Unsupervised setting**\n- The authors claim that the novelty detection settings are unsupervised settings. In other words, only the normal samples are provided and the abnormal samples are not provided.\n- It is unclear how to gather only normal samples. In experiments, the datasets include both normal and abnormal samples (which is critical to evaluate the anomaly detection performances). - So, how to extract only normal samples from the unlabeled data?\n- Please clarify why the gathering of the anomaly samples is prohibited and how to gather only normal samples in practice.\n- Also, \"if\" we can use the gathered anomaly samples as well for constructing the models (like supervised model training), is it better than OC-SVM?\n\n**3. GMM and alternatives**\n- The inference time latency is heavily relied on GMM. \n- As the authors said, the main advantage of GMM is a fairly simple inference procedure which would significantly decrease the latency and memory.\n- In that point of view, is it better to just use GMM instead of using OC-SVM first then convert it to the GMM forms?\n- Note that the prediction performances of GMM are also quite promising in multiple anomaly detection applications.\n- If the kernel is the problem, we can use the kernel density estimator (https://scikit-learn.org/stable/modules/density.html#kernel-density-estimation) directly.\n\n**4. Figure 3**\n- In Figure 3 (Data), even though it is not easy to separable with a linear decision boundary; however, with a simple kernel, we can separate two classes easily.\n- For instance, if we directly use the GMM, we can easily solve this anomaly detection example without having an approximation of OC-SVM.\n- It would be good if the authors can provide some examples that OC-SVM approximation can solve but GMM cannot easily solve in the revised manuscript.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6CY9gfoLcA", "reviewer": "Reviewer_o49b", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper investigates a timely problem of the efficiency of SVM in the inference phase.\n- The proposed algorithm is implemented on real-world IoT devices. \n- The experimental result shows a large improvement in both detection time and space. \n\nWeaknesses\n- The novelty of the paper is a bit limited. The proposed algorithm is built upon the existing Nyström or KJL approach with a well-established GMM model. \n- Why does the paper fit the normal data into a GMM model? Why not use other unsupervised clustering algorithms?\n- It is hard to understand the relationship between retrained AUC and the test time speed up in Table 2, 3, and 4. \n- The paper evaluates the efficiency of one-class SVMs on DDoS attacks. However, since DDoS attacks generate a large amount of traffic in a short time, it is unclear whether the proposed algorithm can be efficient enough to detect the attacks in real time. It would be better if the paper could compare the speed of DDoS attacks with the speed of detection.\n- The paper is not well-organized and hard to follow. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces an efficient one-class SVM algorithm for anomaly detection, aiming to reduce the detection time and space. The new algorithm first leverages Nyström or KJL equation to extract embedded data for high-dimensional mapping and then fits the normal data into a Gaussian Mixture Model for anomaly detection. The experimental results show that the algorithm can speed up the detection time.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- Explain why to use GMM in the algorithm.\n- Provide a figure showing the relationship between retrained AUC and the test time speed.\n- Compare the detection time with the generation time of DDoS attacks.\n- Reorganize the paper.\n", "broader_impact_concerns": "The paper does not raise any broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an efficient implementation of One-Class Support Vector Machines (OCSVM). The authors first apply the Nystrom and (Gaussian) Sketching approaches as used in the SVM literature, and then use GMM models for the detection. Reviewers highlighted the importance of the problem and significance of the results. However, there are several concerns needed to be addressed by the authors, e.g., (a) the authors say OCSVM is a SOTA method, but it is hard to agree, and (b) why not use GMM instead of OCSVM? AE carefully read the authors' response as well as the revised draft. Overall, AE thinks the authors provide reasonable answers (and supporting additional experimental results), and such concerns should not be a reason to reject the paper (under the TMLR's acceptance policy). Nevertheless, to be accessed by a broader audience, AE strongly recommend the authors to provide more motivations and intuitions for OCSVM and GMM. AE also finds many typos, which should be fixed before publishing the draft, e.g., (c) OCSVM has the performance in AUC than others (performance -> better performance?), and (d) we notice that GMM has a higher AUC than OCSVM over different training sizes (GMM -> KDE?).
Decision: Accept (poster)