# Anomaly Detection for IOT Systems Using Active Learning
## 3 Data Collection
### Data Processing  
The protocol for conducting data processing utilizing the UNSW-NB15 dataset for anomaly detection in IoT systems through active learning involves the following steps:  
* Data cleaning: The data cleaning process includes identifying and removing duplicate records and the appropriate treatment of missing or erroneous data entries. It is possible to employ techniques such as one-hot or label encoding to convert categorical data into numerical form. The dataset under consideration exhibits a complete absence of missing or null elements. To complete the data cleaning process, it is necessary to convert the categorical variables in the dataset using label encoding. The dataset's categorical variables, namely protocol, state, service, and attack_category, are presented in Table 3.  
Figure 8: State-type distribution. * Feature Selection: Choose the characteristics that have the utmost significance in identifying abnormalities. Consider the concept of domain competence and the importance of feature significance metrics such as mutual information and correlation. To enhance the outcomes, it is possible to select specific traits that exhibit a strong correlation. However, previous studies have predominantly employed a limited number of characteristics. This study aims to construct an anomaly detection model using all available data, encompassing highly and minimally linked features. * Feature Scaling: To ensure comparability, it is necessary to standardize the scales of the features. Standardization and min-max scaling are widely used techniques in data pre-processing. To ensure consistency within our dataset, we utilized traditional scalar operations. The revised data frame is presented in Table 5.
* Train/Test Split: Generate training and testing datasets by partitioning the available data. The testing set will be utilized to evaluate the efficacy of the anomaly detection model after its training on the training set. The data frame consisting of 45 columns will be initially partitioned into two separate frames:
* 1: Qualities. * 2: Education. The initial 43 feature variables will be included in the features data frame. Two variables are used to represent the classes in our study: one is binary, while the other is multi-dimensional. In the present investigation, we will exclude the multi-label target variable and solely focus on binary labels. Subsequently, the dataset will be partitioned into a training set including 70% of the data and a testing set comprising the remaining 30%, using [PERSON]'s train-test split procedure. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Proto** & **State** & **Service** & **Attack\_Cat** \\ \hline
117 & 4 & 0 & 6 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Label encoded categorical variables. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Proto** & **State** & **Service** & **Attack\_Cat** \\ \hline UDP & INT & - & Normal \\ UDP & INT & - & Normal \\ UDP & INT & - & Normal \\ UDP & INT & - & Normal \\ UDP & INT & - & Normal \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Categorical variables. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**0** & **1** & **2** & **3** & **4** & **5** & **6** \\ \hline
1.457197 & \(-\)0.110851 & 0.088625 & \(-\)0.674406 & \(-\)2.050741 & \(-\)0.094586 & \(-\)0.134511 \\ \(-\)1.176180 & \(-\)0.213728 & 0.410563 & 0.274329 & 0.932695 & \(-\)0.124455 & \(-\)0.151816 \\
0.569545 & \(-\)0.213729 & \(-\)3.881935 & \(-\)0.674406 & 0.932695 & \(-\)0.124455 & \(-\)0.151816 \\ \(-\)0.081899 & \(-\)0.206895 & 0.088625 & \(-\)0.674406 & \(-\)0.559023 & 0.204110 & 0.263504 \\ \(-\)0.035069 & \(-\)0.090130 & 0.088625 & \(-\)0.674406 & \(-\)0.559023 & \(-\)0.064716 & \(-\)0.099901 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Standardized features data frame. * X_train size = (57,632, 43)
* y_train size = (57,632)
* X_test size = (24,700, 43)
* y_test size = (24,700)  
The initial record consists of many rows, while the subsequent record has multiple variables within each data frame.