# Anomaly Detection for IOT Systems Using Active Learning
## 3 Data Collection
### Exploratory Data Analysis (EDA)
* Correlation Matrix: A correlation matrix can provide insights into the relationships between different properties within a dataset. Adopting this approach will facilitate a more comprehensive understanding of the critical attributes that hold the most significance in the anomaly identification process while distinguishing those that may be superfluous or lacking in substance. Below, Figure 4 shows the correlation matrix for selected features of the UNSW data frame. Figure 2: Attack distribution by category. Figure 4: Correlation matrix for selected features of the UNSW data frame. Figure 3: Top five protocols. Correlation is a robust descriptive measure that provides valuable insights into the degree of association between two variables. The strength of the association between variables can be assessed using the correlation coefficient. A correlation coefficient close to 1 indicates a high degree of correlation, whereas a value below 1 or negative suggests a weak relationship between the variables. In the figure above, only significant features were depicted, with those presented in blue hues indicating a strong correlation, while those shown in red hues denote a weak correlation. Additionally, the correlation coefficient is displayed on each correlated box, providing a quantifiable correlation assessment. * Box Plots of Feature Distributions: The examination of box plots depicting the distributions of individual characteristics enables comprehension of the variability and distribution of the data. It can facilitate the identification of abnormal data points and outliers. As depicted in Figure 5, outliers exhibit significant divergence from the remaining data points within the sample. The presence of outliers within the characteristics could indicate atypical or deviant patterns in the network traffic under investigation within the framework of this code. A significantly elevated value for the \"sbytes\" or \"dbytes\" attribute indicates the presence of huge network packets. Identifying and analyzing outliers can facilitate an understanding of network traffic patterns and the detection of anomalous behavior. Additional research may be necessary to ascertain the underlying source of any observed outliers, as outliers might arise from measurement errors or other factors unrelated to network traffic behavior. * Distribution of Label Classes: The equilibrium of the dataset can be ascertained by examining a bar plot that exhibits the distribution of the label classes, namely, normal or assault. Considering an imbalanced dataset is crucial when creating a machine learning model for anomaly detection. Figure 5: Box plot of the distributions of selected features for outliers. The chart's vertical axis represents the quantity of each service, while the horizontal axis enumerates the many types of services. To facilitate the identification of the most commonly utilized services, the bars in the chart depicting the frequency of each service are organized in descending order. This visual representation may aid network administrators in prioritizing the protection of commonly utilized services and implementing measures to defend them against potential vulnerabilities. The dataset indicates that DNS and HTTP are the most commonly utilized services. * State Distribution: The state distribution of a dataset characterizes the various states that a network connection may encounter over its lifespan. Established, Syn_sent, Fin_Wait1, Fin_Wait2, Time_Wait, Close, and other states are illustrative of such conditions. Figure 8 displays the distribution of state types. Examining the distribution of states can offer valuable insights into the dynamics of a network and the characteristics of the traffic inside the dataset. Through the examination of state distributions, it becomes possible to discern potential anomalies and correlations that may exist between particular states and the various categories of attacks.