# Trustworthy AI and Federated Learning for Intrusion Detection in 6G-Connected Smart Buildings
## 6 Data Pre-Processing and Engineering Module
### Data Pre-Processing  
The data pre-processing phase is minimal and primarily focused on data formatting. Unlike traditional data pre-processing phases, this approach follows the principles of adaptability and zero-touch automation. This ensures that the pre-processing stage remains flexible enough to accommodate diverse datasets without requiring human intervention. Pre-processing was implemented as a Python script that takes as input the csv files of the six sensors (GPS, fridge, weather, thermostat, garage door, and motion light) and the csv file of the network features extracted by Zeek. As illustrated in Figure 2, the pre-processing operation is composed of steps, the specific nature of which is contingent upon whether the data originate from a sensor or the network. The sensor data are numerical (e.g., temperature of the fridge) or boolean (e.g., phone signal of the garage door). The only data pre-processing step involves the spotting and formatting of malformed data as follows. * For boolean features, malformed values, such as \"False\", \"false\", \"f\", \"True\", \"true\", and \"t\", are converted into a common binary representation (0,1). * For numerical features, malformed values such as strings are dropped. Network data are both categorical (e.g., HTTP methods) and numerical (e.g., connection duration). The related operations are described as follows. * Categorical features are converted into integer features using a custom _label encoder_[55], which iterates and collects all the possible values for a particular feature to create a string:int mapping, which is used for conversion. The different mappings are kept in a file (mapping.txt) that can be accessed to load the label encoders and use them in the future to convert other data while keeping the mapping. * For numerical features, malformed values such as strings are dropped, and wide-range values (e.g., duration, bytes, pkts) are converted to logarithm scale to be efficiently represented in a narrower range of values. A common pre-processing phase is the conversion of the date:time pair in the sensor data and the timestamp (UNIX) of the network data into a common timestamp \"YYYY/MM/DD hh:mm:ss\" format. After these steps, all the data are scaled with a _robust scaler_[56], which scales the data by removing the median and scaling according to the quantile range, preserving outliers. The IDS is designed with the strategic decision to adopt a binary labeling approach for the sensors. This approach enables the IDS to function effectively as an anomaly detection system, focusing on the identification of deviations from normal behavior within the sensor data streams. In contrast, the network-based component of the IDS has a broader scope, encompassing six of the eight types of attacks present in the ToN-IoT dataset. This decision was based on an automated examination of the dataset, which revealed significant differences in the number of packets across the various attack categories and excluded the unbalanced attack instances. As can be observed in Table 3, the instances of MITM and ransomware are significantly lower than those of other attacks. While the decision to exclude unbalanced instances diminishes the overall robustness and applicability of the developed IDS, this approach was adopted to concentrate the experimental efforts on the development of a general data engineering unit that is not specifically tailored to the ToN-IoT dataset. It is important to note that several state-of-the-art techniques, such as the Synthetic Minority Over-sampling Technique (SMOTE) [57] and generative models like  
Figure 2: Data pre-processing steps. Generative Adversarial Networks (GANs) [58], could have been employed to address this issue. This will be the focus of future work aimed at enhancing the capabilities of the data engineering unit. In summary, Figure 2 shows the network feature extraction process delineated in Section 6.1 and the data pre-processing process described in Section 6.2.