# An Assessment of Environmental RF Noise Due to IoT Deployment
## 6 Preparing Data for Analysis  
### Dealing with Big Data  
The concept of big or large data is defined relative to the computer hardware and software systems employed to operate this data. Factors including the data volume, required storage, processing capability, and data format dictate the difficulty levels a machine may face to carry out data-intensive tasks. Big data often leads to computer infrastructure struggling to process and store the given data timeously. Big data is no longer valid when the machine can effortlessly process and store a volume of data [30; 31]. Scientific measurements are one of the generators of big data. They are often made at a high time resolution and start to become excessive when they involve two or three dimensions of space. Dealing with big data requires additional hardware and software capabilities to help ensure seamless and fast computing [30]. Frequently, memory and CPU are the components exposing machine weaknesses. Some software operates by loading the entire volume of data in the RAM. Processing would take a long time or eventually fail as the dataset size is larger than the PC memory. Distributed computing has successfully addressed big data by providing the advantage of completing challenging tasks by parallel processing [32].  
The literature reveals that the options for cloud computing, multiple parallels, and distributed computing come with a cost, time, and effort. Implementing these methods would contradict the strategy set in this paper to offer simplified and cost-effective solutions. [PERSON](r) proposed some techniques for efficient computer memory usage [12; 33; 34]. Furthermore, divide-and-conquer is one of the fundamental approaches for data reduction, which consists of decomposing extensive data into small partitions, applying the required processes to individual partitions, and then recombining the individual results together [11; 35]. Inspired by this, this research adopted a data-wrangling approach to transforming the raw dataset from 10 billion rows (30 GB) to a reduced dataset of 10 million rows (30 MB), suitable for computational analysis. The techniques in this approach required, in the first instance, to exclude every nil value, since the interest in this study was solely on non-zero levels. Secondly, the average values were calculated by data segmentation: splitting non-zero samples to produce 10 million segments, applying the arithmetic average on data in each segment, and merging the 10 million averaged results to bind a reduced dataset [12]. Figure 8 illustrates this data reduction method. Programmatically, Algorithm 1 is a procedural code snippet in Python to implement data reduction for DUT_433_Tx0 data using Pandas and NumPy libraries. The arithmetic mean, median, and peak characterize the radio noise. The metric used depends on the continuous or impulsive nature of the noise. The mean is convenient for Gaussian noise. One of the specifications for white Gaussian noise definition is a zero mean value. On the contrary, the median is not affected by pulses, and is therefore a good statistical metric when analyzing non-Gaussian noise. Peak values are arbitrary [28; 36]. Knowing that IoT products are configured for short duration air-time (in order of milliseconds), it should justly imply that these products would generate impulsive radio noise. With that in mind, this paper employs the arithmetic average. This paper suggests that the myriad IoT radio activities continuously occupy the environment by considering the random and multiple RF transmissions, the scalability, and the number of deployable IoT units. In aggregate, the radio receiver does not experience the essence of impulsive noise.