## 5 Methodology and Development
### Distributed Preprocessing  
In order to mitigate the resource consumption of the preprocessing task, we defined an architecture following the distributed computing model. The process was done by dividing the functions into a matrix of relations which is capable of interpreting the dependencies between the functions, taking into consideration some variables, i.e., coupling, cohesion, recursion, idempotence, etc. After that, the code was split into different pieces of code; these pieces of code were separated into different daemons that are executed remotely by different machines. These pieces of code (units) are units that were separated from a sequential disposition to a parallel disposition. When doing this, the code must be modified to correctly compile the preprocessing function in parallel, while protecting critical writing/reading operations to provide reliable operations of software [44]. The platform used to execute preprocessing distributed functions was hosted on Google Cloud, selecting basic machines, named n1-4, which provide a four-core central processing unit (CPU) and 15 GB of random-access memory (RAM). The instances were connected between a shared file system based on network file system (NFS) to share the software components. As can be seen in Figure 1, the machines need to be connected with each other, in order to be able to exchange data. Parameters and variables are executed just as if they were a single computer executing a program. Another important detail is the presence of a distributed filesystem; different machines must access the same filesystem to make sure they share the same input files. \begin{table}
\begin{tabular}{c c} \hline \hline
**Feature** & **Encode** \\ \hline Timestamp & Dropped \\ Duration of the flow & Z-score normalization \\ Source/Destination IP address & Index encode \\ Source/Destination port & No encode \\ Protocol & Index encode \\ Flags & Index encode \\ Forwarding status & Dropped \\ Type of service & Index encode \\ Number of packets in the flow & Z-score normalization \\ Number of bytes transmitted & Z-score normalization \\ Result of the flow & Binary/Multilabel \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Summary of applied encodings to UGR’16 features. ```
1:preprocess_dataset.py
2:open full dataset file
3:drop incomplete rows
4:for every machine available:
5: create one thread per CPU core
6:START OF DISTRIBUTED PREPROCESING FUNITION
7:while not processed features available do:
8:# Code executed in parallel at each machine
9:for every free thread in parallel do:
10: feature to preprocess = random feature from the dataset not processed yet
11:process_piece(feature to preprocess):
12:if feature is duration or number of packets or bytes:
13: code each value as normal distribution of all
14: if feature is type of traffic:
15: code as type of traffic with int
16: else:
17: assign each distinct value a number
18: save column into final dataset
19: Give result to machine one
20: Continue
21:END OF DISTRIBUTED PREPROCESING FUNITION
22:continue
23:write final dataset into file
```  
**Algorithm 1** Preprocessing UGR’16 dataset  
Algorithm 1 presents how the dataset is preprocessed. This task is done by columns because it is mandatory to consider the different values in each feature (each feature correspond with a column). This allows building an index array of the different elements, which is used to calculate the normal distribution of the values that shape the feature or just to encode them in the desired way as required in each case. The whole process is done in an agile way, dividing the different functions correctly and protecting critical reading/writing operations to provide reliable operation of the software that will be executed in parallel in the above-mentioned machines. One of the available threads (launched continuously) manages the execution. This first thread starts by opening the full dataset file and the different columns corresponding with each feature identified.