# Data-Based Modelling of Chemical Oxygen Demand for Industrial Wastewater Treatment
## 2 Materials and Methods
### Data Pre-Processing  
The data were pre-processed using MATLAB(r) software. The purpose of data pre-processing was to process the available data into the most complete form so that it could be used for modelling purposes. This included multiple steps. Firstly, variables that were constant (such as set point values for certain variables) were removed from the datasets. Variables were also removed if they included many Not a Number (NaN) values. Such variables contained no useful information from the modelling perspective. The NaN values were replaced with interpolated values. Removal of NaN values from the data is important because they can cause issues with mathematical operations and modelling methods later. In this study, linear interpolation was employed to replace NaN values, utilising either the last known value or the next known value. The choice between these options depended on factors such as whether the variable began or ended with a NaN value. Removal of NaN values was followed by an automatic outlier detection method. Outlier detection is an important step in data analysis and modelling. Firstly, it helps ensure data quality by identifying and addressing data errors, leading to higher data integrity and reliability. It also enables accurate statistical analysis by preventing distortions in data distribution and calculations of statistical measures. The 'quartiles' method was used to identify outlier points automatically [27]. In this method, data elements that are 1.5 interquartile range (\(IQR\)) below the lower quartile or above the upper quartile are automatically classified as outliers. The \(IQR\) can be calculated as in Equation (1):  
\[IQR=Q_{3}-Q_{1}, \tag{1}\]  
where \(Q_{3}\) represents the upper quartile (75 per cent of values from lowest to highest) and \(Q_{1}\) the lower quartile (25 per cent of values from lowest to highest). After detecting the points that are above or below 1.5 \(IQR\) of their respective quartile, the points were marked as outliers and changed to NaN values. This was performed so that the locations of these points would not go missing during deletion, as the removal of values from different parts between datasets would lead to discontinuity with the data timestamps if removed directly. Usually, during this part of data pre-processing, data timestamps would also have to be fixed. However, the timestamps did not include any errors or multiple values, which is why the timestamps could be ignored during the data pre-processing and modelling as every data point was recorded at steady one-minute intervals. Next, data points from the dataset, which were clearly outliers (such as negative pH values), were changed to NaN values. Other outlier points were detected by manually inspecting the data for possible outliers. Possible outlier points were left in the dataset if it was unclear whether the point was an outlier or a correct reading. The NaN values were then replaced with interpolated values similarly to before. Once all outliers and NaN values were removed and interpolated from the data, the data were standardised. Standardisation is performed so that every variable uses the same common scale and can be performed with many different formulas. However, since most of the variables in the datasets were close to normally distributed, the standardisation was performed using the standard score formula (Equation (2)) [28]:  
\[Z_{i}=\frac{x_{i}-\overline{x}}{S} \tag{2}\]  
where \(\overline{x}\) is the mean, and \(S\) is the standard distribution of the variable being standardised. The standard score \(Z_{i}\) represents how many standard deviations the actual value \(x_{i}\) differs from the variable mean. The datasets were then sampled utilising a moving median [29], where a median from a certain point window is used to represent all the data points from that window. The data points that were used to calculate the median were removed afterward, and only one point remained to represent all the removed values.