# Analysing Urban Traffic Patterns with Neural Networks and COVID-19 Response Data
## 3 Methodology and Data
### Methodology of the Study
#### 3.2.1 Methodology of Data Preparation  
According to the CRISP-DM methodology, preparing a high-quality modelling matrix containing data that will lead us to the solution before creating the predictive model is essential. However, it is crucial for these data to be of the highest possible quality. This means it is necessary to thoroughly check the data quality, correctness of values, relevance, and admissibility of values and to eliminate or correct any errors in the data. It is widely known that the data preparation phase is the most time-consuming in the entire data mining project, possibly taking up more than 80% of the total project time. Similarly, in the case of this study, after a thorough examination of the data, we found errors likely caused by incorrect functioning or complete failure of communication from the sensor (or sensors). Such communication failures resulted in data errors where the sensor reported zero counts for all types of vehicles (cars, vans, trucks, and unknown) for several consecutive 15-min time intervals, and subsequently, upon the restoration of communication functionality, the sensor reported cumulative vehicles numbers for all the missing time intervals. An example is provided in Figure 6. This specific communication failure with the sensor lasted for 76 consecutive 15-min intervals, and the diagram displays only a few zero-reported vehicle numbers. Subsequently, upon communication restoration, the sensor reported cumulative values for individual types of vehicles for all the missing time intervals, as is visible in the last displayed row. Since using such erroneously reported values during the sensors' failures would lead to a distortion of the predictive model being created, it was necessary to address these values in more detail. The first step involved manually identifying individual outages of all the sensors. This was not an automated task, as each case required expert evaluation to determine whether it was a failure or an acceptable value. In this way, all 72 target variables needed to be checked, and cases considered as communication failures needed to be marked. Subsequently, after identifying failures and incorrectly reported values, we proceeded to decide on the next steps for handling these values in the analysis. One option for dealing with incorrect values is to delete them from the dataset. However, in this case, this would entail a significant loss of information as there were multiple communication failures; moreover, omitting some observations would disrupt the temporal continuity in the data. Therefore, only correcting the erroneous values, i.e., replacing them with estimated corrected values, was considered. We aimed to utilise the information we had in the rest of the dataset by using the cumulative vehicle numbers for all the time intervals during the failure. These cumulative vehicle numbers thus needed to be distributed across all the time intervals during the communication failure. One of the options considered was replacing all zero values during the failure with the average value obtained from the cumulative numbers of vehicles. However, this solution would introduce inaccuracies into the subsequent model of mobility because, for example, if the failure lasted during both day and night hours, the zero values reported during the failure in the night hours, when vehicle numbers are naturally low and often zero, would be replaced with the average value. Moreover, the same value would be estimated during daytime hours, including peak traffic times. Therefore, we sought a more sophisticated solution for replacing incorrect values during the sensors' failures. Finally, we opted for estimating these values by creating regression models using the information on the cumulative numbers of vehicles. For each sensor, vehicle type, and direction, a linear regression model of the dependency of the vehicle numbers of that type on the specified explanatory variables (listed in Table 3 above) was created.