## 5 Results
### Evaluation of the Artifacts
#### 5.2.2 Efficiency Evaluations  
Iteration 1: Offline accuracy of ML algorithms was evaluated for facilitating decision making of multi access routing. The evaluation was performed with a training set, which was collected with an emulation (see Appendix A), where two vehicles moved for a duration of 20 min, and six access points routed data. The size of the data set was 143,751 samples. The offline accuracy of RandomForestRegressor and River is provided in Table 1.  
Figure 13 presents the error of incremental ML with River as a function of data samples. Two (RSSI and bitrate) and three (RSSI, bitrate, and SNR) feature sets were experimented with. The results indicate that River, with both feature sets, could facilitate decision making of multi-access routing with a reasonable accuracy. Incremental learning with River took ~2040 samples until error was lower than 10% (with RSSI and bitrate-feature set). Thus, a reasonable accuracy should be achieved in less than 3.4 min from the beginning of the emulation (10 samples/second were collected). In the end of the experiment, error/SMAPE was ~6.6% with both feature sets (Table 1); sklearn's RandomForestRegressor also indicated that RSSI and bit rate were the most significant features (see Table 1) from an accuracy point of view.  
Iteration 2: Evaluation of efficiency in decision making regarding routing of the data transfers/communication. Efficiency of the architecture was evaluated based on online experiments in the emulator (see Appendix A for the detailed testing procedure). Particularly, a RandomForestRegressor-based model (see Table 1) was trained offline with sklearn, and the model was deployed to RedisAI inference engine. Emulated vehicles fetched online predictions from RedisAI by providing RSSI, bit rate, and signal-to-noise ratio (SNR) to the prediction service as input parameters.  
Two different emulation scenarios were executed (6 min and 1 h), where two vehicles moved according to pre-calculated routes, and six access points enabled data transfer towards the control center. The OWAMP tool was utilized for measuring OWD in emulation. OWD and predictions were saved in InfluxDB and matched together afterwards based on timestamps (every 100 ms) for error/accuracy evaluation. Additionally, an emulation was  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Algorithm** & **MLP Regressor** & **RandomForestRegressor** & **River** \\ \hline Training features & RSSI, bitrate, and SNR & RSSI, bitrate, and SNR & RSSI, bitrate, and SNR \\  & & & \\ Algorithm configuration & default & n\_estimators = 100 & seed = random, algorithm = \\  & & & AdaptiveRandomForesRegressor \\ Accuracy metric & Stratified K-fold (10) & Stratified K-fold (10) & Symmetric MAPE \\  & & RSSI = 0.0386, & \\ Feature importances & - & bit rate = 0.946, & - \\  & & SNR = 0.015 & \\ Accuracy & 0.91\(\pm\) 0.07 & 0.935 \(\pm\) 0.05 & RSSI, bitrate, and SNR: 6.648\% \\ \hline \hline \end{tabular}
\end{table}
Table 1: Offline accuracy/efficiency of different ML algorithms.  
Figure 13: Error (symmetric MAPE) of incremental ML with River as a function of data samples. Measurements of the first 5000 samples are provided for illustrating until SMAPE was <10 %.  
performed with a reference case, which utilized a heuristic RSSI-based model for decision making.  
Figure 14 presents the OWD of vehicle 2 in the short (6 min) emulation experiment. The ML-based model achieved \(\sim\)11.5% lower OWD, when compared to the RSSI-based model.  
The results of all emulation experiments are summarized in Table 2. The results indicate that the ML-based model achieved \(\sim\)5-13% lower OWD, when compared to the reference model. The prediction error was \(\sim\)5.4-7.5%, which is within the same range as the offline accuracy (0.935 \(\pm\) 0.05). Thus, it seems that a ML-based model can facilitate decision making of online routing with a decent accuracy, which improves OWD in comparison to the heuristic approach.