# FlockAI: A Testing Suite for ML-Driven Drone Applications
## 4 Modeling and Implementation
### Drone Configuration
From this, we immediately observe that the FIGs algorithm enables the drone to stay in the air for an additional 2 min and 34 s. Based on these results, and for the specific use-case in hand, despite the fact that the FIGs algorithm can boost mission completion time by over 2 min, this comes with a significant hit in terms of accuracy. Clearly, the 35% boost in accuracy for the CNN algorithm is of most importance, even if the 35% additional accuracy translates to 2.5 min less in flight time. Thus, by being able to rapidly reproduce and replay emulated testbeds, the ML practitioner is able to answer Q4 and having answered all four questions, he/she can now take more informative decisions. ## 6 Related Work

As drone technology is now just growing out of its infancy, the majority of the benchmarking frameworks for drones have been designed to support the testing of various hardware capabilities and/or flight control and state estimation. For example, Drones-Bench [33] is a system for stress testing in a controlled (and grounded) environment the hardware components of real drones during take-off and landing. The system aims to detect faults in the hardware by measuring the thrust force control action, the power consumption of the motorized components, and the attitude stability. \begin{table}
\begin{tabular}{c c c} \hline \hline  & **CNN** & **FIGs** \\ \hline
**Sequence** & **Inf. Delay (img count)** & **Inf. Delay (img count)** \\ \hline seq1 & 23 & - \\ seq2 & 26 & 29 \\ seq3 & 29 & 29 \\ seq4 & 29 & 29 \\ seq5 & 23 & 28 \\ seq6 & 24 & 23 \\ seq7 & 26 & - \\ seq8 & 27 & - \\ seq9 & 26 & - \\ seq10 & 26 & 29 \\ seq11 & 29 & 29 \\ seq12 & 28 & 29 \\ seq13 & 25 & - \\ seq14 & 26 & 28 \\ seq15 & 28 & 29 \\ seq16 & 28 & - \\ seq17 & 26 & 29 \\ seq18 & 27 & 29 \\ seq19 & 28 & - \\ seq20 & - & - \\ \hline \hline \end{tabular}
\end{table} 
Table 4: ML algorithm inference delay (in images of camera feed). \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Algorithm** & **Accuracy (\%)** & **Median Inference Time (ms)** & **Flight Time (s)** \\ \hline CNN & 95 & 1361 & 1488 \\ FIGs & 60 & 274 & 1642 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: ML algorithm accuracy vs. performance for on-board inference. On the other hand, jMavSim [34] is a simulation tool aimed at simplifying the design and testing of autonomous flight control (e.g., mission planning) for drones that adopt the PX4 autopilot firmware. The jMavSim programming environment is in Java and both monitoring and energy consumption data (for the motorized components) depend on external tools that interface between the PX4 autopilot when the testing adopts a hardware-in-the-loop configuration. In turn, Rotor5 [35] is a modular framework to design algorithms for drone flight control and state estimation, which can then be tested in the Gazebo simulator to debug errors that could potential lead to drone collisions and crashes. Another tool is MRS [36], an open-source framework with various drone templates (i.e., dji, tarot drones) focusing on the design and testing of user-developed C++ algorithms for drone flight control and state estimation. Unfortunately, both Rotor5 and MRS lack energy profiling and do not provide any monitoring insights for the system utilization and communication overhead of the drone from the execution of user algorithms. Nonetheless, a handful of frameworks are now aiming to provide users with the ability to deploy and test algorithms over drone settings. Specifically, UTSim [37] is a framework for testing over a simulator (Unity) different algorithms for air traffic integration. In particular, users specify, via C# scripts, the properties of the environment, including the number of drones and their configuration (i.e., speed, destination), and specify the algorithm used for path planning and collision avoidance. The simulator, at the end, outputs a log file with the content capturing data, such as the number of sent and received messages, the number of detected objects and collided drones.