# FlockAI: A Testing Suite for ML-Driven Drone Applications
## 4 Modeling and Implementation
### Drone Configuration
In turn, AirSim [38] is a framework built on top of the Unreal game engine, to provide users with the ability to simulate a drone flight under various environmental conditions. These include, wind effects, GPS signal degradation, and physical phenomena, such as drag and gravity alterations. Moreover, the authors state that AirSim was proposed for developers to generate (annotated) training data to be used by ML algorithms. However, no facilitation of the execution of ML algorithms on the drone itself is provided. Moreover, [PERSON] et al. introduce FANETSim [25], a java-based simulation tool that enables the testing of drone-to-ground station and drone-to-drone communication overhead. Overhead is examined in terms of message counts and energy consumption. However, the energy model only considers energy relevant to flying the drone (\(E_{motors}\)) which is modeled with drone velocity being the only dependent variable. Finally, Deepbots [39] is a framework of abstractions enabling Reinforcement Learning (RL) over Webots by acting as a middleware between the simulator and the OpenAI Gym. Specifically, users extend a set of functions for an AI agent to run in Webots. However, this only includes RL parameterization such as how rewards are updated, how to calculate the next step and how to process a new observation. Hence, leaving drone configuration, performance monitoring, and even energy modeling, as burdening tasks for the user. ## 7 Discussion

Table 5 provides a concise overview to serve as a guide for performing a qualitative comparison of the aforementioned works. From this overview, one first observes that the majority of the frameworks support the deployment and testing of user-designed algorithms for flight control and state estimation. The execution settings of these algorithms varies with some frameworks opting for remote execution (i.e., jMavSim, UTSim, AirSim) and some opting for in place execution (autopiloting). However, no framework other than FlockAI, supports both execution modes. As shown in experiment scenario 2, evaluating when to perform on-board or remote inference is of critical importance for ML-driven drone applications as inference can impact flight time. Still, this may be justified by the fact that, other than Deepbots, FlockAI is the only drone testing framework that supports ML algorithm evaluation. In regards to monitoring, the majority of the introduced works provide metrics for flight control and a few provide network/communication data, but only FlockAI provides insights in regards to system utilization (i.e., cpu, mem, i/o). Moreover, energy profiling for the majority of the works is limited to only evaluating the motorized

components impact. However, our evaluation has shown the importance of the processing and communication overhead to the total energy deficit. Finally, a note on limitations of FlockAI. The current version does not support the configuration of environmental conditions in the simulation, as opposed to AirSim, and FlockAI only supports drone controller programming in Python, despite the fact that webots can run C++ and Java. ## 8 Conclusions and Future Work

In this article, we have introduced FlockAI: an open and extensible framework facilitating the rapid prototyping of reproducible experiment scenarios for ML-driven drone applications evaluated on top of a robotic simulator. Specifically, FlockAI embeds a powerful modeling framework for designing experiment benchmarks so that ML practitioners can configure drone resource capabilities, enable sensing and communication modules, deploy their ML algorithm(s) that will run on the drone itself and finally, monitor the perfor

mance and impact of the ML-driven tasks. Through runtime monitoring, ML practitioners are able to evaluate various key performance indicators of their ML-driven applications including flight performance, resource utilization, communication overhead, and energy consumption.