Package: aerialist
AERIALIST: UAV Test BenchAerialist(unmanned AERIAL vehIcle teST bench) is a novel test bench for UAV software that automates all the necessary UAV testing steps: setting up the test environment, building and running the UAV firmware code, configuring the simulator with the simulated world properties, connecting the simulated UAV to the firmware and applying proper UAV configurations at startup, scheduling and executing runtime commands, monitoring the UAV at runtime for any issues, and extracting the flight log file after the test completion.With Aerialist, we aim to provide researchers with an easy platform to automate tests on both simulated and real UAVs, allowing them to do experiments required to overcome the UAV simulation-based testing challenges.Table of ContentsIntroductionUAV TestsDemo VideoGetting StartedDocker Test ExecutionLocal Test ExecutionKubernetes Test ExecutionUsageTest Description FileCommand Line InterfacePython APIReferencesLicenseContactsIntroductionThe below figure demonstrates Aerialist's software architecture, with the current implementation supporting UAVs powered byPX4-Autopilot(a widely used open-source UAV firmware).The input is aTest Descriptionfile, which defines the UAV and environment configurations and the test steps.
TheTest Runnersubsystem (that abstracts any dependencies to the actual UAV, its software platform, and the simulation environment) prepares the environment for running the test case as described in the test description.After setting up the simulation environment (if testing a simulated UAV), the Test Runner connects to the (simulated or physical) UAV and configures it according to the startup instructions. Then, it sends runtime commands, monitors the UAV's state during the flight, and extracts flight logs at the end of the test for future analysis. Each module is detailed in theArchitecture Documents.UAV TestsThe de-facto testing standard of UAVs relies onmanually written system-level teststo test UAVsin the field.
These tests are defined assoftware configurations(using parameters, config files, etc.) in a specificenvironmentsetup (e.g., obstacles placement, lighting conditions) and a set of runtimecommands.
The runtime commands received during the UAV flight (e.g., from a remote controller) make the UAV fly with a specific human observablebehavior(e.g., trajectory, speed, distance to obstacles).Hence, Aerialist models a UAV test case with the following set oftest propertiesand uses aYAMLstructure to describe the test.Drone: Software configurations of the UAV model, including allAutopilot parametersand configuration files (e.g., mission plan) required to set up the drone for the test.Environment: Simulation settings such as the used simulator, physics of the simulated UAV, simulation world (e.g., surface material, UAV’s initial position), surrounding objects (e.g., obstacles size, position), weather conditions (e.g., wind, lighting), etc.Commands: Timestamped external commands from the ground control station (GCS) or the remote controller (RC) to the UAV during the flight (e.g., change  flight mode, go in a specific direction, enter mission mode).Expectation (optional): a time series of certain sensor readings that the test flights are expected to follow closely.Demo VideoTake a look at theDemo Videoto get more insights on the test execution methods introduced later.Getting StartedYou can execute UAV test cases with Aerialist in three different ways.Docker Test Execution(Recommended): Execute Test Cases in pre-built Docker containers without the need to install PX4 dependencies on your machine.
This is the recommended option for most use cases and supports headless simulation (without the graphical interface).Local Test Execution: Execute Test Cases using PX4 dependencies installed on the same machine.
This allows you to easily develop new functionalities and run the UAV simulators with the graphical interface so you can visually follow the UAV behavior.
This is only recommended if you prefer to get more insights into UAV behavior.Kubernetes Test Execution: You can also deploy your test execution to a Kubernetes cluster for more scale.
This option is only recommended if you are using Aerialist to conduct large-scale experiments on test generation for drones.Docker Test ExecutionUsing Docker containers with pre-configured PX4 dependencies to execute test cases is the simplest and recommended way of executing UAV tests with Aerialist.
Aerialist's Docker image is hosted onDockerhub.Using Docker Container's CLIRequirements:DockerThis has been tested onWindows, Ubuntu and macOS with x86-64 processors.You may need to rebuild the docker image if you are using another OS or architecture.docker run -it skhatiri/aerialist bashYou can now use theCommand Line Interfacein the container's bash.checkpython3 aerialist exec --helpNote:The .env for the docker image comes fromtemplate.env. You can customize them usingenvironment variablesfor the Docker container.Using Host's CLIAlternatively, you can use the CLI on your local machine and instruct Aerialist to execute test cases inside a docker container.
This gives you more flexibility since the test results (flight logs) are directly stored on the host machine and you don't lose them.Requirements:DockerPython ≥ 3.8Your user should be able to run docker commands withoutsudo.check hereThis has been tested onUbuntu and macOS with x86-64 processors.You may need to rebuild the docker image if you are using another OS or architecture.Clone this repository andcdinto its root directorypip3 install -r requiremetns.txtCreate a file named.envin the repository's root directory. Then copy and customize the contents oftemplate.envinto it.Create a folder namedresultsin the repository's root directory.You can use the dockerfile to build a Docker image with all the requirements, or instead pull the latest image from the Image repository.docker build . -t skhatiri/aerialistordocker pull skhatiri/aerialistYou can now instruct Aerialist to execute test cases inside a docker containerJust add--agent dockerto the command line or update the test-description file (agent.engine:docker).You can now use theCommand Line Interfacein your local bash.checkpython3 aerialist exec --helpLocal Test ExecutionNote:InstallingPX4-Autopilot,PX4-Avoidanceand their requirements including ROS and Gazebo could be problematic in many cases. We only suggest this full installation to users who need direct visual access to the simulator or are curious to visually monitor the UAVs during the flight. Otherwise, test execution, extracting the flight logs, and plotting them can be achieved by thedocker exectionas well.We have prepared a ready-to-use virtual machine (download link) to help users onboaord fast. You can move on to usingdockerized test executionsif you don't need the simulation visualizations any more.If you prefer to run the simulations and PX4 on your own machine, followPX4 installation guide.Requirements:Ubuntu 18We have prepared abashsriptto automate all the steps of installing PX4 and Aerialist in one shot.Follow the instructions.You can also follow astep-by-step guideif needed.Clone this repository and cd into its root directorypip3 install -r requiremetns.txtCreate a file named.envin the repository's root directory. Then copy and customize the contents oftemplate.envinto it.UpdatePX4_HOMEandRESULTS_DIRvariables according to your installation.You can now use theCommand Line Interface.Kubernetes Test ExecutionAerialist can also deploy test executions on a Kubernetes cluster to facilitate running tests in the cloud. Specifically, as can be seen in the below figure, Aerialist can run multiple executions of the same test case in isolated Kubernets pods in parallel, and gather test results for further processing.This feature is specifically helpful when performing test generation tasks, where UAV's flight could be subject to non-determinism and multiple simulations are required.
Follow ourK8S Deployment Guideline.UsageTest Description FileUsing a predefinedtest-description yaml fileis the easiest way to define the test case.# template-test.yamldrone:port:sitl# type of the drone to connect to {sitl, ros, cf}#params: #PX4 parameters : https://docs.px4.io/main/en/advanced_config/parameter_reference.html# {parameter_name}: {parameter_value} #(keep datatype -> e.g, 1.0 for float, 1 for int)# CP_DIST: 1.0# POS_MOD: 2.5params_file:samples/flights/mission1-params.csv#csv file with the same structure as abovemission_file:samples/flights/mission1.plan# input mission file addresssimulation:simulator:ros# the simulator environment to run {gazebo,jmavsim,ros}speed:1# the simulator speed relative to real timeheadless:true# whether to run the simulator headlessobstacles:-size:# Object 1 size in l,w,hl:5w:5h:5position:# Object 1 position in x,y,z and it's rotationx:-8y:3z:0r:0# home_position: # home position to place the drone [lat,lon,alt]test:commands_file:samples/flights/mission1-commands.csv# runtime commands file addressassertion:log_file:samples/flights/mission1.ulg# reference log file address# variable: trajectory # reference variables to compareCommand-Line InterfaceTest ExecutionYou can utilize the toolkit with the following command line options:Make sure you are at the root directory of the repository:cd Aerialist/IMPORTANT NOTE: The following commands are assuming that you are using Aerialist directly on your machine (not inside a docker container).--agent dockeris asking Aerialist to create a docker container and execute test cases there as detailedabove.If you are executing the commandinside a docker container, exclude--agent dockerto let the test run locally there.The simplest way to execute a UAV test by Aerialist is by the following command:python3 aerialsit exec --test [test-file.yaml] --agent dockerReplaying a pre-recorded manual flight (RC commands are extracted and replayed from a .ulg log)python3 aerialist exec --test samples/tests/manual1.yaml --agent dockerExecuting an autonomous mission with a box-shaped obstacle in the environmentpython3 aerialist exec --test samples/tests/mission1.yaml --agent dockerMore sample tests can be foundhereWhen test exection is finished, you will have test results inresults/folder:Flight Log of the executed test (.ulg)Plot of the flight trajectory, as seen belowYou can usepython3 aerialist exec --helpanywhere to get help onother possible arguments.Plotting Executed Test CasesYou can plot flight trajectory of the executed test cases using the following comnmand. plots are stored inresults/folder.python3 aerialist plot --test [test-file.yaml] --log [test-log.ulg]Using Aerialist in Your CodeYou can integrate Aerialist's Python package in your own code and directly define and execute UAV test cases with it.
This can be specifically useful when you are working on test generation approaches for UAVs. An example of such usage of Aerialist can be found inSurrealist.pip3 install git+https://github.com/skhatiri/Aerialist.gitCreate an instance ofDroneTestclass and define your test caseconfigure and execute the test case using your preferred test execution agent (LocalAgent,DockerAgent,K8sAgent)Take a look at ourcode snippetsfor more details and sample codes.ReferencesIf you use this tool in your research, please cite the following papers:Sajad Khatiri, Sebastiano Panichella, and Paolo Tonella, "Simulation-based Test Case Generation for Unmanned Aerial Vehicles in the Neighborhood of Real Flights,"In 2023 IEEE 16th International Conference on Software Testing, Verification and Validation (ICST)PreprintExperiments Dataset@inproceedings{khatiri2023simulation,
  title={Simulation-based test case generation for unmanned aerial vehicles in the neighborhood of real flights},
  author={Khatiri, Sajad and Panichella, Sebastiano and Tonella, Paolo},
  booktitle={2023 16th IEEE International Conference on Software Testing, Verification and Validation (ICST)},
  year={2023},
}LicenseThe software we developed is distributed under MIT license. See thelicensefile.ContactsFeel free to use theDiscussionssection to ask your questions and look for answers.You can also contact us directly using email:Sajad Khatiri (Zurich University of Applied Sciences) -mazr@zhaw.ch