Package: opensbt
OpenSBT - A Modular Framework for Search-based Testing of Automated Driving SystemsIntroOpenSBT provides a modular and extandable code base to facilitate search-based testing of automated driving systems. It provides interfaces to integrate search algorithms, fitness/criticality functions and simulation environments in a modular way. It allows to visualize testing results and analyse the critical behaviour of the tested system.An introductory video of OpenSBT can be found here:https://www.youtube.com/watch?v=qi_CTTzrk5s.OverviewOpenSBT builds uponPymoo0.6.0 and extends internal optimization related models, such asProblem,Result, to tailor heuristic search algorithms for testing ADS systems.InstallationOpenSBT requires Python to be installed and its compatibality has been tested with Python 3.8. OpenSBT can be run as a standalone application or can be imported as a library.Installation instructions are available in the followingjupyter notebook.UsageYou can find several tutorials asjupyter notebookswhich explain step-by-step of how to use OpenSBT. In these tutorials, we have integrated:A simplified SUT simulated in very simplistic simulator (linear motion planning, no GPU required)A real-world FMI-based AEB agent developed in thefortiss Mobility labwhich is simulated inCARLAusing the simulator adapterCARLA Runner Extension.Note: We have also implemented asimulator adapterfor testing Simulink-based SUTs in Prescan.OutputOpenSBT produces several result artefacts. All artefacts are written into theresultsfolder in a folder named as the problem name.
OpenSBT generates the following outputs:TypeDescriptionExampleDesign Space PlotVisualization of all evaluated test cases in the input space + of predicted critical regions using the decision tree algorithm, pairwise. Constraints of derived regions are stored in CSV filebounds_regions.csvand the learned tree intree.pdfScenario 2D VisualizationVisualization of traces of the ego vehicle and adversaries in a two-dimensional GIF animationObjective Space PlotVisualization of fitness values of evaluated test cases, pairwise.All TestcasesCSV file of all test inputs of all evaluated testcasesall_testcases.csvAll Critical TestcasesCSV file of all critical test inputs of all evaluated testcasesall_critical_testcases.csvCalculation PropertiesCSV file of all experiment configuration parameters (e.g. algorithm parameters, such as population size, number iterations; search space, fitness function etc..).calculation_properties.csvEvaluation ResultsCSV file containing performance values of the algorithm, e.g., number critical test cases found in relation to all evaluations, execution time.summary_results.csvApplication Use CasesOpenSBT has been already applied:For a replication experiment to replicate the results of a surrogate assisted testing technique, s.here.In an industrial case study to validate a systems behaviour for different operating scenarios, s.hereand here:https://doi.org/10.1007/978-3-031-46002-9_15ContributionIf you like to contribute please contact one of the developers listed below or create a pull request. If you face any issue with OpenSBT feel free to create an issue or contact the developers.AcknowledgementsOpenSBT has been developed byLev Sorokin,Tiziano MunaroandDamir Safinwithin theFOCETA Project. Special thanks go toBrian Hsuan-Cheng Liaoand Adam Molin fromDENSO AUTOMOTIVE Deutschland GmbHfor their valuable feedback and evaluation of OpenSBT on the AVP Case Study in the Prescan simulator.ReferenceIf you use or extend OpenSBT please cite our framework. Here is an example BibTeX entry:@misc{sorokin2023opensbt,title={OpenSBT: A Modular Framework for Search-based Testing of Automated Driving Systems},author={Lev Sorokin and Tiziano Munaro and Damir Safin and Brian Hsuan-Cheng Liao and Adam Molin},year={2023},eprint={2306.10296},archivePrefix={arXiv},primaryClass={cs.SE}}LicenseOpenSBT is licensed under theApache License, Version 2.0.