GABezier at the SBST 2021 Tool Competition
Florian Klück, Lorenz Klampfl, Franz Wotawa
Christian Doppler Laboratory for Quality Assurance Methodologies for Autonomous Cyber-Physical Systems
Institute for Software Technology, Graz University of Technology, Inffeldgasse 16b/2, A-8010 Graz, Austria{fklueck,lklampfl,wotawa}@ist.tugraz.at

Abstract—GABezier is a search-based tool for the automatic
generation of challenging road networks for virtual testing of
an automated lane keep system (ALKS). This paper provides
a brief overview on the tool and summarizes the results ofGABezier’s participation at the first edition of the Cyber-PhysicalSystems Testing Tool Competition. We submitted our tool in two
configurations, namely GABExplore and GABExploit. Especiallythe latter configuration has efficiently generated valid test cases
and triggered many faults.

I. I NTRODUCTION
The ambitious development of Advanced Driver AssistantSystems (ADAS) introduces new challenges to the wellestablished verification and validation methods in the automotive industry as described in [1] and [2] . In the first edition of the Cyber-Physical Systems Testing Tool Competitiondifferent research groups developed automated test generation
tools for virtual testing of self-driving vehicle software. WithGABezier, we submitted a tool for the automatic generationof challenging road networks, combining Bezier Curves andSearch [3]. We submitted two configurations of our tool, an exploratory configuration (GABExplore) that aims to maximize
the number of unique failing test cases and an exploitative
configuration (GABExploit) that aims to maximize the overall
number of failing test cases. Both configurations were able togenerate failure inducing inputs and in particular the latter
configuration was able to expose the highest number offailing test cases [4] during the competition. In this paper weprovide an overview on the GABezier method, summarize theresults achieved in the competition and finally outline present
limitations that we plan to consider as part of our future work.
II. OVERVIEW GAB EZIERGABezier aims to generate challenging road networks forvirtual testing of an ALKS. For our method, road networks
are represented as parametric curves, which are constructedbased on control point sets. The road points described by
the parametric curve form the input for the underlying code
pipeline to automatically generate virtual roads, which are executable in a simulation environment. Furthermore, the obtained
virtual road defines the dynamic driving task the ALKS has toperform from start to end. Changing the location of one control
point has global influence on the resulting road geometry.
During testing, we search for control point arrangementsthat result in challenging road networks. We consider road
networks challenging, when the ALKS equipped vehicle does
not perform the dynamic driving task correctly and leaves

the intended road path. For search, as in our previous work[5], we use a genetic algorithm, where one set of control
points represents one individual in the seed population. For
evaluation of one control point individual, we construct thecorresponding parametric curve and hand over the obtainedroad points to an underlying code-pipeline that was provided
by the organizers of this competition [6]. Here, the final roadis constructed, validated and executed in a virtual environment
namely BeamNG.research [7]. To determine each individual’sfitness, we measure the distance between the ego vehicle andthe center line and outer line of the road. For optimization werely on the standard genetic operators of crossing and mutation
and aim to maximize the fitness function. For the competition
we submitted two configurations of our tool, GABExploit andGABExplore.
a) GABExploit: In the exploitative configuration we randomly generate a seed population and continue the search until
the given time budget is consumed. We aim to maximize thetotal number of failures exposed.
b) GABExplore: In the exploratory configuration, oncea failure inducing input is found we restart the search with anew randomly generate seed population until the given timebudget is consumed. Here, we aim to find more diverse failingtest cases.
III. E XPERIMENTAL S ET-U P AND P ROCEDURE
The goal of the competition is to generate as many diverse
failing test inputs as possible until a given time budget isconsumed. A test input is considered failed, when the resulting
virtual road forces the ALKS equipped vehicle to leave its
designated lane by more than a certain pre-defined tolerance
value. The system under test is an omniscient driving agent
called BeamNG.AI that aims to follow the given road path
while staying close to a pre-defined speed limit. During thecompetition each tool was evaluated in two different experimental set-ups: Default and SBST21. For both configurations,test inputs had to be generated within a given map of size
200 m x 200 m. The Default set-up was executed 5 times
with a time budget of 5 hours per execution, a tolerance
value of 0.95 and no speed limit. The SBST21 set-up wascarried out 10 times with a shorter time budget of 2 hours
per run, a tolerance value of 0.85 and a speed limit of 70
km/h. Each test-run is evaluated regarding the total numberof triggered failures, but also the diversity between failureinducing inputs. Furthermore, the total number of generated
test inputs as well as the share of valid and invalid test inputs

are considered to evaluate the tool performance regarding test
generation efficiency and effectiveness.

TABLE IIB ENCHMARK RESULTS FOR THE SBST21 EXPERIMENT SET- UP.

IV. B ENCHMARK R ESULTS
Table I shows the results obtained for both GAB configurations in the default experimental set-up and Table IIsummarizes the results obtained in the SBST21 experimentset-up. The following abbreviations are used: TR: test run, NTC: test cases generated, V: valid test cases, I: invalid testcases, P: passed test cases, F: failing test cases, E: errors, SP:sparseness, N-SH: number of to sharp test cases, N-I: numberof intersecting test cases.
TABLE IB ENCHMARK RESULTS FOR THE DEFAULT EXPERIMENT SET- UP.
TR12345AVG

N-TC
306
474
461
311
415
393.4V299393
431
309
357
357.8

I
7813025835.6

GABExploit
PF
287
12308
85
311
120
299
10357
0312.4 45.4

E000000SP
4.093
12.78510.795
8.608
9.07

N-SH
7813025835.6

N-I000000TR
12345AVG

N-TC
425
351470
331
329
381.2

V357
325384322
310
339.6

I
68268691941.6

GABExplore
PF
35613223383
1314
8308
2336.6 3E000000SP
19.73121.88718.087
19.9

N-SH
68
2686
91941.6

N-I000000First, we see that both test generator configurations fulfill
their unique purpose, i.e., GABExploit maximizes the amountof failing test cases and GABExplore maximizes the sparseness between failing test cases . In particular, with 120 failingtest cases GABExploit exposed the highest number of failingtests in the default experimental set-up in the course of thiscompetition. However, GABExploit did not show consistent
performance and did not even find one failing test case in
the last run. Furthermore, since this configuration does notpromote searching for diverse failing test cases we see a lower
sparseness of the generated failing test cases, which indicatesan opportunity for improvement considering test efficiency. In
contrast, GABExplore did not generate a comparable amountof failing test cases but the once we generated were unique.
Furthermore, for the default set-up, GABExplore showed
consistent performance over all test runs in finding at least
one failure inducing test input. Similar observations can bemade for the SBST21 experiment set-up, depicted in Table II.
However, here we see that GABExplore is less efficient when
given the smaller time budget. If we compare the results tothe once obtained in the default configuration we see that
GABExplore significantly improves when given more time forthe calculation and search. It is worth noting that we were able
to limit the number of invalid test cases and never generated
intersecting test cases at all.

TR1
2
3
4
5
678910AVG

N-TC
184
126
224
124
153
121
155
123
129
124
146.3

V
160
119
216
119
135
120
129
123
121
114
135.6

I
24
78518
1
26
0
8
1010.7GABExploit
P
F
75
84
101
1889
126
81
38
135
0109
1182
47
123
0
93
28
109
599.7 35.7

E
10
1
0
0
0
0
0
0
00.2

SP
5.462
33.45
12.85411.841
15.466
9.28
9.41
4.563
12.79

N-SH
24
7
8
518
1
26
0
81010.7

N-I
0
0
00
0
000000TR
1234
567
8910AVG

N-TC
137
132
120
127
141
143
33142121151124.7

V
125
123
113
118
13513230139118
126
115.9

I
1297961133
325
8.8

GABExplorePF
12501221112
111801341129
3801380115
3
1260
112.7 0.9E000000
22
0
00
2.2

SP
2.434
18.075
10.25

N-SH
12
97
96
113
3
325
8.8

N-I
0000
0
0
0
0
000

V. C ONCLUSION
This paper reports on the results of GaBezier’s participation
at the first edition of the Cyber-Physical Systems Testing
Tool Competition. In one test run GAExploit achieved thehighest number of failing test cases of all tools in the competition. However, the high similarity between failing test cases,
reduces the overall test efficiency, offering opportunity forimprovement.
R EFERENCES[1] P. Koopman and M. Wagner, “Challenges in autonomous vehicle testingand validation,” SAE International Journal of Transportation Safety,
vol. 4, pp. 15–24, 04 2016.
[2] F. Wotawa, “Testing autonomous and highly configurable systems:
Challenges and feasible solutions,” in Automated Driving: Safer andMore Efficient Future Driving, D. Watzenig and M. Horn, Eds.
Cham: Springer International Publishing, 2017, pp. 519–532. [Online].
Available: ”https://doi.org/10.1007/978-3-319-31895-0-22”[3] F. Klück, L. Klampfl, and F. Wotawa, “Automatic generation of challenging road networks for alks testing based on bezier curves and search,”
2021, arXiv:2103.01288 [cs.SE].
[4] S. Panichella, A. Gambi, F. Zampetti, and V. Riccio, “Sbst tool competition 2021,” in International Conference on Software Engineering,Workshops, Madrid, Spain, 2021. ACM, 2021.
[5] F. Kluck, M. Zimmermann, F. Wotawa, and M. Nica, “Genetic algorithmbased test parameter optimization for adas system testing,” in Proceedings- 19th IEEE International Conference on Software Quality, Reliabilityand Security, QRS 2019, ser. Proceedings - 19th IEEE International
Conference on Software Quality, Reliability and Security, QRS 2019.
United States: Institute of Electrical and Electronics Engineers, 7 2019,
pp. 418–425.
[6] “https://github.com/se2p/tool-competition-av.”[Online].
Available:
https://github.com/se2p/tool-competition-av[7] BeamNG GmbH, “BeamNG.research,” software version 1.3.0.0, 2018-1011, https://www.beamng.gmbh/research.