FRA-UNIted — Team Description 2022
Thomas Gabel, Yalcin Eren, Fabian Sommer,
Alexander Vieth, Eicke Godehardt
Faculty of Computer Science and EngineeringFrankfurt University of Applied Sciences
60318 Frankfurt am Main, Germany{tgabel|godehardt}@fb2.fra-uas.de, {eren|fsommer|avieth}@stud.fra-uas.de

Abstract. The main focus of FRA-UNIted’s effort in the RoboCup soccer simulation 2D domain is to develop and to apply machine learningtechniques in complex domains. In particular, we are interested in applying reinforcement learning methods, where the training signal is only
given in terms of success or failure. In this paper, we review some of ourrecent efforts taken during the past year, also analyzing the impact thatthe ongoing Covid pandemic has been having on our team’s activities.

1

Introduction

The soccer simulation 2D team FRA-UNIted is a continuation of the former
Brainstormers project which has ceased to be active in 2010. The ancestor Brainstormers project was established in 1998 by Martin Riedmiller, starting off witha 2D team which had been led by the first author of this team description paper
since 2005. Our efforts in the RoboCup domain have been accompanied by theachievement of several successes such as multiple world champion and world vicechampion titles as well as victories at numerous local tournaments. Our teamwas re-established in 2015 at the first author’s new affiliation, Frankfurt University of Applied Sciences, reflecting this relocation with the team’s new name
FRA-UNIted.
As a continuation of our efforts in the ancestor project, the underlying andencouraging research goal of FRA-UNIted is to exploit artificial intelligence andmachine learning techniques wherever possible. Particularly, the successful employment of reinforcement learning (RL, [10]) methods for various elements of
FRA-UNIted’s decision making modules — and their integration into the competition team — has been and is our main focus. Moreover, the extended use of
the FRA-UNIted framework in the context of university teaching has moved into
our special focus. So, we aim at employing the 2D soccer simulation domain as afundament for teaching agent-based programming, foundations of multi-agents
systems as well as applied machine learning algorithms.
In this team description paper, we refrain from presenting approaches andideas we already explained in team description papers of the previous years[3]. Instead, we focus on recent changes and extensions to the team as well as
on reporting partial results of work currently in progress. We start this teamdescription paper, however, with a short general overview of the FRA-UNIted
framework. Note that, to this end, there is some overlap with our older teamdescription papers including those written in the context of our ancestor project(Brainstormers 2D, 2005–2010) which is why the interested reader is also referredto those publications, e.g. to [6, 9].
1.1

Design Principles

FRA-UNIted relies on the following basic principles:– There are two main modules: the world module and decision making– Input to the decision module is the approximate, complete world state asprovided by the soccer simulation environment.
– The soccer environment is modeled as a Markovian Decision Process (MDP).
– Decision making is organized in complex and less complex behaviors where
the more complex ones can easily utilize the less complex ones.
– A large part of the behaviors is learned by reinforcement learning methods.
– Modern AI methods are applied wherever possible and useful (e.g. particle
filters are used for improved self localization).

Fig. 1. The Behavior Architecture

1.2

The FRA-UNIted Agent

The decision-making process of the FRA-UNIted agent is inspired by behaviorbased robot architectures. A set of more or less complex behaviors realize theagents’ decision making as sketched in Figure 1. To a certain degree this architecture can be characterized as hierarchical, differing from more complex behaviors,such as “no ball behavior”, to very basic, skill-like ones, e.g. “pass behavior”.
Nevertheless, there is no strict hierarchical sub-divisioning. Consequently, it isalso possible for a low-level behavior to call a more abstract one. For instance,
the behavior responsible for intercepting the ball may, under certain circumstances, decide that it is better to not intercept the ball, but to focus on moredefensive tasks and, in doing so, call the “defensive behavior” and delegating
responsibility for action choice to it.

2Team-Internal Reflection in Regard to the Covid-19
Pandemic

Covid has been a predominant topic in society during the past two years. In thissection, we elaborate on the influence that the ongoing pandemic has had on our
robotic soccer simulation team.
2.1

Team Activity and Statistics

The Corona pandemic has had an enormous impact on our team’s efforts infurther developing our playing capabilities as well as on the implementation andintegration of learning approaches into our competition team. Unfortunately,
we have to recognize that we had to observe a severely declining interest (byuniversity students) in participating in our team’s activities as well as a declined
activity among active team members. There are a number of reasons as well
as a number of observable consequences that all more or less relate to changed
circumstances due to the ongoing Corona situation.
This situation is best reflected by Figure 2 which shows the activity in our
team’s Git repository during recent years. Here, we have split the data into a“Pre-Covid Period” (before Covid hit Germany in March 2020) and a “Covid Period” (since March 2020). Apparently, the (coding-related) activity has dropped
by more than 80% ever since (note that 2018 represents 100% in that statistic).
We are fully aware that quantity and quality are two different things when it
comes to implementing successful soccer simulation playing strategies. High activity in terms of many lines of code added or removed may not necessarily be aguarantuee for improved playing performance. However, a severe decrease in activity like the one shown here, is very likely to be corelated with less innovations
introduced into the code base.

Fig. 2. Activity in FRA-UNIted’s Main Git Repository: The activity in 2018 is referred
to as 100% in this visualization. With the appearance of Covid, the number of changes
to our code base has decreased significantly.

This number is actually in line with a few numbers of further team-related
observations: The number of team meetings (where team members meet, dis-cuss, brainstorm, and code in person) has decreased by approximately the sameamount under Covid. Likewise, in pre-Covid times (counted since 2018) there
were five Master or Bachelor theses submitted and successfully defended that did
relate to FRA-UNIted, to robotic soccer (simulation) and/or to machine learning
topics that bear a strong interconnection to our soccer simulation team. Since
Covid hit the planet, there were just two such theses. Also in line with these observations is the number of people attending our FRA-UNIted kick-off meeting
which we offer two times a year to novices and soccer/coding enthusiasts who
are generally interested in the domain of simulated soccer (a drop of about 70%
had to be recognized) and who, thus, might join our team in the course of thefollowing weeks and months.
A final observation is that our team’s ranking in the various soccer simulation tournaments that took place (in most cases virtually) ever since Covid
has appeared, did not drop at all, as shown by the following enumeration of
rankings. This allows us to conclude that all or at least many of the other teams
which are active in the realm of soccer simulation 2D have had to make similar
experiences.
– Pre-Covid: RoboCup 2018 (#5), RoboCup Asia Pacific 2018 (#3), RoboCup
2019 (#8)– Covid: (Virtual) RoboCup 2020 (#3), RoboCup Japan Open 2020 (#3),
RoboCup Iran Open 2020 (#3), RoboCup 2021 (#8), RoboCup Asia Pacific2021 (#3)
Conclusions We have acknowledged that virtual and pure online events on both,team-internal level as well as on the level of international competitions are not anadequate alternative for a domain where individual and voluntary engagement
are the driving factors for delving into the topic. Personal interaction with otherteam members and getting to know each other in real life, growing by joint work,
training camps or joint tournament participations, used to be driving factorsthat made joining FRA-UNIted an attractive option in pre-Covid times. With
the perspective of lone-working and no real-life RoboCup world championships
tournaments much of this attractivity has gone lost.
2.2

Teaching Experiment: Redevelop the Entire Play Without Ball

Given the difficult situation described in the previous section and knowing wellthat the effort for entering the soccer simulation domain is high, we performed
an experiment in which we made students of a Bachelor level project course(part of the study B.Sc. programme on Computer Science), who had no priorcontact to robotic soccer, redevelop an entire strategic (top-level) behavior, viz
the “no ball behavior” depicted in Figure 1.
The students were provided an intensive initial training on the basics of soccer
simulation 2D and on the FRA-UNIted code base (7 weeks) and were, subsequently, allowed to finish their task within up to 9 more weeks. Furthermore, they
were provided an attenuated version of FRA-UNIted, called FU Base, where thementioned no ball behavior was replaced by an 18-year-old predecessor version,
but where the remaining parts remained intact (world model, coach, goalie aswell as other behaviors on both, the strategic level and on the lower, skill-likelevels). Although just one behavior has been removed1 , this weakened version isof significantly reduced playing strength compared to our team’s full version (average result: 0.6:5.8) and even weaker than Agent2D [1] (average result: 2.1:4.6).
Two student teams (A and B) were formed, consisting of 5 and 6 students,
respectively. The performance of both teams improved in the course of the 9
weeks of time provided (cf. Figure 3 to see how time was allocated across weeks
by the teams), but failed to even reach the strength of FU Base as can be seenfrom the right part of Figure 3.

32
113
108

Inner:
Team A
84

113

Teaching Activities
Initial Training
Coding Noball

800

0

Total Noball Coding
Total Teaching Activities

600
400

679
540

200

160

252

0

Team ATeam B

Average Score Difference

42

Hours of Work (over one semester)

1000

Outer:
Team B−2−4

A vs. Agent2D
A vs. FRA-UNIted
A vs. FU_Base
B vs. Agent2D
B vs. FRA-UNIted
B vs. FU_Base
Average Team A
Average Team B
Joint Average (A and B)

−6

−8

−10

−12

After 3 Weeks

After 6 Weeks

After 9 Weeks

Fig. 3. Left: Total time spent (in hours) by a single student during the entire teachingcourse (averaged over team members). Middle: Coding time for the no ball behavior(total number of hours per team) plus extra times for presentations and meetings(teaching-related). Right: Performance of the resulting teams over time.

Conclusions FRA-UNIted has offered many similar programming projects aspart of the B.Sc./M.Sc. curricula in previous years. In most cases, these yielded
surprisingly well-performing results – however, in any of these prior issues thefocus was on some more specific task (e.g. on dribbling, penalty shoot-outs, thegoalie, or other tasks of similar task complexity) and, crucially, centered aroundsingle-agent development. Since the time spent for the initial training was identical then and now and since now the number of students per team was evendoubled (and, hence, total number of working hours alike, cf. middle part of Figure 3), we conclude that (a) for developing a strategic behavior like the no ballbehavior from scratch a substantial amount of experience is required, (b) thisholds even more so for progressing and optimizing it to a competitive level, (c)this depicts a challenge that can hardly be accomplished by novices with no priorexperience in soccer simulation 2D. We conjecture that the rise in complexity,
when (re)developing some central strategic behavior like the no ball one, mustbe tributed to the fact that it is inherently about multi-agent interaction, coordination and cooperation which is not just much harder to grasp for a machinelearning algorithm, but also for human students learning to program.
1

Though, admittedly an important one since the no ball behavior triggers and/or
implements sub-tasks like intercepting, offering, marking, dueling ball leading opponents, running free, running for passes etc.).

3

Diversified RoboCup Training Protocols on aDistributed Architecture

Training Protocols With the development and implementation of the continuous
integration environment, our RoboCup Team is constantly tested against theprevious world champion or against other teams every night [5]. This process
is producing statistics which are used to asses the performance of our team aswell as the impact of changes made to our teams code. To further build on
this approach, we are currently in the process of developing a training protocol
system, which allows for sophisticated control over the training procedure. A
training protocol is a server sided set of instructions which allows our team toplay against multiple different opponents at the same time, as well as allowing forearly cutoff thresholds which perform an action if, during training, a defined data
point reaches a stable, user defined value. Furthermore these protocols are meant
to be extendable with additional conditions or actions for the training process,such as automatically finding beneficiary values for team behavior parameters
based on the games played under the respective protocol. We investigate a change
to the already existing continuos integration procedure by letting the test PCs
pull the current training protocol from a web server each night, which contains
further instructions for the HLM2 config file, team binaries and match data.

Fig. 4. Utilized Training Workflow

Diagnostics Testing against different opponents has the advantage of potentially
producing a more detailed view on our team’s overall performance, which in thiscase is not only relating to the winning rate, but also various other metrics
such as stamina management, goals shot/received to mention just a few. These
diagnostic capabilities will help to prevent overfitting against only a single opponent. The idea is to also allow for head-to-head comparisons between multiple
opponents on a protocol basis.
First Results By expanding on the continuos integration with diverse opponents
and a training protocol system, we hope to further improve our diagnostic capabilities to gauge our team’s behavior and the impact of changes made. Thisapproach additionally enables us to implement an auto-train system, by adapting
certain protocol values based on incoming match data over a protocol’s run.
2

Hech League Manager by Andreas Hechenblaickner from the Austrian 2D soccer
simulation team KickOffTUG (2004–2010).

4Aggressive Defense Behavior Utilizing Low-Level
Opponent Action Predictions

We started to reimplement our one-on-one behavior, which was presented in2008 (aka NeuroHassle [7]). Due to several adjustments of opposing teams andchanges of the Soccer Server the original behavior became more and more inefficient over the years. For example, omni-directional movements of an agent are
not included in the old behavior. The new behavior should be able to efficiently
conquer the ball from a dribbling opponent moving towards our goal. This task isalso realized with methods of reinforcement learning [10]. In addition, the resultsof our approach on opponent low-level action prediction [4] using a Case-Based
Reasoning (CBR, [8]) approach are utilized to predict the opponent agent’s behavior. So, the state of the game in the next time step can be more accurately
predicted by our agent which enhances the capabilities of the RL algorithm tolearn an even more competitive behavior.
First, we modeled the problem as a Markov Decision Process (MDP). The
state of our agent is described as a nine-dimensional vector. Therefore, the centerof the coordinate system was moved to the opponent’s player and rotated by its
orientation. The state is described by the– distance d between our player and the opponent ball leading player,– velocity (vx and vy component) of our player,– absolute value vo pp of the opponent’s velocity,– position (bx and by component) of the ball,– our player’s body angle α relative to the opponent’s position,– opponent player’s body angle β relative to his direction towards our goal,– value of the strategic angle γ = ∠GOM with G as position of our goal, O
as position of the opponent, and M as the position of our player.
Due the high dimensionality of the problem space we use a neural network toapproximate our value function. The agent is allowed to use dash(power, α) andturn(α) commands with power ∈ [0, 10, . . . , 100] and α ∈ [−180, −165, . . . , 180].
The agent is rewarded when the ball comes into its kickable area or when theirprobability of executing a successful tackling exceeds 0.75. Additionally, theagent obtains a slight punishment for each time step it needs to reach the ball.
This way we enforce learning a time-optimal behavior. The possible startingstates for the training are defined in a similar manner as in [7]. In each episode,both players are starting with a random body angle and velocity. The opponent
player is set in a circle around our player and has the ball in its kick area.
First Results At this stage, the algorithm is tested against the dribbling behavior currently used by our own team, which dribbles with the ball towards a given
target point and which was learned using a reinforcement learning approach aswell (Section 2.2 in [2]). When evaluating our preliminary results, we observedthat the new one-on-one behavior shows an improvement in ball conquering ofapproximately 7% compared to the old behavior. Of course, training and learning against other top teams from recent RoboCup tournaments is one of thenext steps to be taken. Final and stable results for learning this aggressive defense behavior in conjunction with an exploited reliable case-based prediction of
opponent low-level actions are expected in the second quarter of this year andwill hopefully be deployed during the upcoming RoboCup 2022 World Championships tournament in Bangkok (Thailand).

5

Conclusion

In this team description paper we have outlined the characteristics of the FRAUNIted team participating in RoboCup’s 2D Soccer Simulation League. We havestressed that this team is a continuation of the former Brainstormers project,pursuing similar and extended goals in research and development as well as forteaching purposes. Specifically, we have put emphasis on our most recent research
activities and practical implementation of our results.

References
1. Akiyama, H.: Agent2D Base Code. https://osdn.net/projects/rctools/releases/(2012)
2. Gabel, T., Breuer, S., Roser, C., Berneburg, R., Godehardt, E.: FRA-UNIted Team Description 2017 (2017), Supplementary material to RoboCup 2017: Robot
Soccer World Cup XXI
3. Gabel, T., Breuer, S., Sommer, F., Godehardt, E.: FRA-UNIted - Team Description
2019. In: RoboCup 2019: Robot Soccer World Cup XXIII, LNCS (CD Supplement).
Springer, Sydney, Australia (2019)
4. Gabel, T., Godehardt, E.: I Know What You’re Doing: A Case Study on CaseBased Opponent Modeling and Low-Level Action Prediction. In: Workshop on
Case-Based Agents at the International Conference on Case-Based Reasoning(ICCBR-CBA 2015). pp. 13–22. Frankfurt (2015)
5. Gabel, T., Kloeppner, P., Eren, Y., Sommer, F., Breuer, S., Godehardt, E.: FRAUNIted - Team Description 2021. In: RoboCup 2021: Robot Soccer World Cup
XXV, LNCS (CD Supplement). Springer (2021)
6. Gabel, T., Riedmiller, M.: Brainstormers 2D - Team Description 2009. In: L. Iocchi,
H. Matsubara, A. Weitzenfeld, C. Zhou, editors, RoboCup 2009: Robot Soccer
World Cup XII, LNCS (CD Supplement). Springer, Graz, Austria (2009)
7. Gabel, T., Riedmiller, M., Trost, F.: A Case Study on Improving Defense Behavior
in Soccer Simulation 2D: The NeuroHassle Approach. In: L. Iocchi, H. Matsubara,
A. Weitzenfeld, C. Zhou, editors, RoboCup 2008: Robot Soccer World Cup XII,
LNCS. pp. 61–72. Springer, Suzhou, China (2008)
8. Richter, M.: Case-Based Reasoning. A Textbook. Springer, Heidelberg, Germany(2013)
9. Riedmiller, M., Gabel, T.: Brainstormers 2D - Team Description 2007. In: U. Visser,
F. Ribeiro, T. Ohashi and F. Dellaert, editors, RoboCup 2007: Robot Soccer World
Cup XI, LNCS (CD Supplement). Springer, Atlanta, USA (2007)
10. Sutton, R.S., Barto, A.G.: Reinforcement Learning. An Introduction. MIT Press/A
Bradford Book, Cambridge, USA (1998)