Alice2021: Team Description Paper
Fan Wang, YuTang Guo,XingQi Zheng and XinYu Liu
Robotics and Innovation Technology Lab
School of Computer and Information
HeiFei Normal University,P.R.China
Miracle.hftc@gmail.com

Abstract. This team description paper mainly explains the work of Alice2D atthis stage.According to the characteristics of our team, based on the past research,
our team is committed to studying the differences of players' roles andreinforcement learning algorithm,changes to the evaluator,formation optimization.
After many tests, we continuously optimize the team code, which has greatly
improved the performance of the court.
Keywords: Player role differences, Monte Carlo tree algorithm, Decision tree

1 Introduction
Alice2D was founded in 2016, by the Hefei Normal University Student Innovation
Laboratory of a group of robot-loving soccer students. This is the fifth time we apply
for the chance to participate in RoboCup 2D competition. Our team code is based onAgent2D3.1.0[1] which was released by Hidehisa Akiyama. It also includes fragmentsof released code of Helios2008[2]. We sincerely appreciate the assistance fromHidehisa Akiyama and all of the related open source developers. In the 2017 WorldTournament, the team won the seventh place in the regular season and won thechampionship in the challenge, 7th place in the 2018 World Cup regular season.In
China, our team is also named Miracle, winning the third place in China division ofthe 2019 RoboCup World Cup and the third place in China division of the 2021
RoboCup World Cup.

2 Focus of team science
On the basis of manual strategy research, our team made improvement, planned forthe role of agents in the course, and adopted artificial intelligence methods, namelyreinforcement learning and decision tree learning. In the process of reinforcement
learning, by constantly trying actions and observing the rewards of actions, the agent
gradually chooses the beneficial actions in various situations, so as to obtain a highcumulative reward value in the process of interacting with the environment.

3 Player role differences
Robot Soccer players have different positions, including striker, center forward,
midfielder, defender and goalkeeper. Different players have different strategies. For
example, strikers tend to attack, midfielders tend to organize, and defenders tend todefend. Due to time constraints, our team only made improvements on forwards anddefenders, and built different models according to different strategies and movements.
The following are introductions based on player roles.
Forward Player When the striker in the chart on the left side of the position, thegoalkeeper is located in near the goal area, then shot range should be in the range ofthe two red and blue line, the shooting range larger, shooting success rate is high, butif the striker missed shot time, direction along the dotted line to the right, the othergoalkeeper will choose attack, narrowing the scope of a shot. At this time, theshooting range is greatly reduced compared to the Angle of the shot just now, and thesuccess rate is greatly reduced. Therefore, the success rate of the shot will change withthe timing of the shot[3].

Figure 1:Before moving

Figure 2:After moving

In RoboCup2D ， shoot_success(dist，α，p，ε)is[4]：(2-1)α indicates the perspective from which a forward player can shoot，dist indicatesthe shortest distance to shoot ， p indicates the ability value of a forward player ， εindicates the interference factor of the goalkeeper[5]。

(2-2)
According to (2-1) and (2-2), players should try to choose left or right side toattack the goal, and try to avoid attacking the goal (a) in the area directly facing thegoal. This method can be used for reference. On this basis, we optimized the shooting
strategy based on this point. While choosing the left or right side to attack the goal,
we circled with the opposing player to find the best shooting point (b).Practice with100 shots：
Table 1. Shooting training results

Before and after improvement

goals

Percentage of goals scoredbefore improvement4949%

after improvement(a)6565%

after improvement(b)

71

71%

Defender player The quality of defense is the key to victory. The intercept plays animportant role in the defense process. Whether the intercept is successful or not,players have to stop on the floor for several cycles to achieve the actual needs, so thestrategy of intercept is bound to affect the cooperation of the whole team. At this point
you need to change the probability of the tackle in the hope that the tackle will be lesscostly.The traditional player execution strategy is based on the decision tree to selectthe action[7,8]. The execution process of the decision tree is shown in the following
figure:Figure 3:single agent decision-tree

Our team's defense is based on possession and off-possession strategies. Focuses on
producing many different actions for the chain action model, the holder uses theevaluation function to assign different chains to operate on a point, and then selects
the highest point of the chain of operations to be executed.

4 Reinforcement learning algorithm-Monte Carlo algorithm
Our Alice team has studied the combination of Monte Carlo algorithm and action
chain in the past. Monte Carlo algorithm is similar to strategy iteration in dynamic
programming, which also includes strategy iteration and strategy optimization[6].
Monte Carlo algorithm is suitable for learning problems in unknown environment.
Agent2d basic code generates two depth chain action, the basic model of evaluating
the code only consider the first step in the operation, lead to can't choose the optimal
solution, so we change the basic code evaluation model, based on the original to form
the best action chain reaction, we added a chain action depth search depth andevaluation model[9]. The creativity of the algorithm is to follow up on the originalbasis, improve the decision tree for the position of the distance from the goal, such as
outside the penalty area, inside the penalty area, and optimize the movement chain.

Figure 4:Best chain action is our player no.11 pass ball to no.10 and no.10 then pass ball tono.9.Then no.9 can shoot. Because of pass can be generated as the second action in chainact-ion and use Monte carlo tree search to find best

5 Changes to the evaluator
Our team modified the evaluator, including the modification of the goal area, themodification of the actions executed in the previous period and the evaluation of theplayer types. The evaluator was appropriately modified in combination with theformation of our team in order to expect the optimal actions to be executed.

6 Formation optimization
Analyze the RCG file[10] to obtain the data file containing the coordinates of eachplayer in each period. The two-dimensional coordinate source data needed for theinitial experiment can be obtained, and then the source data are mined and analyzed.
The server has been updated, which causes a deviation between the previous server
version and the existing server version. Fix formation parameters, debug and change.

7 Summary and Outlook
Over the course of the year, our team made improvements in combination withformations and evaluators, continued to refine the decision tree for Monte Carloalgorithms, and combined strategy modifications for on-court player types. Today,our team is still exist many problems need to solve, after last time communication
with yushan team learning, inspire the study of digital twin framework[11], but ourteam merely in theory.Our team also attempts and manages to implement a morestable multi-layer action chain based on the relevant principles in TDP ofHELIOS2012, and to optimize the state-space search by means of the addition of
direct pass, short dribble and other actions between the two layers[12].
In the future, our team will continue to scripting language, data mining, data, deeplearning log analysis, etc. On the basis of manual strategy, high level action decision
making is studied.
The RoboCup is a good opportunity to learn from other teams, we will learn thestrengths of other teams, improving our weaknesses. Wish the RoboCup simulation
2D soccer is getting better and better.

References

1. Akiyama,H.:Agent2D Base Code.http://www.rctools.sourceforge.jp.2015
2. Hidehisa Akiyama, Tomoharu Nakashima, HELIOS Base: An Open Source
PackagefortheRoboCupSoccer
2DSimulation.
http://pumath.dl.osdn.jp/rctools/32271
3. Hui Zhou.Research on robocup2d simulation
collaboration[D].Nanjing University ,2013.

footballteamagent

4. Potter M.A,De J.K.Cooperative co-evolution:an architecture for evolving
coadapted subcomponents[J].Evolution Computation,2000,8(1):10-29.
5. Parker,Gary B,Blumenthal J.Punctuated Anytime Learning For Evolving ATeam[C].World Automation 2002Congress Proceedings,2002.

6. Shiyang chen.Research on Robocup_2d High-level Tackling Strategy Based onRein forcement Learning[D]. Anhui University,2014
7. ZHAO Fajun, LI Longshu. Robo Cup regional cooperative strategy based on
multi-Agent Q-learning. Computer Engineering and Applications, 2014, 50（23）：127-130.
8. Xiaofeng Gu,Daiyuan Zhang.Design of catching strategy for robot soccer
game[J]. Computer Applications, 2005, 25(8): 1858-1859.
9. Deng Yong,Xiao Zhou,Siying Chen,Zhengxi Gan,Alice Robocup Simulation 2D
Team Description,ROBOCUP2017 SOCCER SIMULATION 2D LEAGUE,2017
10.YAMAMOTO A,OKA N,Noda I.Imitative learning from the logsof a soccer
simulator[C].The 19th Annual Conference of theJapanese Society for Artificial
Intelligence
11.Zekai Cheng,Ningyu Xie,Sichun Yang,Xingxing She.Application of digital twinframework based on Robocup simulation 2d[J].Journal of Suzhou Universityof Science and Technology,2019
12.H.Akiyama,
H.Shimora,
T.Nakashima,
Y.Narimoto,
K.Yamashita,
HELIOS2012 Team Description Paper, The 16th annual RoboCup
International Symposium, Mexico, 2012.