Explicitly Maintaining Diverse Playing Styles in Self-Play
Authors: Yuan Liu
Ruimin Shen
Miqing Li
Yingfeng Chen
Juan Zou
Changjie Fan
Venue: Submitted to ICLR 2023 (2023)
Keywords: Reinforcement learning
evolutionary algorithm
self-play
diverse playing styles
high skill levels

Abstract: Self-play has proven to be an effective training schema to obtain a high-level agent in complex games through iteratively playing against an opponent from its historical versions. However, its training process may prevent it from generating a well-generalised policy since the trained agent rarely encounters diversely-behaving opponents along its own historical path. In this paper, we aim to improve the generalisation of the policy by maintaining a population of agents with diverse playing styles and high skill levels throughout the training process. Specifically, we propose a bi-objective optimisation model to simultaneously optimise the agents' skill level and playing style. A feature of this model is that we do not regard the skill level and playing style as two objectives to maximise directly since they are not equally important (i.e., agents with diverse playing styles but low skill levels are meaningless). Instead, we create a meta bi-objective model to enable high-level agents with diverse playing styles more likely to be incomparable (i.e. Pareto non-dominated), thereby playing against each other through the training process. We then present an evolutionary algorithm working with the proposed model. Experiments in a classic table tennis game Pong and a commercial role-playing game Justice Online show that our algorithm can learn a well generalised policy and at the same time is able to provide a set of high-level policies with various playing styles.

Under review as a conference paper at ICLR 2023
EXPLICITLY MAINTAINING DIVERSE PLAYING STYLES IN SELF-PLAY
Anonymous authors Paper under double-blind review
ABSTRACT
Self-play has proven to be an effective training schema to obtain a high-level agent in complex games through iteratively playing against an opponent from its historical versions. However, its training process may prevent from generating a well- -ederaeeerll policy since the trained agent rarely encounters diversely-behave opponents along its own historical path. In this paper, we aim to improve the generalisation of the policy by maintaining a population of agents with diverse playing styles and high skill levels throughout the training process. Specifically, we propose bi-objective optimisation model to simultaneously optimise the agents' skill level and playing style. A feature of this model is that we do not regard the skill level and playing style two objectives maximise directly since they are not equally important (i.c., agents with diverse playing styles but low skill levels are meaningless). Instead, we create meta bi-objective model to enable high-level agents with diverse playing styles more likely to be incomparable (i.e. Pareto non-dominated), thereby playing against each other through the training process. We then present evolutionary algorithm working with the proposed model. Experiments in a classic table tennis game Pong and a commercial role- playing game Justice Online show that our algorithm can learn well generalised policy and at the same time is able to provide a set of high-level policies with various playing styles.
INTRODUCTION
Recent years have witnessed impressive results of self-play for Deep Reinforcement Learning (DRL) in sophisticated game environments such as various board (Silver et al. 2016 2017a| Jiang et al., 2019) and video games (Jaderberg et al. 2019 Vinyals et | Berner et al. 2019). The idea behind self-play is using randomly initialised DRL agent to bootstrap itself to high level intelligene by iteratively playing against an opponent from its historical versions (Silver et al. 2016). However, the training process of self-play may prevent it from obtaining well-generalised policy since the trained agent rarely encounters diversely-be bberrsilybe opponents along its own historical path. This easily be taken advantage of by human players. Taking OpenAl Five (Berner et al. 2019) as an example, it has a win rate of 99.4% in more than 7000 Dota 2 open matches but the replay shows that 8 of the top 9 teams that defeated the OpenAl Five are the same team and the policies they use in each game are very similar. This indicates that despite the remarkable high performance, the OpenAl Five agent still not fully comfortable in some circumstances, which can be found and further exploited by human players (e.g., through meta-policies).
Generally, an agent can be identified from two aspects, skill levels and playing styles (Mouret Clune| 2015). These two aspects are crucial for learning a high-level agent in the self-play training process because only playing against opponents with diverse playing styles and appropriate skill levels (i.e., not too low) can maximise the gains of the learning. If one only considers opponents' skill levels, there can be a catastrophic forgetting problem which the agent "forgets" how to play against wide variety of opponents (Hermandez et al. 2019). On the other hand, if one only considers playing styles, there will be a lot of meaningless games that the agent Iearns very little from its far inferior opponents (Laterre al. 2018)
https://arena.opent cireenai.cco/t reeu/essuullt ts

---

Under review as a conference paper at ICLR 2023
Unfortunately, is intrinsically challenging to strike good balance between skill levels and playing styles in self-play. During the training process, the network weights of the DRL agent are usually optimised by a gradient-based method, which progresses along single path that relies the random seed of the network and the environment. At each iteration, the incumbent agent is the only response-police for its historical versions. Such single-path optimisation is very unlikely to experience sufficiently diverse opponents, especially within a sophisticated environment. This may make common self-play algorithms, which use probability function to decide which opponents to consider (e.g., the latest opponent or the past versions (Berner et al. 2019 Oh et al. (2019), unable to generalise their policies, i.e., struggle to cope with opponents that are very different from which they have encountered before.
viable way to introduce diverse playing styles in self-play is to consider the population- approach, where population of agents/opponents are maintained during the training process, with each potentially representing one play style. The population-bane baaedd approach has already been
frequently used in DRL (Jung al. 2020 Carroll et al. 2019 Parker-Holder et al. 2020 Zhao ct al.
202T). For example, Population- -aaseattoo---edddd Training (PBT) (Jaderberg et al. 2017 2019 Li et al.
2019
Liu et al., 2019) optimises population of networks at the same time, allowing for the optimal
hyperparameters and model to be quickly found. Neuroevolution (Heidrich - & Igell 2009)
|Scc et al. 2017: Salimans et al.] 2017. Staanlyy et al., 2019) uses population-basse evolutionary search (e.g., genetic algorithm and evolution strategy) to generate the agents' network parameters
and topology.
In these population-based methods, an interesting idea to promote diversity of the agents' behaviours is to proactively search for "novel" behaviours. This can be very useful since maintaining population of behaviours does not necessarily mean diversifying them over the search space (Jaderberg el al.) 2019).
This is particularly true in sparse/deceptive reward problems (Salimans al.) 2017 Conti et al. 2018)
where the reward function may provide useless/misleading feedback leading to the agent to get stuck and fail learn properly (Lehman & Stanley 2011]] Fcoffett et al.) 2019). Such proactive-novelty
search techniques include novelty search (Conti et al. 2018] Lehman & Stanley| (00TT) intrinsic motivation (Bellemare et al. 2016). count-based exploration (Ostrovski et 2017 Tang et al. (017),
variational information maximisation (Houthoofi et al. 2016), curiosity drivven learning (Baranes) & Oudeyer 2013 Forestier et al. 2017), multi behaviour search (Mouret & Doncieux, 2009 |Shen et al.) (020) and quality- diversity (Cully & Demiris 2018). They, based on the history information in the environment, motivate the agent to visit unexplored states in order to accumulate higher rewards (Conti et al. 2018 [coffet et al., 2019 Guo & Brunskill 2019). For example, the quality- diversity algorithms use domain dependent behaviour characterisations to abstractly describe the agent's behaviour trajectory and encourage the agent to uncover as many diverse behaviour niches as possible, with each niche being represented by its highest-level agent (Mouret & Clune. 2015) Pugh et al., 2016). However, such proactive novelty search may not always be promising since novel behaviours that we search for do not always come with high skill levels. When it comes to population-based self-play, a game could be meaningless when the difference of agents' skill levels is too big, albeit their playing styles being very different. Indeed, what we need effectively is population of ligh and diverse-style agents which play against each other through the training process.
To this end, this paper proposes a novel Bi-Objective (BiO) optimisation model to optimise skill levels and playing styles. One feature of this model is that we do not regard these two aspects as objectives to maximise directly, but rather we create a meta bi-objective model to enable high- level agents with diverse playing styles more likely incomparable (i.e. Pareto nondominated to each other), thus being always kept in the training process. Specifically, in BiO each objective is composed of two components. The first component is related to skill level of the agent, same for the two objectives, while the second component is related to playing style of the agent, we making it completely conflicting for the two objectives. As such, the Pareto optimal solutions in BiO are typically those far away from each other in playing styles but all with reasonably good skill levels (this will be explained in details Section 3).
We propose an evolutionary algorithm to work with the proposed model. We follow the basic framework of multi-objective evolutionary algorithms, but with customized components for self-play.
2

---

Under review as a conference paper at ICLR 2023
We evaluate our algorithm in classic table tennis game Pong and commercial online role- role-playing game Justice Online
It is worth mentioning that the problem here we are dealing with is different from multiobjective ciimmissatio-relltte RL (Yang et al., 2019 Liu & Qian 2021 Xue et al. 2022). such as those in
the Multi- Multi-OOjecttve Reinforcement Learning (MORL) (Moffert & Nowé, 2014 Mossalam et al. 2016). MORL is generalisation of standard reinforcement learning where the scalar reward signal extended to multiple feedback signals, whereas our problem is simultaneously optimise the skill levels and playing styles of the players.
PRELIMINARIES AND RELATED WORK
Each iteration of self-play can be considered as a Multi-agent Markov Decision Process defined by a } is the set of N >> 1 agents, S is the state space observed by all the agents, and A'' is the action space of agent i. Let A == A x x to any state s' S for any joint action a E A. r" S x A x S + R is the reward function that determines the immediate reward received by agent for transition from (s, a) to s 7 E (0, 1] is the discount factor. At step t, each agent E / executes an action aj, according to the system state st. The system then transitions to state St+1, and rewards each agent by (st, ae: St+1). The goal for agent is maximise its own long-term reward (T( E,z [Σ==o 7²1²² (8ts at 8e+ 1++1). })]. by finding the policy of n 7-((/e)).
tuple (N,S, {AA TeEM heW P, {r") hev, 7), where N = [1, {1,..... .
then : x A x S > [0. 1] is the transition probability from any state
PROXIMAL POLICY OPTIMISATION (PPO). PPO (Schulman et al. 2017) is a popular deep policy gradient method where policy updates are computed by surrogate objective regularised by the clipped probability ratios. Inspired by trust region method, the algorithm updates the policy within a close neighhourhood around the previous-iteration policy each time to guarantee monotonic performance improvement. As shown in a clipped surrogate objective CLIP (T):
Ta 8) Toma (8, a,,a),lipp (a s)
T(a 8) Tata (s) -e,,,, + (s,
CLLP (T)
min
where clip(.) removes the incentive of the probability ratio s(als)
[ala(a]a] outside the interval - E, +
(e = 0.2). PPO can sample data from the stable previous-iteration policy Wolds and incrementally refines the policy using multiple steps of stochastic gradient ascent before sampling new data.
MULTI- OBLTI--VBEEIIVEE OPTIMIZATION. Multi-objective optimisation (Matthias 2006) is an optimisa- tion scenario that considers multiple objectives/criteria simultaneously. Without loss of generality, let us consider a maximisation scenario. Formally, a multi-objective optimisation problem can be expressed as:
maximise F(z) (z). fz))""))
(2)
where m is the number of objectives. In the context of multi-objective optimisation, solution æ1 said better than Z2, if and only if x1 is not worse than 22 for all the objectives and better for at least one objective. We call this "better" relation as Pareto dominance, i.e., (Pareto) dominates z2- Two solutions being incomparable means that they are non-dominated to each other. For solution E X, if there is no solution in the solution set X dominating x, then is called Pareto optimal solution in X.
RELATED WORK. Diversifying playing styles of opponents that the agent encounters in the training process is an important topic in self-play. Interesting attempts include using diverse expert data as opponents to enrich the agent's experiences (Silver et 2016, Vinyals
2019 Lowe al.
2020). learning diverse sub-policies and combining them into an ensemble model (Xu et al. 2018). 8). using reward shaping create a series of different playing styles (Oh et al. 2019) and exploring unseen playing styles by domain randomisation (Jaderberg et al. 2019 Berner et al. 2019). However, striking a good balance between playing styles and skill levels a challenging task; some of these methods need extra resources (e.g. expert data (Silver et al.) 2017b)) human hand engincering
https:: htmpss://mmo..cmm ctt. httpm/mm...mmmmee////////tttttt
3

---

Under review as a conference paper at ICLR 2023
(e.g. manual tuning of the reward weights (Oh al. 2019)), and some may degrade in sophisticated environments (Xu et al. 2018).
Another way to maintain diverse opponents in self-play is to consider the population-based training (PBT) (Jaderberg et al. 2019| Vinyals et al. 2019: Hernandez et al.) 2019), where the playing style of each agent in the population is controlled by its own hyperparameterss and reward weights. In such methods, PBT is to meta-optimise the internal rewards and hyperparameters of the agents. Since each agent in the population learns from the experience generated by playing against opponents sampled from the population, more generalised policies can be learned (Bansal et al. 2018; ZZhao et al.) 2021). However, maintaining an agent population does not necessarily mean maintaining diverse playing styles; thus it would be beneficial for this model to proactively explore novel playing styles and then properly maintain them during the training process. We will compare our model with PBT in the experimental studies.
is necessary to note that multi-objective optimisation models, despite not designed for self-play, have been used to diversify agents' behaviours in games (Agapitos et al. 2008 Mouret & Doncieux [009][Shen et al. 2020; Pierrot et al. 2022b). They created one or several behaviour-related objectives for the algorithm to optimise directly. For example, the novelty of the agents' behaviour is considered as an auxiliary objective in Mouret & Doncieux (2009), and opposite behaviours (i.c., aggressive and defensive) of the agents such models is that an agent with a different behaviour from the rest will always be considered Pareto optimal matter how poor its skill level is (since other agent performs better in the corresponding behaviour objective). This may easily lead to the population to have diverse playing-style agents, but their skill levels can be highly variable. We will compare our model with one recent representative of such models in the experimental studies.
considered as two objectives in [hhnn et al. (2020). One main issue of
Lastly, it is worth mentioning that some recent studies formalised skills and styles with differentiable functions and leveraged gradient information to efficiently improve the skill level of agents from different stylistic directions (Fontaine & Nikolaidis [202] Pierrot et al. [022a Tjanaka et al.) 2022). although they are not implemented in the self-play paradigm.
PROPOSED MODEL
Our bi-objective (BiO) optimisation model is designed to optimise skill levels of the agents and at the same time to diversify their playing styles. To do so, each objective is constructed to consist of two components. The first is the agent's skill level, which is shared by the two objectives. The second component is concerned with the agent's 's playing style, which we design to be opposite in the two objectives in order to make diverse style agents likely incomparable (thus as the Pareto optimal agents kept during the search). Formally, the BiO model for an agent with the skill level g(T) and the playing style (TT) is expressed as follows:
maximise [f((TT) = =(( + Kh(TT)
(f2(T) = -x)))
where k is coefficient to make g(x) and ((T)) commensurable. g( is an indicator reflect the agent's 's skill level, and it can be represented by win rate (Chen et al. 2018), Elo score (Jaderberg al., 22019). etc. h(*)) is an indicator to reflect the agent's playing style, and its calculation will be
explained later. The difference between two agents in h(T)) indicates how differently they behave, e.g., aggressively versus defensively.
Figure [ gives an example to illustrate the proposed model, where Figure [a) presents six agents in the original skill-style space and Figure [bb) presents them in the proposed bi-objective space. To help understand the characteristics of the BiO, we provide the following remarks, which can be
derived from the figure as well as Equation 3
A higher skill-level agent will not be dominated by a lower level one in the proposed model. This can be derived from Equation their playing styles l((T)) ) and (TT)) are, л2 will not be better than क on both objectives Si and f2; in the best case for π2, they are non-dominated to each other (e.g., the agents A and B in Fi.
if g higher for the agent n than π2, then whatever
Similar playing style agents tend to be comparable (i.e., dominating or being dominated) even if their skill levels are close. For example, in Figure the agent E's playing style
4

---

Under review as a conference paper at ICLR 2023
18
Skill styles
0.8
Skill levels
0.6
0.6
0.2
0.2
D
-1.2
Defersive
0.2
03.6
0..0
01.5
3.0
0.6
1.0
12
1.4
1.6
Playing styles
Playing styles
(a) The original skill-style space
(b) The proposed BiO space
Figure I: An illustration that the proposed bi-objective model makes diverse style agents with fairly good skill level more likely to be Pareto optimal (i.e., A, B, E). As implemented in this paper, the skill level and playing style are normalized and the coefficient the proposed BiO space where the two meta-ohjectives agents and C have the same playing style, while B has a higher skill level than C. The agents and have similar styles, while E has a higher skill level than F. A' A's style is dissimilar from the others' and the same for D. while A's 's skill level is significantly better than D's. Therefore, if one would like to choose three diverse style agents with good skill levels an opponent population for agent play against, the agents A. B and E be the best choices. This, as shown in Figure [b), is in line with the proposed model, where these three agents are Parcto optimal the bi-objective space.
1. (a) The six agents in the skill-style space. (b) The six agents in
to be maximised. As be seen in Figure Ma), a), the
is similar to F's and their skill levels are fairly close, but E dominates F in the proposed model. In the special case that two agents have the same style, the higher skill level agent always dominates the lower one (e.g. the agents B and C in the figure).
Dissimilar playing style agents tend to be incomparable (i.e., non-dominated to each other) even if one is fairly inferior to the other in skill level. For example, the agent A in Figure] [ which has relatively low skill level, is not dominated by any agent since its style is rather different from the others'. In fact, it can be derived from Equation that the probability of two agents being incomparable increases linearly with the distance of their playing styles.
The coefficient x in the model is a critical parameter which weighs up between agents' skill level and playing style. In our implementation,
simply set 1 after normalising them (i.e. after making (TT) E [o, 1], ())) (r) E [o, 1). is worth mentioning that a different k setting may potentially be more suitable for specific problem.
To sum up, the proposed BiO model enables different style agents with good skill level to be ranked high. As can be seen in Figure [ if one would like to choose three of the six agents as the population of opponents for an agent to play against, then the agents A, B and E will be chosen (since they are Pareto optimal). B and E are chosen because they have higher skill levels than their similar agents (i.e.. C and F, respectively). A is chosen because A is far away from the others in playing style, whereas is not chosen because despite being far away from the others, but its skill level is significantly worse than them too. In fact, the model can be seen to identify representative good agents, but without a need of setting a niche. As such, one can maintain population of high-level agents with diverse playing styles who keep playing against each other throughout the training process.
In BiO, the playing style f(TT) of an agent can be estimated by playing against all the opponents in the pool. Formally, it can be calculated as
K Σ k=1
T Σ t=1
1
h(T))
var (Se, Sc-1)
K
where K is the size of the opponent pool, Tk is the length of the trajectory produced by playing against the opponent Tg, and var (St, $1+1) denotes the relevant state change (e.g. the position change of the agent т) between time steps t and + 1.
Based on the gameplay of games, the calculation of the state change function can generally be classified into two categories, opponent-independdett mode and opponent-dependent mode. In the
5

---

Under review as a conference paper at ICLR 2023
opponent-independen mode, the function can directly be estimated by the change of the agent r's own states. This mode is for games where the players have their own venues, for example the games Pong. Tennis and Blobby Volley. In such games, the movement frequency of the agent и is the main way to reflect its playing style aggressive players frequently change their positions whereas conservative players tend to move slowly and steadily.
In the opponent-dependent mode, the state change function is estimated by not only the agent r's states but the opponent TT Ts That is, the opponent's states are used as reference to estimate the agent's playing style. This mode is for games where the agent and its opponent share the same venue, for example the battle games Justice Online, B&S Arena Battle (Oh et al., 2019) and Toribash (Kanervisto & Hautamāki (019). In such games, the agent's 's playing style needs to be measured by the difference between the agent's movement trajectory and its opponent's movement trajectory aggressive players like close combat, thus the difference typically small, whereas defensive players prefer to move around and tend to stay certain distance from their opponents, thus the difference typically large.
4 OPTIMISATION
In this section, we present an evolutionary algorithm working with the Proximal Policy Optimisation (PPO) (Schulman et al. 2017) to optimise the proposed bi-objective model. Specifically, we use the framework of the classic multi-bbjective evolutionary algorithm NSGA-II (Deb et al.) 2002). but with customized components for self-play DRL, e.g., an evaluation population "frozen" for while allowing the agents play against.
Algorithm gives the main procedure. The procedure starts by initialising the evolutionary population with a set of agents defined by ran- dom neural networks (0) with reward weights (w). Other hyperparameters of the agents (e.g. neural architecture, discount factor and learning rate) are set manually and can be found in Ap- pendix C. Step in the algorithm is to estimate the skill level and playing style of the evolution- ary population. Each of its agents plays against all the agents of the evaluation population and obtains the average skill level and playing style.
Algorithm 1: Algorithm to solve the bi- objective model
Input:
Evolutionary population
P t- (f),w.)..... ..... wx): Evaluation
population E + P; Generation gen +- 0
P Evaluation (, E);
P + Fitness Assiggmentt))))))(()))
while termination condition not met do
P' + V Variation(() P);
foreach (8', u,) E do
0' - PPO(( w P):
Then the agents' fitness (i.e. objective functions) of the proposed BiO model is calculated in Step
p' + Ecaluation E); P' + Fitness Asstgnenss)i
After that, the following steps repeat for each generation of the evolutionary algorithm. Step is to update the population by varying the agents' reward weights. Here, adopt two basic real-valued variation operators, simulated binary crossover and polynomial mutation (Deb 14
P + EnaironmentalSeiectinnn P. P');
gen + gen +
if gen% = then
E P.
P Evaluation PP E);
P < Fitness AsinnssAssimmmeen((P
et al. 2002) (details can be seen in Appendix B). After new agents generated, we use PPO train their network parameters (Steps 5 and 6), where each new agent randomly picks one agent in the parental population to play against and this repeats few times. Then, the new agent population are evaluated against the evaluation population (Step 7) and their objective functions of the BiO model are calculated (Step 8). Next, based on the objective functions, the environmental selection is performed by using the nondominated sorting and crowding distance in NSGA-II (Deb et al. 2002) (details are given in Appendix B) to select the K best solutions from the parental and new populations as the next-generation population (Step 9).
15 return (0, w) which has the highest skill level in
Steps 11 to 14 are update the evaluation population after while (i.e. every freq generations). Introducing an evaluation population that lets the current population play against for estimating the skill level and playing style plays an important role. Fixing the opponents of the agents from different generations enables their fitness comparable; playing against different opponents can easily produce different skill level and playing style even for the same agent. Note that the update frequency

---

Under review as a conference paper at ICLR 2023
parameter freq cannot be set very large/small. A large value may make the skill level of the evaluation population far lower than the current agents', while a very small value may make the fitness of the agents unstable during the evolutionary process.
Finally, after the procedure terminates, the returned agent is the one that has the highest skill in the evolutionary population (Step 15).
5 EXPERIMENTS
We evaluate our model on the Atari game Pong and the commercial game Justice Online. To implement the bi-objective model, the skill level in Pong is estimated by the difference of the final score between the agent and its opponent, and the playing style is done by the move of the agent's paddle. In Justice Online, the skill level is estimated by the difference in the final health point between the agent and its opponent, and the playing style by their average distance in the game. Details about the two environments and their BiO implementations are given in Appendix A.
We consider four competitors to compare with our proposed model: they are PPO (Schulman it al. 2017). DRL strategy based on Population- Based Training (Jaderberg et al. (017)) (denoted by PBT for simplicity). DRL strategy considering multiple rewards (Oh et al. 2019) (denoted
by multi-reward), and a DRL. strategy based on evolutionary multi-objective optimisation (called EMOGI) (Shen et al. 2020). Note that all of these competitors except multi-reward were not designed specifically for self-play. here you used their self-play versions, where the agent is trained by playing against an opponent from its historical versions. PPO is a baseline DRL method, upon which the other algorithms (including ours) based. To alleviate the catastrophic forgetting problem, in PPO we follow the practice in (Berner et al.] (019) to let the agent to play against the latest version with probability of 80% and against past versions with probability of 20%. The multi-reward competitor is PPO working with a multi-reward strategy (Oh et al.) 2019) where multiple rewards are used in training to shape different playing styles (aggressive, balanced and defensive) for generalisation. The PBT competitor is PPO working under the population- based learning environment (Jaderberg al., 2017), where each PPO style agent of the population can exploit information from the rest of the population. Like the proposed BiO, EMOGI (Shen et al., 2020) combines PPO with the evolutionary nulti-obbecctiv optimisation approach, but the objectives are defined based on hierarchical comparison of win rate and two handcrafted playing styles (i.e., aggressive and defensive). More details about these four competitors can be found Appendix C.
For fair comparison, all the four algorithms are run within the same computational budgets (IG frames) and the average win rate of 500 games for each environment is reported. In our algorithm, the frequency of updating the evaluation population freq is set to 5. The hyperparameters of PPO for all the algorithms are set according to (Schulman et al. 2017). Appendix C gives all the parameters and their settings in the experiments.
We aim to evaluate the proposed algorithm through answering the following four questions.
Research Question (RQI): How does BiO perform, comparison with the other algo- rithms, in the same training and evaluation environments when playing against built-in AI?
Research Question 2 (RQ2): How does BiO perform, in comparison with the other algo- rithms, in different training and evaluation environments when against a built-in AI?
Research Question 3 (RQ3): What happens when the agents obtained by BiO and the other algorithms play directly against each other?
Research Question 4 (RQ4): What kind of agents can BiO provide? Do they have diverse playing styles?
The experiment for the question RQ1 is to see the ability of the obtained policy in playing against a rule-based built-in AI player when the training and evaluation environments are the same. The experiment for RQ2 is to see the generalisation of the policy to different environment. Here, we change the start position of the opponent from the central point of the site in the training to a random boundary point in the evaluation. The above two experiments are all about playing against a build-in AI; then one would be interested to know what the agents obtained by all the algorithms play directly against each other. RQ3 is designed to answer this question. The experiment for RQ4 is to
7

---

Under review as a conference paper at ICLR 2023
see if the proposed BiO model can produce diverse playing styles. Since Algorithm I returns only the highest skill-leved policy, one may be curious about what the whole population (i.e., in Step 15 of the algorithm) looks like. RQ4 is for this question.
RQ1: How does BiO perform in the same environment when against the built-in AT? Table T gives the win rate of the policies obtained by the five algorithms. As can be seen in the table, PBT has the best overall performance. This may be attributed to the fact that PBT configures optimal hyperparameters for the training to avoid premature convergence, Our algorithm BiO performs best in Pong and takes the second place in Justice Online, achieving a win rate of over 90% for both games. In contrast, the agents trained by the other three algorithms perform differently in the two environments, especially for Multi-reward which only has close to 20% win rate in Justice Online, but more than 97% win rate in Pong. One explanation is that the built-in AT in Justice Online is rather different from the opponents that the three algorithms' agent (PPO, Multi-reward and EMOGI) encounters in the training, but in Pong they are very similar.
Table 1: Win rate of the policies against the built-in Al in the same training and evaluation environ- ments.
PPO
PBT
Multi-reward
EMOGI
BiO (ours)
Pong
92.8%
99.4% 99.2%
97.4% 20.4%
99.4% 80.8%
99.7%
Justice Online
74.0%
93.0%
How does BiO perform in different environments when against the built-in AI? Table 2 gives the win rate of the policies obtained by the five algorithms. It can be seen that the environmental changes have a greater impact on all the algorithms except BiO. In Pong, the performance of the four peer algorithms (PPO, PBT, Multi-reward and EMOGT) have significant drop, especially the win rate of PPO, PBT and EMOGT being less than 50%. In Justice Online, these four algorithms still cannot obtain a good winning rate, despite slight increase of the Multi-reward's performance compared with the result in the same training and evaluation environments (Table I]. In contrast, our BiO maintains a win rate above 80% when the environment changes, even reaching over 90% win rate in Justice Online. One explanation for this is that the built-in Al is affected very differently by changes in different environments.
RQ2:
Table 2: Win rate of the policies against the built-in Al different training and evaluation environ- ments.
PPO
PBT
Multi-reward
EMOGI
BiO (ours)
Pong
39.8%
19.8% 35.6%
64.5% 39.0%
8.4% 75.0%
84.1%
Justice Online
2.4%
91.0%
What happens when the agents obtained by BiO and the other algorithms play directly against each other? In this experiment, we construct an opponent pool of the agents ever generated, i... one agent from PPO, one from PPT, three from multi-reward (as generates three agents with aggressive, balanced and defensive playing styles respectively). 30 from EMOGI and 30 from BiO (the population size is 30). We then randomly pick one from the opponent pool to be played against agent. This repeats 60 times and the average win rate is reported in Table As shown, BiO achieves a significantly higher win rate than the other four algorithms (except in Justice Online with the same environment where PBT's win rate is slightly higher than BiO's s), indicating better generalisation of the proposed model to different opponents. It is worth noting that the advantage of BiO over its competitors is more evident when training and testing in different environments. This implies the importance of maintaining a set of high skill-level agents with diverse styles during the training, which provides the agent with opportunities to play against very different opponents.
RQ3:
RQ4: Can BiO provide diverse playing styles? Since our algorithm (Algorithm in Section 4) returns only the highest skill-level agent in the final population, one may ask how the remaining agents look like. Figure b)) plots the final population obtained by the algorithm under the Justice Online environment in the space of skill level and playing style. As can be seen, these agents have similar skill level and diverse playing styles. This indicates that our algorithm can provide agents with very different behaviours. These agents should all have good generalisation as they encountered the same set of various opponents in the training.
8

---

Under review as a conference paper at ICLR 2023
Table 3:
Win rate of the policies against pool of the opponents obtained by all the algorithms in the
same and different training-evain cealluttcn environments.
PPO
PBT
Multi-reward
EMOGI
BiO (ours)
Pong (same environment) Justice Online (same environment)
54.6%
59.0%
52.4%
26.9% 55.0%
83.5%
71.0%
79.6%
6.0%
78.0%
Pong (diff. environment)
60.7%
39.3%
70.9%
42.4%
88.8%
Justice Online (diff. environment)
45.9%
45.9%
45.7%
63.7%
81.8%
To get sense of how the results look like in the game environments, Figures ((c) plot the behaviour trajectories three representative agents (aggressive, neural and defensive) tagged in Figure 2(b) nine randomly selected games. As shown, although each agent has very similar behavioural trajectories (orange line) across the nine games, different agents behave rather differently. The aggressive agent towards the opponent, the defensive agent tends avoid confrontation with the opponent and runs around the arena, and the neutral agent stays or walks around some of the arena's boundaries.
Agesta
Denamd egont
(hall 3000
0.0
ogg oo
(Tla 2011
the AXl
Apgrvactve
DDdersdee
Plattig etyle Bol popuiation
16 Fighting
(ggiess stle
(6) Nestral tide
(n) Delendive
Figure 2: An illustration that BiO obtain of agents with diverse play styles under the Justice Online environment. (a) scene the game environment. (b) The final agent population obtained by BiO space of skill level and playing style. (c)(() behaviour trajectories three representative agents tagged in Figure 2(b) on nine randomly selected games, where the orange and blue lines represent the footprints of the agent and opponent, respectively.
CONCLUSION
This paper proposes a bi-objective optimisation model (BiO), working with an evolutionary algorithm, to improve the generalisation of the policy in self-play. BiO maintains an agent population and enables high-level, diverse-style agents more likely to be Pareto optimal. thus playing against each other throughout the training process. The experimental studies have shown its effectiveness and robustness in different environments. In addition to the improvement of the policy's generalisation, by-product from this model is that it can provide a set of high skill-level policies with diverse behaviours.
This work is the first step towards new attempt of balancing performance and behaviour in games, and it could be potentially improved by different ways, for example, tuning the coefficient k of the model for specific problems and developing other population-based algorithms for the model. In addition, despite being used for self- here, it is extendable to other RL settings where the agent's behaviour can be properly measured.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Alexandros Agapitos, Julian Togelius, Simon M. Lucas, Jurgen Schmidhuber, and Andreas Konstan- tinidis. Generating diverse opponents with multiobjective evolution. 2008 IEEE Symposium Computational Intelligence and Games, pp. 135-142, 2008. doi: 10. (..109/CCG..00085555555
Trapit Bansal, Jakub Pachocki, Szymon Sidor, Ilya Sutskever, and Igor Mordatch. Emergent complexity via multi-agent competition. In 6th International Conference Learning Rep- resentations, ICLR, Vancouver, BC, Canada, April 2018. OpenReview.net. URL https
//openrevien woteenreview.nnt/oorumme wttp://eooooommmdommaaaaaae
Adrien Baranes and Pierre Yves Oudeyer. Active learning of inverse models with intrinsically motivated goal exploration in robots. Roboties and Autonomous Systems, 61(1):49-77 January 2013.
Mare G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Rémi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, NIPS, pp. 1471 1471-1479, Barcelona, Spain, December 2016.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowiez, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement learning. CoRR, abs/19 1ess1912.06680, 2019.
Micah Carroll, Rohin Shah, Mark K. Ho, Tom Griffiths, Sanjit A. Seshia, Pieter Abbeel, and Anca D. Dragan. On the utility of learning about humans for human-AI coordination. In Advances in Neural Informaionn Processing Systems, NIPS, pp. 5175-5186, Vancouver, BC, Canada, December 2019.
Yutian Chen, Aja Huang, Ziyu Wang, Ioannis Antonoglou, Julian Schrittwieser, David Silver, and Nando de Freitas. Bayesian optimization AlphaGo. CoRR, abs/18 ////1/8220665555 2018.
Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Improving exploration in evolution strategies for deep reinforcement learning via population of novelty-seeking agents. In Advances in Neural Information Processing Systems, NIPS, pp. 5032-5043, Montréal, Canada, December 2018.
Antoine Cully and Yiannis Demiris. Quality and diversity optimization: A unifying modular framework. IEEE Transactions on Evolutionary Computation, 22(2):245-259, April 2018
Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and TAMT Meyarivan. A fast and elitist multiobjec- tive genetic algorithm: NSGA-II. IEEE transactions on evolutionary computation, 6(2):182-197, April 2002.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Go-explore: new approach for hard-explonation problems. CoRR, abs/1901.10095,, 2019. URL http:
1/arxiv.oot wttp////////bbooooooeo 10999
10995
Matthew Fontaine and Stefanos Nikolaidis. Differentiable quality diversity. Advances in Neural Information Processing Systems, NIPS, 34:10040-10052, 2021.
Sébastien Forestier, Yoan Mollard, and Pierre-Yves Oudeyer. Intrinsically motivated goal exploration processes with automatic curriculum learning. CoRR, abx//708.00299) 2017. URL http:
//arxiv.org/abs/
02190
Zhaohan Daniel Guo and Emma Brunskill. Directed exploration for reinforcement learning. CoRR, abs/1906.0780s 2019. http://tpp///aaairoooooo/aa////a
wtt.://rg/abba//gg 07805
Verena Heidrich-Meisned and Christian Igel. Neuroevolution strategies for episodic reinforcement learning. Journal of Algorithms, 1224)1152-166, October 2009.
10

---

Under review as a conference paper at ICLR 2023
Daniel Hernandez, Kevin Denamganai, Yuan Gao, Peter York, Sam Devlin, Spyridon Samothrakis, and James Alfred Walker. generalized framework for self- training. In 2019 JEEE Conference Games (CoG), pp. 1-8, London, United Kingdom, August 2019. 8848006.
doi: 10.1109/CIG.2019.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. VIME: variational information maximizing exploration. In Advances in Neural Information Processing Systems, NIPS, pp. 1109-1117, Barcelona, Spain, December 2016.
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M. Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukeuoglu. Population based training of neural networks. CoRR, awss/711.099300 2017.
URL http: org/ abs /ess/1711 1111/11111111 09846
Max Jaderberg, Wojciech M Czarnecki, lain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattic, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, et al. Human- level performance in 3d multiplayer games with population-base based reinforcement learning. Science, 364(6443):859 865, 2019.
Qiqi Jiang, Kuangzheng Li, Boyao Du, Hao Chen, and Hai Fang. Deltadou: Expert-level doudizhu AI through self-play. In Proceedings of the Twenty-Eights International Joint Conference Artificial Intelligence, IJCAI-19, pp. 1265-12222 1271, Macao, China, July 2019. International Joint Conferences on Artificial Intelligence Organization. doi: 00244963ijcai.201199// URLhttps
7/doi..og/10. 249.3/..99331111 1ttp:///ccaa....11.......
Whiyoung Jung, Giseung Park, and Youngchul Sung. Population-guided parallel policy search for reinforcement learning. In 8th International Conference Learning Representations, ICLR, Addis Ababa, Ethiopia, April 2020. URL https://///......t.. httpss///ppeneevee...o wtwp://vwwww..eeeeoooummm...... [JeIND4KWH
Anssi Kanervisto and Ville Hautamäki. Torille: Learning environment for hand-to-hand combat. In IEEE Conference on Games, CoG 2019, pp. 1-8, London, United Kingdom, August 2019. IEEE,
Alexandre Laterre, Yunguan Fu, Mohamed Khalil Jabri, Alain-Sam Cohen, David Kas, Karl Ha- ijar, Torbjorn S. Dahl, Amine Kerkeni, and Karim Beguir. Ranked reward: Enabling self-play reinforcement learning for combinatorial optimization. CoRR, abs/1800.01677 2018. URL
http: org. 1007//066/180001672
Joel Lehman and Kenneth O. Stanley. Evolving a diversity of virtual creatures through novelty search and local competition. 13th Annual Genetic and Evolutionary Computation Conference, GECCO, pp. 211-218, Dublin, Ireland, July 2011. ACM.
Ang Li, Ola Spyra, Sagi Perel, Valentin Dalibard, Max Jaderberg, Chenjie Gu, David Budden, Tim Harley, and Pramod Gupta. A generalized framework for population based training. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Duta Mining, KDD, pp. 1791-1799, Anchorage, AK, USA, August 2019. ACM.
Liu and C. Qian. Prediction guided meta- letanlea for multi-objective reinforcement learning. IEEE Congress on Evolutionary Computation, CEC, pp. 2171-2178, 2021.
Siqi Liu, Guy Lever, Josh Merel, Saran Tunyasuvunakool, Nicolas Heess, and Thore Graepel. Emergent coordination through competition. In 7th International Conference on Learning Rep- resentations, ICLR, New Orleans, LA, USA, May 2019. URL htattss://openzevi.cm net/
Eorum?1d=BkGBB jR5mm
jR55km
Ryan Lowe, Abhinav Gupta, Jakob Foerster, Douwe Kiela, and Joelle Pineau. On the interaction between supervision and self-play in emergent communication, arXiv preprint arXiv:2002.0109; 2020.
Ehrgott Matthias. Multicriteria optimization. Springer Science & Business Media, 2006.
11

---

Under review as a conference paper at ICLR 2023
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Proceedings of the 33nd International Conference Machine Learning, ICML, pp. 1928-1937, New York City, NY, USA, June 2016.
Kristof Van Moffaert and Ann Nowé. Multi-objective reinforcement learning using sets of pareto dominating policies. The Journal of Machine Leurning Reseurch, 15(1):3483-351,, January 2014.
Hossam Mossalam, Yannis M. Assael, Diederik M. Roijers, and Shimon Whiteson, Multi-objeetive deep reinforcement learning. CoRR, abs/1610.027077, 2016. URL /ttp:://///rr abs/
02707
Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites. CoRR, has/1504.04909, 2015. |ttpp::://arxxv wwt.:/orggabbbs....e abs/ab/4//1.//
04909
Jean-Baptiste Mouret and Stéphane Doncieux. Using behavioral exploration objectives to solve deceptive problems in neuro-evolution. In Genetic and Evolutionary Computation Conference, GECCO, pp. 627-634, Montreal, Québec, Canada, July 2009. ACM.
Inseok Oh, Seungeun Rho, Sangbin Moon, Seongho Son, Hyoil Lee, and Jinyun Chung. Creat- ing pro-level Al for real-time fighting game with deep reinforcement learning. arXiv preprint arXiv: 2019.
Georg Ostrovski, Mare G. Bellemare, Aäron den Oord, and Rémi Munos. Count-based exploration with neural density models. In Proceedings of the 34th International Conference on Machine Learning, ICML, volume 70, pp. 2721-2730, Sydney, NSW, Australia, August 2017.
Jack Parker-Holder, Aldo Pacchiano, Krzysztof Choromanski, and Stephen Roberts. Effective diversity in population- reinforcement learning. CoRR, abs/2002.00632, 2020.
Krzysztof Pawelezyk. Michal Kawulok, and Jakub Nalepa. Gemetically-trainee deep neural networks In Proceedings of the Genetic and Evolutionary Computation Conference Companion, GECCO, pp. 63 64, Kyoto, Japan, July 2018. ACM.
Thomas Pierrot, Valentin Macé, Felix Chalumeau, Arthur Flajolet, Geoffrey Cideron, Karim Beguir, Antoine Cully, Olivier Sigaud, and Nicolas Perrin-Gilbert. Diversity policy gradient for sample efficient quality-diversity optimization. ICLR Workshop on Agent Learning 2022a.
Thomas Pierrot, Guillaume Richard, Karim Beguir, and Antoine Cully. Multi-objective quality diversity optimization. arXiv preprint arXiv:2202.03057 2022b.
Justin K. Pugh, Lisa B. Soros, and Kenneth O. Stanley, Quality diversity: A new frontier for evolutionary computation. Front. Robotics and AI, 3:40, July 2016.
Tim Salimans, Jonathan Ho, Xi Chen, and llya Sutskever. Evolution strategies as a scalable alternative to reinforcement learning. CoRR, abs/ 0as5//033.3664 2017.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.cn 77.044777 2017. URL http:://ax....org///
1707. 06347
Ruimin Shen, Yan Zheng, Jianye Hao, Zhaopeng Meng, Yingfeng Chen, Changjie Fan, and Yang Liu. Generating behavior-diverse game ais with evolutionary multi-objective deep reinforcement learning. In International Joint Conferences on Artificial Intelligence, IJCAI, 2020. URL https desites.goo. cww wwtp///weeeeeaaa/--eaeeaeeee
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonogiou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484 4899, January 2016. doi: na. 16961.
12

---

Under review as a conference paper at ICLR 2023
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy P. Lillicrap, Karen Simonyan, and Demis Hassabis. Mastering chess and shogi by self- with a general reinforcement learning algorithm. CoRR, 111.0171,,,115, 2017a. URL htt.ttttt./ax.l..or....
abs/ 01815
David Silver, Julian Schrittwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel. and Demis Hassabis. Mastering the game of go without human knowledge. Nature, 550:354-35555
October 2017b.
Kenneth Stanley, Jeff Clune, Joel Lehman, and Risto Miikkulainen. Designing neural networks through neuroevolution. Nature Machine Intelligence, 111):44-35, January 2019.
Felipe Petroski Such, Vashisht Madhavan, Edoardo Conti, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Deep neuroevolution: Genetic algorithms are competitive alternative for training deep neural networks for reinforcement learning. CoRR, abs/ abs/1772.00567, 2017.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. #exploration: A study of count-based exploration for deep reinforcement learning. In Advances in Neural Information Processing Systems, NIPS, pp. 2753- 2762, Long Beach, CA, USA, December 2017.
Bryon Tjanaka, Matthew C Fontaine, Julian Togelius, and Stefanos Nikolaidis. Approximating gradi- ents for differentiable quality diversity in reinforcement learning. arXiv preprint 2ivv:2202.036666 2022.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czarnekki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft II using agett reinforcement learning. Nature, 575(7782)3350- 354, October 2019.
Huazhe Xu, Keiran Paster, Qibin Chen, Haoran Tang, Pieter Abbeel, Trevor Darrell, and Sergey Levine. Hierarchical deep reinforcement learning agent with counter self-play on competitive games.
www. for um? wmw.mmmddddoeeerre.... September 2018.
K. Xue, J. Xu, L. Yuan, M. Li, C. Qian, Z. Zhang, and Y. Yu. Multi-agent dynamic algorithm configuration. Advances in Neural Information Processing Systems, NIPS, 2022.
Runzhe Yang. Xingyuan Sun, and Karthik Narasimhan. A generalized algorithm for multi-objective reinforcement learning and policy adaptation. In Advances in Neural Information Processing Systems, NIPS, pp. 14610-14621, Vancouver, BC, Canada, December 2019.
Rui Zhao, Jinming Song, Hu Haifeng, Yang Gao, Yi Wu, Zhongqian Sun, and Yang Wei. Max- imum entropy population based training for zero-shot human-ai coordination. arXiv preprint 1r2iv:21 2021.
A ENVIRONMENTS
This section describes the two environments Pong and Justice Online and the implementation of the bi-objective model BiO in them, i.e., how to determine the skill level function g(T) and the playing style function ((T)) in the two environments.
PONG. Pong is two-player game that simulates table tennis, in which each player controls an in-game paddle by moving it vertically across the left or right side of the screen. There are three actions that the players can take: move up the paddle, move down the paddle, or stay still. Points are earned when one fails to return the ball to the other. The goal of the game is for each player to reach 20 points before the opponent. The self-play version of Pong used in the experiments is accessible at https::///// h/tp:////gt/bb/////// com/ x xinghai-sunn www.ssuundddddddeeeeeeee deep-ri/blob http://eer//bbbll//aattte///ee selfp hal__pallllnnggggl md
To estimate the skill level of an agent in Pong, we consider the average difference of the final scores between the agent and its opponent in the opponent pool. Formally, the skill level function g(T) of
13

---

Under review as a conference paper at ICLR 2023
the agent is calculated as:
1 K
score ScOre + 20
(5)
40
k=
where K is the size of the opponent pool (i.e. the population size in our algorithm) and score, is the score the agent æ at the end of the game.
The playing style in Pong is in the opponent- iinneennt-ddeeeddent mode. It can be reflected by the move frequency of the agent's paddle. As such, the state change function varna (when æ plays against mi) in the playing style function (i.e. Equation (3) in the text)
K
1 K
h(T))
varny
(Se, (Se, 8c-1)
(6)
can be defined as
0, if the position of IT's paddle in state 8141 is the same in 8t.
varxx (Sts $141)
1, otherwise
where Ti denotes the length of the trajectory produced by playing against the opponent Tu.
JUSTICE ONLINE. Justice Online Suppliess duels between players called "Arena Battles". The arena battle a two-player zero-sum game, where two players fight against each other to decrease the opponent's Health Point (HP) is non-trivial task. First, it is a real-time game which requires two players to make decisions simultaneously under imperfect information. At any time of the game, neither player knows what the opponent's s current skills and skill levels are likely to be. Second, the game has a massive state space. An agent must make various skills, moves and targeted decisions simultaneously. leading to huge number of possible states.
the goal within fixed amount of time. This game
To estimate the skill level of an agent, we consider the average difference of the final HP values between the agent and its opponent in the opponent pool, expressed as:
K
g
HP.
HPP,
(8)
K
k=1
where K is the size of the opponent pool and HPa is the HP value of the agent at the end of the game.
The playing style in Justice Online is in the opponent-dependent mode. It needs to be measured by the relative position (i.e. distance) between the agent and its opponent. Aggressive agents like close combat and their distance to the opponent is small, whereas defensive agents like to stay in a certain distance from their opponents. We thus define the var=, function Equation 6
var (St, 8(+1) = dish++1(5 (T,TTK)
(9)
where diste+1 (π, Ty) denotes the Euclidean distance between the agents π and Ty at step t + 1. Note that both the skill level and playing style values normalized into the range of [0, in implementation.
B ALGORITHM DETAILS
This section first describes how to generate agents in our algorithm and then explains how to constitute the new population based on the old population and newly-gon wwwly-generated agents.
B. AGENT GENERATION
In the proposed algorithm, variable parameters of the agents are the network weight & and the reward weight w; other hyperparameters (e.g- neural architecture, discount factor and learning rate) are fixed and set manually (their settings can be found in Appendix C). Here, we consider basic variation operators from evolutionary algorithm for searching for promising 0 and w. For 0, we use the single-point crossover (Pawelczyk et al.) 2018)) to recombine two networks. It randomly selects a hidden layer two networks and swaps part of them to produced two new networks. Figure illustrates this operator.
https:///ghg wtt../gghiuhhccooo htw.b//.co///eeeeeee-xiiiieeee
14

---

Under review as a conference paper at ICLR 2023
Purent
Parent
Children
Childwen
Figure 3: An illustration of the single-point crossover for generating neural networks.
B.2 ENVIRONMENTAL SELECTION
use the classic NSGA-II environmental selection procedure (Deb et al., 2002) to select the K best solutions as the next-generation population from the union of the old population P and the new generated solutions P', as shown in Algorithm 2 First, we divide the union into different fronts (F,, ..... F.... ......) where the solutions in the same front critical front F; is found such that [i U F2 U... F,-1 S K and FF U E2 U... U U F.I K; correspondingly the first 1 fronts (F1, F2...., Fi-1) make up the new population P. Now if the size of P is less than the population capacity K, we calculate the crowding distance of the solutions in F; and them in descending order. Next, choose the first K PP] solutions of F; and put them into P. Now, P is the population to be returned.
nondominated to each other. Then, the
Algorithm 2: wtweerrooomeettaSteettttttPP" Deb al 2002)
Input: Population capacity K;
Fi, F2...,,F... -Nonominatddor(PPPP))
1/ Partition P into different nondominated fronts FI, F2.... F.... and find the critical front F, where 0 U...U.UUURR <F
P UU..UUUFFF
if [P < < then
P, + Crowding (Fstaneee
// Calculate crowding distance of
each solution in F,
F ;
// Sort in the desending order accroding to
crowding distance
P PUR K |
:/]
Choose the first K KPlPl solutions of F
return P
C EXPERIMENT DETAILS
This section details all paramettrs/hyyerppaaeeeess used the experiments. We first list the PPO- associated hyperparameters (Mnih et 2016) which are shared by all the four algorithms. This is followed by the parametershhyperarameeee specific to each algorithm. Finally we give the reward settings the four algorithms for the two environments Pong and Justice Online.
PARAMETERS FOR PPO USED IN ALL THE ALGORITHMS.
Discount Rate (y 0.99). This parameter controls how much the RL agent cares about rewards in the immediate future relative to those in the distant future.
Weights for Loss Function ([1, 0.5, 0.01).. The loss function of PPO consists of three terms, the policy loss (actor), the value loss (critic), and an entropy loss. Their weights were assigned 1, 0.5 and 0.01, respectively.
15

---

Under review as a conference paper at ICLR 2023
Learning Rate for Adam (0.00025). This parameter controls the learning rate of the loss function.
Neural Architecture [256, 128, 128). The network consists of three hidden layers, of which the first one has 256 nodes and the other two have 128 nodes. For the image input (such as Pong game), we stack 4 layers of 3 x 3 convolutions before the fully connected layers.
PARAMETERS FOR PPO-SP.
Opponent Pool Update Frequency (1, 000 episodes). This parameter controls the update frequency of the opponent pool in self-play, i.e., how often the algorithm sends its agent into the opponent pool.
Opponent Change Frequency (3 episodes). This parameter controls how long the agent plays against one opponent.
PARAMETERS FOR THE MULTI-REWARD ALGORITHM.
Opponent Pool Update Frequency (1, 000 episodes). This parameter controls the update frequency of the opponent pool in self-play.
Opponent Change Frequency (3 episodes). This parameter controls how long the agent plays against one opponent.
Opponent Selection Probability. In the multi-reward algorithm, the most recent 10 models of each playing style are selected to play against the agent with probability p, while the other past versions are selected uniformly with probability 1 p. As practiced in (Oh et 2019), p was set gradually decreased from 0.8 to 0.1 with the progress of the training.
PARAMETERS FOR THE PROPOSED ALGORITHM.
Population Size (K = 30). This parameter controls the size the population in the evolution-
ary process.
Single-Point Crossover Probability (0.5). This parameter controls the probability of performing the recombination of two networks.
Simulated Binary Crossover Probability (1.0). This parameter controls the probability of performing the recombination of two agents' reward weights. The distribution index
Tec 20 was used (Deb et al. 2002).
Polynomial Mutation Probability (pm. = 0.05). This parameter controls the probability of performing the disturbance an agent's reward weights. The distribution index 7/m- 20 was used (Deb et al. 2002).
PPO Training Budget (Pong 600, 000, Justice Online 1, 000, 000). This parameter controls the training overheads (frames) of PPO one generation of the evolutionary algorithm.
REWARD SETTINGS PONG. In Pong, the internal reward step for the algorithms PPO and PPO-SP was set to r = if the agent wins at that step; otherwise r, = 0. As for the algorithms and BiO, different playing styles of agents are considered explicitly or implicitly. So a factor to reflect the agent's 's playing style was also included in the reward function. Formally, the internal reward at step is expressed as
= wir, + wort
skill
style
(10)
where
r
pekill
1,
if the agent wins at step otherwise
(11)
and
11
the position the agent's paddle in state s, is different from that in 8,-1 otherwise
style
(12)
0.
16

---

Under review as a conference paper at ICLR 2023
w1 and W2 are two weight parameters. Among them, Wa is to control agents' playing styles. In the multi-reward algorithm, wi was set to across the three styles, while w2 was set to 0.01, and -0.01 for the busy, neutral and lazy styles, respectively. In our algorithm, W2 and W2 are generated by the evolutionary search (i.e. simulated binary crossover and polynomial mutation) within the range of wi E [0, 1] and W2 [-0.01, 0.01].
REWARD SETTINGS JUSTICE ONLINE. In Justice Online, the goal of the agent is to decrease the opponent's 's Health Point (HP) while trying to keep its own HP non-decren non-decreasing. Therefore, the HP margin can be used as the internal reward. Specifically, in PPO and PPO-SP we considered the following internal reward at step / when the agent æ plays against its opponent TA-
T = (HP." HP + (HP - H
(13)
where P. denotes the HP value of the agent step L. The term PH HP ) is to encourage the agent to defend the opponent's 's attack, and the term (HP,T - HP. is to encourage the agent to attack the opponent.
As for the algorithms multi-reward and BiO, since different playing styles of agents are considered, we added a term to reflect the agent's s playing style. Specifically, we used the following internal reward at step t when the agent plays against its opponent TT.
= w> HP + w2((H
HP,,") + (1d - (x), (14)
where d ) denotes the Euclidean distance between the agents π and Tr at step I. Among the three weight parameters, wh and w2 concerned with the skill level while wy controls the playing style. In the multi-reward algorithm, set to 1, 0 and -1 for the aggressive. neutral and defensive styles, respectively. all the weights are generated by the evolutionary search within the range of wn E [0, 1J. W2 E [0, 1] and Wt E [-1, 1]. Table and Table 5 summarize the reward weight settings of the multi-reward algorithm and the proposed algorithm in the two environments, respectively.
and W2 were set to 1 across the three styles, while wi was
our algorithm.
Table 4: Reward weights of each style in the multi-reward algorithm.
Environments
Playing styles
Reward weights
busy
W1 = 1, w2 = 0.01
Pong
neutral
W1 1, 1022 0
lazy
W1 1, W12 0.01
aggressive
Wi 1, W2 1, W3
Justice Online
neutral
W1 1, 02 1, 103
defensive
W1 W2 1, W3
Table 5: The range of reward weights for the search in the proposed algorithm.
Environments
Weight range
Pong
Wi E [0, 1]. wy e [-0.01, 0.01]
Justice Online
E [0, 1). w2 E [0, 1]. wa E [-1.1]
17


Reviews:
[{"review_id": "oHBwoucLWm", "reviewer": "Reviewer_mGvT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a bi-objective optimization method for learning strategies for playing games from a pool of diverse agents. The method is population-based in the sense that it maintains a population of agents that attempts to optimize for a bi-objective function. Such a function accounts for the diversity of play, which is game-dependent, and winning rate. \n\nThe system was evaluated in the games of Pong and Justice Online, where the proposed method was competitive in some settings and stronger in others with respect to other population-based agents and RL baselines. ", "summary_of_the_review": "Paper deals with a challenging and important problem, but misses an important part of the literature that could be used as baseline in the experiments. The empirical design seems to be problematic as it might be giving advantage to the method introduced in this paper. ", "strength_and_weaknesses": "Strength\n\nThis paper deals with a challenging topic, which is the one of learning strategies for playing complex games. The paper works with the underlying hypothesis that one is able to learn stronger strategies while keeping a population of diverse agents. While other population-based works dealt with similar research question, the paper does a good job reviewing some of these works. The experiments in Pong and Justice Online seem to support this underlying assumption.\n\nWeaknesses\n\nA key weakness of the paper is to not discuss the paper \"A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning\" by Lanctot et al (2017). The paper talks vaguely about \"self play\" without explaining exactly which self-play algorithm they are referring to. Lanctot et al. describe a general framework for multiagent RL that allows one to instantiate algorithms such as Iterated-Best Response, Fictitious Play, and Double Oracle. As far as I know these are all self-play algorithms. In particular, Fictitious Play can be seem as a population-based algorithm and should probably be used as a baseline in the experiments. Why not consider these algorithms as baselines? \n\nThe paper is also constructed under the assumption that \"maintaining an agent population does not necessarily mean maintaining diverse playing styles.\" (see Related Work section). However, population-based agents such as AlphaStar were designed to learn a diverse set of playing styles. It isn't clear from the experiments whether the proposed method is able to produce \"more diverse agents\" than algorithms such as AlphaStar. \n\nAnother weakness is related to how opponents are sampled to generate the results shown in Table 3. The experiments starts with a pool of agents formed by 1 PPO agent, 1 PPT (this is probably a typo and authors meant PBT) agent, 3 Multi-Reward Agents, 30 EMOGI agents, and 30 BiO agents. Then, an opponent is randomly sampled from this pool of agents for each of the evaluated methods. The agents play a game and the result is stored. The process is repeated 60 times and the average win rate is reported in Table 3. \n\nThis experimental design is problematic because it gives an advantage to EMOGI and BiO as they represent a much bigger share of the pool of agents. That way, both EMOGI and BiO are likely to face, during test, agents they were trained with. Despite this significant disadvantage, PBT is still competitive with BiO. This makes me wonder whether BiO is bringing something new in comparison to other PBT agents. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "fbiBmEPNtK", "reviewer": "Reviewer_FdvX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to optimize a population of agents generated through self-play to cover a multi-objective Pareto front among objectives for play style and performance (skill level). Play style is quantified in terms of state changes during play against various opponents. The optimization algorithm adapts the NSGA-II evolutionary algorithm to use a population generated through self-play. Empirical studies on two games shows benefits to the proposed algorithm when the starting state of a game is randomized and relatively strong performance against a pool of other agents (in both cases compared other RL or multi-objective optimization algorithms).", "summary_of_the_review": "\nThe technical novelty of the paper is the incremental change to NSGA-II adapted to self-play. Empirical results are strong on what is reported, but the base of comparison is narrow: two domains and without comparison to a related class of multi-objective optimization algorithms (quality-diversity). The style results show some diversity, but in a relatively narrow sense: navigating to a fixed point or circling around a space (with no results reported on Pong). I greatly appreciate the conceptual simplicity of the algorithm and see promise in the results. But the lack of a breadth of empirical results makes it hard to see the results as robust to a wide variety of competitive RL tasks. Ultimately the results are promising and with further validation would make a potentially useful contribution to the set of competitive RL algorithms.", "strength_and_weaknesses": "## Strengths\n\n1. Improves over widely used baselines. The study results demonstrate benefits relative to reasonable baselines in two different domains.\n2. Straightforward to test and use. The algorithm is not complex or difficult to implement, making it a potentially strong addition to the set of multi-objective optimization approaches used in RL training.\n\n\n## Weaknesses\n\n1. Some missing experiment details. Were opponents weighted by algorithm type for RQ3? More details below.\n2. Few domains. Pong is a relatively simple baseline domain with very limited degrees of freedom for agent actions. Justice Online is clearly complex and valuable. It would help to include results from one or two other benchmark competitive RL domains to better clarify the applicability and power of BiO over baselines. Right now it is hard to tell how much BiO is particularly tailored to Justice Online compared to being generally powerful.\n3. Missing baseline: domain randomization. This is a minor point, but one of the main differences in performance relates to performing poorly in \"different environments\" (RQ2). The widely used practical solution is to randomize agent starting state during training. The results would benefit from reporting how all algorithms perform when applying randomization to the starting states during training. How much does BiO improve over this simple baseline augmentation?\n4. No comparison to quality-diversity algorithms. The paper would benefit from comparing to a representative algorithm from the class of quality-diversity algorithms, as they are widely used for RL tasks (references below). MAP-Elites is one classic candidate.\n\n\n## Feedback & Questions\n- Q: For RQ3 the experiment picks a random opponent 60 times from a pool of 65 possible opponents (as I understand from the text). This seems likely to be highly imbalanced as 30 opponents are from EMOGI and 30 from BiO. Were the opponents selected so that 1/5 of the time an opponent comes from each of the algorithms? It would also help the results to include a matchup table reporting winrates of each algorithm against each variant, not only the average as reported in Table 3.\n\n- Table 3: Why might PPO do so well in the Justice Online different environment (45.9% win rate) compared to the built-in AI (2.4% win rate in Table 2)? This may be a minor anomaly, but is surprising.\n\n- Q: How would this approach scale to games where playstyle has high dimensionality? The experiments only examine a single dimension of style (distance to opponent). As dimensionality increases the algorithm would likely need to maintain a larger population and would sample each opponent less frequently. What do these scaling properties look like? Is there a test environment or setup that could examine this case?\n\n- Q: Are there other arenas in Justice Online? The \"different environments\" setup is a change in initial position. How well does a baseline PPO algorithm do if trained with randomized opponent starting positions? Why does the starting position change cause such large differences in performance? This seems like the agents overfit to a very narrow range of conditions, rather than learning the \"real\" task, likely due to self-play having little incentive to mirror. \n\nreferences\n- MAP-Elites\n\t- Jean-Baptiste Mouret and Jeff Clune.  Illuminating search spaces by mapping elites. arXiv preprint arXiv:1504.04909, 2015.\n- Other quality-diversity approaches\n\t- Pourchot, Aloïs, and Olivier Sigaud. \"CEM-RL: Combining evolutionary and gradient-based methods for policy search.\" arXiv preprint arXiv:1810.01222 (2018).\n\t- Khadka, Shauharda, and Kagan Tumer. \"Evolution-guided policy gradient in reinforcement learning.\" Advances in Neural Information Processing Systems 31 (2018).\n\t- Khadka, Shauharda, Somdeb Majumdar, Tarek Nassar, Zach Dwiel, Evren Tumer, Santiago Miret, Yinyin Liu, and Kagan Tumer. \"Collaborative evolutionary reinforcement learning.\" In International conference on machine learning, pp. 3341-3350. PMLR, 2019.\n\t- Parker-Holder, Jack, Aldo Pacchiano, Krzysztof M. Choromanski, and Stephen J. Roberts. \"Effective diversity in population based reinforcement learning.\" Advances in Neural Information Processing Systems 33 (2020): 18050-18062.\n\t- Jung, Whiyoung, Giseung Park, and Youngchul Sung. \"Population-guided parallel policy search for reinforcement learning.\" International Conference on Learning Representations, 2020.\n\t- Fontaine, Matthew, and Stefanos Nikolaidis. \"Differentiable quality diversity.\" Advances in Neural Information Processing Systems 34 (2021): 10040-10052.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VxlGkvbXdJ", "reviewer": "Reviewer_1h9y", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a population-based self-play RL approach (with PPO for training in the inner loop, and an evolutionary approach in the outer loop) that trains a population of agents for games that are high-skill but also diverse in terms of playing style. First, it summarises \"playing style\" as a single scalar, which ranges between 0 (\"defensive\") and 1 (\"aggressive\"). Then, it aims to optimise a multi-objective (specifically, 2-objective) problem, where one objective adds the playing style to the skill level, and the other objective subtracts the playing style from the skill level. The approach is evaluated in 2 games (Pong and Justice Online).", "summary_of_the_review": "A good paper in general, with a few important details that require some more clarification.", "strength_and_weaknesses": "**Strengths**:\n- Majority of the paper well-written and easy to read.\n- Interesting and simple approach, empirical results look promising.\n\n**Weaknesses**:\n- While I listed the majority of the paper being easy to read as a strength above, at the same time there is a weakness in that a few parts require more clarification. I provide detailed comments below.\n- The experiments needs some more details provided, such that we can better quantify the significance of the results. See detailed comments below.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject