Multi-Critic Actor Learning: Teaching RL Policies to Act with Style
Authors: Siddharth Mysore
George Cheng
Yunqi Zhao
Kate Saenko
Meng Wu
Venue: ICLR 2022 Poster (2022)
Keywords: Reinforcement Learning
Multi-Style Learning
Multi-Task Learning
Actor-Critic

Abstract: Using a single value function (critic) shared over multiple tasks in Actor-Critic multi-task reinforcement learning (MTRL) can result in negative interference between tasks, which can compromise learning performance. Multi-Critic Actor Learning (MultiCriticAL) proposes instead maintaining separate critics for each task being trained while training a single multi-task actor. Explicitly distinguishing between tasks also eliminates the need for critics to learn to do so and mitigates interference between task-value estimates. MultiCriticAL is tested in the context of multi-style learning, a special case of MTRL where agents are trained to behave with different distinct behavior styles, and yields up to 56% performance gains over the single-critic baselines and even successfully learns behavior styles in cases where single-critic approaches may simply fail to learn. In a simulated real-world use case, MultiCriticAL enables learning policies that smoothly transition between multiple fighting styles on an experimental build of EA’s UFC game.

Published as a conference paper at ICLR 2022
MULTI-CRITIC ACTOR LEARNING: TEACHING RL POLICIES TO ACT WITH STYLE
Siddharth Mysore
George Cheng & Yunqi Zhao Electronic Arts
Boston University & Electronic Arts
Kate Saenko
Meng Wu Electronic Arts
Boston University & MIT-IBM Watson Al Lab
ABSTRACT
Using single value function (critic) shared over multiple tasks in Actor-Critic multi-task reinforcement learning (MTRL) can result in negative interference be- tween tasks, which can compromise learning performance. Multi-Critic Actor Learning (MultiCriticALL) proposes instead maintaining separate critics for each task being trained while training a single multi talti-task actor. Explicitly distinguish- ing between tasks also eliminates the need for critics learn do and mitigates interference between task-value estimates. MultiCriticAL. is tested in the context of multi-style learning, a special case of MTRL where agents are trained be- have with different distinct behavior styles, and yields up to 56% performance gains over the single-critic baselines and even successfully learns behavior styles in cases where single-critic approaches may simply fail to learn. In simulated real- world use case, MultiCriticAL enables learning policies that smoothly transi- tion between multiple fighting styles on an experimental build of EA's UFC game.
INTRODUCTION
Reinforcement Learning (RL) offers an interesting means by which to develop interesting behaviors in a variety of controls settings. The work presented in this paper is primarily contextualized by our efforts to develop RL techniques for teaching RL-policies to behave with multiple distinct and specific styles for game-play. RL. can be useful in developing control policies for game-play and has been demonstrated as being capable of learning human-like or even superhuman play. most notably beating top-ranked human players games such Dota (Berner et al.) 2019). StarCraft (Vinyals) let al. 2019), Chess (Silver et al.) 2017), and Go (Silver et al. 2016). RL-based control in games offers wide array of potential applications, including testing (Zhao et al., 2020; Ariyurek et al.) 2021 game design (Gisslén al. 2021), providing competition for human players (Berner et al. 2019), or simply as a means to develop more interesting caue--pay& Al behav- be notorious however for training aesthetically unappealing policies, but prior work has demonstrated that, with careful shaping of the reward functions, is possible to derive highly specific desirable behavior (Peng et al. 2018|Myson et al. 20211b) Machine learning (ML) solutions to game-play AI can however represent a signifi- cant increase in compute requirements over their heuristic-based counterparts, especially when using deep learning techniques. If a learned control policy could represent multiple desirable styles and offered controllable way transition between the learned styles, this could reduce the total com- pute burden of the ML--based controller, making it more practically viable. Ideally, developing an MI. model would constitute learning a single, low-complexity policy that would model and transi- tion smoothly between multiple behavior styles. Reduced model complexity and deploying just single policy network enables greater compute efficiency through hardware parallelism and reduced applications, such as videogames.
202 Gordillo et al.
iors (Zhao et al.) 2020 Alonso et al. 2020). RL algorithms
memory costs, which is beneficial in
We explore the problem of mutli-style RL, a special case of the more commonly explored prob- lem of multi-task RL (MTRL). Whereas MTRL techniques typically seek to solve wide array of RL problems under a single learning campaign, multi-style RL adds the extra constraint that the 'tasks' in question all have identical system dynamics but would be characterized by different

---

Published as a conference paper at ICLR 2022
Typical Actor-Critic RL
MultiCriticAL
Cnticc
Gritic Optimization
Critie, IE Mayyadl
Multi-Eritc Optimication
Cosinaiians 8. Tuak Ercodins
Ti Tin westiie & &Takk Excosiin
Policy Optiiaztionn
Policy Optimization
Stske Rewards
TasksStyle
Rewards
Figure I: MultiCriticAL breaks from the common practice of using a single unified critic in MTRL and instead uses separate critics for each task learned. The proposed method is used successfully train multiple distinct behavior styles in various games including Pong and UFC.
ward signals corresponding to the different desired styles of behavior. We focus specifically on applications of Actor-Critic RI. techniques, which, in addition to being compatible with continuous action-space control, allow for reduced runtime computational cost as only the actor functions are required for inference, while the critics can often be disregarded after training. |Mysore et al. (202 and Andrychowicz et al. (2021) have also demonstrated that it is possible to train compact and per- formant policy networks through careful consideration of the actor and critic network architectures.
A common baseline approach to MTRL is to employ one-hot task encoding to delineate between tasks. However, in a number of multi-style cases considered this was not sufficient to enable successful multi-style policy-learning. We suspected that the similarity of game-play states visited between each trained style interfered with each style's value optimization. In surveying existing MTRL literature for advancements in the field of multi-task Actor-Critic RL methods, we noticed surprising hole. There are studies that employ single actors and single critics (Finn et al. 2017 Yang Sodhani et al. 2021: Nichol et al. 2018), multiple actors and multiple crities (Andreas)
al. 2020
et al.
2017 Teh et al. 2017 Rusu Rusu et al. 2015 2016 Huang et al. 2017), or multiple actors and a
single critic (Yang et al. 2011) Dewangan al. 2018). Curiously, seemingly no prior work explores
using a single policy with distinct, per-task value estimation, i.e. single actor with multiple critics.
This paper proposes Multi- Actor Learning (MultiCriticALL), single-actor, multi-critic frame- work for multi-task Actor-Critie optimization, and evaluates it on variety of multi-style learning problems. The core idea of MultiCriticALL is to maintain separate per-style (or task) critic functions for each style being learned. By separating the critic MultiCriticAL would be able to avoid negative interference in the learned values for different styles. Our results show that MultiCriticAL consis- tently trains more performant policies, compared to its single-critic counterparts, succeeds in cases where the single-critic methods fail and achieves between 20-45% improvement in more traditional multi-task settings involving multi-level game-play. Additionally, we demonstrate the utility of Mul- tiCriticAL. in a use-case more reflective of real-world application, in an experimental build for EA's UFC Knockout Mode, where we train policies to act with multiple specific and distinct behavior styles. This work is also amongst the first to study the efficacy of MTRL algorithms across broad range of multi-style problems, an aspect of MTRL receiving limited prior attention.
PRELIMINARIES
The core reinforcement learning (RL) problem is framed as a finite-horizon Markov decision pro- cess, (S, A, P.r,y), with state transition dynamics P S.A + S, for states E S and actions A, and reward function r S.A + R. The RL objective to train policy, : S A such that the expected sum of discounted rewards, E,, [Σ, 7"rss, az)]. with discount factor is max- imized for all trajectories, Tr, sampled on policy æ. RL typically assumes that both the transition dynamics and reward dynamics encompassed by black-box environment. RL agents interact with the environment to observe the state response and generated rewards, but where specific reward functions and transition dynamics are not assumed known. Solving the RL optimization problem requires estimating expected rewards and using these approximations to optimize the action policies.
Among the different techniques for tackling RL optimization, we mainly consider Actor-Criti meth- ods, whose key characteristic is their separation of the 'actors', representing the RL policy, and the 'critic', representing the value-function estimator that learns estimate future rewards. This sep-
2

---

Published as a conference paper at ICLR 2022
aration allows Actor-Critic RL to take advantage of the improved sample-efficieng of value-based techniques, such as Q-learning (Watkins & Dayan 1992) while taking advantage of policy-based techniques, such as policy gradients (Sutton et al. 2000), for improved robustness to stochasticity and to enable learning in continuous action domains. Examples of prominent contemporary algo- rithms in this class of RL methods include TRPO (Schulman et al., 2013), PPO (Schulman al.)
2017), DDPG (Lillicrap et al. 2016), TD3 (Fujimoto et al.) 2018), and SAC (Haarnoja et al. 2018))
In algorithms based on Q-learning, such as DDPG, TD3, and SAC, the policy optimization criteria Jxg for a policy, π, parameterized by 0, is proportional to the Q-value:
(1) where s' is the next state reached from state s in response to action o, and R(r) is the discounted sum of rewards over trajectory т. For algorithms based on a more traditional policy gradient optimization, such as TRPO and PPO, policies are optimized proportional to the advantage function, A and a measure of policy divergence:
Juo & Q" (s, a) = E [R(TT.)) ) 9o = s, ag = aj = Eg [r(s, a) + +0 (s)s))
Jup & log(Te(a(s)) (s,a)) = (og(TT(alss) (Q VFo(ss)
(2)
where the faluecfunction VT(s) = E,, [R(T=)] so = s| = E, ((s) + +~V= For critic networks parameterized by ø. either a state-value estimator, Vs Q-value (state-action-vallue) estimator, Qo are optimized with optimization criteria Jvz or JØz respectively:
Jvz x VI (s) - (r(s, + (s' ~ P(s, (())x( Joz x 110 a) - (r(s,a) + Q3(s' ~
(3)
(4)
While specific details around how actors and crities are optimized vary between algorithms, the above represents the fundamentals of optimization criteria upon which we shall build our discussion.
3 RELATED WORK
We broadly categorize multi-task RL (MTRL) techniques as either single-policy, if the same policy function activations are used to determine actions over all tasks, or multi-policy, if more separate ac- tivations (in the case of nuuti-headed policies) or policy functions are used. Multi-police techniques range from training multiple policies and selecting or mixing between them (Andreas et al.) [017]] Peng et al. 2018 Dewangan et al. 2018 8, Yang et al. 2017 Huang et al., 2017). to simultancously or iteratively training additional policies while sharing representation between policies (Rusu al,, 2015 Teh et al. 2017: Rusu et al. 2016). Techniques requiring multiple policy networks are limited by their storage and computational costs during inference, and multi-headed policies can limit our ability to smoothly transition between behaviors, which makes them less attractive in the context of RL for game-play. Therefore, we focus instead on pongle-policy learning.
Single-policy MTRL techniques train single policy function to solve multiple tasks, without relying on information from previously trained policies. Model-free meta-learning techniques have been proposed improve multi-task generalization, in the absence of task specificity (Finn et al.) 2017 Nichol et al. 2018), but where explicit task distinction can be exploited, a common approach has been exploit task encoding to distinguish between tasks. A simple, yet effective approach this problem has been to extend state representations with one-hot task encoding (Peng et al. 2018 Yu al. 2020 Yang et al. 2020), thus allowing policies to distinguish between individual tasks and decide actions in a aesk-appropriate way. Recent work has also sought to enhance the performance and generalizability of such single-policy techniques through automated task encoding (Sodhani et al. 2021) or through noft-modularization (Yang et al.) 2020), which encourages the development of distinct information pathways per task within actor networks.
Our work considers the special MTRL case of multi-style learning. Multi-style learning requires agents to learn different behaviors for highly similar states while providing potentially highly dis- similar rewards for each style. This variability in the reward signals can have adverse impact on the quality of learned sufficiently distinguish between styles. We hypothesize that negative interference more strongly im- pacts funutiinn learning in multi-style tasks as task similarity may make specific style-values harder distinguish. Multi-style learning is however aspect of MTRL that has not received much
when using a single value estimator if the estimator cannot
3

---

Published as a conference paper at ICLR 2022
prior consideration. Peng al. (2018) explore the most similar setting to work where agents were required to learn different movement styles animation tasks. Their work tested single-policy MTRL and found that policies were only able to learn a limited number of styles, requiring the authors to resort to a multi-policy approach.
Most works considering the Actor-Critic MTRL problem focus predominantly on how the actors, ie the policy-functions, are optimized, with less attention given to critics. The problem of negative interference between tasks in the broader context of MTRL one that has been highlighted and studied before (Yang et al. 2020; Parisotto et al. 2015 Teh et al. 2017 Standley et al., (020) however, but has similarly been approached as policy optimization problem. A number of multi- policy techniques were noted to maintain separate valuations per policy (Teh et al.] 2017; Andreas et al. 2017 Peng et al., 2018), though some work has also suggested that sharing critics can allow optimization to better capture useful shared knowledge to improve policy optimization (Dewangan al. 2018 Yang et al.) 2017). similar idea dominates single-policy techniques, where all the
studied methods maintain single value-function estimator across all tasks (Nichol et al. 20018] Yu
cct al. 2020; Peng et al. 2018 Yang et al., 2020; Sodhani et al. 202T). While task-encoding may allow separate talkesalues to be learned implicilly by the critics, this never found to be explicitly designed. Our work alternatively frames the multi-style learning framework as one that explicitly learns distinct per-style values to propose a value-learning optimization scheme that is better able to accommodate disparate reward functions.
MULTI-CRITIC ACTOR LEARNING
The policy functions, i.e. the 'actor' part of Actor-Critic frameworks, represent the core decision- making processes of RL agents and the success or failure of any algorithm rests on the policies' abilities to learn useful mappings. The learned crities have limited utility after training and learned value-function estimators are also known for being poor approximations of the true value-function they attempt to capture (Ilyas et al., 2020), so it makes sense that prior work has focused more on how policies are constructed and trained. The limitations of value learning are however exactly why the specifics of velue-network constructions and optimization are important to consider. A critical element in the performance of RL algorithms be the representational capacity of the learned malue-functions (Mysore et al.) 2021a AAnryychowccz et al.] (02T). It, therefore, stands to reason, that better structuring how value-networks represent the learned values can be important component of improving palue-function learning and therefore policy performance.
A common formulation of single-critic MTRL optimization involves including a task-encoding, Ω to distinguish between learned tasks and to specify task rewards and updating the state representation (s, S2. The optimization criteria in Equations I are correspondingly updated to use in place of s. The resulting MTRL optimization criteria typically represent rewards and learned values by scalar values. Practically, the value- ffllee-fnntions are learned by a single network with a single output node regardless of the number of tasks. A single learned value-function representing multiple values may enforce continuity between the learned values, which may not necessarily reflect the true value functions and could compromise the quality of learned values and resulting policies.
Value Optimization for Multi-Critic Actor Learning By separating the learned values per style, assumptions of continuity between style-values can be dismissed and style (or task) values can be learned explicitly, without relying on the crities to inherently learn the distinctions. We propose Multi-Critic Actor Learning (MultiCriticclI) with style-conditioned value optimization criteria:
Jvgj E,4;
(r:(s, (8)) + (z P(s, (5)]] Euu [O% (s, a) - (ri(s, a) 27, (s' ~ a(s.a), n ((5)))]
V
(5)
J JQy l2
x
(6)
where, for M styles, the task/style encoding, Ω wwo..... wM] and wi E [0, 1]. The policy op- timization criteria requires the value functions to be continuous with respect to the states and task encoding, and this property can be maintained by appropriately selecting the value functions when computing the policy optimization criteria. The specifics of the exact formulation of the optimiza- tion criteria would vary in accordance with the base Actor-Critic algorithm employed, with Equa-
4

---

Published as a conference paper at ICLR 2022
tions 1 and respectively being updated as:
Ina Enir or Jm & (u,
Practically, multi multi-valued critics may be represented by separate independent critic networks multi- network (MN) MultiCriticAL- - or single base critic network with multiple heads, where each head represents a different learned value multi-headed (MH) MultiCriticAL. While the latter allows for some learned representations be shared between the value functions, the former may be more ben- eficial when sharing is likely to lead to undesirable interference. This is visually shown in Figure] and extended Figure 14 in Appendix C
Relationship to MTRL and MORL DQNs MultiCriticAL shares theoretical underpinnings with existing DQN techniques tackle MTRL and multi-objective RL (MORL) by learning task/objective-specifee values (Parisotto et al. 2015 Moffaert & Nowē 2014: D Eramo et al.) 2020 Rusu et al., 2016). Learned DQN policies and value functions share single combined architectural backbone, forcing value estimation machinery to be retained at inference time. By extending the the- ory to Actor-Critic methods, we both enable utility in continuous control and the practical benefits of disentangling the representational complexity of the actors from the often greater representational burden placed on the crities MMysore al. 2021a). (Additional discussion in Appendix B
5 EVALUATION
We evaluate the proposed MultiCriticAL. method in series of multi-style RL problems. We first test our method on two basic multi-style learning environments designed as simple and light- demonstrations of the types of problems seek to address: (i) learning to follow simple shapes, where different styles correspond to different shapes, (ii) playing a modified game of Pong where the player's paddle is allowed to rotate and agents are trained play aggressively or defensively against a stock AL. While relatively simple, these environments help highlight the efficacy of MultiCriticAL over the more typical single-critic MTRL framework. Additionally, ing to play multiple levels of the Sega genesis games Sonic the Hedgehog, and Sonie the Hedge- hog where the agent is trained play the different levels of each game. following the reward structure devised for the Gym Retro contest (Nichol et al. 2018). Finally, we demonstrate a prac- tical use-case for MultiCriticAL in an experimental build of EA's UFC Knockout Mode fighting game where RL agents are integrated into the game controls and trained with different fighting styles. Appendix [A provides learning curves and additional visualizations for the results discussed in this section.
test MultiCriticAL.
learn-
Baselines We train MultiCriticAI. with two popular contemporary actor-critic algorithms: PPO (Schulman et al.) 2017) and SAC (Haarnoja et al.) 2018), which represent Actor-Critic al- gorithms based on both value iteration and Q-learning respectively. These are compared against Multi-task PPO (MTPPO) and Multi-task SAC (MTSAC), which augment the state information with a one-hot task encoding and were also proposed as baselines by Yu et al. (202). While we recognize that these are not state-of-the-art MTRL algorithms, they share common critic opti- mization structure with more recent advancements in policy (actor) optimization (Sodhani et al.) 2021 Yang et al. 2020), and we aim demonstrate the fundamental utility of MultiCriticAl. in the absence of additional policy optimization tricks. Training code is based on OpenAl's '' Spinning Up (Achiam| 2018) and provided in the Supplementary Material for the multi-style environments. As the Sonic the Hedgehog games require paid access and the development environment for UFC is proprietary, game code is not provided, though Appendix detaiis how they were adapted.
Seeding To account for the variability in RL, performance is averaged over 15 random seeds.
5.1 PATH FOLLOWING
Environment Design and Reward Shaping In the path following environment, given its current position in space, the current phase of motion E [o, 1], and an indicator on which shape to reproduce, the agent is expected to follow trajectory that generates the desired shape. Three shapes are tested:
5

---

Published as a conference paper at ICLR 2022
SAC
PPO
duted NAMMOORSAL ME
OSR UHA IN RICHEAN PHARLBOTTOAL
Figure 2: The path-following task requires agents to learn a Circular, Square, or Triangular path. MultiCriticAL agents can learn follow all 3 shapes with better performance compared to the more typical single-critic multi-task value-learning frameworks of MTSAC and MTPPO.
Circle, Square and Triangle. This simple problem adds no additional complexity related to gener- alization or complex system dynamics and simply expects policies to learn mapping function that encodes the desired shape. The multi-style objective is for the agents to accurately produce any of the three required shapes, given the task encoding. Agents between the agent's current and desired position in space. The rewards are cast to be positive in the range of [0, 1] per step. Appendix offfrs further details, with code in Supplementary Material.
rewarded based the absolute error
The pathing behavior learned for each algorithm over the three shapes is shown in Fig- ure 5.1 While single-style SAC and PPO are able to learn follow each shape, performances of MTSAC and MTPPO in the multi-style setting are compromised. This is most prominent with MT- SAC. MTPPO fares better but performance in the multi-style setting is still notably worse. Adapted to both algorithms, MultiCriticAL more consistently learns to follow all three shapes. It should also be noted that larger networks were needed for the PPO algorithm to solve this task, using two hidden layers with 32 neurons each, as opposed to SAC's 8 each. This likely contributes to the im- proved performance of PPO in the multi-task setting as the networks being used have higher relative representational capacity, but are still outperformed by MultiCriticAL.
Results
5.2 PONG WITH ROTATABLE PADDLES
Environment Design and Reward Shaping We use a modified game of Pong to explore a multi-style learning problem where one style is represented by sparse reward, and the other a dense reward, anticipating that this might be case where single-crite falue-function estimation may fail. Pong typi- cally allows only single degree of freedom for paddle movement and does not normally command nuanced strategies. Adding an extra degree of free- dom by allowing the player's paddles rotate introduces increased complex- ity to the game and facilitates the training of different behavior styles, For this game, we consider two primary styles of play: (i) aggressive with the objective of winning the game as quickly as possible, and (ii) defensive where the objective is to avoid losing while prolonging the game as long possible. In a real game setting, these could be analogous to hard- and easy- mode Al respectively, where the former tries give the opponent challenge and the latter may help the opponent learn to play. Our implementation is based on the Pong environment provided by the Pygame Learning Environ- ment (PLE) (Tasfi 2016). For simplicity, the basic opponent AI cannot rotate its paddle and uses heuristic-based Pong controller provided by PLE. The Aggressive style rewards winning as quickly as possible while the Defensive style encourages avoiding losing and ensuring that the opponent can keep receiving the ball to prolong the game (additional details in Appendix D)
Figure 3: Behav- ior samples for Ag- gressive and Defen- sive play in Pong.
Results We first established baseline performance by training SAC and PPO on the Aggressive and Defensive play-styles. Figure 3 shows samples of play strategies adopted for each style, where the Aggressive agent exploits paddle rotation to bounce the ball off the side-wall to outmaneuver the opponent. Conversely, Defensive agents attempt to return the ball directly to the opponent, exploiting the paddle rotation to angle the ball back towards the opponent. As desired, aggressive play yields high win-rates, with plays lasting significantly shorter than defensive play, which would win approximately half of the games but play longer games (videos in Supplementary Material).

---

Published as a conference paper at ICLR 2022
Table 1: Learning different play-styles on Pong with rotatable player paddles
Agent
Single style
MN MultiCritcAA.
MultiCriticAA
Aggressine Setting - Style = (1. 0)
Keward f
0.80 = 0.06
0.20 = 0.36
0.85 +: 0.01
0.80 + 0.03
Win-rate f
94 + 2%
10 + 13%
80 3%
84 = 8%
SAC
Play-time
280 = 102
170 =:
253 35
291 + 55
Delensive Peiennive--eitiing Style (0, 17
Reward f Win-rate
162.75 1 12.15
166.05 1 13.04
176.80 1 8.50
165.59 + 13.01
51 3%
50 5%
30
5%
12%
Play-time
1536 1: 113
1571 d: 122
1661 4: 53
1560 4: 154
Aggressive-Setting Style
Reward f
0.70 = 0.12
0.30 = 0.36
0.83 + 0,08
0.70
Win-rase f
81 + 10%
32 + 23%
79 2%
15%
PPO
Play-time
586 + 98
241 111
483 44
330 + 98
Dejenstve-Netong - Style [0, 1;
Reward T
101.35 17.90
35.86 16.08
92.75 1: 12.30
72.60 14.06
Win-rate
54 + 7%
41 = 3%
40 3%
61 = 8%
Play-time
726 121
Table 2: Interpolating between aggressive and defensive Pong play-styles - Style (0.5, 0.5)
SAC
PPO
Agent
MN MulMiiiiCL
MH MuliCriccA
MulMulliCLLiAAL
MH-MulliCricAL
Training with Binary Style Selection
Win-zate Play-time
=
64 = 13%
64 = 4%
78 = 5%
770 + 148
765 + 113
615 + 164
541 + 67
Training with Explicir Style Interpolation
Win-rate
73 = 6%
72 8%
71 = 3%
70 2%
Play-time
834 + 108
793 + 120
621 + 158
602 + 106
Multi-style performance is tested by introducing a one-hot style encoder and randomly uniformly sampling either defensive or aggressive play for each new training episode. We observed that both MTSAC and MTPPO failed to learn to properly distinguish between play-styles. MTSAC agents often defaulted to highly defensive play (likely due to the over-representation of the dense defensive rewards in the replay buffers) while MTPPO agents tended to learn intermediate behavior that was neither aggressive nor defensive. MultiCriticAL. agents were however able to successfully learn to distinguish between the styles with both base algorithms, with performance comparable to single- style agents. Table and Figure A.2.T show performance statisties averaged over 20 test episodes per seed for each algorithm.
We also test how MultiCriticAL agents would respond to changes in the style encoding at test time. As shown in Table 2 when provided intermediate (0.5,0.5) encoding, the agents presented with behavior statistics that were almost halfway between that of their aggressive and defensive statis- tics. This even without explicit conditioning, when agents were trained with purely binary style indicators. When explicitly interpolating between styles in training (by providing a multi- dimensional style-reward, with aggressive and defensive rewards scaled by their encoding weight), both MultiCriticAL. SAC and PPO agents yielded performance statistics that more closely matched the mid-points between their aggressive and defensive behaviors.
5.3 SONIC THE HEDGEHOG
Training Setup To test MultiCriticAL in more established multi-task setting, we tested two Sonic the Hedgehog games, using the Gym-retro wrappers designed for the games during the Gym- retro contest (Nichol et al. 2018). using the same setup configuration developed for the competition. MultiCriticAI. is set up to maintain separate critic for each trained level. Sufficient experimen- tation for statistical rigor on this benchmark required hardware parallelism, which our code only supports for PPO. SAC is therefore omitted for this benchmark.
Results Performance on 17 of the main levels for each game, averaged over 10 tests per agent, per level, are presented in Figure A.2.2 with per-level tabulated numerical rewards provided in Ap- pendix A Sonic the Hedgehog appears to be a game where learning shared representations over multiple levels helps overall performance, for both MultiCriticAL and MTPPO, though this obser- vation does not hold for Sonic the Hedgehog 2, where MTPPO agents perform worse on average than agents trained on single-levels. MultiCriticAL agents tend to perform comparable or better than independently trained and MTPPO agents in almost all cases, however, with 35% and 20% improvements on Sonic the Hedgehog and 56% and 40% improvement on Sonic the Hedgehog 2 of MN-MuliiCriiiAAl and MH--MultiCrtticAL over MTPPO respectively.
7

---

Published as a conference paper at ICLR 2022
prrprterties grmmm giver Mernaacces
Figure 4: Evaluated over two Sonic games with 17 training levels per game, where levels function similarly but have significantly different layouts and visual design, MultiCriticAL offers consistently superior performance to the single-critic baseline and achieves comparable, if not better performance to agents trained specifically for each level - additional data and visualization Appendix A.2.2
Pogressivve
||
Defensive
Neuttal
Moviigg Blocking
Figure 5: We show 8 seconds of play with one representative frame per second for the different styles of play for the MultiCriticAL agent trained to play UFC. The RL agent has black hair while the stock Al has white hair. All styles are captured for the same RL agent, trained to play with multiple behavior styles. As desired, the agent more frequently goes on the offensive when set to Aggressively. prioritizes constant blocking and avoiding damage on Defensive, standing its ground and blocking incoming attacks Blocking and moving around the ring for Moving.
5.4 UFC ENVIRONMENT
For use case more reflective of real-world application, we trained AI fighters in an experimental build of UFC Knockout Mode. gym-like (Brockman et al. 2016) wrapper was built for the game environment enable an RI. agent to play against the game' stock AI. Note that this is not a feature of the commercially released game but is a modification made for research purposes.
Environment design and Reward Shaping The RL agent observes information about the player's and opponent's position, velocity, health, and actions being taken, stacked over 8 steps. The RL- agent is able control 11 basic discrete high-level actions including forward and backward move- ment, high or low blocking, left or right attacking (with straights, uppercuts, or hooks), and is also allowed take no action. The RL-agent conditioned over 2 sets of 2 playing styles: aggressive or defensive and blocking or moving. During training, aggression vs. defensiveness, and blocking vs. moving are set on [0, 1] scales with (0.5, 0.5) representing neutral style which is meant to balance behavior from both scales. For all styles, the agent is rewarded for winning and is penalized for every strike request. For the aggressive style, the agent is additionally rewarded for reducing opponent health. For the defensive style, the agent gets small positive reward for each step that remains alive, is penalized for losing health, and is rewarded for reducing opponent health (though not much as with the aggressive style). For the blocking style, the agent is rewarded for every successful block, and for the moving style, the agent is rewarded for maintaining a positive velocity. Details are provided on the state and action spaces, reward tuning, and training in Appendix [D.]
8

---

Published as a conference paper at ICLR 2022
Table 3: MTPPO vs. Multi-headed MultiCriticAI-PPP on UFC
Dominum Style
Style Settin
Normalized Per Action Style Rewand x10"
MIRO
MIl-CulCCTacAL PPO
Aggression
Blocking
Moving
Appresion
Deiendiennes
Blacing
Moving
Aggressive
(0.0
7.08
0,00
0.10 + 0.05
1.17 + 0.15
7.16 1.61
0.00
0.05 + 0.02
2.19
0.21
Defencive
(1..0.5)
0.00
20.08 1 0.30
4.31 1 0.41
1.19
0.00
20.80 4 0.07
4.27 4 0.37
2.33
0.17
Nestral
(0.5, 0.5)
2.84 = 0.90
14.15
0.74
1.32
4 0.50
1.93 4 0.07
2.16 2 0.15
10.80
= 0.12
1.68
4 1.68
2:24
0.15
Blocking
(0.5.000
2.25
3.10
8.78.
0.00
+ 1.14
11.56
1.32
1.97
2.32
0.00
Moving
[0.5.100 3.07
= 1.01
14.78
1.13
0.00
4.19 2 0.43
1.75 1.25
10.13 = 0.88
0.00
4.50 = 0.20
Results MultiCriticAL allows PPO agents to successfully learn and smoothly transition between the different fighting styles. The behavior styles are distinguished visually and Figure attempts to capture this through frames from captured gameplay with videos included in the Supplcmentary Material. Reward statistics for each style are presented in Table 3 and Figure A.2.3 Aggressive and defensive plays are the most distinct but there are also observable differences between control that prioritizes blocking and moving, with the blocking-styy while moving prompts agents to move around the ring more. When comparing MultiCriticAA. to MTPPO, we note that, while rewards at the extremes of style distinction are similar, MultiCriticAA, appears to be better at more consistently maintaining intermediate styles (when a setting is at 0.5). Visually, it appears that MTPPO also tended to favor blocking and defense over learning more nuanced behavior (Table 6
conditioning agents to hold their ground
5.5 DISCUSSION
While explicit separation of the learned value functions per style under the MultiCriticAL framework offers demonstrable and consistent improvement over the single-critic basclines, it must be noted that MTSAC and MTPPO baselines do not necessarily reflect the state-of-the-ar in MTRL, but they do reflect how value optimization is typically handled in contemporary literature. It is likely that utilizing some of the improvements to policy optimization techniques discussed in Section 3 may contribute to improved performance of policies trained using single-critic calue-optimization, how- ever the same may also further improve the performance of agents trained with MultiCriticA.I and remains to be studied. Curiously, MN-MultiCriticcAL consistently outperformed JIH-MultiCriticAAl on all the tested tasks, which calls into question the idea that shared representation important and/or useful (Dewangan et al., 2018 Yang et al., (017) and is an aspect that may warrant further study, An interesting extension could to extend modularization (Yang et al. 2020 Andreas al,, 2017) to value-function optimization. Learning with curricula (Bengio al. 2009; Graves et al. 2017 Mysore et al. 2019; Andreas et al. 2017) has also been demonstrated to (sometimes dramat- curricula also
like potential natural fit for multi-critic optimization. Recent work (Mysore et al., 2021a la Andrychowicz et al., 2021 Ilyas
ically) improve RL performance and multi-style/tan
et al. 2020 has also demonstrated that value function capacity be key bottleneck in RL perfor- mance. Increasing the network capacity of a single-critic falue-function may improve the efficacy of the critic, but this is not guaranteed and there are few principled ways determine what a sufficient network capacity may be. Significantly increased computational resources may be required for the lcarnabllity of the multi-style problems under a single-crite framework. MultiCriticAAL also requires increased compute during training, particularly CL--MultCrriticAL, but MH MH-MultiCriticAl allows for a bounded increase computational load (< 3% as tested- see Appendix B) with an acceptable compromise in relative performance while still outperforming single-critic value learning.
6 CONCLUSION
We study Actor-Crine optimization in multi-style RL problems, where agents are required to behave with different styles the same environments, our work represents one of the first explorations of the application of RL to a wider array of multi-style problems. We hypothesized that limitations of single-actor, single-critic MTRL frameworks may result in interference between the reward sig- nals for different styles due to high similarity in visited states and to mitigate this, proposed Multi-Critic Actor Learning (MultiCriticAAL), which separates critics into per-task value-function estimators, thus avoiding interference between style rewards and reduces learning complexity by not requiring critics to learn to distinguish between tasks. MultiCriticAL consistently outperforms the single-critic MTRL baselines, achieving up to 56% improvement in multi-task performance and successfully learning multiple behavior styles where single-critic methods would fail.

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
As noted in Section training code is based upon the open-source OpenAl Spinning Up code- base (Achiam 2018). with modifications made to support MultiCriticAL. for PPO and SAC. The modified code-base is included in the Supplementary Material for this paper, along with the code for the custom environments for Path Following (presented in Section 5.1) and Pong with rotatable paddles (presented in Section 5.2). The README files included with the code detail installation and use of the code. For the Sonie the Hedgehog games, reproducing results would require acquiring the games' ROM files and linking them to an install of gym-retro, as detailed by Nichol et al. (2018) Also included in Appendices D and D are additional details for the setup of the training environments discussed in the main body of the paper as well as training configurations used. We recognize that tests on the UFC environment may not be independently reproducible due to requiring access to proprietary development software and we therefore mainly discuss it as demonstration of the practical utility for our proposed MultiCriticAL method, rather than as a benchmark task.
REFERENCES
Joshua Achiam. Spinning Up in Deep Reinforcement Learning. 2018
Eloi Alonso, Maxim Pete, David Goumard, and Joshua Romoff Deep reinforcement learning for navigation in aaa video games. NeuriPS Workshop on Challenges of Real-World RI., 2020.
Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. International Conference on Machine Learning, pp. 166-175. PMLR, 2017.
Marein Andrychowiez, Anton Raichuk, Piotr Stańezyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, Leonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, and Olivier Bachem. What matters for on-policy deep actor-critic methods? large-scale study. In International Conference on Learning Representations, 2021.
S. Ariyurek, A. Betin-Can, and E. Surer. Automated video game testing using synthetic and human- like agents. IEEE Transactions on Games, 13(1):00-67, 2021. doi: 10.1 21110/99.22999997557777
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference machine learning, pp. 41-48, 2009.
Christopher Berner et al. Dota 2 with large scale deep reinforcement learning. CoRR, abs/1912.06680,, 2019.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv:1606.07 0154000000050, 2016
Carlo D'Framo, Davide Tateo, Andrea Bonarini, Marcello Restelli, and Jan Peters. Sharing knowl- edge in multi-task deep reinforcement learning. International Conference on Learning Repre- sentations, 2020. URL
rkgpv2VFvr
Parijat Dewangan, S Phaniteja, K Madhava Krishna, Abhishek Sarkar, and Balaraman Ravindran. Digrad: Multi-task reinforcement learning with shared actions. arXiv preprint arXiv: 100,10633,, 2018.
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https: 2017.
//qithub. www.:/ppeaaa/aaseeeieooo..
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leamiig for fast adaptation of deep networks. In International Conference on Machine Learning, pp. 1126-1135. PMLR, 2017.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning, pp. 1587-1596, 2018.
Linus Gisslén, Andy Eakins, Camilo Gordillo, Joakim Bergdahl, and Konrad Tollmar. Adversarial reinforcement learning for procedural content generation. IEFE Conference Games, 2021.
10

---

Published as a conference paper at ICLR 2022
Camilo Gordillo, Joakim Bergdahl, Konrad Tollmar, and Linus Gisslén. Improving playtesting coverage via curiosity driven reinforcement learning agents. IEEE Conference on Games, 2021.
Alex Graves, Marc G Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu. Automated curriculum learning for neural networks. In international conference on machine learning, pp. 1311-1320. PMLR, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. International Conference on Machine Learning (ICML), 2018.
Hado Hasselt. Double q-learning. Advances in neural information processing systems, 23:2613- 2621, 2010.
Zhewei Huang, Shuchang Zhou, BoEr Zhuang, and Xinyu Zhou. Learning to run with actor-critic ensemble. arXiv preprint arXiv: (7t.:::77200888777 2017.
Andrew Hyas, Logan Engstrom, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. A closer look at deep policy gradients. In International Con- ference on Learning Representations, 2020.
Timothy Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. Interna- fional Conference on Learning Representations, 2016.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 518(544))529-5333 2015.
Kristof Van Moffaert and Ann Nowé. Multi-objective reinforcement learning using sets of pareto dominating policies. Journal of Machine Learning Research, 15(107):3663-369, 2014. URL
http::/ fmwr wwtpg//aper//aaeeee wwwpaaaarrsa.//.aaaaaaaapa http://////aaoofaaaa..aa...
Siddharth Mysore, Robert Platt, and Kate Saenko. Reward-guided curriculum for robust reinforce- ment learning. In Workshop on Multi-task and Lifelong Reinforcement Learning at ICML, 2019.
Siddharth Mysore, Bassel Mabsout, Renato Mancuso, and Kate Saenko. Honey, shrunk the actor: A case study preserving performance with smaller actors in actor-critic rl. IEEE Conference on Games, 2021a.
Siddharth Mysore, Bassel Mabsout, Kate Saenko, and Renato Mancuso. How to train your quadro- tor: A framework for consistently smooth and responsive flight control via reinforcement learning. Systems (TCPS), 5(4): 2021b.
ACM Transactions on
Alex Nichol, Vicki Pfau, Christopher Hesse, Oleg Klimov, and John Schulman. Gotta learn fast: A new benchmark for generalization rl. arXiv preprint I 003700, 2018.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems, 29:4026 4034, 2016.
Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimie: Deep multitask and
transfer reinforcement learning. arXiv preprint 1rXiv:151..0633 2015.
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic: Example- guided deep reinforcement learning of physics-based character skills. ACM Transactions on Graphics (TOG), 3-14) 2018.
Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell, Policy distil- lation. arXiv preprint arXiv:1511.06229. 2015.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv: arXiv:1606.0444 04676, 2016.
11

---

Published as a conference paper at ICLR 2022
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning, Lille, France, 07-09 Jul 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. 1oxivv1707.0660 2017.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with general reinforcement learning algorithm. arXiv preprint arXiv: 2017.
Shagun Sodhani, Amy Zhang, and Joelle Pineau. Multi-task reinforcement learning with context- based representations. arXiv preprint arXiv:2102.06177 2021.
Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? In International Conference on Machine Learning, pp. 2122-911.. PMLR, 2020.
Denis Steckelmacher, Hélène Plisnier, Diederik Roijers, and Ann Nowé. Sample-effiiiet model- free reinforcement learning with off- off-polliy critics. arXiv preprint 0rXiv:1903.041999 2019.
Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in neural informa- fion processing systems, 2000.
Norman Tasfi.
Pygame learning environment.
http:://github.com//stttt
2016.
Yee Whye Teh, Victor Bapst, Wojeiech Marian Czameeki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. arXiv preprint arXiv: 10X.:0177..04755 2017.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforeement learning with double q- learning. Proceedings of the AAAI conference on artificial inselligence, volume 30, 2016.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czameeki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev. et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782)350--544 2019.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pp. 1995-2003. PMLR, 2016.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4):279-23) 2922 1992.
Ruihan Yang, Huazhe Xu, Yi Wu, and Xiaolong Wang. Multi-task reinforcement learning with soft modularization. arXiv preprint arXiv:2003. 13661, 2020.
Zhaoyang Yang. Kathryn E Merrick, Hussein A Abbass, and Lianwen Jin. Multi-task deep rein- forcement learning for continuous action control. In IJCAI, volume 17, pp. 3301-3307, 2017.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta- detaawworl: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, pp. 1094-1100. PMLR, 2020.
Y. Zhao, I. Borovikov, F. de Mesentier Silva, A. Beirami, J. Rupert, C. Somers, J. Harder, J. Kolen, J. Pinto, R. Pourabolghasem, J. Pestrak, H. Chaput, M. Sardari, L. Lin, S. Narravula, N. Aghdaie, and K. Zaman. Winning is not everything: Enhancing game development with intelligent agents. IEEE Transactions Games, 12(2), 2020. doi: 10. wtt.T/...2...999990
12

---

Published as a conference paper at ICLR 2022
A ADDITIONAL DATA AND RESULTS
A.1 LEARNING CURVES
A.1.1 PATH FOLLOWING
Figure 6: Learning Curves for all tested SAC variants on the Path Following benchmark.
Figure 7: Learning Curves for all tested PPO variants on the Path Following benchmark.
A.1.2 PONG
Pong
Figure 8: Learning Curves for all tested SAC variants on the Pong benchmark.
13

---

Published as a conference paper at ICLR 2022
-221
Figure 9: Learning Curves for all tested PPO variants on the Pong benchmark.
A.1.3 SONIC THE HEDGEHIOG
sudgetin Asttge Leaning Garva
Hodgalog Anerggo Carce
Okg save
300
125
Average Learning Curves the Sonie the Hedgehog benchmark games. Represented here is the averaged reward over the 17 levels of the game as training progresses. We chose this representation in the interest providing a more compact representation instead including 34 separate figures, with one for each level, as that would both require a lot of space and be more difficult to parse.
Figure 10:
14

---

Published as a conference paper at ICLR 2022
A.2
ALTERNATIVE RESULTS REPRESENTATION
A.2.1 PONG
PONG AGGRESSIVE REWARDS
PONG AGGRESSIVE EPISODE LENGTH
15
1530
1400
1230
0.5
1000
800
0.5
600
400
-3.5
SAC
FPO
SAC
PPO
PONG DEFENSIVE REWARDS
DEFENSIVE
EPISODE LENGTH
10 Single
MT w/ ont-hot
MN MutiCHiiAL
Mth Mu MCIISAAA
inyoestye
MT wf one hox
WWMMuh REAL
MH MutiiriAA
2000
1800
1400
150
1200
100
200
Figure 11: Box and Whisker plots for Pong Results presented in Section 5
A.2.2 SONIC THE HEDGEHOGG
Table Rewards achieved Sonic the Hedgehog (1)
Level
Single-levei Policy
MTPPO
MN-MulsCCiticAL
MH-MMlliCrnkcL
GHZI
37.41 + 13.81
38. 38.70=2 = 23.41
42.30 = 5.32
40.72 = 6.04
GH72
GreenHHllZoneAcz2
17.60 + 9.66
26.40 + 10.62
42.26 + 2.07
32.63 1 6.01
GHZ3
GreenHilZZoneAN
7.22 4.07
9.37 = 6.58
38.85
25.18
12.02 5.54
MZ1
MarbleZone.Actt
28. + 5.45
26.20 + 11.30
39.93
13.30
40. + 8.45
MarbleZone.Aa
22.84
M73
MurhleZone.Acll
27.31 1: 2.36
22.64 1: 7.04
29.82 1: 0.68
28.50 1: 0.17
SYZI
Spring Spring YardZone Actl
2.73
10.02
9.16 - 5.14
SYZ2
Spring YardZone Act2
5.55 3.71
11.59 + 2.85
18.65 3.41
13. 6.06
SY73
Spring YandZone.Aat
19.67 +: 2.19
17.17 + 1.58
22.82 4.99
26.14 2.37
LZI
LabyrinthZone Act!
22.50
5.00
6.400
25.16
LZ2
LabytintZZoneeco
29.52 + 0.40
29.65 + 0.19
29.59
0.31
29. 16 1: 0.06
LabyrinhZane
25.70
0.13
25.82
SIZII
Starl Act!
35.03 + 0.51
35.15 1.06
12.38 47.80
39.79 + 4.38
SL.22
Aet
6.76 0.14
13.05 + 4.62
17.04 3.21
SLZ3
StuLieghiZene.Ai1
14. + 6.95
26.37 5.01
30.40 = 6.39
30.20 1.00
SBZ
ScmppraainZon..e Acti
8.03 1.65
10.73 + 4.52
9.92 + 4.13
11.68 + 6.11
SB22
Scmapraan/oon.Acct
8.54 1.27
Average
19.12 2.03
20.48 2221
27.39 2.65
21.73 14.33111 1.31
15

---

Published as a conference paper at ICLR 2022
Table 5: Rewards achieved Sonic the Hedgehog
Level
Single Policy
MullCCiii.i.
MII.MuliiCHR:AL
EHZ1
EmneraldHillZoee.Ao
18.98 : 10.01
43.72 + 17.08
65.82
15.48
44.03 = 6,07
BIZZ2
Eineralal Ac12
21.82 2.37
18.43 1 10.09
28.31
6.27
23.63 13.28
CP71
ChemicalPlamt7mme..e Actl
26.58 3.29
28.21 2.01
37.48
10.18
33.65 11.64
CFZ2
ChemiclPlantZone..
28.82 4.32
18.84 8.90
+4.60
27.01 5.00 5.00
AquaticRuinZene.Aci
24.28
ARZ2
AquniicRinZZoneAccii Act2
37.40 + 6.89
24.12 2.85
39.41 7.30
33.15 10.23
CNZ
CasinoNighiZonee
17.12 0.97
12.02 3.51
20.34
13.65 = 1.99
HTZ1
HiilTTopZone.Ac
7.82 0.29
6.36 2.72
7.78 = 0.85
5.65 0.57
HTZ2
HiilTTapZoneAatz)
6.44 4.23
4.22 2.81
19.89 2.96 2.96
21.30 3.08
MCZI
MysticCaveZone Actl
10.25
7.42 1.14
9.74 1.44
MCZ2
MystioCaveZone. Act2
8.57 2.31
5.00
8.75 = 3.32
11.96 = 1.59
OflOceanZZone.Aee
O0Z2
OilOccanZone.Aoe Act2
0.90
9.92
13.72 - 4.16
14.06 + 3.22
MZ1
MenopolisZone.Ae
14.64
6.07 1.94
14.39
2.58
14.03 3.52
MZ2
MeimpoliszaneeAcr
12.79 2.68
11.50 2.53
13.12
0.37
16.90
7.06
M73
Metmpolis7ane.Aee Act3
6.49 + 2.27
10.31 1.59
11.75
2.29
17.34 5.16
WFZ
WingFortressoone
27.43 17.3+0.0
0.44
8.98 24.81
5.53
Average
18.90 1.71
13.73 = 1.06
21
2.44
19.22 = 1.90
B tnltre ENTPO I MNAASRRCOO E AUROEMSI
SONIC THE HEDGEHOG
Figure 12: Sonie Results from Tables 4 ann visualized
A.2.3 UFC REWARD AND ACTION STATISTICS
Table 6: Action probabilities for MTPPO and MH-MultiCriticcL--PPO on UFC
Dominant Style
No-action
Strike
Black
Move
Win Rate (%)
MTPPO
Aggressive
3.80
32.99
3.60
59.61
96
Defensive
ties)
Neutral
7.48
20.97
24.12
47.42
100
Blocking
8.31
20.89
27.21
43.59
Moving
6.46
26.75
15.91
50.89
88
MH MultiCriticA.. PPO
Aggressive
15.78
Defensive
0.52
4.94
30.20)
64.34
12 (56% ties)
Neutral
(4% ties)
Blocking 0.49
14.88
23.92
60.71
88
Moving 0.59
8.70
20.88
69.82
64 (8% ties)
We observe from Figure A.2.3 and Table 6 that MultiCriticAL. more consistently learns delineated styles that consistent with the behaviors that the rewards try encourage. Note that, with method, it is, in fact, the aggressive style that wins most fights consistently and the defensive style learns block and move not lose as quickly, whereas MTPPO tends seems equate defensive play with just blocking, and this also contributes to higher rate of losing. We see as well that the intermediate settings correspond more balanced blend of action probabilities between extremes when compared to MTPPO.
16

---

Published as a conference paper at ICLR 2022
AGGRESSION SCORES
DEFENSIVENESS SCORES
MTMO
B KHH MuLiCH
MTTPO
BI MH MutSONIALL
0.20
0.025
0.012
0.008
0.015
0.23
0.004
Aggrosiiee
Deferaive
Noutral
Bledkng
Moxingg
Aggressne
Delena
Neatral
Blokkng
Moxing
Dominant style
Dominant
BLOCKING SCORES
MOVEMENT SCORES
MatSOIOUAL
MuRSCHOKAL
0.005
0.004
Momalted
0.000
Apd sseive
Deforsive
Nestral
Blocking
Moving
Asgressie
Dclonsive
Neatrol
Blocking
Moving
Dominast Style
Dominant Style
Figure 13: Box and Whisker plots for UFC style rewards presented in Section
A.3 PONG ACTION INTERPOLATION WITH MULTI-POLICY COMPOSITION
As an exercise in curiosity, we also consider what would happen we attempted to derive interme- diate styles by interpolating between actions in the Pong game. This was meant test approach similar to the multi-policy action composition technique used by Peng et al. (2018). Unlike in their case where actions were weighted according to each actor's estimated critic value, we instead di- rectly interpolate between actions proposed by pre- pre-trained aggressive and defensive agents. We take this approach the training for either style would not offer the value functions any meaningful information about the other unseen style(s), prohibiting the form of value comparison employed by Peng et al. We hypothesized that it possible and even likely that policies would destructively interfere with each other if the longer-term goals of constituent policies conflict with each other and this is exactly what was observed in testing. Except for intermediate styles very close to the learned extremes, interpolating between actions proposed by the aggressive and defensive agents caused near-complete failure in the composite policy's 's control.
Composite policies were built by pairing randomly selected pre-trained aggressive policy with a randomly selected defensive policy. On a (0.5,0.5) 'middle' setting. SAC composite policies achieved average win-rate of 30% + 5% (compared the approximately 95% and 50% of the constituent policies) with an average Play-time of 213 + 89 steps (often losing the game quickly). Similarly, composite PPO also achieves low average win-rate of 15% + 3% and similarly lose the game quickly. A high win-rate would not be expected, as the defensive policies are explicitly trained avoid winning, but both policy classes trained to avoid losing. Given the high loss-rate, composing policies in this way, for this multi-style task, can only be regarded as a failure. By con- trast however, as we showed in Table 2 MultiCriticAL capable of learning to interpolate between styles, both when and not explicitly trained to do so.
This experiment, by no means, constitutes exhaustive testing. and cannot be used to claim that all, or even most, forms of multi-policy policy composition would be ineffective and is therefore not included the main body of the paper. We do believe however that it helps highlight a cireumstance where policy composition can fail in cases where long-term strategies for different target behavior styles may interfere. Considered more generally, there are no guarantees that actions proposed by distinct policies would be similar enough, that weighted sum them would even make sense, as cooperation between policies is not something that is explicitly trained for. One might benefit from explicit conditioning, though this outside the scope of our work.
17

---

Published as a conference paper at ICLR 2022
B RELATING MULTICRITICAL TO DQNS AND MULIT- LEARNING
The idea of using multiple Q functions to guide the training of a single policy is not strictly new to the broader context of RL and has been used in notably single-task RL for improving the stability of learned value functions. These include techniques such as Double Q-learning (Hasselt, 2010
Van Hasselt et al. 2016). Dueling Q-networks (Wang et al. 2016). and Bootstrapped DQNs (Os- band et al.) 2016)] Steckelmacher et al., (019). There are also applications in multi-task learning, though we only encountered them in works considering DQN frameworks that, in effect, learn task- specific Q values for single policy in DQN architectures (Rusu et al. 2016: 2015) D'Eramo et al.) 2020). As DQNs jointly encode both the action policies and the value representation the same network, however, they do not share the practical benefits offered by Actor-Critic Critic methods (and con- sequently MultiCriticAL) in separating the actors and critics. We believe that, practically, this is a meaningful distinction DQN methods are (i) limited to discrete action spaces and (ii) need the full representational and computational complexity the value estimation to preserved at inference time after training, which computationally inefficient. As shown by /Adrychowiczz and Mysore et al. (2021a), oftentimes, it appears that the value function often bears the higher bur- den of required complexity to enable successful policy learning. Noting this, there significant practical ad advantage to being able to just train smaller, less computationally expensive actor-networks of actor-critic methods and disregard the value estimation networks at inference time. Given that our work is largely motivated by application to video games, where runtime resource use is an important consideration, this was an important distinction.
al. (2021)
C MULTI-NETWORK vs. MULTI-HEADED TRAINING FRAMEWORK OVERVIEW
Typical Critic RL
Multi MUSCHIAAL
Multi headed MuleCriicAL
Crittt
Stste Crites
Style Critic
Gritic Optimoatian
Multi-Criic
Multi-Crtic
Gprimbat
Oprimoation
Polliy Optimiuntion
Pollcy
Pollcy Optimuutt
Optimioation
Actor
Ervy
Actor
Actor
Figure 14: MN-MultiCriticAL and MHlliCtiCt frameworks compared. The main differ- ences between the multi-network (MN) and multi-headed (MH) is in how the critic functions ultimately represented by the neural network(s). the MN case, separate and independent net- works are maintained for each style/task value function, whereas the MH configuration allows value functions to share a common backbone and instead utilizes different network heads (i.e. output nodes) for each learned value. While the former appears to offer superior performance in most of the tested cases, the latter can be less computationally expensive and may incur reduced training costs. It should be noted, however, that neither architecture needs to impact the actor, thus preserv- ing run- run-time inference cost.
18

---

Published as a conference paper at ICLR 2022
D ADDITIONAL ENVIRONMENT SETUP DETAILS
D.1 PATH FOLLOWING
Observed State Agents observe 4-dimensional state with the current (z, y) coordinates of the agent in space and the current phase [0, represented by [sin(2px), cos(2pmr)
Goal generation Goals, y are generated based on the current phase of motion. They are as follows:
Circle: g(p)
cos(
Vp E [0,
0.5,
2 0.25
10.5,155 4pl
0.25 0.5
Square: s(p)
[2.5 4p, 0.5)
5
-0..5 3.5
0,75
[L.5p, 0.5 - 3p]
p ps 1/3
Triangle: g(p)
[1.5 3p. 0.5
1/3 S p 2/3
1.5 + 1.5p 1 -
2/35
Reward design For current state and desired state g: r= 1- 1- g/)l log(max(As)) where max(As) is the difference between the extents of the space bounds.
D.2 PONG
Observed State Agents observe 8-dimensional state with the current vertical y positions of both the player and opponent, the player's velocity, the paddle angle, and the ball's (x, y) position and velocity.
Reward design Aggressive play is conditioned with a sparse reward: +1 on win and -1 on loss. Agents are meant to learn that the best way to maximize cumulative discounted rewards is to win as quickly as possible. Defensive play is conditioned with dense rewards: -1 on loss, -1 for bouncing the ball against walls, +0.5 for receiving the ball, +0.1 if the opponent successfully receives the ball, and +0.1 I for every time-step of play. Defensive rewards encourage avoiding losing and ensuring that the opponent can receive the ball in order to prolong the game.
D.3 SONIC THE HEDGEHOG
Reward scaling As recommended by Nichol et 2018), we scale rewards by factor of 0.01 in order to encourage more stable learning with PPO.
D.4 EA UFC DEVELOPMENT ENVIRONMENT
Environment The UFC environment an experimental build of UFC. During training simula- tion, superfluous elements of the game, such audio, spectators, etc. are disabled. Socket com- munication is used to interface between the game-code and the python-based RL algorithms. The environment wrapper controls the game simulation and extracts and processes state information be- fore communicating it through the socket to the python RL code and interprets requests from the python code trigger inputs in the game. During training. agents trained for 240 epochs with 4000 sample interactions collected per epoch. Training is run in increments of 30 epochs to improve simulation and memory management stability. Episodes terminated when the agent runs the max episode length or if the health differential between the agent and opponent is over 40 units.
19

---

Published as a conference paper at ICLR 2022
State and Action Space for the UFC environment as 30-dimensional observations for the RL agent:
The environment returns the following data
Position: (X,Z) coordinates in the ring
Velocity: scalar of trajectory velocity
Health: [0,100] float value
Action active: high-level action indicator
Strike type: indicator type strike performed
Strike segment: indicates the current stage of striking
Ticks to event: time until next event in game ticks
Target hit location: the region of the opponent's s body the agent is targeting (upper or lower)
Block successful: indicates when strike is successfully blocked
The agent is allowed to take the following actions:
No action
(Strike) Straight: left or right, single request
(Strike) Uppercut: left or right, single request
(Strike) Hook: left or right, single request
Block: high or low, needs to be held to sustain blocking
Move: forward backward relative current opponent position
Reward Design
The reward breakdown for the environment as follows:
Standard to all styles Victory reward: +1; Strike request penalty: -0.001
Aggressive style successful attack reward: + +|Heththyppnnell /100
Defensive style survival reward: +0.02 per step; attack reward: +| +|AHcalthopooomm/2222
health loss penalty: - -AHclthhagm
Blocking style block reward: +0.01 per successful block
Movement style velocity reward: +0.05v
E TRAINING CONFIGURATION
Training code is based on OpenAl's Spinning Up (Achiam 2018) and is provided in the zip folder included as additional supplementary material with our main paper submission.
With the exception of hidden layer configuration, we use the same default hyperparameters for each of the algorithms used. Different hidden layer configurations were selected for each environment such that they would not be needlessly large but would consistently solve the problem in the single- style cases
Network Configuration Hidden layer configurations for both the actors and critics per environ- ment:
Path following: PPO [64,64], SAC [8,8]
Pong: PPO [64,64], SAC [32,32]
Sonic the Hedgehog 1 and 2: PPO [256,256]. Note additionally that the CNN feature tractor is shared between the actors and all critics in any particular training run, though they are not shared between separate training based on the the extractor used by Mnih et al. (2015) as it is the same configuration used in the Retro-contest (Nichol al. 2018).
The CNN feature extractor configured
The above network configurations correspond to the following numbers trainable weights for each of the critic network(s).
20

---

Published as a conference paper at ICLR 2022
Table 7: Trainable Critic parameters per task
Single style
MullMiCChiieeL.. vs. (S)
SAC
Path Following
137
161
483
163
7 11.24% 0.77%
Pong
5061
7109
14218
7164
PPO
Path Following
4737
14211
4867
2.74%
Pong
683353
76544
1533090
76802
1 0.33%
201
? 2.04%
Misc. details Additionally, a frame wrapper is used with Pong and Sonic games to augment state information with information from the last 4 observed states. This wrapper is based on the FrameStack wrapper provided OpenAl Baselines (Dhariwal et al. (017).
Training time determined by running training until performance plateaued, with an additional buffer given to multi-task training to account for the multi-task problem setting, with initial tests conducted to ensure that performance indeed plateaued.
Run configurations Configurations for training per environment are as follows:
Path following
Single-style PPO Trained for 100 epochs with 4000 steps per epoch and 90 steps per episode
Single-style SAC Trained for epochs with 4000 steps per epoch and 90 steps per episode
Multi-style MTPPO Trained for 300 epochs with 4000 steps per epoch and steps per episode
Multi-style MTSAC Trained for 150 epochs with 4000 steps per epoch and 90 steps per episode
Multi-style MultiCriticAA--PO Trained for 300 epochs with 4000 steps per epoch and 90 steps per episode
Multi-style MultiCriticAL-SAAC Trained for 150 epochs with 4000 steps per epoch and 90 steps per episode
Pong with rotatable paddles
Single-style PPO Trained for 150 epochs with 4000 steps per epoch and variable length episodes
Single-style SAC Trained for 50 epochs with 4000 steps per epoch and variable length episodes
Multi-style MTPPO Trained for 250 epochs with 4000 steps per epoch and variable length episodes
Multi-style MTSAC-TTainne for 150 epochs with 4000 steps per epoch and variable length episodes
Multi-style MultiCriticAL--PO- Trained for 250 epochs with 4000 steps per epoch and variable length episodes
Multi-style MultiCriticAL-SAC- Trained for 150 epochs with 4000 steps per epoch and variable length episodes
Sonic the Hedgehog (both and 2)
Single-style PPO Trained for 30 epochs with 10000 steps per epoch and up to 2500 steps per episode, until Sonic's death, whichever occurs first
Multi-style MTPPO Trained for 300 epochs with 10000 steps per epoch and up to 2500 steps per episode, or until Sonic's death, whichever occurs first
Multi-style Mult-OriticAL-PPO Trained for 300 epochs with 10000 steps per epoch and up to 2500 steps per episode, until Sonie's death, whichever occurs first
21


Reviews:
[{"review_id": "nOXH6l4WDFn", "reviewer": "Reviewer_mdph", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed ideas are presented clearly and the exposition on related work throughout the paper is useful in guiding the reader.\n\nThe main technical contribution is the idea of training separate critics or value heads while maintaining a single, shared actor for multiple tasks. As the authors pointed out in the rebuttal, the choice of a single actor is significant for many compute-sensitive applications (e.g. video games), where inference can be done in parallel (e.g. computing the actions of several NPCs executing different “behaviours” in a single forward pass). I think the paper’s claims are interesting and somewhat surprising since they suggest that negative inference disproportionately affects value learning over policy learning. Furthermore, this challenges the need for the explicit separation of policies for different tasks, which is a common technique in the multi-task RL literature, e.g. [1,2].\n\nThe experiments consider several unique, well-motivated environments and tasks. Following the paper’s revisions, I find that the results convincingly demonstrate the claims and address the majority of my initial concerns. However, a comparison against a state-of-the-art multi-actor, multi-critic algorithm could have strengthened the experiments. Currently, the “single-style” baseline is meant to fill this role, however it is difficult to compare this to the authors’ proposed approach in terms of sample efficiency since: (1) the single-style is only trained on a single task vs multiple simultaneous tasks (2) single-style does not observe sample efficiency gains from shared multi-task training. \n\nAdditional comments:\n- Tables and graphs need to report the meaning of the error margins. i.e. are they reporting standard error, 90% confidence intervals, or something else? \n- Typos: “mutli” (second paragraph, first sentence), “mulit” (appendix B header)\n\n\n[1] Yang, Zhaoyang et al. “Multi-Task Deep Reinforcement Learning for Continuous Action Control.” IJCAI (2017).\n\n[2] Teh, Yee Whye et al. “Distral: Robust multitask reinforcement learning.” NIPS (2017).\n", "summary_of_the_paper": "This paper proposes a single-actor, multi-critic approach to address the well-known problem of negative interference in multi-task RL (MTRL). The work focuses on a special case of MTRL described as “multi-style RL”, where the goal is to learn several distinct behaviours under the same environment dynamics. Experiments were conducted on a wide variety of environments: a path following domain, Pong, Sonic the Hedgehog, and a UFC fighting game. Results provide evidence that the proposed approach achieves a better final performance than a single-actor, single-critic MTRL baseline.", "summary_of_the_review": "Following the rebuttal and the paper revisions, I vote to accept. The paper demonstrates that several distinct behaviours can be successfully learned with a novel single-actor, multi-critic setup. While the technique is simple and the main idea shares some motivation with previous work, I believe this will be a valuable contribution to the multi-task RL literature.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "lDR40Uw4HU1", "reviewer": "Reviewer_Lh1z", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is very well written, and to me, easy to understand and to follow. It is also well-motivated, and the related work section seems quite complete. It seems that the authors did not believe themselves when they found out that single-actor multiple-critics settings are rare in multi-task RL, and spent a considerable amount of time looking for these papers.\n\nSpeaking of related work, it cites the Actor-Mimic paper [3], in which N critics are trained on N tasks, and then the actor is distilled to be good at all these tasks. The different with this paper, I think, is that the Actor-Mimic actor does not observe a task descriptor. Am I right? I have the feeling that the Actor-Mimic may deserve a bit more than an entry in a citation list in the background section, as it seems quite close to the main contribution of this paper.\n\nOne small remark regarding the experiments is that the authors focus on the impact of multiple critics (which is well-motivated). Our of curiosity, I would have been interested to see at least one experiment that compares MultiCriticAL with the state of the art (actor-based) of multi-task RL, just to see what fancy actors allow to do in comparison with multiple critics.\n\nRegarding the related work section, already very complete, a mention of Bootstrapped DQN could be added (with a mention that it is not multi-task) in case the paper is attacked on the novelty of multiple critics. Multiple critics are common in single-task RL, to provide uncertainty measures and good exploration [1, 2], but I'm not aware of an application to multi-task RL.\n\nOverall, I really like this paper, and recommend accepting it. It identifies an interesting problem and omission in the literature, and proposes a simple solution that performs well in many experiments.\n\n[1]: Osband, I., Blundell, C., Pritzel, A., & Van Roy, B. (2016). Deep exploration via bootstrapped DQN. Advances in neural information processing systems, 29, 4026-4034.\n[2]: Steckelmacher, D., Plisnier, H., Roijers, D. M., & Nowé, A. (2019). Sample-Efficient Model-Free Reinforcement Learning with Off-Policy Critics. arXiv preprint arXiv:1903.04193.\n[3]: Parisotto, E., Ba, J. L., & Salakhutdinov, R. (2015). Actor-mimic: Deep multitask and transfer reinforcement learning. arXiv preprint arXiv:1511.06342.", "summary_of_the_paper": "The paper considers the setting in which a single RL policy has to be learned for several tasks (or styles), to be selected at run-time after the agent has been trained on all the tasks. The core of the paper is the identification of a gap in the literature, where it has not been tried to train several critics and a single actor for multi-task RL, with each critic seeing only one task to be learned. The paper proposes to do just that, and shows promising experimental results in a variety of environments.", "summary_of_the_review": "The paper is well-written, easy to follow, and well-position regarding related work (with a note on the Actor-Mimic). The proposed method is simple, yet unique and performs well. An extra experiment that compares MultiCriticAL with policy-based multi-task approaches would have been nice, but I overall still recommend accepting this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "f009QISRgTm", "reviewer": "Reviewer_Y35C", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "strengths:\n\n1. The method is straightforward, extending the existing deep RL method via multiple value networks.\n\nweaknesses:\n\n1. The method can not generate different styles under the same environment, or put another way, this method can not generate different styles under the same reward function.  We only have one task or environment in many practical applications, but we want to generate behaviors with different styles. However, this paper can not generate multi-style behaviors when receiving the same reward signal. The algorithm proposed in this paper is more like a multi-task algorithm instead of a multi-style algorithm.\n\n**update**\nI have read the response of the authors and increased my score. I still think the title of this paper is misleading.", "summary_of_the_paper": "This paper wants to propose a simple method to deal with multi-task (style) problems.", "summary_of_the_review": "This paper proposed a simple method for multi-task problems. The experimental results are convincing, but the usage of this method is limited. Because this method can only be used for environments with multiple reward functions, and their approach can not generate multi-style behaviors for the same task. I think it would be more reasonable for the author to reformat their paper as a multi-task paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "d7faUXxx0Cn", "reviewer": "Reviewer_kVWj", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1) The problem of multi-objective RL has many important application in the real world, such as games and robotics. This paper tackles the exact challenges in multi-objective RL.\n2) The proposed algorithm is simple, but effective.\n3) The examples in the evaluation (e.g. Pong and the boxing game) demonstrates an useful application of the proposed method: designing AI for games.\n\nWeakness:\n1) One important reference in multi-objective RL is missing: [1] \"Multi-Objective Reinforcement Learning using Sets of Pareto Dominating Policies, Van Moffaert et al., JMLR, 2014\". The proposed algorithm is very similar to Section 2.2.1 in that paper. The difference is that this paper extends the same multi-critic-single-actor idea to Deep Reinforcement Learning. However, the claim in this paper that \"seemingly no prior work explores the use of a single actor with multiple critics\" is not true. The existence of [1] significantly reduced the novelty and technical contributions of this paper.\n\nAdditional questions and comments:\n1) To train the multi-critic-single-actor (eq. 5, 6, 7, 8), do you need to specify a fixed set of weights w? Or are the w randomly sampled during the training stage?\n\n2) Most of the results are shown in a tabular form. It would be great to visualize the learning process with learning curves. Learning curves gives a lot more insights than the final reward, such as learning speed, sample efficiency, and the progress of learning different styles. It could also be a way to validate the claim that the proposed algorithm mitigates the negative interference between tasks.\n\n3) Section 5.3, \"MTPPO agents perform worse on average than agents trained on single-levels\". Are the agents \"trained on single-levels\" just trained on one out of the 17 levels, or trained on all levels but the policy does not take the one-hot encoding of the level as an additional input?", "summary_of_the_paper": "This paper proposes to extend the actor-critic frame to tackle multi-objective (multi-task) reinforcement learning. The key idea is to learn multiple critics that correspond to different reward functions. Then a single policy is optimized for a weighted combination of these critics. The paper applies the multi-objective RL to learning different styles of completing tasks, such as aggressive or defensive style in a boxing game. The paper shows that the proposed algorithm can beat several multi-task learning baselines.", "summary_of_the_review": "The paper proposes a simple and yet effective algorithm for multi-objective reinforcement learning. The paper is well written and the results are convincing. However, it misses an important prior work that has similar high-level ideas. The examples in the Evaluation Section of this paper show its great potential in real-world applications, especially for designing AI in video games.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Vz2VtiV1s76", "reviewer": "Reviewer_eF9d", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\n\n1) Simple yet novel approach\n\t- Exploring multiple critics for multiple reward functions for a single task is a natural idea that has not been done before.\n\t- There are clear ways to take these ideas further when needing to combine behaviors and author/control the behavior of RL agents trained with this method.\n2) Intriguing qualitative results\n\t- The UFC game videos are promising for the way behaviors are distinguished, even if the quantitative analysis is less clear. Without baselines it's hard to know if this would not obtain with other methods, but it's clearly promising.\n\n\n# Weaknesses\n\n1) Lack of statistical rigor in results\n\t- The results do not compare algorithms using tests for statistical significance of differences. This makes it hard to tell which results hold up under repeated trials.\n\t- From a glance the error bars in the Sonic results all overlap for most algorithms (Figure 4).\n\t- Similar overlaps seem to obtain for Pong (Table 1). For example, the SAC Defensive setting play time of MT w/One-hot is 1446±442, which overlaps the MN-MultiCriticAL 1637±113.\n\t- The UFC results do not report any measure of variance.\n\t- Action: Provide statistical testing of these differences. Or consider reporting other metrics recommended for RL evaluation (for example: https://arxiv.org/abs/2108.13264)\n2) Lack of baselines in the UFC domain\n\t- The authors note that the domain is meant for demonstration only and not as a benchmark. I found the qualitative results shown most compelling for this case and so was disappointed to not have baseline comparisons (acknowledging the costs for doing these experiments). \n\t- Action: Adding these baseline comparisons would be a great improvement to the paper. I recognize this may not be feasible, but want to indicate to the authors that this would be a strong demonstration of the work. An alternative may be to consider some other simple continuous domain (walker, half-cheetah, &c.) that more clearly demonstrates the potential to learn stylized behavior.\n3) No direct test of the interference hypothesis\n\t- One of the more interesting ideas in the paper is that MTRL may suffer from interference between task heads when sharing a backbone. In the methods this is the difference between the MH and MN variants of MultiCriticAL.\n\t- No tests in the paper or analysis investigate this claim.\n\t- Action: It would strengthen the paper to show statistical tests that clearly demonstrate this claim.\n4) Some missing technical details\n\t- Below are some minor details that should be easy to address to improve the paper\n\t1) Why do the single task networks perform so well? Are they only trained and tested in a single task setting? This was not explained in the methodology.\n\t2) What is the evidence that UFC agent \"smoothly transition between the different fighting styles\"? The results show different styles but do not directly assess transitioning between styles.\n\t3) How does one interpret Table 3 given agents were trained on joint parameter combinations, rather than the single dimensions shown? Are these roughly the extreme quadrants and center of a 2D grid?", "summary_of_the_paper": "The paper introduces a variant of actor-critic reinforcement learning algorithms where multiple critics are trained in a reduced version of multi-task training where a task allows multiple reward functions. Two variants are proposed for handling multiple reward functions: one where a backbone value function is trained with multiple heads and one where completely distinct value functions are trained. Evaluation considers three domains: tracing different shapes, Pong with aggressive-defensive styles, and Sonic the Hedgehog game levels (where levels are styles). Results are ambiguous as to the improvement of the architecture over classical methods. Some qualitative results are shown for a fighting game trained with the multi-critic approach.", "summary_of_the_review": "The paper presents a technique with promise for training agents to perform tasks in a variety of ways. The current empirical results are ambiguous when comparing the proposed method to baseline methods, leading me to recommend rejection.\n\nMy score would increase if the authors can provide clear (statistically) conclusive results of superior performance of the proposed method.\n\n**Update**\nI have increased my score in light of the additional experiment seeds added and clearer differences in performance from the resulting reductions in variance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors investigate a multi-task RL actor-critic technique, where a single actor is used while multiple critics are trained (one per task, where each task corresponds to a different reward function). Experiments on several environments demonstrate that this method works quite well in practice.

All reviewers found the proposed approach sensible and effective, in spite of its simplicity. The main concerns were:
- Lack of novelty: although this is indeed not a particularly original idea, the specific instantiation in the actor-critic setup is novel and well motivated
- Some confusing / unconvincing experimental results: after receiving this feedback, the authors were able to upload a new revision that addressed the main concerns
- Focusing on the "multi-style" aspect when this is essentially a multi-task algorithm: although I agree that framing it as a specific case of multi-task learning would make sense and would probably make more appealing to multi-task RL researchers, I do not consider this to be a major issue

In spite of being a relatively straightforward paper, I believe it is good to have strong empirical evaluation of such basic techniques disseminated to the research community, and I thus recommend acceptance, in accordance with reviewers' recommendations after the discussion period.
Decision: Accept (Poster)