Year: 2021

Abstract:
Using a single value function (critic) shared over multiple tasks in Actor-Critic multi-task reinforcement learning (MTRL) can result in negative interference between tasks, which can compromise learning performance. Multi-Critic Actor Learning (MultiCriticAL) proposes instead maintaining separate critics for each task being trained, while training a single multi-task actor. Explicitly distinguishing between tasks also eliminates the need for critics to learn to do so and mitigates interference between task-value estimates. MultiCriticAL is tested in the context of multi-style learning, a special case of MTRL where agents are trained to behave with different distinct behavior styles, and yields up to 56% performance gains over the single-critic baselines and even successfully learns behavior styles in cases where single-critic approaches may simply fail to learn. To simulate a real-world use-case, MultiCriticAL is tested on an experimental build of EA’s UFC game, where it enables learning a policy that can smoothly transition between multiple fighting styles.

heading: 1 INTRODUCTION
text: Reinforcement Learning (RL) offers an interesting means by which to develop interesting behaviors in a variety of controls settings. The work presented in this paper is primarily contextualized by our efforts to develop RL techniques for teaching RL-policies to behave with multiple distinct and specific styles for game-play. RL can be be useful in developing control policies for game-play and has been demonstrated as being capable of learning human-like or even superhuman play, most notably beating top-ranked human players in games such as Dota 2 (Berner et al., 2019), StarCraft (Vinyals et al., 2019), Chess (Silver et al., 2017), and Go (Silver et al., 2016). RL-based control in games offers a wide array of potential applications, including testing (Zhao et al., 2020; Ariyurek et al., 2021; Gordillo et al., 2021), game design (Gisslén et al., 2021), providing competition for human players (Berner et al., 2019), or simply as a means to develop more interesting game-play AI behaviors (Zhao et al., 2020; Alonso et al., 2020). RL algorithms can be notorious however for training aesthetically unappealing policies, but prior work has demonstrated that, with careful shaping of the reward functions, it is possible to derive highly specific desirable behavior (Peng et al., 2018; Mysore et al., 2021b). Machine learning (ML) solutions to game-play AI can however represent a significant increase in compute requirements over their heuristic-based counterparts, especially when using deep learning techniques. If a learned control policy could represent multiple desirable styles and offered a controllable way to transition between the learned styles, this could reduce the total compute burden of the ML-based controller, making it more practically viable. Ideally developing an ML model would constitute learning a single, low-complexity policy that would model and transition smoothly between multiple behavior styles. Reduced model complexity and deploying just a single policy network enables greater compute efficiency through hardware parallelism and reduced memory costs, which is beneficial in resource-constrained applications, such as videogames.
We explore the problem of mutli-style RL, a special case of the more commonly explored problem of multi-task RL (MTRL). Whereas MTRL techniques typically seek to solve a wide array of RL problems under a single learning campaign, multi-style RL adds the extra constraint that the ‘tasks’ in question all have identical system dynamics but would be characterized by different reward signals corresponding to the different desired styles of behavior. We focus specifically on applications of Actor-Critic RL techniques, which, in addition to being compatible with continuous action-space control, allow for reduced runtime computational cost as only the actor functions are
required for inference, while the critics can often be disregarded after training. Mysore et al. (2021a) and Andrychowicz et al. (2021) have also demonstrated that, it is possible to train compact and performant policy networks through careful consideration of the actor and critic network architectures.
A common baseline approach to MTRL is to employ one-hot task encoding to delineate between tasks. In a number of multi-style cases considered however, this was not sufficient to enable successful multi-style policy-learning. We suspected that the similarity of game-play states visited between each trained style interfering with each style’s value optimization. In surveying existing MTRL literature for advancements in the field of multi-task Actor-Critic RL methods, we noticed a surprising hole. There are studies which employ single actors and single critics (Finn et al., 2017; Yang et al., 2020; Sodhani et al., 2021; Nichol et al., 2018), multiple actors and multiple critics (Andreas et al., 2017; Teh et al., 2017; Rusu et al., 2015; 2016; Huang et al., 2017), or multiple actors and a single critic (Yang et al., 2017; Dewangan et al., 2018). Curiously, seemingly no prior work explores using a single policy with distinct, per-task value estimation, i.e. a single actor with multiple critics.
This paper proposes Multi-Critic Actor Learning (MultiCriticAL), a single-actor, multi-critic framework for multi-task Actor-Critic optimization and evaluate it on a number of multi-style learning problems. The core idea of MultiCriticAL is to maintain separate per-style (or task) critic functions for each style being learned. By separating the critic MultiCriticAL would be able to avoid negative interference in the learned values for different styles. Our results show that MultiCriticAL consistently trains more performant policies, compared to its single-critic counterparts, succeeds in cases where the single-critic methods fail and achieves between 20-45% improvement in more traditional multi-task settings involving multi-level game-play. Additionally, we demonstrate the utility of MultiCriticAL in a use-case more reflective of real-world application, in an experimental build for EA’s UFC Knockout Mode, where we train policies to act with multiple specific and distinct behavior styles. This work is also amongst the first to study the efficacy of MTRL algorithms across a broad range of multi-style problems, an aspect of MTRL receiving limited prior attention.

heading: 2 PRELIMINARIES
text: The core reinforcement learning (RL) problem is framed as a finite-horizon Markov decision process, 〈S,A, P, r, γ〉, with state transition dynamics P : S,A → S, for states s ∈ S and actions a ∈ A, and reward function r : S,A → R. The RL objective is to train a policy, π : S → A such that the expected sum of discounted rewards, Eτπ [∑ i γ ir(si, ai) ] , with discount factor γ, is maximized for all trajectories, τπ , sampled on policy π. RL typically assumes that both the transition dynamics and reward dynamics are encompassed by a black-box environment. RL agents interact with the environment to observe the state response and generated rewards, but where specific reward functions and transition dynamics are not assumed known. Solving the RL optimization problem requires estimating expected rewards and using these approximations to optimize the action policies.
Among the different techniques for tackling RL optimization, We mainly consider Actor-Critic methods, whose key characteristic is their separation of the ‘actors’, representing the RL policy, and the ‘critic’, representing the value-function estimator that learns to estimate future rewards. This separation allows Actor-Critic RL to take advantage of the improved sample-efficiency of value-based techniques, such as Q-learning (Watkins & Dayan, 1992), while taking advantage of policy-based techniques, such as policy gradients (Sutton et al., 2000), for improved robustness to stochasticity
and to enable learning in continuous action domains. Examples of prominent contemporary algorithms in this class of RL methods include TRPO (Schulman et al., 2015), PPO (Schulman et al., 2017), DDPG (Lillicrap et al., 2016), TD3 (Fujimoto et al., 2018), and SAC (Haarnoja et al., 2018).
In algorithms based on Q-learning, such as DDPG, TD3 and SAC, the policy optimization criteria Jπθ for a policy, π, parameterized by θ, is proportional to the Q-value: Jπθ ∝ Qπθ (s, a) = E [R(τπθ )|s0 = s, a0 = a] = Es′ [r(s, a) + γQπθ (s′, πθ(s′))] (1) where s′ is the next state reached from state s in response to action a, andR(τ) is the discounted sum of rewards over trajectory τ . For algorithms based on a more traditional policy gradient optimization, such as TRPO and PPO, policies are optimized proportional to the the the advantage function, Aπθ and a measure of policy divergence:
Jπθ ∝ log(πθ(a|s))Aπθ (s, a) = log(πθ(a|s)) (Qπθ (s, a)− V πθ (s)) (2) where the value-function V π(s) = Eτπ [R(τπ)|s0 = s] = Es′ [r(s, π(s)) + γV π(s′)]. For critic networks parameterized by φ, either a state-value estimator, Vφ, or Q-value (state-action-value) estimator, Qφ are optimized with optimization criteria JV πφ or JQπφ respectively:
JV πφ ∝ ||V π φ (s)− (r(s, π(s)) + V πφ (s′ ∼ P (s, π(s)))|| (3) JQπφ ∝ ||Q π φ(s, a)− (r(s, a) +Qπφ(s′ ∼ P (s, a), π(s′)))|| (4)
While specific details around how actors and critics are optimized vary between algorithms, the above represents the fundamentals of optimization criteria upon which we shall build our discussion.

heading: 3 RELATED WORK
text: We broadly categorize multi-task RL (MTRL) techniques as either single-policy, if the same policy function activations are used to determine actions over all task, or multi-policy, if more separate activations (in the case of multi-headed policies) or policy functions are used. Multi-policy techniques range from training multiple policies and selecting or mixing between them (Andreas et al., 2017; Peng et al., 2018; Dewangan et al., 2018; Yang et al., 2017; Huang et al., 2017), to simultaneously or iteratively training additional policies while sharing representation between policies (Rusu et al., 2015; Teh et al., 2017; Rusu et al., 2016). Techniques requiring multiple policy networks are limited by their storage and computational costs during inference, and multi-headed policies can limit our ability to smoothly transition between behaviors, which makes them less attractive in the context of RL for game-play. We therefore focus instead on single-policy learning.
Single-policy MTRL techniques train a single policy function to solve multiple tasks, without relying on on information from previously trained policies. Model-free meta-learning techniques have been proposed to improve multi-task generalization, in the absence of task specificity (Finn et al., 2017; Nichol et al., 2018), but where explicit task distinction can be exploited, a common approach has been to exploit task encoding to distinguish between tasks. A simple, yet effective approach to this problem has been to extend state representations with one-hot task encoding (Peng et al., 2018; Yu et al., 2020; Yang et al., 2020), thus allowing policies to distinguish between individual tasks and decide actions in a task-appropriate way. Recent work has also sought to enhance performance and generalizability of such single-policy techniques through automated task encoding (Sodhani et al., 2021) or through soft-modularization (Yang et al., 2020), which encourages the development of distinct information pathways per task within actor networks.
Our work considers the special MTRL case of multi-style learning. Multi-style learning requires agents to learn different behaviors for highly similar states while providing potentially highly dissimilar rewards for each style. This variability in the reward signals can have an adverse impact on the quality of learned value-functions when using a single value estimator if the estimator cannot sufficiently distinguish between styles. We hypothesize that negative interference more strongly impacts value-function learning in multi-style tasks as task similarity may make specific style-values harder to distinguish. Multi-style learning is however an aspect of MTRL that has not received much prior consideration. Peng et al. (2018) explore the most similar setting to our work – where agents were required to learn different movement styles in animation tasks. Their work tested single-policy MTRL and found that policies were only able to learn a limited number of styles, requiring the authors to resort to a multi-policy approach.
Most works considering the Actor-Critic MTRL problem focus predominantly on how the actors, i.e the policy-functions, are optimized, with less attention given to critics. The problem of negative interference between tasks in the broader context of MTRL is one that has been highlighted and studied before (Yang et al., 2020; Parisotto et al., 2015; Teh et al., 2017; Standley et al., 2020) however, but has similarly been approached as a policy optimization problem. A number of multipolicy techniques were noted to maintain separate valuations per policy Teh et al. (2017); Andreas et al. (2017); Peng et al. (2018), though some work has also suggested that sharing critics can allow optimization to better capture useful shared knowledge to improve policy optimization (Dewangan et al., 2018; Yang et al., 2017). A similar idea dominates single-policy techniques, where all the studied methods maintain a single value-function estimator across all tasks (Nichol et al., 2018; Yu et al., 2020; Peng et al., 2018; Yang et al., 2020; Sodhani et al., 2021). While task-encoding may allow separate task-values to be learned implicitly by the critics, this was never found to be explicitly designed. Our work alternatively frames the multi-style learning framework as one that explicitly learns distinct per-style values to propose a value-learning optimization scheme that is better able to accommodate disparate reward functions.

heading: 4 MULTI-CRITIC ACTOR LEARNING
text: The policy functions, i.e. the ‘actor’ part of Actor-Critic frameworks, represent the core decisionmaking processes of RL agents and the success or failure of any algorithm rests on the policies’ abilities to learn useful mappings. The learned critics have limited utility after training and learned value-function estimators are also known for being poor approximations of the true value-functions they attempt to capture (Ilyas et al., 2020), so it makes sense that prior work has focused more on how policies are constructed and trained. The limitations of value learning are however exactly why the specifics of value-network constructions and optimization are important to consider. A critical element in the performance of RL algorithms can be the representational capacity of the learned value-functions (Mysore et al., 2021a; Andrychowicz et al., 2021). It therefore stands to reason, that better structuring how value-networks represent the learned values can be an important component of improving value-function learning and therefore policy performance.
A common formulation of single-critic MTRL optimization involves including a task-encoding, Ω, to distinguish between learned tasks and to specify task rewards and updating the state representation s̄ = 〈s,Ω〉. The optimization criteria in Equations 1-4 are correspondingly updated to use s̄ in place of s. The resulting MTRL optimization criteria typically represent rewards and learned values by scalar values. Practically, the value-functions are learned by a single network with a single output node, regardless of the number of tasks. A single learned value-function representing multiple values may enforce continuity between the learned values, which may not necessarily reflect the true value functions and could compromise the quality of learned values and resulting policies.
Value Optimization for Multi-Critic Actor Learning By separating the learned values per style, assumptions of continuity between style-values can be dismissed and style (or task) values can be learned explicitly, without relying on the critics to inherently learn the distinctions. We propose Multi-Critic Actor Learning (MultiCriticAL) with a style-conditioned value optimization criteria:
JVπφ |Ω ∝ ∣∣∣∣∑ iωi [ V πφi(s)− (ri(s, π(s̄)) + V π φi(s ′ ∼ P (s, π(s̄)) ]∣∣∣∣ (5)
JQπφ|Ω ∝ ∣∣∣∣∑ iωi [ Qπφi(s, a)− (ri(s, a) +Q π φi(s ′ ∼ P (s, a), π(s̄′))) ]∣∣∣∣ (6)
where, for M styles, the task/style encoding, Ω = [ω0, . . . , ωM ] and ωi ∈ [0, 1]. The policy optimization criteria requires the value functions to be continuous with respect to the states and task encoding, and this property can be maintained by appropriately selecting the value functions when computing the policy optimization criteria. The specifics of the exact formulation of the optimization criteria would vary in accordance with the base Actor-Critic algorithm employed, with Equations 1 and 2 respecitvely being updated as:
Jπθ ∝ ∑ iωiQ πθ φi (s, a) (7)
or Jπθ ∝ log(πθ(a|s̄)) ∑ iωiA πθ φi (s, a) (8)
Practically, multi-valued critics may be represented by separate independent critic networks – multinetwork (MN) MultiCriticAL – or a single base critic network with multiple heads, where each head
represents a different learned value – multi-headed (MH) MultiCriticAL. While the latter allows for some learned representations to be shared between the value functions, the former may be more beneficial when sharing is likely to lead to undesirable interference. This is visually shown in Figure 1 and extended in Figure 13 in Appendix C.
Relationship to MTRL and MORL DQNs MultiCriticAL shares theoretical underpinnings with existing DQN techniques to tackle MTRL and multi-objective RL (MORL) by learning task/objective-specific values (Parisotto et al., 2015; Moffaert & Nowé, 2014; D’Eramo et al., 2020; Rusu et al., 2016). Learned DQN policies and value functions share a single combined architectural backbone, forcing value estimation machinery to be retained at inference time. By extending the theory to Actor-Critic methods, we both enable utility in continuous control and the practical benefits of disentangling the representational complexity of the actors from the often greater representational burden placed on the critics (Mysore et al., 2021a). (Additional discussion in Appendix B).

heading: 5 EVALUATION
text: We evaluate the proposed MultiCriticAL method in a series of multi-style RL problems. We first test our method on two basic multi-style learning environments designed as simple and light-weight demonstrations of the types of problems we seek to address: (i) learning to follow simple shapes, where different styles correspond to different shapes, (ii) playing a modified game of Pong where the player’s paddle is allowed to rotate and agents are trained to play aggressively or defensively against a stock AI. While relatively simple, these environments help highlight the efficacy of MultiCriticAL over the more typical single-critic MTRL framework. Additionally, we test MultiCriticAL on learning to play multiple levels of the Sega genesis games Sonic the Hedgehog, and Sonic the Hedgehog 2, where the agent is trained to play the different levels of each game, following the reward structure devised for the Gym Retro contest (Nichol et al., 2018). Finally, we demonstrate a practical use-case for MultiCriticAL in an experimental build of EA’s UFC Knockout Mode fighting game where RL agents are integrated into the game controls and trained with different fighting styles. Appendix A provides learning curves and additional visualizations for the results discussed in this section.
Baselines We train MultiCriticAL with two popular contemporary actor-critic algorithms: PPO (Schulman et al., 2017) and SAC (Haarnoja et al., 2018), which represent Actor-Critic algorithms based on both value iteration and Q-learning respectively. These are compared against Multi-task PPO (MTPPO) and Multi-task SAC (MTSAC), which augment the state information with a one-hot task encoding and were also proposed as baselines by Yu et al. (2020). While we recognize that these are not state-of-the-art MTRL algorithms, they share a common critic optimization structure with more recent advancements in policy (actor) optimization (Sodhani et al., 2021; Yang et al., 2020), and we aim to demonstrate the fundamental utility of MultiCriticAL in the absence of additional policy optimization tricks. Training code is based on OpenAI’s Spinning Up (Achiam, 2018) and is provided in the Supplementary Material for the multi-style environments. As the Sonic the Hedgehog games require paid access and the development environment for UFC is proprietary, game code is not provided, though the Appendix D details how they were adapted.
Seeding To account for the variability in RL, performance is averaged over 15 random seeds.

heading: 5.1 PATH FOLLOWING
text: Environment Design and Reward Shaping In the path following environment, given is current position in space, the current phase of motion ∈ [0, 1], and an indicator on which shape to reproduce, the agent is expected to follow a trajectory that generates the desired shape. Three shapes are tested: Circle, Square and Triangle. This simple problem adds no additional complexity related to generalization or complex system dynamics and simply expects policies to learn a mapping function that encodes the desired shape. The multi-style objective is for the agents to accurately produce any of the three required shapes, given the task encoding. Agents are rewarded based on the absolute error between the agent’s current and desired position in space. The rewards are cast to be positive in the range of [0, 1] per step. Appendix D offers further details, with code in Supplementary Material.
Results The pathing behavior learned for each algorithm over the three shapes is show in Figure 5.1. While single-style SAC and PPO are able to learn to follow each shape, performance of MTSAC and MTPPO in the multi-style setting is compromised. This is most prominent with MTSAC. MTPPO fares better but performance in the multi-style setting is still notably worse. Adapted to both algorithms, MultiCriticAL demonstrates more consistently learns to follow all three shapes. It should also be noted that larger networks were needed for the PPO algorithm to solve this task, using two hidden layers with 32 neurons each, as opposed to SAC’s 8 each. This likely contributes to the improved performance of PPO in the multi-task setting as the networks being used have higher relative representational capacity, but is still outperformed by MultiCriticAL.

heading: 5.2 PONG WITH ROTATABLE PADDLES
text: Environment Design and Reward Shaping We use a modified game of Pong to explore a multistyle learning problem where one style is represented by a sparse reward, and the other a dense reward, anticipating that this might be a case where single-critc value-function estimation may fail. Pong typically allows only a single degree of freedom for paddle movement and does not normally command nuanced strategies. Adding an extra degree of freedom by allowing the player’s paddles to rotate introduces increased complexity to the game and facilitates the training of different behavior styles. For this game, we consider two primary styles of play: (i) aggressive – with the objective of winning the game as quickly as possible, and (ii) defensive – where the objective is to avoid losing while prolonging the game as long as possible. In a real game setting, these could be analogous to hard- and easy-mode AI respectively, where the former tries to give the opponent a challenge and the latter may help the opponent learn to play. Our implementation is based on the Pong environment provided by the Pygame Learning Environment (PLE) (Tasfi, 2016). For simplicity, the basic opponent AI cannot rotate its paddle and uses a heuristic-based Pong controller provided by PLE. The Aggressive style rewards winning as quickly as possible while the Defensive style encourages avoiding losing and ensuring that the opponent can keep receiving the ball to prolong the game. Additional details are provided in Appendix D.
Results We first established baseline performance by training SAC and PPO on the Aggressive and Defensive play-styles. Figure 3 shows samples of play strategies adopted for each style, where the Aggressive agent exploits paddle rotation to bounce the ball off the side wall to outmaneuver the opponent. Conversely, Defensive agents attempt to return the ball directly to the opponent, exploiting the paddle rotation to angle the ball back towards the opponent. As desired, aggressive play yields high win-rates, with plays lasting significantly shorter than defensive play, which would win approximately half of the games but play longer games (videos in Supplementary Material).
Multi-style performance is tested by introducing a one-hot style encoder and randomly uniformly sampling either defensive or aggressive play for each new training episode. We observed that both MTSAC and MTPPO failed to learn to properly distinguish between play-styles. MTSAC agents often defaulted to highly defensive play (likely due to the over representation of the dense defensive rewards in the replay buffers) while MTPPO agents tended to learn intermediate behavior that was neither aggressive, nor defensive. Multi-
CriticAL agents were however able to successfully learn to distinguish between the styles with both base algorithms, with performance comparable to single-style agents. Table 1 and Figure A.2.1, show performance statistics averaged over 20 test episodes per seed for each algorithm.
We also test how MultiCriticAL agents would respond to changes in the style encoding at test time. As shown in Table 2, when provided an intermediate (0.5,0.5) encoding, the agents presented with behavior statistics that were almost halfway between that of their aggressive and defensive statistics. This was even without explicit conditioning, when agents were trained with purely binary style indicators. When explicitly interpolating between styles in training (by providing a multidimensional style-reward, with aggressive and defensive rewards scaled by their encoding weight), both MultiCriticAL SAC and PPO agents yielded performance statistics that more closely matched the mid-points between their aggressive and defensive behaviors.

heading: 5.3 SONIC THE HEDGEHOG
text: Training Setup To test MultiCriticAL in a more established multi-task setting, we tested two Sonic the Hedgehog games, using the Gym-retro wrappers designed for the games during the Gymretro contest (Nichol et al., 2018), using the same setup configuration developed for the competition. MultiCriticAL is setup to maintain a separate critic for each trained level. Sufficient experimentation for statistical rigor on this benchmark required hardware parallelism, which our code only supports for PPO. SAC is therefore omitted for this benchmark.
Results Performance on 17 of the main levels for each game, averaged over 10 tests per agent, per level, are presented in Figure 5.3, with per-level tabulated numerical rewards provided in Appendix A. Sonic the Hedgehog appears to be a game where learning shared representations over multiple levels helps overall performance, for both MultiCriticAL and MTPPO, though this observation does not hold for Sonic the Hedgehog 2, where MTPPO agents perform worse on average than agents trained on single-levels. MultiCriticAL agents tend to perform comparable or better than independently trained and MTPPO agents in almost all cases however, with 35% and 20% improvements on Sonic the Hedgehog and 56% and 40% improvement on Sonic the Hedgehog 2 of MN-MultiCriticAL and MH-MultiCriticAL over MTPPO respectively.

heading: 5.4 UFC ENVIRONMENT
text: For a use-case more reflective of real-world application, we trained AI fighters in an experimental build of UFC Knockout Mode. A gym-like (Brockman et al., 2016) wrapper was built for the game environment to enable an RL agent to play against the game’s stock AI. Note that this is not a feature of the commercially released game but is a modification made for research purposes.
Environment design and Reward Shaping The RL agent observes information about the player’s and opponent’s position, velocity, health, and actions being taken, stacked over 8 steps. The RLagent is able to control 11 basic discrete high-level actions including forward and backward movement, high or low blocking, left or right attacking (with straights, uppercuts or hooks), and is also allowed to take no action. The RL-agent is conditioned over 2 sets of 2 playing styles: aggressive or defensive and blocking or moving. During training, aggression vs. defensiveness, and blocking vs. moving are set on [0, 1] scales with (0.5, 0.5) representing a neutral style which is meant to balance behavior from both scales. For all styles, the agent is rewarded for winning and is penalized for every strike request. For the aggressive style, the agent is additionally rewarded for reducing the opponent health. For the defensive style, the agent gets a small positive reward for each step that it remains alive, is penalized for losing health and is rewarded for reducing the opponent’s health (though not as much as with the aggressive style). For the blocking style, the agent is rewarded for every successful block and for the moving style, the agent is rewarded for maintaining a positive velocity. More specifics on the state and action spaces, specifics on reward tuning, and training are provided in Appendix D.4.
Results MultiCriticAL allows PPO agents to successfully learn and smoothly transition between the different fighting styles. The behavior styles are distinguished visually and Figure 5 attempts to capture this through frames from captured gameplay with videos included in the Supplementary Material. Reward statistics for each style are presented in Table 3 and Figure A.2.3. Aggressive and defensive play are the most distinct but there are also observable differences between control that prioritizes blocking and moving, with the blocking-style conditioning agents to hold their ground while moving prompts agents to move around the ring more. When comparing MultiCriticAL to MTPPO, we note that, while rewards at the extremes of style distinction are similar, MultiCriticAL appears to be better at more consistently maintaining intermediate styles (when a setting is at 0.5). Visually, it appears that MTPPO also tended to favor blocking and defense over learning more nuanced behavior (Table 6).

heading: 5.5 DISCUSSION
text: While explicit separation of the learned value functions per style under the MultiCriticAL framework offers a demonstrable and consistent improvement over the single-critic baselines, it must be noted that MTSAC and MTPPO baselines do not necessarily reflect the state-of-the-art in MTRL, but they do reflect how value optimization is typically handled in contemporary literature. It is likely that utilizing some of the improvements to policy optimization techniques discussed in Section 3, may contribute to improved performance of policies trained using single-critic value-optimization, however the same may also further improve the performance of agents trained with MultiCriticAL and remains to be studied. Curiously, MN-MultiCriticAL consistently outperformed MH-MultiCriticAL on all the tested tasks, which calls into question the idea that shared representation is important and/or useful (Dewangan et al., 2018; Yang et al., 2017) and is an aspect that may warrant further study. An interesting extension could be to extend modularization (Yang et al., 2020; Andreas et al., 2017) to value-function optimization. Learning with curricula (Bengio et al., 2009; Graves et al., 2017; Mysore et al., 2019; Andreas et al., 2017) has also been demonstrated to (sometimes dramatically) improve RL performance and multi-style/task curricula also seem like a potential natural fit for multi-critic optimizaiton. Recent work (Mysore et al., 2021a; Andrychowicz et al., 2021; Ilyas et al., 2020) has also demonstrated that value function capacity can be a key bottleneck in RL performance. Increasing the network capacity of a single-critic value-function may improve the efficacy of the critic, but this is not guaranteed and there are few principled ways to determine what a sufficient network capacity may be. Significantly increased computational resources may be required for the learnability of the multi-style problems under a single-critc framework. MultiCriticAL also requires increased compute during training, particularly MN-MultiCriticAL, but MH-MultiCriticAL allows for a bounded increase to computational load (< 3% as tested - see Appendix E) with an acceptable compromise in relative performance while still outperforming single-critic value learning.

heading: 6 CONCLUSION
text: We study Actor-Critic optimization in multi-style RL problems, where agents are required to behave with different styles in the same environments, our work represents one of the first explorations of the application of RL to a wider array of multi-style problems. We hypothesized that limitations of single-actor, single-critic MTRL frameworks may result in interference between the reward signals for different styles due to high similarity in visited states and to mitigate this, we proposed Multi-Critic Actor Learning (MultiCriticAL), which separates critics into per-task value-function estimators, thus avoiding interference between style rewards and reduces learning complexity by not requiring critics to learn to distinguish between tasks. MultiCriticAL consistently outperforms the single-critic MTRL baselines, achieving up to 56% improvement in multi-task performance and successfully learning multiple behavior styles where single-critic methods would fail.

heading: REPRODUCIBILITY STATEMENT
text: As noted in Section 5, training code is based upon the open-source OpenAI Spinning Up codebase (Achiam, 2018), with modifications made to support MultiCriticAL for PPO and SAC. The modified code-base is included in the Supplementary Material for this paper, along with the code for the custom environments for Path Following (presented in Section 5.1) and Pong with rotatable paddles (presented in Section 5.2). The README files included with the code detail installation and use of the code. For the Sonic the Hedgehog games, reproducing results would require acquiring the games’ ROM files and linking them to an install of gym-retro, as detailed by Nichol et al. (2018). Also included Appendices D and E are additional details for the setup of the training environments discussed in the main body of the paper as well as training configurations used. We recognize that tests on the UFC environment may not be independently reproducible due to requiring access to proprietary development software and we therefore mainly discuss it as a demonstration of the practical utility for our proposed MultiCriticAL method, rather than as a benchmark task.

heading: A ADDITIONAL DATA AND RESULTS

heading: A.1 LEARNING CURVES

heading: A.1.1 PATH FOLLOWING
text: A.1.2 PONG

heading: A.1.3 SONIC THE HEDGEHOG

heading: A.2 ALTERNATIVE RESULTS REPRESENTATION
text: A.2.1 PONG

heading: A.2.2 SONIC THE HEDGEHOG

heading: A.2.3 UFC REWARD AND ACTION STATISTICS
text: We observe from Figure A.2.3 and Table 6 that MultiCriticAL more consistently learns delineated styles that are consistent with the behaviors that the rewards try to encourage. Note that, with our method, it is in fact the aggressive style that wins most fights consistently and the defensive style learns to block and move to not lose as quickly, whereas MTPPO tends seems to equate defensive play with just blocking, and this also contributes to a higher rate of losing. We see as well that the intermediate settings correspond to a more balanced blend of action probabilities between extremes when compared to MTPPO.

heading: MTPPO

heading: A.3 PONG ACTION INTERPOLATION WITH MULTI-POLICY COMPOSITION
text: As an exercise in curiosity, we also consider what would happen if we attempted to derive intermediate styles by interpolating between actions in the Pong game. This was meant to test an approach similar to the multi-policy action composition technique used by Peng et al. (2018). Unlike in their case where actions were weighted according to each actor’s estimated critic value, we instead directly interpolate between actions proposed by pre-trained aggressive and defensive agents. We take this approach as the training for either style would not offer the value functions any meaningful information about the other unseen style(s), prohibiting the form of value comparison employed by Peng et al. We hypothesized that it is possible and even likely that policies would destructively interfere with each other if the longer-term goals of constituent policies conflict with each other and this is exactly what was observed in testing. Except for intermediate styles very close to the the learned extremes, interpolating between actions proposed by the aggressive and defensive agents caused a near-complete failure in the composite policy’s control.
Composite policies were built by pairing a randomly selected pre-trained aggressive policy with a randomly selected defensive policy. On a (0.5,0.5) ‘middle’ setting, SAC composite policies achieved an average win-rate of 30% ± 5% (compared to the approximately 95% and 50% of the constituent policies) with an average Play-time of 213 ± 89 steps (often losing the game quickly). Similarly, PPO composite also achieve a low average win-rate of 15% ± 3% and similarly lose the game quickly. A high win-rate would not be expected, as the defensive policies are explicitly trained to avoid winning, but both policy classes are trained to avoid losing. Given the high loss-rate, composing policies in this way, for this multi-style task, can only be regarded a failure. By contrast however, as we showed in Table 2, MultiCriticAL is capable of learning to interpolate between styles, both when and not explicitly trained to do so.
This experiment, by no means, constitutes exhaustive testing, and cannot be used to claim that all, or even most, forms of multi-policy policy composition would be ineffective and is therefore not included in the the main body of the paper. We do believe however that it helps highlight a circumstance where policy composition can fail in cases where long term strategies for different target behavior styles may interfere. Considered more generally, there are no guarantees that actions proposed by distinct policies would be similar enough, that a weighted sum over them would even make sense, as cooperation between policies is not something that is explicitly trained for. One might benefit from explicit conditioning, though this was outside the scope of our work.

heading: B RELATING MULTICRITICAL TO DQNS AND MULIT-Q LEARNING
text: The idea of using multiple Q functions to guide the training of a single policy is not strictly new to the broader context of RL and has been used in notably single-task RL for improving stability of learned value functions. These include techniques such as Double Q-learning (Hasselt, 2010; Van Hasselt et al., 2016), Dueling Q-networks (Wang et al., 2016) and Bootstrapped DQNs (Osband et al., 2016; Steckelmacher et al., 2019). There are also applications in multi-task learning, though we only encountered them in works considering DQN frameworks that, in effect, learn task-specific Q values for a single policy in DQN architectures (Rusu et al., 2016; 2015; D’Eramo et al., 2020). As DQNs jointly encode both the action policies and the value representation in the same network
however, they do not share the practical benefits offered by Actor-Critic methods (and consequently MultiCriticAL) in separating the actors and critics. We believe that, practically, this is a meaningful distinction as DQN methods are (i) limited to discrete action spaces and (ii) need the full representational and computational complexity of the value estimation to be preserved at inference time after training, which is computationally inefficient. As shown by Andrychowicz et al. (2021) and Mysore et al. (2021a), oftentimes, it appears that the value function often bears the higher burden of required complexity to enable successfuly policy learning. Noting this, there is a significant practical advantage to being able to just train smaller, less computationally expensive actor networks of actor-critic methods and disregard the value estimation networks at inference time. Given that our work is largely motivated by application to video games, where runtime resource use is an important consideration, this was an important distinction.

heading: C MULTI-NETWORK VS. MULTI-HEADED TRAINING FRAMEWORK OVERVIEW

heading: D ADDITIONAL ENVIRONMENT SETUP DETAILS

heading: D.1 PATH FOLLOWING
text: Observed State Agents observe a 4-dimensional state with the current (x, y) coordinates of the agent in space and the current phase ρ ∈ [0, 1] represented by [sin(2ρπ), cos(2ρπ)]
Goal generation Goals, g are generated based on the current phase of motion. The are as follows for:
• Circle: g(ρ) = [sin(2ρπ), cos(2ρπ)] ∀ρ ∈ [0, 1]
• Square: g(ρ) =  [4ρ− 0.5, 0.5] ρ ≤ 0.25 [0.5, 1.5− 4ρ] 0.25 ≤ ρ ≤ 0.5 [2.5− 4ρ,−0.5] 0.5 ≤ ρ ≤ 0.75 [−0.5, 4ρ− 3.5] 0.75 ≤ ρ ≤ 1
• Triangle: g(ρ) =  [1.5ρ, 0.5− 3ρ] ρ ≤ 1/3 [1.5− 3ρ,−0.5] 1/3 ≤ ρ ≤ 2/3 [−1.5 + 1.5ρ, 1.5ρ− 1.5] 2/3 ≤ ρ ≤ 1
Reward design For current state s and desired state g: r = 1− log(|s−g|)/ log(max(∆s)) where max(∆s) is the difference between the extents of the space bounds.

heading: D.2 PONG
text: Observed State Agents observe an 8-dimensional state with the current vertical y positions of both the player and opponent, the player’s velocity, the paddle angle, and the ball’s (x, y) position and velocity.
Reward design Aggressive play is conditioned with a sparse reward: +1 on a win and -1 on loss. Agents are meant to learn that the best way to maximize cumulative discounted rewards is to win as quickly as possible. Defensive play is conditioned with dense rewards: -1 on loss, -1 for bouncing the ball against walls, +0.5 for receiving the ball, +0.1 if the opponent successfully receives the ball and +0.1 for every time-step of play. Defensive rewards encourage avoiding losing and ensuring that the opponent can receive the ball in order to prolong the game.

heading: D.3 SONIC THE HEDGEHOG
text: Reward scaling As recommended by Nichol et al. (2018), we scale rewards by a factor of 0.01 in order to encourage more stable learning with PPO.

heading: D.4 EA UFC DEVELOPMENT ENVIRONMENT
text: Environment The UFC environment is an experimental build of UFC. During training simulation, superfluous elements of the game, such as audio, spectators, etc. are disabled. Socket communication is used to interface between the game-code and the python-based RL algorithms. The environment wrapper controls the game simulation and extracts and processes state information before communicating it through the socket to the python RL code and interprets requests from the python code to trigger inputs in the game. During training, agents are trained for 240 epochs with 4000 sample interactions collected per epoch. Training is run in increments of 30 epochs to improve simulation and memory management stability. Episodes are terminated when the agent runs out the max episode length or if the health differential between the agent and opponent is over 40 units.
State and Action Space for the UFC environment The environment returns the following data as 30-dimensional observations for the RL agent:
• Position: (X,Z) coordinates in the ring • Velocity: scalar of trajectory velocity • Health: [0,100] float value • Action active: high-level action indicator • Strike type: indicator of type of strike performed • Strike segment: indicates the current stage of striking • Ticks to event: time until next event in game ticks • Target hit location: the region of the opponent’s body the agent is targeting (upper or lower) • Block successful: indicates when a strike is successfully blocked
The agent is allowed to take the following actions:
• No action • (Strike) Straight: left or right, single request • (Strike) Uppercut: left or right, single request • (Strike) Hook: left or right, single request • Block: high or low, needs to be held to sustain blocking • Move: forward or backward relative to current opponent position
Reward Design The reward breakdown for the environment is as follows:
• Standard to all styles – Victory reward: +1; Strike request penalty: -0.001 • Aggressive style – successful attack reward: +|∆Healthopponent|/100 • Defensive style – survival reward: +0.02 per step; attack reward: +|∆Healthopponent|/200,
health loss penalty: −∆Healthagent/50 • Blocking style – block reward: +0.01 per successful block • Movement style – velocity reward: +0.05v

heading: E TRAINING CONFIGURATION
text: Training code is based on OpenAI’s Spinning Up (Achiam, 2018) and is provided in the zip folder included as additional supplementary material with our main paper submission.
With the exception of hidden layer configuration, we use the same default hyperparameters for each of the algorithms used. Different hidden layer configurations were selected for each environment such that they would not be needlessly large but would consistently solve the problem in the singlestyle cases
Network Configuration Hidden layer configurations for both the actors and critics per environment:
• Path following: PPO [64,64], SAC [8,8] • Pong: PPO [64,64], SAC [32,32] • Sonic the Hedgehog 1 and 2: PPO [256,256]. Note additionally that the CNN feature ex-
tractor is shared between the actors and all critics in any particular training run, though they are not shared between separate training runs. The CNN feature extractor was configured based on the the extractor used by Mnih et al. (2015) as it is the same configuration used in the Retro-contest (Nichol et al., 2018).
The above network configurations correspond to the following numbers of trainable weights for each of the critic network(s).

heading: SAC

heading: PPO
text: Misc. details Additionally, a frame wrapper is used with Pong and the Sonic games to augment state information with infromation from the last 4 observed states. This wrapper is based on the FrameStack wrapper provided in OpenAI Baselines (Dhariwal et al., 2017).
Training time was determined by running training until performance plateaued, with an additional buffer given to multi-task training to account for the multi-task problem setting, with initial tests conducted to ensure that performance indeed plateaued.
Run configurations Configurations for training per environment are as follows:

heading: Path following
text: • Single-style PPO – Trained for 100 epochs with 4000 steps per epoch and 90 steps per episode
• Single-style SAC – Trained for 50 epochs with 4000 steps per epoch and 90 steps per episode
• Multi-style MTPPO – Trained for 300 epochs with 4000 steps per epoch and 90 steps per episode
• Multi-style MTSAC – Trained for 150 epochs with 4000 steps per epoch and 90 steps per episode
• Multi-style MultiCriticAL-PPO – Trained for 300 epochs with 4000 steps per epoch and 90 steps per episode
• Multi-style MultiCriticAL-SAC – Trained for 150 epochs with 4000 steps per epoch and 90 steps per episode
Pong with rotatable paddles
• Single-style PPO – Trained for 150 epochs with 4000 steps per epoch and variable length episodes
• Single-style SAC – Trained for 50 epochs with 4000 steps per epoch and variable length episodes
• Multi-style MTPPO – Trained for 250 epochs with 4000 steps per epoch and variable length episodes
• Multi-style MTSAC – Trained for 150 epochs with 4000 steps per epoch and variable length episodes
• Multi-style MultiCriticAL-PPO – Trained for 250 epochs with 4000 steps per epoch and variable length episodes
• Multi-style MultiCriticAL-SAC – Trained for 150 epochs with 4000 steps per epoch and variable length episodes

heading: Sonic the Hedgehog (both 1 and 2)
text: • Single-style PPO – Trained for 30 epochs with 10000 steps per epoch and up to 2500 steps per episode, or until Sonic’s death, whichever occurs first
• Multi-style MTPPO – Trained for 300 epochs with 10000 steps per epoch and up to 2500 steps per episode, or until Sonic’s death, whichever occurs first
• Multi-style MultiCriticAL-PPO – Trained for 300 epochs with 10000 steps per epoch and up to 2500 steps per episode, or until Sonic’s death, whichever occurs first

Review summaries:
This paper proposes to extend the actor-critic frame to tackle multi-objective (multi-task) reinforcement learning. The key idea is to learn multiple critics that correspond to different reward functions. Then a single policy is optimized for a weighted combination of these critics. The paper applies the multi-objective RL to learning different styles of completing tasks, such as aggressive or defensive style in a boxing game. The paper shows that the proposed algorithm can beat several multi-task learning baselines.

This paper proposes a single-actor, multi-critic approach to address the well-known problem of negative interference in multi-task RL (MTRL). The work focuses on a special case of MTRL described as “multi-style RL”, where the goal is to learn several distinct behaviours under the same environment dynamics. Experiments were conducted on a wide variety of environments: a path following domain, Pong, Sonic the Hedgehog, and a UFC fighting game. Results provide evidence that the proposed approach achieves a better final performance than a single-actor, single-critic MTRL baseline.

The paper introduces a variant of actor-critic reinforcement learning algorithms where multiple critics are trained in a reduced version of multi-task training where a task allows multiple reward functions. Two variants are proposed for handling multiple reward functions: one where a backbone value function is trained with multiple heads and one where completely distinct value functions are trained. Evaluation considers three domains: tracing different shapes, Pong with aggressive-defensive styles, and Sonic the Hedgehog game levels (where levels are styles). Results are ambiguous as to the improvement of the architecture over classical methods. Some qualitative results are shown for a fighting game trained with the multi-critic approach.

The paper considers the setting in which a single RL policy has to be learned for several tasks (or styles), to be selected at run-time after the agent has been trained on all the tasks. The core of the paper is the identification of a gap in the literature, where it has not been tried to train several critics and a single actor for multi-task RL, with each critic seeing only one task to be learned. The paper proposes to do just that, and shows promising experimental results in a variety of environments.