Multi-Critic Actor Learning: Teaching RL Policies to Act with Style

1 INTRODUCTION . Reinforcement Learning ( RL ) offers an interesting means by which to develop interesting behaviors in a variety of controls settings . The work presented in this paper is primarily contextualized by our efforts to develop RL techniques for teaching RL-policies to behave with multiple distinct and specific styles for game-play . RL can be be useful in developing control policies for game-play and has been demonstrated as being capable of learning human-like or even superhuman play , most notably beating top-ranked human players in games such as Dota 2 ( Berner et al. , 2019 ) , StarCraft ( Vinyals et al. , 2019 ) , Chess ( Silver et al. , 2017 ) , and Go ( Silver et al. , 2016 ) . RL-based control in games offers a wide array of potential applications , including testing ( Zhao et al. , 2020 ; Ariyurek et al. , 2021 ; Gordillo et al. , 2021 ) , game design ( Gisslén et al. , 2021 ) , providing competition for human players ( Berner et al. , 2019 ) , or simply as a means to develop more interesting game-play AI behaviors ( Zhao et al. , 2020 ; Alonso et al. , 2020 ) . RL algorithms can be notorious however for training aesthetically unappealing policies , but prior work has demonstrated that , with careful shaping of the reward functions , it is possible to derive highly specific desirable behavior ( Peng et al. , 2018 ; Mysore et al. , 2021b ) . Machine learning ( ML ) solutions to game-play AI can however represent a significant increase in compute requirements over their heuristic-based counterparts , especially when using deep learning techniques . If a learned control policy could represent multiple desirable styles and offered a controllable way to transition between the learned styles , this could reduce the total compute burden of the ML-based controller , making it more practically viable . Ideally developing an ML model would constitute learning a single , low-complexity policy that would model and transition smoothly between multiple behavior styles . Reduced model complexity and deploying just a single policy network enables greater compute efficiency through hardware parallelism and reduced memory costs , which is beneficial in resource-constrained applications , such as videogames . We explore the problem of mutli-style RL , a special case of the more commonly explored problem of multi-task RL ( MTRL ) . Whereas MTRL techniques typically seek to solve a wide array of RL problems under a single learning campaign , multi-style RL adds the extra constraint that the ‘ tasks ’ in question all have identical system dynamics but would be characterized by different reward signals corresponding to the different desired styles of behavior . We focus specifically on applications of Actor-Critic RL techniques , which , in addition to being compatible with continuous action-space control , allow for reduced runtime computational cost as only the actor functions are required for inference , while the critics can often be disregarded after training . Mysore et al . ( 2021a ) and Andrychowicz et al . ( 2021 ) have also demonstrated that , it is possible to train compact and performant policy networks through careful consideration of the actor and critic network architectures . A common baseline approach to MTRL is to employ one-hot task encoding to delineate between tasks . In a number of multi-style cases considered however , this was not sufficient to enable successful multi-style policy-learning . We suspected that the similarity of game-play states visited between each trained style interfering with each style ’ s value optimization . In surveying existing MTRL literature for advancements in the field of multi-task Actor-Critic RL methods , we noticed a surprising hole . There are studies which employ single actors and single critics ( Finn et al. , 2017 ; Yang et al. , 2020 ; Sodhani et al. , 2021 ; Nichol et al. , 2018 ) , multiple actors and multiple critics ( Andreas et al. , 2017 ; Teh et al. , 2017 ; Rusu et al. , 2015 ; 2016 ; Huang et al. , 2017 ) , or multiple actors and a single critic ( Yang et al. , 2017 ; Dewangan et al. , 2018 ) . Curiously , seemingly no prior work explores using a single policy with distinct , per-task value estimation , i.e . a single actor with multiple critics . This paper proposes Multi-Critic Actor Learning ( MultiCriticAL ) , a single-actor , multi-critic framework for multi-task Actor-Critic optimization and evaluate it on a number of multi-style learning problems . The core idea of MultiCriticAL is to maintain separate per-style ( or task ) critic functions for each style being learned . By separating the critic MultiCriticAL would be able to avoid negative interference in the learned values for different styles . Our results show that MultiCriticAL consistently trains more performant policies , compared to its single-critic counterparts , succeeds in cases where the single-critic methods fail and achieves between 20-45 % improvement in more traditional multi-task settings involving multi-level game-play . Additionally , we demonstrate the utility of MultiCriticAL in a use-case more reflective of real-world application , in an experimental build for EA ’ s UFC Knockout Mode , where we train policies to act with multiple specific and distinct behavior styles . This work is also amongst the first to study the efficacy of MTRL algorithms across a broad range of multi-style problems , an aspect of MTRL receiving limited prior attention . 2 PRELIMINARIES . The core reinforcement learning ( RL ) problem is framed as a finite-horizon Markov decision process , 〈S , A , P , r , γ〉 , with state transition dynamics P : S , A → S , for states s ∈ S and actions a ∈ A , and reward function r : S , A → R. The RL objective is to train a policy , π : S → A such that the expected sum of discounted rewards , Eτπ [ ∑ i γ ir ( si , ai ) ] , with discount factor γ , is maximized for all trajectories , τπ , sampled on policy π. RL typically assumes that both the transition dynamics and reward dynamics are encompassed by a black-box environment . RL agents interact with the environment to observe the state response and generated rewards , but where specific reward functions and transition dynamics are not assumed known . Solving the RL optimization problem requires estimating expected rewards and using these approximations to optimize the action policies . Among the different techniques for tackling RL optimization , We mainly consider Actor-Critic methods , whose key characteristic is their separation of the ‘ actors ’ , representing the RL policy , and the ‘ critic ’ , representing the value-function estimator that learns to estimate future rewards . This separation allows Actor-Critic RL to take advantage of the improved sample-efficiency of value-based techniques , such as Q-learning ( Watkins & Dayan , 1992 ) , while taking advantage of policy-based techniques , such as policy gradients ( Sutton et al. , 2000 ) , for improved robustness to stochasticity and to enable learning in continuous action domains . Examples of prominent contemporary algorithms in this class of RL methods include TRPO ( Schulman et al. , 2015 ) , PPO ( Schulman et al. , 2017 ) , DDPG ( Lillicrap et al. , 2016 ) , TD3 ( Fujimoto et al. , 2018 ) , and SAC ( Haarnoja et al. , 2018 ) . In algorithms based on Q-learning , such as DDPG , TD3 and SAC , the policy optimization criteria Jπθ for a policy , π , parameterized by θ , is proportional to the Q-value : Jπθ ∝ Qπθ ( s , a ) = E [ R ( τπθ ) |s0 = s , a0 = a ] = Es′ [ r ( s , a ) + γQπθ ( s′ , πθ ( s′ ) ) ] ( 1 ) where s′ is the next state reached from state s in response to action a , andR ( τ ) is the discounted sum of rewards over trajectory τ . For algorithms based on a more traditional policy gradient optimization , such as TRPO and PPO , policies are optimized proportional to the the the advantage function , Aπθ and a measure of policy divergence : Jπθ ∝ log ( πθ ( a|s ) ) Aπθ ( s , a ) = log ( πθ ( a|s ) ) ( Qπθ ( s , a ) − V πθ ( s ) ) ( 2 ) where the value-function V π ( s ) = Eτπ [ R ( τπ ) |s0 = s ] = Es′ [ r ( s , π ( s ) ) + γV π ( s′ ) ] . For critic networks parameterized by φ , either a state-value estimator , Vφ , or Q-value ( state-action-value ) estimator , Qφ are optimized with optimization criteria JV πφ or JQπφ respectively : JV πφ ∝ ||V π φ ( s ) − ( r ( s , π ( s ) ) + V πφ ( s′ ∼ P ( s , π ( s ) ) ) || ( 3 ) JQπφ ∝ ||Q π φ ( s , a ) − ( r ( s , a ) +Qπφ ( s′ ∼ P ( s , a ) , π ( s′ ) ) ) || ( 4 ) While specific details around how actors and critics are optimized vary between algorithms , the above represents the fundamentals of optimization criteria upon which we shall build our discussion . 3 RELATED WORK . We broadly categorize multi-task RL ( MTRL ) techniques as either single-policy , if the same policy function activations are used to determine actions over all task , or multi-policy , if more separate activations ( in the case of multi-headed policies ) or policy functions are used . Multi-policy techniques range from training multiple policies and selecting or mixing between them ( Andreas et al. , 2017 ; Peng et al. , 2018 ; Dewangan et al. , 2018 ; Yang et al. , 2017 ; Huang et al. , 2017 ) , to simultaneously or iteratively training additional policies while sharing representation between policies ( Rusu et al. , 2015 ; Teh et al. , 2017 ; Rusu et al. , 2016 ) . Techniques requiring multiple policy networks are limited by their storage and computational costs during inference , and multi-headed policies can limit our ability to smoothly transition between behaviors , which makes them less attractive in the context of RL for game-play . We therefore focus instead on single-policy learning . Single-policy MTRL techniques train a single policy function to solve multiple tasks , without relying on on information from previously trained policies . Model-free meta-learning techniques have been proposed to improve multi-task generalization , in the absence of task specificity ( Finn et al. , 2017 ; Nichol et al. , 2018 ) , but where explicit task distinction can be exploited , a common approach has been to exploit task encoding to distinguish between tasks . A simple , yet effective approach to this problem has been to extend state representations with one-hot task encoding ( Peng et al. , 2018 ; Yu et al. , 2020 ; Yang et al. , 2020 ) , thus allowing policies to distinguish between individual tasks and decide actions in a task-appropriate way . Recent work has also sought to enhance performance and generalizability of such single-policy techniques through automated task encoding ( Sodhani et al. , 2021 ) or through soft-modularization ( Yang et al. , 2020 ) , which encourages the development of distinct information pathways per task within actor networks . Our work considers the special MTRL case of multi-style learning . Multi-style learning requires agents to learn different behaviors for highly similar states while providing potentially highly dissimilar rewards for each style . This variability in the reward signals can have an adverse impact on the quality of learned value-functions when using a single value estimator if the estimator can not sufficiently distinguish between styles . We hypothesize that negative interference more strongly impacts value-function learning in multi-style tasks as task similarity may make specific style-values harder to distinguish . Multi-style learning is however an aspect of MTRL that has not received much prior consideration . Peng et al . ( 2018 ) explore the most similar setting to our work – where agents were required to learn different movement styles in animation tasks . Their work tested single-policy MTRL and found that policies were only able to learn a limited number of styles , requiring the authors to resort to a multi-policy approach . Most works considering the Actor-Critic MTRL problem focus predominantly on how the actors , i.e the policy-functions , are optimized , with less attention given to critics . The problem of negative interference between tasks in the broader context of MTRL is one that has been highlighted and studied before ( Yang et al. , 2020 ; Parisotto et al. , 2015 ; Teh et al. , 2017 ; Standley et al. , 2020 ) however , but has similarly been approached as a policy optimization problem . A number of multipolicy techniques were noted to maintain separate valuations per policy Teh et al . ( 2017 ) ; Andreas et al . ( 2017 ) ; Peng et al . ( 2018 ) , though some work has also suggested that sharing critics can allow optimization to better capture useful shared knowledge to improve policy optimization ( Dewangan et al. , 2018 ; Yang et al. , 2017 ) . A similar idea dominates single-policy techniques , where all the studied methods maintain a single value-function estimator across all tasks ( Nichol et al. , 2018 ; Yu et al. , 2020 ; Peng et al. , 2018 ; Yang et al. , 2020 ; Sodhani et al. , 2021 ) . While task-encoding may allow separate task-values to be learned implicitly by the critics , this was never found to be explicitly designed . Our work alternatively frames the multi-style learning framework as one that explicitly learns distinct per-style values to propose a value-learning optimization scheme that is better able to accommodate disparate reward functions .

Summary: This paper proposes a single-actor, multi-critic approach to address the well-known problem of negative interference in multi-task RL (MTRL). The work focuses on a special case of MTRL described as “multi-style RL”, where the goal is to learn several distinct behaviours under the same environment dynamics. Experiments were conducted on a wide variety of environments: a path following domain, Pong, Sonic the Hedgehog, and a UFC fighting game. Results provide evidence that the proposed approach achieves a better final performance than a single-actor, single-critic MTRL baseline.