Multi-Critic Actor Learning: Teaching RL Policies to Act with Style

1 INTRODUCTION . Reinforcement Learning ( RL ) offers an interesting means by which to develop interesting behaviors in a variety of controls settings . The work presented in this paper is primarily contextualized by our efforts to develop RL techniques for teaching RL-policies to behave with multiple distinct and specific styles for game-play . RL can be be useful in developing control policies for game-play and has been demonstrated as being capable of learning human-like or even superhuman play , most notably beating top-ranked human players in games such as Dota 2 ( Berner et al. , 2019 ) , StarCraft ( Vinyals et al. , 2019 ) , Chess ( Silver et al. , 2017 ) , and Go ( Silver et al. , 2016 ) . RL-based control in games offers a wide array of potential applications , including testing ( Zhao et al. , 2020 ; Ariyurek et al. , 2021 ; Gordillo et al. , 2021 ) , game design ( Gisslén et al. , 2021 ) , providing competition for human players ( Berner et al. , 2019 ) , or simply as a means to develop more interesting game-play AI behaviors ( Zhao et al. , 2020 ; Alonso et al. , 2020 ) . RL algorithms can be notorious however for training aesthetically unappealing policies , but prior work has demonstrated that , with careful shaping of the reward functions , it is possible to derive highly specific desirable behavior ( Peng et al. , 2018 ; Mysore et al. , 2021b ) . Machine learning ( ML ) solutions to game-play AI can however represent a significant increase in compute requirements over their heuristic-based counterparts , especially when using deep learning techniques . If a learned control policy could represent multiple desirable styles and offered a controllable way to transition between the learned styles , this could reduce the total compute burden of the ML-based controller , making it more practically viable . Ideally developing an ML model would constitute learning a single , low-complexity policy that would model and transition smoothly between multiple behavior styles . Reduced model complexity and deploying just a single policy network enables greater compute efficiency through hardware parallelism and reduced memory costs , which is beneficial in resource-constrained applications , such as videogames . We explore the problem of mutli-style RL , a special case of the more commonly explored problem of multi-task RL ( MTRL ) . Whereas MTRL techniques typically seek to solve a wide array of RL problems under a single learning campaign , multi-style RL adds the extra constraint that the ‘ tasks ’ in question all have identical system dynamics but would be characterized by different reward signals corresponding to the different desired styles of behavior . We focus specifically on applications of Actor-Critic RL techniques , which , in addition to being compatible with continuous action-space control , allow for reduced runtime computational cost as only the actor functions are required for inference , while the critics can often be disregarded after training . Mysore et al . ( 2021a ) and Andrychowicz et al . ( 2021 ) have also demonstrated that , it is possible to train compact and performant policy networks through careful consideration of the actor and critic network architectures . A common baseline approach to MTRL is to employ one-hot task encoding to delineate between tasks . In a number of multi-style cases considered however , this was not sufficient to enable successful multi-style policy-learning . We suspected that the similarity of game-play states visited between each trained style interfering with each style ’ s value optimization . In surveying existing MTRL literature for advancements in the field of multi-task Actor-Critic RL methods , we noticed a surprising hole . There are studies which employ single actors and single critics ( Finn et al. , 2017 ; Yang et al. , 2020 ; Sodhani et al. , 2021 ; Nichol et al. , 2018 ) , multiple actors and multiple critics ( Andreas et al. , 2017 ; Teh et al. , 2017 ; Rusu et al. , 2015 ; 2016 ; Huang et al. , 2017 ) , or multiple actors and a single critic ( Yang et al. , 2017 ; Dewangan et al. , 2018 ) . Curiously , seemingly no prior work explores using a single policy with distinct , per-task value estimation , i.e . a single actor with multiple critics . This paper proposes Multi-Critic Actor Learning ( MultiCriticAL ) , a single-actor , multi-critic framework for multi-task Actor-Critic optimization and evaluate it on a number of multi-style learning problems . The core idea of MultiCriticAL is to maintain separate per-style ( or task ) critic functions for each style being learned . By separating the critic MultiCriticAL would be able to avoid negative interference in the learned values for different styles . Our results show that MultiCriticAL consistently trains more performant policies , compared to its single-critic counterparts , succeeds in cases where the single-critic methods fail and achieves between 20-45 % improvement in more traditional multi-task settings involving multi-level game-play . Additionally , we demonstrate the utility of MultiCriticAL in a use-case more reflective of real-world application , in an experimental build for EA ’ s UFC Knockout Mode , where we train policies to act with multiple specific and distinct behavior styles . This work is also amongst the first to study the efficacy of MTRL algorithms across a broad range of multi-style problems , an aspect of MTRL receiving limited prior attention . 2 PRELIMINARIES . The core reinforcement learning ( RL ) problem is framed as a finite-horizon Markov decision process , 〈S , A , P , r , γ〉 , with state transition dynamics P : S , A → S , for states s ∈ S and actions a ∈ A , and reward function r : S , A → R. The RL objective is to train a policy , π : S → A such that the expected sum of discounted rewards , Eτπ [ ∑ i γ ir ( si , ai ) ] , with discount factor γ , is maximized for all trajectories , τπ , sampled on policy π. RL typically assumes that both the transition dynamics and reward dynamics are encompassed by a black-box environment . RL agents interact with the environment to observe the state response and generated rewards , but where specific reward functions and transition dynamics are not assumed known . Solving the RL optimization problem requires estimating expected rewards and using these approximations to optimize the action policies . Among the different techniques for tackling RL optimization , We mainly consider Actor-Critic methods , whose key characteristic is their separation of the ‘ actors ’ , representing the RL policy , and the ‘ critic ’ , representing the value-function estimator that learns to estimate future rewards . This separation allows Actor-Critic RL to take advantage of the improved sample-efficiency of value-based techniques , such as Q-learning ( Watkins & Dayan , 1992 ) , while taking advantage of policy-based techniques , such as policy gradients ( Sutton et al. , 2000 ) , for improved robustness to stochasticity and to enable learning in continuous action domains . Examples of prominent contemporary algorithms in this class of RL methods include TRPO ( Schulman et al. , 2015 ) , PPO ( Schulman et al. , 2017 ) , DDPG ( Lillicrap et al. , 2016 ) , TD3 ( Fujimoto et al. , 2018 ) , and SAC ( Haarnoja et al. , 2018 ) . In algorithms based on Q-learning , such as DDPG , TD3 and SAC , the policy optimization criteria Jπθ for a policy , π , parameterized by θ , is proportional to the Q-value : Jπθ ∝ Qπθ ( s , a ) = E [ R ( τπθ ) |s0 = s , a0 = a ] = Es′ [ r ( s , a ) + γQπθ ( s′ , πθ ( s′ ) ) ] ( 1 ) where s′ is the next state reached from state s in response to action a , andR ( τ ) is the discounted sum of rewards over trajectory τ . For algorithms based on a more traditional policy gradient optimization , such as TRPO and PPO , policies are optimized proportional to the the the advantage function , Aπθ and a measure of policy divergence : Jπθ ∝ log ( πθ ( a|s ) ) Aπθ ( s , a ) = log ( πθ ( a|s ) ) ( Qπθ ( s , a ) − V πθ ( s ) ) ( 2 ) where the value-function V π ( s ) = Eτπ [ R ( τπ ) |s0 = s ] = Es′ [ r ( s , π ( s ) ) + γV π ( s′ ) ] . For critic networks parameterized by φ , either a state-value estimator , Vφ , or Q-value ( state-action-value ) estimator , Qφ are optimized with optimization criteria JV πφ or JQπφ respectively : JV πφ ∝ ||V π φ ( s ) − ( r ( s , π ( s ) ) + V πφ ( s′ ∼ P ( s , π ( s ) ) ) || ( 3 ) JQπφ ∝ ||Q π φ ( s , a ) − ( r ( s , a ) +Qπφ ( s′ ∼ P ( s , a ) , π ( s′ ) ) ) || ( 4 ) While specific details around how actors and critics are optimized vary between algorithms , the above represents the fundamentals of optimization criteria upon which we shall build our discussion . 3 RELATED WORK . We broadly categorize multi-task RL ( MTRL ) techniques as either single-policy , if the same policy function activations are used to determine actions over all task , or multi-policy , if more separate activations ( in the case of multi-headed policies ) or policy functions are used . Multi-policy techniques range from training multiple policies and selecting or mixing between them ( Andreas et al. , 2017 ; Peng et al. , 2018 ; Dewangan et al. , 2018 ; Yang et al. , 2017 ; Huang et al. , 2017 ) , to simultaneously or iteratively training additional policies while sharing representation between policies ( Rusu et al. , 2015 ; Teh et al. , 2017 ; Rusu et al. , 2016 ) . Techniques requiring multiple policy networks are limited by their storage and computational costs during inference , and multi-headed policies can limit our ability to smoothly transition between behaviors , which makes them less attractive in the context of RL for game-play . We therefore focus instead on single-policy learning . Single-policy MTRL techniques train a single policy function to solve multiple tasks , without relying on on information from previously trained policies . Model-free meta-learning techniques have been proposed to improve multi-task generalization , in the absence of task specificity ( Finn et al. , 2017 ; Nichol et al. , 2018 ) , but where explicit task distinction can be exploited , a common approach has been to exploit task encoding to distinguish between tasks . A simple , yet effective approach to this problem has been to extend state representations with one-hot task encoding ( Peng et al. , 2018 ; Yu et al. , 2020 ; Yang et al. , 2020 ) , thus allowing policies to distinguish between individual tasks and decide actions in a task-appropriate way . Recent work has also sought to enhance performance and generalizability of such single-policy techniques through automated task encoding ( Sodhani et al. , 2021 ) or through soft-modularization ( Yang et al. , 2020 ) , which encourages the development of distinct information pathways per task within actor networks . Our work considers the special MTRL case of multi-style learning . Multi-style learning requires agents to learn different behaviors for highly similar states while providing potentially highly dissimilar rewards for each style . This variability in the reward signals can have an adverse impact on the quality of learned value-functions when using a single value estimator if the estimator can not sufficiently distinguish between styles . We hypothesize that negative interference more strongly impacts value-function learning in multi-style tasks as task similarity may make specific style-values harder to distinguish . Multi-style learning is however an aspect of MTRL that has not received much prior consideration . Peng et al . ( 2018 ) explore the most similar setting to our work – where agents were required to learn different movement styles in animation tasks . Their work tested single-policy MTRL and found that policies were only able to learn a limited number of styles , requiring the authors to resort to a multi-policy approach . Most works considering the Actor-Critic MTRL problem focus predominantly on how the actors , i.e the policy-functions , are optimized , with less attention given to critics . The problem of negative interference between tasks in the broader context of MTRL is one that has been highlighted and studied before ( Yang et al. , 2020 ; Parisotto et al. , 2015 ; Teh et al. , 2017 ; Standley et al. , 2020 ) however , but has similarly been approached as a policy optimization problem . A number of multipolicy techniques were noted to maintain separate valuations per policy Teh et al . ( 2017 ) ; Andreas et al . ( 2017 ) ; Peng et al . ( 2018 ) , though some work has also suggested that sharing critics can allow optimization to better capture useful shared knowledge to improve policy optimization ( Dewangan et al. , 2018 ; Yang et al. , 2017 ) . A similar idea dominates single-policy techniques , where all the studied methods maintain a single value-function estimator across all tasks ( Nichol et al. , 2018 ; Yu et al. , 2020 ; Peng et al. , 2018 ; Yang et al. , 2020 ; Sodhani et al. , 2021 ) . While task-encoding may allow separate task-values to be learned implicitly by the critics , this was never found to be explicitly designed . Our work alternatively frames the multi-style learning framework as one that explicitly learns distinct per-style values to propose a value-learning optimization scheme that is better able to accommodate disparate reward functions .

Summary: The paper introduces a variant of actor-critic reinforcement learning algorithms where multiple critics are trained in a reduced version of multi-task training where a task allows multiple reward functions. Two variants are proposed for handling multiple reward functions: one where a backbone value function is trained with multiple heads and one where completely distinct value functions are trained. Evaluation considers three domains: tracing different shapes, Pong with aggressive-defensive styles, and Sonic the Hedgehog game levels (where levels are styles). Results are ambiguous as to the improvement of the architecture over classical methods. Some qualitative results are shown for a fighting game trained with the multi-critic approach.