Robust Reinforcement Learning using Adversarial Populations

1 INTRODUCTION . Developing controllers that work effectively across a wide range of potential deployment environments is one of the core challenges in engineering . The complexity of the physical world means that the models used to design controllers are often inaccurate . Optimization based control design approaches , such as reinforcement learning ( RL ) , have no notion of model inaccuracy and can lead to controllers that fail catastrophically under mismatch . In this work , we aim to demonstrate an effective method for training reinforcement learning policies that are robust to model inaccuracy by designing controllers that are effective in the presence of worst-case adversarial noise in the dynamics . An easily automated approach to inducing robustness is to formulate the problem as a zero-sum game and learn an adversary that perturbs the transition dynamics ( Tessler et al. , 2019 ; Kamalaruban et al. , 2020 ; Pinto et al. , 2017 ) . If a global Nash equilibrium of this problem is found , then that equilibrium provides a lower bound on the performance of the policy under some bounded set of perturbations . Besides the benefit of removing user design once the perturbation mechanism is specified , this approach is maximally conservative , which is useful for safety critical applications . However , the literature on learning an adversary predominantly uses a single , stochastic adversary . This raises a puzzling question : the zero-sum game does not necessarily have any pure Nash equilibria ( see Appendix C in Tessler et al . ( 2019 ) ) but the existing robust RL literature mostly appears to attempt to solve for pure Nash equilibria . That is , the most general form of the minimax problem searches over distributions of adversary and agent policies , however , this problem is approximated in the literature by a search for a single agent-adversary pair . We contend that this reduction to a single adversary approach can sometimes fail to result in improved robustness under standard parametrizations of the adversary policy . The following example provides some intuition for why using a single adversary can decrease robustness . Consider a robot trying to learn to walk east-wards while an adversary outputs a force representing wind coming from the north or the south . For a fixed , deterministic adversary the agent knows that the wind will come from either south or north and can simply apply a counteracting force at each state . Once the adversary is removed , the robot will still apply the compensatory forces and possibly become unstable . Stochastic Gaussian policies ( ubiquitous in continuous control ) offer little improvement : they can not represent multi-modal perturbations . Under these standard policy parametrizations , we can not use an adversary to endow the agent with a prior that a strong wind could persistently blow either north or south . This leaves the agent exploitable to this class of perturbations . The use of a single adversary in the robustness literature is in contrast to the multi-player game literature . In multi-player games , large sets of adversaries are used to ensure that an agent can not easily be exploited ( Vinyals et al. , 2019 ; Czarnecki et al. , 2020 ; Brown & Sandholm , 2019 ) . Drawing inspiration from this literature , we introduce RAP ( Robustness via Adversary Populations ) : a randomly initialized population of adversaries that we sample from at each rollout and train alongside the agent . Returning to our example of a robot perturbed by wind , if the robot learns to cancel the north wind effectively , then that opens a niche for an adversary to exploit by applying forces in another direction . With a population , we can endow the robot with the prior that a strong wind could come from either direction and that it must walk carefully to avoid being toppled over . Our contributions are as follows : • Using a set of continuous robotics control tasks , we provide evidence that a single adversary does not have a consistent positive impact on the robustness of an RL policy while the use of an adversary population provides improved robustness across all considered examples . • We investigate the source of the robustness and show that the single adversary policy is exploitable by new adversaries whereas policies trained with RAP are robust to new adversaries . • We demonstrate that adversary populations provide comparable robustness to domain randomization while avoiding potential failure modes of domain randomization . 2 RELATED WORK . This work builds upon robust control ( Zhou & Doyle , 1998 ) , a branch of control theory focused on finding optimal controllers under worst-case perturbations of the system dynamics . The Robust Markov Decision Process ( R-MDP ) formulation extends this worst-case model uncertainty to uncertainty sets on the transition dynamics of an MDP and demonstrates that computationally tractable solutions exist for small , tabular MDPs ( Nilim & El Ghaoui , 2005 ; Lim et al. , 2013 ) . For larger or continuous MDPs , one successful approach has been to use function approximation to compute approximate solutions to the R-MDP problem ( Tamar et al. , 2014 ) . One prominent variant of the R-MDP literature is to interpret the perturbations as an adversary and attempt to learn the distribution of the perturbation under a minimax objective . Two variants of this idea that tie in closely to our work are Robust Adversarial Reinforcement Learning ( RARL ) ( Pinto et al. , 2017 ) and Noisy Robust Markov Decision Processes ( NR-MDP ) ( Tessler et al. , 2019 ) which differ in how they parametrize the adversaries : RARL picks out specific robot joints that the adversary acts on while NR-MDP adds the adversary action to the agent action . Both of these works attempt to find an equilibrium of the minimax objective using a single adversary ; in contrast our work uses a large set of adversaries and shows improved robustness relative to a single adversary . A strong alternative to the minimax objective , domain randomization , asks a designer to explicitly define a distribution over environments that the agent should be robust to . For example , ( Peng et al. , 2018 ) varies simulator parameters to train a robot to robustly push a puck to a target location in the real world ; ( Antonova et al. , 2017 ) adds noise to friction and actions to transfer an object pivoting policy directly from simulation to a Baxter robot . Additionally , domain randomization has been successfully used to build accurate object detectors solely from simulated data ( Tobin et al. , 2017 ) and to zero-shot transfer a quadcopter flight policy from simulation ( Sadeghi & Levine , 2016 ) . The use of population based training is a standard technique in multi-agent settings . Alphastar , the grandmaster-level Starcraft bot , uses a population of `` exploiter '' agents that fine-tune against the bot to prevent it from developing exploitable strategies ( Vinyals et al. , 2019 ) . ( Czarnecki et al. , 2020 ) establishes a set of sufficient geometric conditions on games under which the use of multiple adversaries will ensure gradual improvement in the strength of the agent policy . They empirically demonstrate that learning in games can often fail to converge without populations . Finally , Active Domain Randomization ( Mehta et al. , 2019 ) is a very close approach to ours , as they use a population of adversaries to select domain randomization parameters whereas we use a population of adversaries to directly perturb the agent actions . However , they explicitly induce diversity using a repulsive term and use a discriminator to generate the reward . 3 BACKGROUND . In this work we use the framework of a multi-agent , finite-horizon , discounted , Markov Decision Process ( MDP ) ( Puterman , 1990 ) defined by a tuple 〈Aagent × Aadversary , S , T , r , γ〉 . Here Aagent is the set of actions for the agent , Aadversary is the set of actions for the adversary , S is a set of states , T : Aagent × Aadversary × S → ∆ ( S ) is a transition function , r : Aagent × Aadversary × S → R is a reward function and γ is a discount factor . S is shared between the adversaries as they share a state-space with the agent . The goal for a given MDP is to find a policy πθ parametrized by θ that maximizes the expected cumulative discounted reward Jθ = E [ ∑T t=0 γ tr ( st , at ) |πθ ] . The conditional in this expression is a short-hand to indicate that the actions in the MDP are sampled via at ∼ πθ ( st , at−1 ) . We denote the agent policy parametrized by weights θ as πθ and the policy of adversary i as π̄φi . Actions sampled from the adversary policy π̄φi will be written as ā i t. We use ξ to denote the parametrization of the system dynamics ( e.g . different values of friction , mass , wind , etc . ) and the system dynamics for a given state and action as st+1 ∼ fξ ( st , at ) . 3.1 BASELINES . Here we outline prior work and the approaches that will be compared with RAP . Our baselines consist of a single adversary and domain randomization . 3.1.1 SINGLE MINIMAX ADVERSARY . Our adversary formulation uses the Noisy Action Robust MDP ( Tessler et al. , 2019 ) in which the adversary adds its actions onto the agent actions . The objective is max θ E [ T∑ t=0 γtr ( st , at + αāt ) |πθ , π̄φ ] min φ E [ T∑ t=0 γtr ( st , at + αāt ) |πθ , π̄φ ] ( 1 ) where α is a hyperparameter controlling the adversary strength . This is a game in which the adversary and agent play simultaneously . We note an important restriction inherent to this adversarial model . Since the adversary is only able to attack the agent through the actions , there is a restricted class of dynamical systems that it can represent ; this set of dynamical systems may not necessarily align with the set of dynamical systems that the agent may be tested in . This is a restriction caused by the choice of adversarial perturbation and could be alleviated by using different adversarial parametrizations e.g . perturbing the transition function directly . 3.1.2 DYNAMICS RANDOMIZATION . Domain randomization is the setting in which the user specifies a set of environments which the agent should be robust to . This allows the user to directly encode knowledge about the likely deviations between training and testing domains . For example , the user may believe that friction is hard to measure precisely and wants to ensure that their agent is robust to variations in friction ; they then specify that the agent will be trained with a wide range of possible friction values . We use ξ to denote some vector that parametrizes the set of training environments ( e.g . friction , masses , system dynamics , etc. ) . We denote the domain over which ξ is drawn from as Ξ and use P ( Ξ ) to denote some probability distribution over ξ . The domain randomization objective is max θ Eξ∼P ( Ξ ) [ Est+1∼fξ ( st , at ) [ T∑ t=0 γtr ( st , at ) |πθ ] ] st+1 ∼ fξ ( st , at ) at ∼ πθ ( st ) ( 2 ) Here the goal is to find an agent that performs well on average across the distribution of training environment . Most commonly , and in this work , the parameters ξ are sampled uniformly over Ξ .

Summary: This paper proposes to improve robustness in reinforcement learning via a population of diverse adversaries, where previous works mainly focus on the use a single adversary to mitigate the problem that the trained policy could be highly exploitable by the adversary. Specifically, at each iteration, it randomly selects an adversary from the population for rollouts, and it is trained by PPO. Experiments are conducted on 3 MuJoCo environments in comparison with vanilla PPO, domain randomization.