Maximum Entropy RL (Provably) Solves Some Robust RL Problems

1 INTRODUCTION Many real-world applications of reinforcement learning ( RL ) require control policies that not only maximize reward but continue to do so when faced with environmental disturbances , modeling errors , or errors in reward specification . These disturbances can arise from human biases and modeling errors , from non-stationary aspects of the environment , or actual adversaries acting in the environment . A number of works have studied how to train RL algorithms to be robust to disturbances in the environment ( e.g. , Morimoto & Doya ( 2005 ) ; Pinto et al . ( 2017 ) ; Tessler et al . ( 2019 ) ) . However , designing robust RL algorithms requires care , typically requiring an adversarial optimization problem and introducing additional hyperparameters ( Pinto et al. , 2017 ; Tessler et al. , 2019 ) . Instead of designing a new robust RL algorithm , we will instead analyze whether an existing RL method , Max- Ent RL , offers robustness to perturbations . One widely used class of RL methods is based on the maximum entropy principle , where the expected reward objective is additionally augmented with an entropy maximization term ( Ziebart et al. , 2008 ) . Prior work has conjectured that such algorithms should learn robust policies ( Haarnoja et al. , 2017 ; Huang et al. , 2019 ) , on account of the close connection between maximum entropy methods and robustness in supervised learning domains ( Grünwald et al. , 2004 ; Ziebart , 2010 ) . However , despite decades of research on maximum entropy RL methods ( Kappen , 2005 ; Todorov , 2007 ; Toussaint , 2009 ; Theodorou et al. , 2010 ) , a formal characterization of the robustness properties of such approaches has proven elusive . To our knowledge , no prior work formally proves that MaxEnt RL methods are robust , and no prior work characterizes the disturbances against which these methods are robust . Showing we could obtain robust policies from existing MaxEnt RL methods , which already constitute a significant portion of the RL methods in use today ( Abdolmaleki et al. , 2018 ; Haarnoja et al. , 2018a ; Vieillard et al. , 2020 ) , would be useful because it would enable practitioners to leverage existing , tried-and-true methods to solve robust RL problems . Stochastic policies , of the sort learned with MaxEnt RL , inject noise into the actions during training , thereby preparing themselves for deployment in environments with disturbances . For example , in the robot pushing task shown in Fig . 1 , the policy learned by MaxEnt RL pushes the white puck to the goal using many possible routes . In contrast , ( standard ) RL learns a deterministic policy , always using the same route to get to the goal . Now , imagine that this environment is perturbed by adding the red barrier in Fig . 1 . While the policy learned by ( standard ) RL always collides with this obstacle , the policy learned by MaxEnt RL uses many routes to solve the task , and some fraction of these routes continue to solve the task even when the obstacle is present . While a number of prior works have articulated the intuition that the stochastic policies learned via MaxEnt RL should be robust to disturbances ( Levine , 2018 ; Abdolmaleki et al. , 2018 ; Lee et al. , 2019 ) , no prior work has actually shown that MaxEnt RL policies are provably robust , nor characterized the set of disturbances to which they are robust . Applications of MaxEnt RL methods to problems that demand robustness are likely significantly hampered by a lack of understanding of when such methods are robust , what kinds of reward functions should be used to obtain the desired type of robustness , and how the task should be set up . The goal in our work is to make this notion precise , proving that MaxEnt RL is already a robust RL algorithm , and deriving the robust set for these policies . The main contribution of this paper is a theoretical characterization of the robustness of existing MaxEnt RL methods . Specifically , we show that the MaxEnt RL objective is a nontrivial lower bound on the robust RL objective , for a particular robust set . Importantly , these two objectives use different reward functions : to learn a policy that maximizes a reward function on a wide range of dynamics functions , one should apply MaxEnt RL to a pessimistic version of that reward function . MaxEnt RL is not robust with respect to the reward function it is trained on . As we will show , the robust set for these methods is non-trivial , but in some important special cases can be quite intuitive . To our knowledge , our results are the first to formally show robustness of standard MaxEnt RL methods to dynamics perturbations and characterize their robust set . Importantly , our analysis and experiments highlight that robustness is only achieved for relatively large entropy coefficients . We validate our theoretical results on a set of illustrative empirical experiments . 2 RELATED WORK . The study of robustness in controls has a long history , with robust or H∞ control methods proposing provably robust solutions under various assumptions on the true dynamics ( Zhou et al. , 1996 ; Doyle et al. , 2013 ) . RL offers an appealing alternative , since robust RL methods can in principle learn robust policies without knowing the true dynamics . However , policies learned by standard RL algorithms often flounder in the face of environmental disturbances ( Rajeswaran et al. , 2016 ; Peng et al. , 2018 ) . The problem of learning robust policies that achieve high reward in the face of adversarial environmental disturbances , often called robust RL , has been well studied in the literature ( Bagnell et al. , 2001 ; Nilim & Ghaoui , 2003 ; Morimoto & Doya , 2005 ; Pinto et al. , 2017 ; Tessler et al. , 2019 ; Russel & Petrik , 2019 ; Kamalaruban et al. , 2020 ; Russel et al. , 2020 ; 2021 ; Derman et al. , 2021 ) . Prior robust RL methods are widely-applicable but often require many additional hyperparameters or components . For example , Bagnell et al . ( 2001 ) modify the Bellman backup in a Q-learning by solving a convex optimization problem in an inner loop , and Tessler et al . ( 2019 ) trains an additional adversary policy via RL . Robust RL is different from robust control in the traditional sense , which focuses on stability independent of any reward function ( Zhou et al. , 1996 ; Doyle et al. , 2013 ) . Robust RL problem involves estimating a policy ’ s returns under “ similar ” MDPs , a problem that has been studied in terms of distance metrics for MDPs ( Lecarpentier et al. , 2020 ) . Robust RL is different from maximizing the average reward across many environments , as done by methods such as domain randomization ( Sadeghi & Levine , 2016 ; Rajeswaran et al. , 2016 ; Peng et al. , 2018 ) . Closely related to robust RL are prior methods that are robust to disturbances in the reward function ( Hadfield-Menell et al. , 2017 ; Bobu et al. , 2020 ; Michaud et al. , 2020 ) , or aim to minimize a cost function in addition to maximizing reward ( Chow et al. , 2017 ; Achiam et al. , 2017 ; Chow et al. , 2019 ; Carrara et al. , 2019 ; Tang et al. , 2019 ; Srinivasan et al. , 2020 ; Thananjeyan et al. , 2020 ) . Robust RL is also different from the problem of learning transferable or generalizable RL agents ( Lazaric , 2012 ; Zhang et al. , 2018 ; Justesen et al. , 2018 ; Cobbe et al. , 2019 ) , which focuses on the average-case performance on new environments , rather than the worst-case performance . 3 PRELIMINARIES . We assume that an agent observes states st , takes actions at ∼ π ( at | st ) , and obtains rewards r ( st , at ) . The initial state is sampled s1 ∼ p1 ( s1 ) , and subsequent states are sampled st+1 ∼ p ( st+1 | st , at ) . We will use pπ ( τ ) to denote the probability distribution over trajectories for policy π. Episodes have T steps , which we summarize as a trajectory τ , ( s1 , a1 , · · · , sT , aT ) . Without loss of generality , we can assume that rewards are undiscounted , as any discount can be addressed by modifying the dynamics to transition to an absorbing state with probability 1− γ . The standard RL objective is : arg max π Eτ∼pπ ( τ ) [ T∑ t=1 r ( st , at ) ] , where pπ ( τ ) = p1 ( s1 ) T∏ t=1 p ( st+1 | st , at ) π ( at | st ) dτ is the distribution over trajectories when using policy π . In fully observed MDPs , there always exists a deterministic policy as a solution ( Puterman , 2014 ) . The MaxEnt RL objective is to maximize the sum of expected reward and conditional action entropy . Formally , we define this objective in terms of a policy π , the dynamics p and the reward function r : JMaxEnt ( π ; p , r ) , Eat∼π ( at|st ) , st+1∼p ( st+1|st , at ) [ T∑ t=1 r ( st , at ) + αHπ [ at | st ] ] , ( 1 ) whereHπ [ at | st ] = ∫ A π ( at | st ) log 1 π ( at|st ) dat denotes the entropy of the action distribution . The entropy coefficient α balances the reward term and the entropy term ; we use α = 1 in our analysis . As noted above , the discount factor is omitted because it can be subsumed into the dynamics . Our main result will be that maximizing the MaxEnt RL objective ( Eq . 1 ) results in robust policies . We quantify robustness by measuring the reward of a policy when evaluated on a new reward function r̃ or dynamics function p̃ , which is chosen adversarially from some set : max π min p̃∈P̃ , r̃∈R̃ Ep̃ ( st+1|st , at ) , π ( at|st ) [ T∑ t=1 r̃ ( st , at ) ] . This robust RL objective is defined in terms of the sets of dynamics P̃ and reward functions R̃ . Our goal is to characterize these sets . The robust RL objective can be interpreted as a two-player , zero-sum game . The aim is to find a Nash equilibrium . Our goal is to prove that MaxEnt RL ( with a different reward function ) optimizes a lower bound this robust objective , and to characterize the robust sets P̃ and R̃ for which this bound holds . 4 MAXENT RL AND ROBUST CONTROL . In this section , we prove the conjecture that MaxEnt RL is robust to disturbances in the environment . Our main result is that MaxEnt RL can be used to maximize a lower bound on a certain robust RL objective . Importantly , doing this requires that MaxEnt RL be applied to a different , pessimistic , version of the target reward function . Before presenting our main result , we prove that MaxEnt RL is robust against disturbances to the reward function . This result is a simple extension of prior work , and we will use this result for proving our main result about dynamics robustness

Summary: This work analyzes the implicit robustness of maximum entropy RL. They show, theoretically, that there exists some robust MDP for which the optimal solution is obtained by maximizing the max-ent objective. Empirically they show that indeed SAC learns behavior which is in a sense robust.