MAXIMUM ENTROPY RL (PROVABLY) SOLVES SOME ROBUST RL PROBLEMS
Year: 2021

Abstract:
Many potential applications of reinforcement learning (RL) require guarantees that the agent will perform well in the face of disturbances to the dynamics or reward function. In this paper, we prove theoretically that maximum entropy (MaxEnt) RL maximizes a lower bound on a robust RL objective, and thus can be used to learn policies that are robust to some disturbances in the dynamics and the reward function. While this capability of MaxEnt RL has been observed empirically in prior work, to the best of our knowledge our work provides the first rigorous proof and theoretical characterization of the MaxEnt RL robust set. While a number of prior robust RL algorithms have been designed to handle similar disturbances to the reward function or dynamics, these methods typically require additional moving parts and hyperparameters on top of a base RL algorithm. In contrast, our results suggest that MaxEnt RL by itself is robust to certain disturbances, without requiring any additional modifications. While this does not imply that MaxEnt RL is the best available robust RL method, MaxEnt RL is a simple robust RL method with appealing formal guarantees.

text: 1 INTRODUCTION
Many real-world applications of reinforcement learning (RL) require control policies that not only maximize reward but continue to do so when faced with environmental disturbances, modeling errors, or errors in reward specification. These disturbances can arise from human biases and modeling errors, from non-stationary aspects of the environment, or actual adversaries acting in the environment. A number of works have studied how to train RL algorithms to be robust to disturbances in the environment (e.g., Morimoto & Doya (2005); Pinto et al. (2017); Tessler et al. (2019)). However, designing robust RL algorithms requires care, typically requiring an adversarial optimization problem and introducing additional hyperparameters (Pinto et al., 2017; Tessler et al., 2019). Instead of designing a new robust RL algorithm, we will instead analyze whether an existing RL method, Max-
Ent RL, offers robustness to perturbations. One widely used class of RL methods is based on the maximum entropy principle, where the expected reward objective is additionally augmented with an entropy maximization term (Ziebart et al., 2008). Prior work has conjectured that such algorithms should learn robust policies (Haarnoja et al., 2017; Huang et al., 2019), on account of the close connection between maximum entropy methods and robustness in supervised learning domains (Grünwald et al., 2004; Ziebart, 2010). However, despite decades of research on maximum entropy RL methods (Kappen, 2005; Todorov, 2007; Toussaint, 2009; Theodorou et al., 2010), a formal characterization of the robustness properties of such approaches has proven elusive. To our knowledge, no prior work formally proves that MaxEnt RL methods are robust, and no prior work characterizes the disturbances against which these methods are robust. Showing we could obtain
robust policies from existing MaxEnt RL methods, which already constitute a significant portion of the RL methods in use today (Abdolmaleki et al., 2018; Haarnoja et al., 2018a; Vieillard et al., 2020), would be useful because it would enable practitioners to leverage existing, tried-and-true methods to solve robust RL problems.
Stochastic policies, of the sort learned with MaxEnt RL, inject noise into the actions during training, thereby preparing themselves for deployment in environments with disturbances. For example, in the robot pushing task shown in Fig. 1, the policy learned by MaxEnt RL pushes the white puck to the goal using many possible routes. In contrast, (standard) RL learns a deterministic policy, always using the same route to get to the goal. Now, imagine that this environment is perturbed by adding the red barrier in Fig. 1. While the policy learned by (standard) RL always collides with this obstacle, the policy learned by MaxEnt RL uses many routes to solve the task, and some fraction of these routes continue to solve the task even when the obstacle is present. While a number of prior works have articulated the intuition that the stochastic policies learned via MaxEnt RL should be robust to disturbances (Levine, 2018; Abdolmaleki et al., 2018; Lee et al., 2019), no prior work has actually shown that MaxEnt RL policies are provably robust, nor characterized the set of disturbances to which they are robust. Applications of MaxEnt RL methods to problems that demand robustness are likely significantly hampered by a lack of understanding of when such methods are robust, what kinds of reward functions should be used to obtain the desired type of robustness, and how the task should be set up. The goal in our work is to make this notion precise, proving that MaxEnt RL is already a robust RL algorithm, and deriving the robust set for these policies.
The main contribution of this paper is a theoretical characterization of the robustness of existing MaxEnt RL methods. Specifically, we show that the MaxEnt RL objective is a nontrivial lower bound on the robust RL objective, for a particular robust set. Importantly, these two objectives use different reward functions: to learn a policy that maximizes a reward function on a wide range of dynamics functions, one should apply MaxEnt RL to a pessimistic version of that reward function. MaxEnt RL is not robust with respect to the reward function it is trained on. As we will show, the robust set for these methods is non-trivial, but in some important special cases can be quite intuitive. To our knowledge, our results are the first to formally show robustness of standard MaxEnt RL methods to dynamics perturbations and characterize their robust set. Importantly, our analysis and experiments highlight that robustness is only achieved for relatively large entropy coefficients. We validate our theoretical results on a set of illustrative empirical experiments.

heading: 2 RELATED WORK
text: The study of robustness in controls has a long history, with robust or H∞ control methods proposing provably robust solutions under various assumptions on the true dynamics (Zhou et al., 1996; Doyle et al., 2013). RL offers an appealing alternative, since robust RL methods can in principle learn robust policies without knowing the true dynamics. However, policies learned by standard RL algorithms often flounder in the face of environmental disturbances (Rajeswaran et al., 2016; Peng et al., 2018). The problem of learning robust policies that achieve high reward in the face of adversarial environmental disturbances, often called robust RL, has been well studied in the literature (Bagnell et al., 2001; Nilim & Ghaoui, 2003; Morimoto & Doya, 2005; Pinto et al., 2017; Tessler et al., 2019; Russel & Petrik, 2019; Kamalaruban et al., 2020; Russel et al., 2020; 2021; Derman et al., 2021). Prior robust RL methods are widely-applicable but often require many additional hyperparameters or components. For example, Bagnell et al. (2001) modify the Bellman backup in a Q-learning by solving a convex optimization problem in an inner loop, and Tessler et al. (2019) trains an additional adversary policy via RL. Robust RL is different from robust control in the traditional sense, which focuses on stability independent of any reward function (Zhou et al., 1996; Doyle et al., 2013). Robust RL problem involves estimating a policy’s returns under “similar” MDPs, a problem that has been studied in terms of distance metrics for MDPs (Lecarpentier et al., 2020).
Robust RL is different from maximizing the average reward across many environments, as done by methods such as domain randomization (Sadeghi & Levine, 2016; Rajeswaran et al., 2016; Peng et al., 2018). Closely related to robust RL are prior methods that are robust to disturbances in the reward function (Hadfield-Menell et al., 2017; Bobu et al., 2020; Michaud et al., 2020), or aim to minimize a cost function in addition to maximizing reward (Chow et al., 2017; Achiam et al., 2017; Chow et al., 2019; Carrara et al., 2019; Tang et al., 2019; Srinivasan et al., 2020; Thananjeyan et al., 2020). Robust RL is also different from the problem of learning transferable or generalizable RL
agents (Lazaric, 2012; Zhang et al., 2018; Justesen et al., 2018; Cobbe et al., 2019), which focuses on the average-case performance on new environments, rather than the worst-case performance.

heading: 3 PRELIMINARIES
text: We assume that an agent observes states st, takes actions at ∼ π(at | st), and obtains rewards r(st,at). The initial state is sampled s1 ∼ p1(s1), and subsequent states are sampled st+1 ∼ p(st+1 | st,at). We will use pπ(τ) to denote the probability distribution over trajectories for policy π. Episodes have T steps, which we summarize as a trajectory τ , (s1,a1, · · · , sT,aT). Without loss of generality, we can assume that rewards are undiscounted, as any discount can be addressed by modifying the dynamics to transition to an absorbing state with probability 1− γ. The standard RL objective is:
arg max π Eτ∼pπ(τ) [ T∑ t=1 r(st,at) ] , where pπ(τ) = p1(s1) T∏ t=1 p(st+1 | st,at)π(at | st)dτ
is the distribution over trajectories when using policy π. In fully observed MDPs, there always exists a deterministic policy as a solution (Puterman, 2014). The MaxEnt RL objective is to maximize the sum of expected reward and conditional action entropy. Formally, we define this objective in terms of a policy π, the dynamics p and the reward function r:
JMaxEnt(π; p, r) , Eat∼π(at|st),st+1∼p(st+1|st,at) [ T∑ t=1 r(st,at) + αHπ[at | st] ] , (1)
whereHπ[at | st] = ∫ A π(at | st) log 1 π(at|st)dat denotes the entropy of the action distribution. The entropy coefficient α balances the reward term and the entropy term; we use α = 1 in our analysis. As noted above, the discount factor is omitted because it can be subsumed into the dynamics.
Our main result will be that maximizing the MaxEnt RL objective (Eq. 1) results in robust policies. We quantify robustness by measuring the reward of a policy when evaluated on a new reward function r̃ or dynamics function p̃, which is chosen adversarially from some set:
max π min p̃∈P̃,r̃∈R̃ Ep̃(st+1|st,at),π(at|st) [ T∑ t=1 r̃(st,at) ] .
This robust RL objective is defined in terms of the sets of dynamics P̃ and reward functions R̃. Our goal is to characterize these sets. The robust RL objective can be interpreted as a two-player, zero-sum game. The aim is to find a Nash equilibrium. Our goal is to prove that MaxEnt RL (with a different reward function) optimizes a lower bound this robust objective, and to characterize the robust sets P̃ and R̃ for which this bound holds.

heading: 4 MAXENT RL AND ROBUST CONTROL
text: In this section, we prove the conjecture that MaxEnt RL is robust to disturbances in the environment. Our main result is that MaxEnt RL can be used to maximize a lower bound on a certain robust RL objective. Importantly, doing this requires that MaxEnt RL be applied to a different, pessimistic, version of the target reward function. Before presenting our main result, we prove that MaxEnt RL is robust against disturbances to the reward function. This result is a simple extension of prior work, and we will use this result for proving our main result about dynamics robustness

heading: 4.1 ROBUSTNESS TO ADVERSARIAL REWARD FUNCTIONS
text: We first show that MaxEnt RL is robust to some degree of misspecification in the reward function. This result may be useful in practical settings with misspecified reward function (Amodei et al., 2016; Clark & Amodei, 2016). Precisely, the following result will show that applying MaxEnt RL to one reward function, r(st,at), results in a policy that is guaranteed to also achieve high return on a range of other reward functions, r̃(st,at) ∈ R̃:
Theorem 4.1. Let dynamics p(st+1 | st,at), policy π(at | st), and reward function r(st,at) be given. Assume that the reward function is finite and that the policy has support everywhere (i.e., π(at | st) > 0 for all states and actions). Then there exists a positive constant > 0 such that the MaxEnt RL objective JMaxEnt is equivalent to the robust RL objective defined by the robust set R̃(π):
min r̃∈R̃(π)
E [∑
t
r̃(st,at) ] = JMaxEnt(π; p, r) ∀π,
where the adversary chooses a reward function from the set
R̃(π) , { r̃(st,at) ∣∣∣∣ Eπ[∑ t log ∫ A exp(r(st,at ′)− r̃(st,at′))dat′ ] ≤ } . (2)
Thus, when we use MaxEnt RL with some reward function r, the policy obtained is guaranteed to also obtain high reward on all similar reward functions r̃ that satisfy Eq. 2. We discuss how MaxEnt RL can be used to learn policies robust to arbitrary sets of reward functions in Appendix A.10. The proof can be found in Appendix A.2, and is a simple extension of prior work (Grünwald et al., 2004; Ziebart et al., 2011). Our proof does not assume that the reward function is bounded nor that the policy is convex. While the proof is straightforward, it will be useful as an intermediate step when proving robustness to dynamics in the next subsection.
The robust set R̃ corresponds to reward functions r̃ that are not too much smaller than the original reward function: the adversary cannot decrease the reward for any state or action too much. The robust set R̃ depends on the policy, so the adversary has the capability of looking at which states the policy visits before choosing the adversarial reward function r̃. For example, the adversary may choose to apply larger perturbations at states and actions that the agent frequently visits.

heading: 4.2 ROBUSTNESS TO ADVERSARIAL DYNAMICS
text: We now show that MaxEnt RL learns policies that are robust to perturbations to the dynamics. Importantly, to have MaxEnt RL learn policies that robustly maximize one a reward function, we will apply MaxEnt RL to a different, pessimistic reward function:
r̄(st,at, st+1) , 1
T log r(st,at) +H[st+1 | st,at]. (3)
The log(·) transformation is common in prior work on learning risk-averse policies (Mihatsch & Neuneier, 2002). The entropy term rewards the policy for vising states that have stochastic dynamics, which should make the policy harder for the adversary to exploit. In environments that have the same stochasticity at every state (e.g., LQ dynamics), this entropy term becomes a constant and can be ignored. In more general settings, computing this pessimistic reward function would require some knowledge of the dynamics. Despite this limitation, we believe that our results may be of theoretical interest, taking a step towards explaining the empirically-observed robustness properties of MaxEnt RL.
To formally state our main result, we must define the range of “similar” dynamics functions against which the policy will be robust. We use the following divergence between two dynamics functions:
d(p, p̃; τ) , ∑ st∈τ log ∫∫ A×S p(st+1 ′ | st,at′) p̃(st+1′ | st,at′) dat ′dst+1 ′. (4)
This divergence is large when the adversary’s dynamics p̃(st+1 | st,at) assign low probability to a transition with high probability in the training environment p̃. Our main result shows that applying MaxEnt RL to the pessimistic reward function results in a policy that is robust to these similar dynamics functions: Theorem 4.2. Let an MDP with dynamics p(st+1 | st,at) and reward function r(st,at) > 0 be given. Assume that the dynamics have finite entropy (i.e.,H[st+1 | st,at] is finite for all states and actions). Then there exists a constant > Hπ∗ [at | st] such that the MaxEnt RL objective with dynamics p and reward function r̄ is a lower bound on the robust RL objective
min p̃∈P̃(π)
JMaxEnt(π; p̃, r) ≥ exp(JMaxEnt(π; p, r̄) + log T ),
where the robust set is defined as P̃ , { p̃(s′ | s,a) ∣∣∣∣ E π(at|st) p(st+1|st,at) [d(p, p̃; τ)] ≤ } . (5)
In defining the robust set, the adversary chooses a dynamics function from the robust set P̃ independently at each time step; our next result (Lemma 4.3) will describe the value of in more detail. The proof, presented in Appendix A.4, first shows the robustness to reward perturbations implies robustness to dynamics perturbations and then invokes Theorem 4.1 to show that MaxEnt RL learns policies that are robust to reward perturbations.
This result can be interpreted in two ways. First, if a user wants to acquire a policy that optimizes a reward under many possible dynamics, we should run MaxEnt RL with a specific, pessimistic reward function r̄(st,at, st+1). This pessimistic reward function depends on the environment dynamics, so it may be hard to compute without prior knowledge or a good model of the dynamics. However, in some settings (e.g., dynamics with constant additive noise), we might assume that H[st+1 | st,at] is approximately constant, in which case we simply set the MaxEnt RL reward to r̄(st,at, st+1) = log r(st,at). Second, this result says that every time a user applies MaxEnt RL, they are (implicitly) solving a robust RL problem, one defined in terms of a different reward function. This connection may help explain why prior work has found that the policies learned by MaxEnt RL tend to be robust against disturbances to the environment (Haarnoja et al., 2018b).
Theorem 4.2 relates one MaxEnt RL problem to another (robust) MaxEnt RL problem. We can also show that MaxEnt RL maximizes a lower bound on an unregularized RL problem.
Corollary 4.2.1. Under the same assumptions as Theorem 4.1 and 4.2, the MaxEnt RL problem is a lower bound on the robust RL objective:
min p̃∈P̃(π),r̃∈R̃(π) Ep̃(st+1|st,at),π(at|st) [∑ t r̃(st,at) ] ≥ exp(JMaxEnt(π; p, r̄) + log T ),
where the robust sets P̃(π) and R̃(π) as defined as in Theorem 4.1 and 4.2.
Our next result analyzes the size of the robust set by providing a lower bound on . Doing so proves that Theorem 4.2 is non-vacuous and will also tell us how to increase the size of the robust set.
Lemma 4.3. Assume that the action space is discrete. Let a reward function r(st,at) and dynamics p(st+1 | st,at) be given. Let π(at | st) be the corresponding policy learned by MaxEnt RL. Then the size of (in Eq. 5) satisfies the following:
= T · E at∼π(at|st), st+1∼p(st+1|st,at) [Hp̃[st+1 | st,at] +Hπ[at | st]] ≥ T · E at∼π(at|st), st+1∼p(st+1|st,at) [Hπ[at | st]] .
This result provides an exact expression for . Perhaps more interesting is the inequality, which says that the size of the robust set is at least as large as the policy’s entropy. For example, if MaxEnt RL learns a policy with an entropy of 10 bits, then ≥ 10. This result immediately tells us how to increase the size of the robust set: run MaxEnt RL with a larger entropy coefficient α. Many popular implementations of MaxEnt RL automatically tune the entropy coefficient α so that the policy satisfies an entropy constraint (Haarnoja et al., 2018c). Our derivation here suggests that the entropy constraint on the policy corresponds to a lower bound on the size of the robust set.
The constraint in the definition of the robust set holds in expectation. Thus, the adversary can make large perturbations to some transitions and smaller perturbations to other states, as long as the average perturbation is not too large. Intuitively, the constraint value is the adversary’s “budget,” which it can use to make large changes to the dynamics in just a few states, or to make smaller changes to the dynamics in many states.

heading: 4.3 WORKED EXAMPLES
text: This section provides worked examples of our robustness results. The aim is to build intuition for what our results show and to show that, in simple problems, the robust set has an intuitive interpretation.
Reward robustness. We present two worked examples of reward robustness. To simplify analysis, we consider the following, smaller robust set, which effectively corresponds to a weaker adversary:{
r(st,at) ∣∣∣∣ log ∫ A exp(r(st,at)− r̃(st,at))dat′ ≤ T } ⊆ R̃. (6)
For the first example, define a 2-armed bandit with the following reward function and corresponding robust set:
r(a) =
{ 2 a = 1
1 a = 2 , R̃ =
{ r̃ ∣∣∣ log ∫
A exp(r(a)− r̃(a))da ≤
} .
Fig. 2 (left) plots the original reward function, r as a red dot, and the collection of reward functions, R̃, as a blue line. In the center plot we plot the expected reward for each reward in R̃ as a function of the action a. The robust RL problem in this setting is to choose the policy whose worst-case reward (dark blue line) is largest. The right plot shows the MaxEnt RL objective. We observe that the robust RL objective and the MaxEnt RL objectives are equivalent.
For the second example, we use a task with a 1D, bounded action spaceA = [−10, 10] and a reward function composed of a task-specific reward rtask and a penalty for deviating from some desired action a∗: r(s,a) , rtask(s,a) − (a − a∗)2. The adversary will perturb this desired action by an amount ∆a and decrease the weight on the control penalty by 50%, resulting in the following reward function: r̃(s,a) , rtask(s,a)− 12 (a− (a
∗ + ∆a))2. In this example, the subset of the robust set in Eq. 6 corresponds to perturbations ∆a that satisfy
∆a2 + 1
2 log(2π) + log(20) ≤ T .
Thus, MaxEnt RL with reward function r yields a policy that is robust against adversaries that perturb a∗ by at most ∆a = O( √ ). See Appendix A.3 for the full derivation.
Dynamics robustness. The set of dynamics we are robust against, P̃ , has an intuitive interpretation as those that are sufficiently close to the original dynamics p(st+1 | st,at). This section shows that, in the case of linear-Gaussian dynamics (described at the end of this subsection), this set corresponds to a bounded L2 perturbation of the next state.
Because the robust set in Theorem 4.2 is defined in terms of the policy, the adversary can intelligently choose where to perturb the dynamics based on the policy’s behavior. Robustness against this adversary also guarantees robustness against an adversary with a smaller robust set, that does not depend on the policy:{
p̃(st+1 | st,at) ∣∣∣∣ log ∫∫
A×S elog p(st+1|st,at)−log p̃(st+1|st,at)dat ′dst+1 ′ ≤ T
} ⊆ P̃.
We will use this subset of the robust set in the following worked example.
Consider an MDP with 1D, bounded, states and actions st,at ∈ [−10, 10]. The dynamics are p(st+1 | st,at) = N (st+1;µ = Ast +Bat, σ = 1) the reward function is r(st,at) = ‖st‖22, and episodes have finite length T . Note that the dynamics entropyH[st+1 | st,at] is constant. We assume
the adversary modifies the dynamics by increasing the standard deviation to √
2 and shifts the bias by an amount β, resulting in the dynamics p̃(st+1 | st,at) = N (st+1;µ = Ast +Bat + β, σ = √ 2). The robust set defined in Theorem 4.2 specifies that the adversary can choose any value of β that satisfies 12β 2 + log(8 √ π) + log(20) ≤ . To apply MaxEnt RL to learn a policy that is robust to any of these adversarial dynamics, we would use the pessimistic reward function specified by Theorem 4.2: r̄(st,at, st+1) = 2T log ‖st‖2. See Appendix A.7 for the full derivation.

heading: 4.4 LIMITATIONS OF ANALYSIS
text: We identify a few limitations of our analysis that may provide directions for future work. First, our definition of the robust set is different from the more standard H∞ and KL divergence constraint sets used in prior work. Determining a relationship between these two different sets would allow future work to claim that MaxEnt RL is also robust to these more standard constraint sets (see Lecarpentier et al. (2020)). On the other hand, MaxEnt RL may not be robust to these more conventional constraint sets. Showing this may inform practitioners about what sorts of robustness they should not except to reap from MaxEnt RL. A second limitation is the construction of the augmented reward: to learn a policy that will maximize reward function r under a range of possible dynamics functions, our theory says to apply MaxEnt RL to a different reward function, r̄, which includes a term depending on the dynamics entropy. While this term can be ignored in special MDPs that have the same stochasticity at every state, in more general MDPs it will be challenging to estimate this augmented reward function. Determining more tractable ways to estimate this augmented reward function is an important direction for future work.

heading: 5 NUMERICAL SIMULATIONS
text: This section will present numerical simulations verifying our theoretical result that MaxEnt RL is robust to disturbances in the reward function and dynamics function. We describe our precise implementation of MaxEnt RL, standard RL, and all environments in Appendix B.
Comparison with prior robust RL methods. We compare MaxEnt RL against two recent robust RL methods, PR-MDP and NR-MDP (Tessler et al., 2019). These methods construct a two-player game between a player that chooses actions and a player that perturbs the dynamics or actions. This recipe for robust RL is common in prior work (Pinto et al., 2017), and we choose to compare to Tessler et al. (2019) because it is a recent, high-performing instantiation of this recipe. We also include the DDPG baseline from Tessler et al. (2019). We evaluate all methods on the benchmark tasks used in Tessler et al. (2019), which involve evaluating in an environment where the masses are different from the training environment. We ran MaxEnt RL with both small and large entropy coefficients, evaluating the final policy for 30 episodes and taking the average performance. We repeated this for 5 random seeds to obtain the standard deviation. The results shown in Fig. 3 suggest that MaxEnt RL with a large entropy coefficient α is at least competitive, if not better, than prior purpose-designed robust RL methods. Note that MaxEnt RL is substantially simpler than PR-MDP and NR-MDP.
Intuition for why MaxEnt RL is robust. To build intuition into why MaxEnt RL should yield robust policies, we compared MaxEnt RL (SAC (Haarnoja et al., 2018a)) with standard RL (TD3 (Fujimoto et al., 2018)) on two tasks. On the pusher task, shown in Fig. 1, a new obstacle was added during evaluation. On the button task, shown in Fig. 4b, the box holding the button was moved closer or further away from the robot during evaluation. Note that the robust RL objective corresponds to an adversary choosing the worst-case disturbances to these environments.
Fig. 1 (right) and Fig. 4b (center) show that MaxEnt RL has learned many ways of solving these tasks, using different routes to push the puck to the goal and using different poses to press the button. Thus,
when we evaluate the MaxEnt RL policy on perturbed environments, it is not surprising that some of these strategies continue to solve the task. In contrast, the policy learned by standard RL always uses the same strategy to solve these tasks, so the agent fails when a perturbation to the environment makes this strategy fail. Quantitatively, Fig. 4a and 4b (right) show that the MaxEnt RL policy is more robust than a policy trained with standard RL.
In many situations, simply adding noise to the deterministic policy found by standard RL can make that policy robust to some disturbances. MaxEnt RL does something more complex, dynamically adjusting the amount of noise depending on the current state. This capability allows MaxEnt RL policies to have lower entropy in some states as needed to ensure high reward. We study this capability in the 2D navigation task shown in Fig. 5. The agent starts near the top left corner and gets reward for navigating to the bottom right hand corner, but incurs a large cost for entering the red regions. The policy learned
by MaxEnt RL has low entropy initially to avoid colliding with the red obstacles, and then increases its entropy in the second half of the episode. To study robustness, we introduce a new “L”-shaped obstacle for evaluation. The policy learned by MaxEnt RL often navigates around this obstacle, whereas the policy from standard RL always collides with this obstacle. Adding independent Gaussian noise to the actions from the standard RL policy can enable that policy to navigate around the obstacle, but only at the cost of entering the costly red states.
Testing for different types of robustness. Most prior work on robust RL focuses on changing static attributes of the environment, such as the mass or position of objects (Tessler et al., 2019; Kamalaruban et al., 2020). However, our analysis suggests that MaxEnt RL is robust against a wider range of perturbations, which we probe in our next set of experiments.
First, we introduced perturbations in the middle of an episode. We took the pushing task shown in Fig. 1 and, instead of adding an obstacle, perturbed the XY position of the puck after 20 time steps. By evaluating the reward of a policy while varying the size of this perturbation, we can study the range of disturbances to which MaxEnt RL is robust. We measured the performance of MaxEnt RL policies trained with different entropy coefficients α. The results shown in Fig. 6 indicate all methods perform well on the environment without any disturbances, but only the MaxEnt RL trained with the largest entropy coefficient is robust to larger disturbances. This experiment supports our theory that the entropy coefficient determines the size of the robust set.
Our analysis suggests that MaxEnt RL is not only robust to random perturbations, but is actually robust against adversarial perturbations. We next compare MaxEnt RL and standard RL in this
setting. We trained both algorithms on a peg insertion task shown in Fig. 7 (left). Visualizing the learned policy, we observe that the standard RL policy always takes the same route to the goal, whereas the MaxEnt RL policy uses many different routes to get to the goal. For each policy we found the worst-case perturbation to the hole location using CMA-ES (Hansen, 2016). For small perturbations, both standard RL and MaxEnt RL achieve a success rate near 100%. However, for perturbations that are 1.5cm or 2cm in size, only MaxEnt RL continues to solve the task. For larger perturbations neither method can solve the task. In summary, this experiment highlights that MaxEnt RL is robust to adversarial disturbances, as predicted by the theory.
Finally, our analysis suggests that MaxEnt RL is also robust to perturbations to the reward function. To test this theoretical result, we apply MaxEnt RL on four continuous control tasks from the standard OpenAI Gym (Brockman et al., 2016) benchmark. We compare to SVG-0 (Heess et al., 2015) (which uses stochastic policy gradients) and to fictitious play (Brown, 1951). In the RL setting, fictitious play corresponds to modifying standard RL to use an adversarially-chosen reward function for each Bellman update. We evaluate the policy on an adversarially chosen reward function, chosen from the set defined in Equation 2. The analytic solution for this worst-case reward function is r̃(st,at) = r(st,at)− log π(at | st). Both MaxEnt RL and standard RL can maximize the cumulative reward (see Fig. 11 in Appendix B), but only MaxEnt RL succeeds as maximizing the worst-case reward, as shown in Fig. 8. In summary, this experiment supports our proof that MaxEnt RL solves a robust RL problem for the set of rewards specified in Theorem 4.1.

heading: 6 DISCUSSION
text: In this paper, we formally showed that MaxEnt RL algorithms optimize a bound on a robust RL objective. This robust RL objective uses a different reward function than the MaxEnt RL objective. Our analysis characterizes the robust sets for both reward and dynamics perturbations, and provides intuition for how such algorithms should be used for robust RL problems. To our knowledge, our work is the first to formally characterize the robustness of MaxEnt RL algorithms, despite the fact that numerous papers have conjectured that such robustness results may be possible. Our experimental evaluation shows that, in line with our theoretical findings, simple MaxEnt RL algorithms perform competitively with (and sometimes better than) recently proposed adversarial robust RL methods on benchmarks proposed by those works.
Of course, MaxEnt RL methods are not necessarily the ideal approach to robustness: applying such methods still requires choosing a hyperparameter (the entropy coefficient), and the robust set for MaxEnt RL is not always simple. Nonetheless, we believe that the analysis in this paper, represents an important step towards a deeper theoretical understanding of the connections between robustness and entropy regularization in RL. We hope that this analysis will open the door for the development of new, simple algorithms for robust RL.

heading: ETHICS STATEMENT
text: While we believe that making RL algorithms more robust will be generally useful, we do not believe that our theoretical analysis has any direct ethical implications.

heading: REPRODUCIBILITY STATEMENT
text: Complete proofs and experimental details can be found in the appendix.

heading: A PROOFS

heading: A.1 USEFUL LEMMAS
text: Before stating the proofs, we recall and prove two (known) identities.

heading: Lemma A.1.
text: Ep(x,y)[− log p(x | y)] = min f(x,y) Ep(x,y)
[ −f(x, y) + log
∑ x′ ef(x ′,y)
] . (7)
This identity says that the negative entropy function is the Fenchel dual of the log-sum-exp function. This identity can be proven using calculus of variations:
Proof. We start by finding the function f(x, y) that optimizes the RHS of Eq. 7. Noting that the RHS is a convex function of f(x, y), we can find the optimal f(x, y) by taking the derivative and setting it equal to zero:
d
df(x, y)
( Ep(x,y)[−f(x, y)] + log
∑ x′ ef(x ′,y)
) = −p(x, y) + e
f(x,y)∑ x′ e f(x′,y) .
Setting this derivative equal to zero, we see that solution f∗(x, y) satisfies
ef ∗(x,y)∑
x′ e f∗(x′,y)
= p(x, y).
Now, we observe that p(y) = 1:
p(y) = ∑ x p(x, y)
= ∑ x ef ∗(x,y)∑ x′ e f∗(x′,y)
=
∑ x e
f∗(x,y)∑ x′ e f∗(x′,y) = 1.
We then have
p(x | y) = p(x, y)
* 1p(y)
= ef ∗(x,y)∑
x′ e f∗(x′,y)
.
Taking the log of both sides, we have log p(x | y) = f∗(x, y)− log ∑ x′ ef ∗(x′,y).
Multiplying both sides by -1 and taking the expectation w.r.t. p(x, y), we obtain the desired result: Ep(x,y)[− log p(x | y)] = Ep(x,y)[−f∗(x, y)] + log ∑ x′ ef ∗(x′,y)
= min f(x,y) Ep(x,y)[−f(x, y)] + log ∑ x′ ef(x ′,y).
This lemma also holds for functions with more variables, a result that will be useful when proving dynamics robustness.

heading: Lemma A.2.
text: Ep(x,y,z)[− log p(x, y | z)] = min f(x,y,z) Ep(x,y,z) −f(x, y, z) + log ∑ x′,y′ ef(x ′,y′,z)  . Proof. Simply apply Lemma A.1 to the random variables (x, y) and z.

heading: A.2 PROOF OF THEOREM 4.1
text: This section will provide a proof of Theorem 4.1.
Proof. We start by restating the objective for robust control with rewards:
min r̃∈R̃(π) E at∼π(at|st) st+1∼p(st+1|st,at) [∑ t r̃(st,at) ] . (8)
We now employ the KKT conditions (Boyd et al., 2004, Chpt. 5.5.3). If the robust set R̃ is strictly feasibly, then there exists a constraint value ≥ 0 (the dual solution) such that the constrained problem is equivalent to the relaxed problem with Lagrange multiplier λ = 1:
min r̃ E at∼π(at|st) st+1∼p(st+1|st,at) [∑ t r̃(st,at) + log ∫ A exp(r(st,at)− r̃(st,at))dat′ ] .
To clarify exposition, we parameterize r̃ by its deviation from the original reward, ∆r(st,at) , r(st,at)− r̃(st,at). We also define ρπt (st,at) as the marginal distribution over states and actions visited by policy π at time t. We now rewrite the relaxed objective in terms of ∆r(st,at):
min ∆r E at∼π(at|st) st+1∼p(st+1|st,at) [∑ t r(st,at)−∆r(st,at) + log ∫ A exp(∆r(st,at))dat ′ ]
= E at∼π(at|st) st+1∼p(st+1|st,at) [∑ t r(st,at) ]
+ min ∆r E at∼π(at|st) st+1∼p(st+1|st,at) [∑ t −∆r(st,at) + log ∫ A exp(∆r(st,at))dat ′ ] .
= E at∼π(at|st) st+1∼p(st+1|st,at) [∑ t r(st,at) ]
+ min ∆r ∑ t Eρπt (st,at) [ −∆r(st,at) + log ∫ A exp(∆r(st,at))dat ′ ] .
= E at∼π(at|st) st+1∼p(st+1|st,at) [∑ t r(st,at)− log π(at | st) ] .
The last line follows from applying Lemma A.1.
One important note about this proof is that the KKT conditions are applied to the problem of optimizing the reward function r̃, which is a convex optimization problem. We do not require that the policy π(at | st) or the dynamics p(st+1 | st,at) be convex.

heading: A.3 WORKED EXAMPLE OF REWARD ROBUSTNESS
text: We compute the penalty for Example 1 in Section 4.3 using a Gaussian integral: PENALTY(r̃, s) = log ∫ 10 −10 exp(r(s,a)− r̃(s,a))da
= log ∫ 10 −10 exp(−(a− a∗)2 + 1 2 (a− (a∗ + ∆a))2)da
= log ∫ 10 −10 exp(−1 2 (a− (a∗ + ∆a))2 + ∆a2)da
= ∆a2 + 1
2 log(2π) + log(20).

heading: A.4 PROOF OF THEOREM 4.2
text: Proof. We now provide the proof of Theorem 4.2. The first step will be to convert the variation in dynamics into a sort of variation in the reward function. The second step will convert the constrained robust control objective into an unconstrained (penalized) objective. The third step will show that the penalty is equivalent to action entropy.
Step 1: Dynamics variation → reward variation. Our aim is to obtain a lower bound on the following objective:
min p̃∈P̃(π) E at∼π(at|st) st+1∼p̃(st+1|st,at) [∑ t r(st,at) ] .
We start by taking a log-transform of this objective, noting that this transformation does not change the optimal policy as the log function is strictly monotone increasing.
logE at∼π(at|st) st+1∼p̃(st+1|st,at) [∑ t r(st,at) ] .
Note that we assumed that the reward was positive, so the logarithm is well defined. We can write this objective in terms of the adversarial dynamics using importance weights.
log E at∼π(at|st) st+1∼p(st+1|st,at) [(∏ t p̃(st+1 | st,at) p(st+1 | st,at) )∑ t r(st,at) ]
= logE at∼π(at|st) st+1∼p(st+1|st,at)
[ exp ( log (∑ t r(st,at) ) + ∑ t log p̃(st+1 | st,at)− log p(st+1 | st,at) )] (a) ≥ E at∼π(at|st) st+1∼p(st+1|st,at) [ log (∑ t r(st,at) ) + ∑ t log p̃(st+1 | st,at)− log p(st+1 | st,at) ]
= E at∼π(at|st) st+1∼p(st+1|st,at)
[ log ( 1
T ∑ t r(st,at)
) + log T +
∑ t log p̃(st+1 | st,at)− log p(st+1 | st,at) ] (b) ≥ E at∼π(at|st) st+1∼p(st+1|st,at) [∑ t 1 T log r(st,at) + log p̃(st+1 | st,at)− log p(st+1 | st,at) ] + log T. (9)
Both inequalities are applications of Jensen’s inequality. As before, our assumption that the rewards are positive ensures that the logarithms remain well defined. While the adversary is choosing the dynamics under which we will evaluate the policy, we are optimizing a lower bound which depends on a different dynamics function. This step allows us to analyze adversarially-chosen dynamics as perturbations to the reward.
Step 2: Relaxing the constrained objective To clarify exposition, we will parameterize the adversarial dynamics as a deviation from the true dynamics:
∆r(st+1, st,at) = log p(st+1 | st,at)− log p̃(st+1 | st,at). (10) The constraint that the adversarial dynamics integrate to one can be expressed as∫
S p(st+1 | st,at)e−∆r(st+1,st,at)︸ ︷︷ ︸
p̃(st+1|st,at)
dst+1 = 1
Using this notation, we can write the lower bound on the robust control problem (Eq. 9) as follows:
min ∆r E at∼π(at|st), st+1∼p(st+1|st,at) [∑ t 1 T log r(st,at)−∆r(st+1, st,at) ] + log T (11)
s.t. E at∼π(at|st), st+1∼p(st+1|st,at) [∑ t log ∫∫ A×S e∆r(st+1 ′,st,at ′)dat ′dst+1 ′ ] ≤ (12)
and ∫ S p(st+1 | st,at)e−∆r(st+1,st,at)︸ ︷︷ ︸
p̃(st+1|st,at)
dst+1 = 1 ∀st,at. (13)
The constraint in Eq. 12 is the definition of the set of adversarial dynamics P̃ , and the constraint in Eq. 13 ensures that p̃(st+1 | st,at) represents a valid probability density function. Note that ∆r(st+1, st,at) = 0 is a strictly feasible solution to this constrained optimization problem for > 0. Note also that the problem of optimizing the function ∆r(st,at) is a convex optimization
problem. We can therefore employ the KKT conditions (Boyd et al., 2004, Chpt. 5.5.3). If the robust set P̃ is strictly feasibly, then there exists ≥ 0 (the dual solution) such that the set of solutions p̃ to the constrained optimization problem Eq. 11 are equivalent to the set of solutions to the following relaxed objective with Lagrange multiplier λ = 1:
min ∆r E at∼π(at|st), st+1∼p(st+1|st,at) [∑ t 1 T log r(st,at)−∆r(st+1, st,at) (14)
+ log ∫∫ A×S e∆r(st+1 ′,st,at ′)dat ′dst+1 ′ ] + log T
s.t. ∫ S p(st+1 | st,at)e−∆r(st+1 ′,st,at ′)dst+1 = 1 ∀st,at. (15)
This step does not require that the policy π(at | st) or the dynamics p(st+1 | st,at) be convex. Our next step is to show that the constraint does not affect the solution of this optimization problem. For any function ∆r(st+1, st,at), we can add a constant c(st,at) and obtain the same objective value but now satisfy the constraint. We construct c(st,at) as
c(st,at) = log ∫ S p(st+1 | st,at)e−∆r(st+1 ′,st,at ′)dst+1 = 1 ∀st,at.
First, we observe that adding c(st,at) to ∆r does not change the objective:
E at∼π(at|st), st+1∼p(st+1|st,at) [∑ t 1 T log r(st,at)− (∆r(st+1, st,at) + c(st,at))
+ log ∫∫ A×S e∆r(st+1 ′,st,at ′)+c(st,at)dat ′dst+1 ′ ] + log T
= E at∼π(at|st), st+1∼p(st+1|st,at) [∑ t 1 T log r(st,at)−∆r(st+1, st,at)− c(st,at)
+ log ( ec(st,at) ∫∫ A×S e∆r(st+1 ′,st,at ′)dat ′dst+1 ′ )] + log T
= E at∼π(at|st), st+1∼p(st+1|st,at) [∑ t 1 T log r(st,at)−∆r(st+1, st,at)− c(st,at) + c(st,at)
+ log (∫∫ A×S e∆r(st+1 ′,st,at ′)dat ′dst+1 ′ )] + log T.
Second, we observe that the new reward function ∆r(st+1, st,at) + c(st,at) satisfies the constraint in Eq. 15: ∫
S p(st+1 | st,at)e−(∆r(st+1 ′,st,at ′)−c(st,at))dst+1
= e−c(st,at) ∫ S p(st+1 | st,at)e−∆r(st+1 ′,st,at ′)dst+1
=
∫ S p(st+1 | st,at)e
−∆r(st+1′,st,at′)dst+1∫ S p(st+1 | st,at)e−∆r(st+1 ′,st,at′)dst+1 = 1.
Thus, constraining ∆r to represent a probability distribution does not affect the solution (value) to the optimization problem, so we can ignore this constraint without loss of generality. The new, unconstrained objective is
min ∆r E a∼π(at|st), s′∼p(st+1|st,at) [∑ t 1 T log r(st,at)−∆r(st+1, st,at)
+ log ∫∫ A×S e∆r(st+1,st,at)dat ′dst+1 ′ ] + log T. (16)
Step 3: The penalty is the Fenchel dual of action entropy. We define ρπt (st,at, st+1) as the marginal distribution of transitions visited by policy π at time t. We now apply Lemma A.2 to Eq. 16:
min ∆r E a∼π(at|st), s′∼p(st+1|st,at) [∑ t 1 T log r(st,at)−∆r(st+1, st,at)
+ log ∫∫ A×S e∆r(st+1,st ′,at ′)dat ′dst+1 ′ ] + log T (17)
= E a∼π(at|st), s′∼p(st+1|st,at) [∑ t 1 T log r(st,at) ] + log T
+ min ∆r E a∼π(at|st), s′∼p(st+1|st,at) [∑ t −∆r(st+1, st,at) + log ∫∫ A×S e∆r(st+1,st ′,at ′)dat ′dst+1 ′ ]
= E a∼π(at|st), s′∼p(st+1|st,at) [∑ t 1 T log r(st,at) ] + log T
+ min ∆r ∑ t Eρπt (st,at,st+1) [ −∆r(st+1, st,at) + log ∫∫ A×S e∆r(st+1,st ′,at ′)dat ′dst+1 ′ ]
= E a∼π(at|st), s′∼p(st+1|st,at) [∑ t 1 T log r(st,at)− log p(at, st+1 | st) ] + log T
= E a∼π(at|st), s′∼p(st+1|st,at) [∑ t 1 T log r(st,at)− log π(at | st)− log p(st+1 | st,at) ] + log T
Summary. We have thus shown the follow: min p̃∈P̃ log JMaxEnt(π; p̃, r) ≥ JMaxEnt(π; p, r̄) + log T.
Taking the exponential transform of both sides, we obtain the desired result.

heading: A.5 ROBUSTNESS TO BOTH REWARDS AND DYNAMICS
text: While Theorem 4.2 is phrased in terms of perturbations to the dynamics function, not the reward function, we now discuss how this result can be used to show that MaxEnt RL is simultaneously robust to perturbations in the dynamics and the reward function. Define a modified MDP where the reward is appended to the observation. Then the reward function is the last coordinate of the observation. In this scenario, robustness to dynamics is equivalent to robustness to rewards.

heading: A.6 HOW BIG IS THE ROBUST SET ( )?
text: Our proof of Theorem 4.2 used duality to argue that there exists an for which MaxEnt RL maximizes a lower bound on the robust RL objective. However, we did not specify the size of this . This raises the concern that might be arbitrarily small, even zero, in which case the result would be vacuous. In this section we provide a proof of Lemma 4.3, which provides a lower bound on the size of the robust set.
Proof. Our proof proceeds in three steps. We first argue that, at optimality, the constraint on the adversarial dynamics is tight. This will allow us to treat the constraint as an equality constraint, rather than an inequality constraint. Second, since this constraint holds with equality, then we can rearrange the constraint and solve for in terms of the optimal adversarial dynamics. The third step is to simplify the expression for .
Step 1: The constraint on ∆r holds with equality. Our aim here is to show that the constraint on ∆r in Eq. 12 holds with equality for the optimal ∆r (i.e., that which optimizes Eq. 11).1 The objective
1In the case where there are multiple optimal ∆r, we require that the constraint hold with equality for at least one of the optimal ∆r.
in Eq. 11 is linear in ∆r, so there must exist optimal ∆r at the boundary of the constraint (Rockafellar, 1970, Chapter 32).
Step 2: Solving for . Since the solution to the constrained optimization problem in Eq. 11 occurs at the boundary, the constraint in Eq. 12 holds with equality, immediately telling us the value of :
= E at∼π(at|st), st+1∼p(st+1|st,at) [∑ t log ∫∫ A×S e∆r(st+1 ′,st,at ′)dat ′dst+1 ′ ]
= T · Eρ(st) [ log ∫∫ A×S e∆r(st+1 ′,st,at ′)dat ′dst+1 ′ ] , (18)
where ∆r is the solution to Eq. 17. This identity holds for all states st. The second line above expresses the expectation over trajectories as an expectation over states, which will simplify the analysis in the rest of this proof. The factor of T is introduced because we have removed the inner summation.
Step 3: Simplifying the expression for . To better understand this value of , we recall that the following identity (Lemma A.2) holds for this optimal ∆r:
e∆r(st,at,st+1)∫∫ A×S e ∆r(st,at′,st+1′)dat′, dst+1′ = ρ(at, st+1 | st) ∀st,at, st+1.
We next take the log(·) of both sides and rearrange terms:
log ∫∫ A×S e∆r(st,at ′,st+1 ′)dat ′, dst+1 ′ = ∆r(st,at, st+1)− log ρ(at, st+1 | st).
Next, we substitute the definition of ∆r (Eq. 10) and factor log ρ(at, st+1 | st):
log ∫∫ A×S e∆r(st,at ′,st+1 ′)dat ′, dst+1 ′
= log p(st+1 | st,at)− log p̃(st+1 | st,at)− log p(st+1 | st,at)− log π(at | st) = − log p̃(st+1 | st,at)− log π(at | st).
Substituting this expression into Eq. 18, we obtain a more intuitive expression for :
= T · E at∼π(at|st), st+1∼p(st+1|st,at) [− log p̃(st+1 | st,at)− log π(at | st)]
= T · E at∼π(at|st), st+1∼p(st+1|st,at) [Hp̃[st+1 | st,at] +Hπ[at | st]]
≥ T · E at∼π(at|st), st+1∼p(st+1|st,at) [Hπ[at | st]] .
The last line follows from our assumption that the state space is discrete, so the entropyHp̃[st+1 | st,at] is non-negative. This same result will hold in environments with continuous state spaces as long as the (differential) entropyHp̃[st+1 | st,at] is non-negative.

heading: A.7 WORKED EXAMPLE OF DYNAMICS ROBUSTNESS
text: We calculate the penalty using a Gaussian integral:
log ∫∫ A×S p(st+1 | st,at) p̃(st+1 | st,at) datdst+1
= log ∫∫ A×S 2 exp(−1 2 (st+1 − (Ast +Bat))2 + 1 4 (st+1 − (Ast +Bat − β))2)datdst+1
= log ∫∫ A×S 2 exp(−1 4 (st+1 − (Ast +Bat − β))2 + 1 2 β2)datdst+1
= log ( 2 √ 4π ∫ 10 −10 exp( 1 2 β2)dat ) = 1
2 β2 + log(8
√ π) + log(20).
We make two observations about this example. First, the penalty would be infinite if the adversary dynamics p̃ had the same variance as the true dynamics, p: the adversary must choose dynamics with higher variance. Second, the penalty depends on the size of the state space. More precisely, the penalty depends on the region over which the adversary applies their perturbation. The adversary can incur a smaller penalty by applying the perturbation over a smaller range of states.

heading: A.8 TEMPERATURES
text: Many algorithms for MaxEnt RL (Haarnoja et al., 2018a; Fox et al., 2015; Nachum et al., 2017) include a temperature α > 0 to balance the reward and entropy terms:
JMaxEnt(π, r) = Eπ [ T∑ t=1 r(st,at) ] + αHπ[at | st].
We can gain some intuition into the effect of this temperature on the set of reward functions to which we are robust. In particular, including a temperature α results in the following robust set:
Rαr = { r(st,at) + αust(at) | ust(at) ≥ 0 ∀st,at and ∫ A e−ust (at)dat ≤ 1 ∀st } = { r(st,at) + ust(at) | ust(at) ≥ 0 ∀st,at and ∫ A e−ust (at)/αdat ≤ 1 ∀st } . (19)
In the second line, we simply moved the temperature from the objective to the constraint by redefining ust(at)→ 1αust(at). We visualize the effect of the temperature in Fig. 9. First, we fix a reward function r, and plot the robust set Rαr for varying values of α. Fig. 9 (left) shows the somewhat surprising result that increasing the temperature (i.e., putting more weight on the entropy term) makes the policy less robust. In fact, the robust set for higher temperatures is a strict subset of the robust set for lower temperatures: α1 < α2 =⇒ Rα2r ⊆ Rα2r . This statement can be proven by simply noting that the function e− x α is an increasing function of α in Equation 19. It is important to recognize that being robust against more reward functions is not always desirable. In many cases, to be robust to everything, an optimal policy must do nothing.
We now analyze the temperature in terms of the converse question: if a reward function r′ is included in a robust set, what other reward functions are included in that robust set? To do this, we take a reward function r′, and find robust sets Rαr that include r
′, for varying values of α. As shown in Fig. 9 (right), if we must be robust to r′ and use a high temperature, the only other reward functions to which we are robust are those that are similar, or pointwise weakly better, than r′. In contrast, when using a small temperature, we are robust against a wide range of reward functions, including those that are highly dissimilar from our original reward function (i.e., have higher reward for some actions, lower reward for other actions). Intuitively, increasing the temperature allows us to simultaneously be robust to a larger set of reward functions.

heading: A.9 MAXENT SOLVES ROBUST CONTROL FOR REWARDS
text: In Sec. 4.1, we showed that MaxEnt RL is equivalent to some robust-reward problem. The aim of this section is to go backwards: given a set of reward functions, can we formulate a MaxEnt RL problem such that the robust-reward problem and the MaxEnt RL problem have the same solution? Lemma A.3. For any collection of reward functions R, there exists another reward function r such that the MaxEnt RL policy w.r.t. r is an optimal robust-reward policy for R:
arg max π
Eπ [ T∑ t=1 r(st,at) ] +Hπ[a | s] ⊆ arg max π min r′∈R Eπ [ T∑ t=1 r′(st,at) ] .
We use set containment, rather than equality, because there may be multiple solutions to the robustreward control problem.
Proof. Let π∗ be a solution to the robust-reward control problem:
π∗ ∈ arg max π min ri∈R Eπ [ T∑ t=1 ri(st,at) ] .
Define the MaxEnt RL reward function as follows:
r(st,at) = log π ∗(at | st).
Substituting this reward function in Equation 1, we see that the unique solution is π = π∗.
Intuitively, this theorem states that we can use MaxEnt RL to solve any robust-reward control problem that requires robustness with respect to any arbitrary set of rewards, if we can find the right corresponding reward function r for MaxEnt RL. One way of viewing this theorem is as providing an avenue to sidestep the challenges of robust-reward optimization. Unfortunately, we will still have to perform robust optimization to learn this magical reward function, but at least the cost of robust optimization might be amortized. In some sense, this result is similar to Ilyas et al. (2019).

heading: A.10 FINDING THE ROBUST REWARD FUNCTION
text: In the previous section, we showed that a policy robust against any set of reward functions R can be obtained by solving a MaxEnt RL problem. However, this requires calculating a reward function r∗ for MaxEnt RL, which is not in general an element in R. In this section, we aim to find the MaxEnt reward function that results in the optimal policy for the robust-reward control problem. Our main idea is to find a reward function r∗ such that its robust set, Rr∗ , contains the set of reward functions we want to be robust against, R. That is, for each ri ∈ R, we want
ri(st,at) = r ∗(st,at) + ust(at) for some ust(at) satisfying ∫ A e−ust (at)dat ≤ 1 ∀st.
Replacing u with r′ − r∗, we see that the MaxEnt reward function r must satisfy the following constraints: ∫
A er ∗(st,at)−r′(st,at)dat ≤ 1 ∀st ∈ S, r′ ∈ R.
We define R∗(R) as the set of reward functions satisfying this constraint w.r.t. reward functions in R:
R∗(R) , { r∗ ∣∣∣∣ ∫ A er ∗(st,at)−r′(st,at)dat ≤ 1 ∀st ∈ S, r′ ∈ R } Note that we can satisfy the constraint by making r∗ arbitrarily negative, so the set R∗(R) is nonempty. We now argue that all any applying MaxEnt RL to any reward function in r∗ ∈ R∗(R) lower bounds the robust-reward control objective. Lemma A.4. Let a set of reward functions R be given, and let r∗ ∈ R∗(R) be an arbitrary reward function belonging to the feasible set of MaxEnt reward functions. Then
JMaxEnt(π, r ∗) ≤ min
r′∈R Eπ [ T∑ t=1 r′(st,at) ] ∀π ∈ Π.
Note that this bound holds for all feasible reward functions and all policies, so it also holds for the maximum r∗:
max r∗∈R∗(R)
JMaxEnt(π, r ∗) ≤ min
r′∈R Eπ [ T∑ t=1 r′(st,at) ] ∀π ∈ Π.
Defining π∗ = arg maxπ JMaxEnt(π, r ∗), we get the following inequality:
max r∗∈R∗(R),π∈Π
JMaxEnt(π, r ∗) ≤ min
r′∈R Eπ∗ [ T∑ t=1 r′(st,at) ] ≤ max π∈Π min r′∈R Eπ [ T∑ t=1 r′(st,at) ] . (20)
Thus, we can find the tightest lower bound by finding the policy π and feasibly reward r∗ that maximize Equation 20:
max r,π
Eπ [ T∑ t=1 r(st,at) ] +Hπ[at | st] (21)
s.t. ∫ A er(st,at)−r ′(st,at)da ≤ 1 ∀ st ∈ S, r′ ∈ R.
It is useful to note that the constraints are simply LOGSUMEXP functions, which are convex. For continuous action spaces, we might approximate the constraint via sampling. Given a particular policy, the optimization problem w.r.t. r has a linear objective and convex constraint, so it can be solved extremely quickly using a convex optimization toolbox. Moreover, note that the problem can be solved independently for every state. The optimization problem is not necessarily convex in π.

heading: A.11 ANOTHER COMPUTATIONAL EXPERIMENT
text: This section presents an experiment to study the approach outlined above. Of particular interest is whether the lower bound (Eq 20) comes close to the optimal minimax policy.
We will solve robust-reward control problems on 5-armed bandits, where the robust set is a collection of 5 reward functions, each is drawn from a zero-mean, unit-variance Gaussian. For each reward function, we add a constant to all of the rewards to make them all positive. Doing so guarantees that the optimal minimax reward is positive. Since different bandit problems have different optimal minimax rewards, we will normalize the minimax reward so the maximum possible value is 1.
Our approach, which we refer to as “LowerBound + MaxEnt”, solves the optimization problem in Equation 21 by alternating between (1) solving a convex optimization problem to find the optimal reward function, and (2) computing the optimal MaxEnt RL policy for this reward function. Step 1 is done using CVXPY, while step 2 is done by exponentiating the reward function, and normalizing it to sum to one. Note that this approach is actually solving a harder problem: it is solving the robust-reward control problem for a much larger set of reward functions that contains the original set
of reward functions. Because this approach is solving a more challenging problem, we do not expect that it will achieve the optimal minimax reward. However, we emphasize that this approach may be easier to implement than fictitious play, which we compare against. Different from experiments in Section 4.1, the “LowerBound + MaxEnt” approach assumes access to the full reward function, not just the rewards for the actions taken. For fair comparison, fictitious play will also use a policy player that has access to the reward function. Fictitious play is guaranteed to converge to the optimal minimax policy, so we assume that the minimax reward it converges to is optimal. We compare against two baselines. The “pointwise minimum policy” finds the optimal policy for a new reward function formed by taking the pointwise minimum of all reward functions: r̃(at) = minr∈R r(at). This strategy is quite simple and intuitive. The other baseline is a “uniform policy” that chooses actions uniformly at random.
We ran each method on the same set of 10 robust-reward control bandit problems. In Fig. 10, we plot the (normalized) minimax reward obtained by each method on each problem, as well as the average performance across all 10 problems. The “LowerBound + MaxEnt” approach converges to a normalized minimax reward of 0.91, close to the optimal value of 1. In contrast, the “pointwise minimum policy” and the “uniform policy” perform poorly, obtaining normalized minimax rewards of 0.56 and 0.60, respectively. In summary, while the method proposed for converting robustreward control problems to MaxEnt RL problems does not converge to the optimal minimax policy, empirically it performs well.

heading: B EXPERIMENTAL DETAILS

heading: B.1 DYNAMICS ROBUSTNESS EXPERIMENTS (FIG. 4)
text: We trained the MaxEnt RL policy using the SAC implementation from TF Agents (Guadarrama et al., 2018) with most of the default parameters (unless noted below).
Fig. 4a We used the standard Pusher-v2 task from OpenAI Gym (Brockman et al., 2016). We used a fixed entropy coefficient of 1e− 2 for the MaxEnt RL results. For the standard RL results, we used the exact same codebase to avoid introducing any confounding factors, simply setting the entropy coefficient to a very small value 1e− 5. The obstacle is a series of three axis-aligned blocks with width 3cm, centered at (0.32, -0.2), (0.35, -0.23), and (0.38, -0.26). We chose these positions to be roughly along the perpendicular bisector of the line between the puck’s initial position and the goal. We used 100 episodes of length 100 for evaluating each method. To decrease the variance in the results, we fixed the initial state of each episode:
qpos = [0., 0., 0., 0., 0., 0., 0.,−0.3,−0.2, 0., 0.], qvel = [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.].
Fig. 4b We used the SawyerButtonPressEnv environment from Metaworld (Yu et al., 2020), using a maximum episode length of 151. For this experiment, we perturbed the environment by modifying the observations such that the button appeared to be offset along the Y axis. We recorded the average performance over 10 episodes. For this environment we used an entropy coefficient of 1e1 for MaxEnt RL and 1e− 100 for standard RL.2
Fig. 6 We used the standard Pusher-v2 task from OpenAI Gym (Brockman et al., 2016). We modified the environment to perturb the XY position of the puck at time t = 20. We randomly sampled an angle θ ∼ Unif[0, 2π] and displaced the puck in that direction by an amount given by the disturbance size. For evaluating the average reward of each policy on each disturbance size, we used 100 rollouts of length 151.
Fig. 5 We used a modified version of the 2D navigation task from Eysenbach et al. (2019) with the following reward function:
r(st,at) = ‖st − (8, 4)T ‖2 − 10 · 1(st ∈ Sobstacle). Episodes were 48 steps long.
2We used a larger value for the entropy coefficient for standard RL in the previous experiment to avoid numerical stability problems.
Fig. 7 We used the peg insertion environment from Eysenbach et al. (2017). Episodes were at most 200 steps long, but terminated as soon as the peg was in the hole. The agent received a reward of +100 once the peg was in the hole, in addition to the reward shaping terms described in Eysenbach et al. (2017).

heading: B.2 REWARD ROBUSTNESS EXPERIMENTS (FIG. 8)
text: Following Haarnoja et al. (2018a), we use an entropy coefficient of α = 1/5. In Fig. 8, the thick line is the average over five random seeds (thin lines). Fig. 11 shows the evaluation of all methods on both the expected reward and the minimax reward objectives. Note that the minimax reward can be computed analytically as
r̃(st,at) = r(st,at)− log π(at | st).

heading: B.3 BANDITS (FIG. 10)
text: The mean for arm i, µi, is drawn from a zero-mean, unit-variance Gaussian distribution, µi ∼ N (0, 1). When the agent pulls arm i, it observes a noisy reward ri ∼ N (µi, 1). The thick line is the average over 10 random seeds (thin lines).

heading: C ROBUST RL ABLATION EXPERIMENTS
text: We ran additional ablation experiments to study whether simple modifications to the robust RL baseline from Section 5 (Tessler et al., 2019) would improve results. These experiments will help discern whether the good performance of MaxEnt RL, relative to the PR-MDP and NR-MDP baselines, comes from using a more recent RL algorithm (SAC instead of DDPG), or from the entropy regularization. We used the following ablations:
1. larger network: We increased the width of all neural networks to 256 units, so that the network size the exactly the same as for the MaxEnt RL method.
2. dual critic: Following TD3 (Fujimoto et al., 2018), we added a second Q function and used the minimum over two (target) Q functions. This change affects not only the actor and critic updates, but also the updates to the adversary learned by PR/NR-MDP.
3. more exploration: We increased the exploration noise from 0.2 to 1.0.
We implemented these ablations by modifying the open source code released by Tessler et al. (2019).
The results, shown in Fig. 12, show that these changes do not significantly improve the performance of the NR-MDP or PR-MDP. Perhaps the one exception is on the Hopper-v2 task, where PR-MDP with the larger networks is now the best method for large relative masses. The dual critic ablation generally performs worse than the baseline. We hypothesize that this poor performance is caused by using the (pessimistic) minimum over two Q functions for updating the adversary, which may result
in a weaker adversary. This experiment suggests that incorporating the dual critic trick into the action robustness framework may require some non-trivial design decisions.

Review summaries:
The authors work on the important problem of robustness guarantees in RL algorithms. They analyze the robustness of max entropy RL w.r.t. dynamics uncertainties, and reward function uncertainties.

The main goal of the paper is to show that optimizing a MaxEnt objective is equivalent to solve a robust problem with uncertainty on the dynamics. The authors provide a theorem which illustrate that solving MaxEnt with a modified reward is equivalent to solve a robust version of MaxEnt w.r.t. an uncertain dynamics. The authors provide some simple examples to show the soundness of their results, and they provide a numerical simulations on more complex problems to experimentally validate their claims.

This work analyzes the implicit robustness of maximum entropy RL. They show, theoretically, that there exists some robust MDP for which the optimal solution is obtained by maximizing the max-ent objective. Empirically they show that indeed SAC learns behavior which is in a sense robust.