Maximum Entropy RL (Provably) Solves Some Robust RL Problems
Authors: Benjamin Eysenbach
Sergey Levine
Venue: ICLR 2022 Poster (2022)
Keywords: reinforcement learning
robustness
maximum entropy

Abstract: Many potential applications of reinforcement learning (RL) require guarantees that the agent will perform well in the face of disturbances to the dynamics or reward function. In this paper, we prove theoretically that maximum entropy (MaxEnt) RL maximizes a lower bound on a robust RL objective, and thus can be used to learn policies that are robust to some disturbances in the dynamics and the reward function. While this capability of MaxEnt RL has been observed empirically in prior work, to the best of our knowledge our work provides the first rigorous proof and theoretical characterization of the MaxEnt RL robust set. While a number of prior robust RL algorithms have been designed to handle similar disturbances to the reward function or dynamics, these methods typically require additional moving parts and hyperparameters on top of a base RL algorithm. In contrast, our results suggest that MaxEnt RL by itself is robust to certain disturbances, without requiring any additional modifications. While this does not imply that MaxEnt RL is the best available robust RL method, MaxEnt RL is a simple robust RL method with appealing formal guarantees.

Published as a conference paper at ICLR 2022
MAXIMUM ENTROPY RL (PROVABLY) SOLVES SOME ROBUST RL PROBLEMS
Benjamin Eysenbach Cameegie Mellon University, Google Brain beysembal cs cmu. edu
Sergey Levine
UC Berkeley, Google Brain
ABSTRACT
Many potential applications of reinforcement learning (RL) require guarantees that the agent will perform well in the face of disturbances the dynamics or reward function. In this paper, we prove theoretically that maximum entropy (MaxEnt) RL maximizes a lower bound on a robust RL objective, and thus can be used learn policies that robust to some disturbances in the dynamics and the reward function. While this capability of MaxEnt RL has been observed empirically in prior work, the best of knowledge and theoretical characterization of the MaxEnt RL. robust set. While a number of prior robust RL algorithms have been designed to handle similar disturbances to the reward function or dynamics, these methods typically require additional moving parts and hyperparameters results suggest that MaxEnt RL by itself is robust to certain disturbances, without requiring any additional modifications. While this does not imply that MaxEnt RL is the best available robust RI. method, MaxEnt RI. is simple robust RI. method with appealing formal guarantees.
work provides the first rigorous proof
top of base RL algorithm. In contrast, our
INTRODUCTION
Many real-works applications of reinforcement learning (RL) require control policies that not only maximize reward but continue to do so when faced with environmental disturbances, modeling errors, or errors in reward specifica- tion. These disturbances can arise from human biases and modeling errors, from non- non- stationary aspects of the environment, or actual adversaries acting in the environment. A number of works have studied how train RL algorithms to robust to disturbances in the environment (e.g., Morimoto & Doya (2005); Pinto et al. (2017); Tessler et (2119)) However, designing robust RL algorithms requires care, typically requiring an adversarial optimization problem and intro- ducing additional hyperparameters (Pinto et al.) 2017: Tessler al. 2019). Instead of designing a new robust RL algorithm, we will instead an- alyze whether an existing RL method, MaxEnt RL, offers robustness to perturbations. MaxEnt RL methods are based on the maximum entropy principle, and augmented the expected reward objective with an entropy maximization term (Ziebart et al. 2008). Prior work has conjectured that such algorithms should learn robust policies (Haarnoja ct al. 2017 Huang et al., 2019). on account of the close connection between maximum entropy
Original Dynamics
Perturbed Dynamics
Standard
MaxdEnt
goal
Figure 1: MaxEnt RL is robust to disturbances. (Left) We applied standard RL and MaxEnt RL manipulation task without obstacles, but added ob- stacles (red squares) during evaluation. We then plot the position of the object when evaluating the learned poli- cies (Center) on the original environment and (Right) the environment with obstacle. The stochastic policy learned by MaxEnt RL often navigates around the obstacle, whereas the deterministic policy from standard RL almost always collides with the obstacle.
methods and robustness in supervised learning domains (Grünwald et al. 2004 Ziebart 2010).
However, despite decades of research on maximum entropy RL methods (Kappen 2005 Todorov
2007| Toussaint 2009: Theodorou et al. 2010), a formal characterization of the robustness properties of such approaches has proven clusive. To our knowledge, no prior work formally proves that MaxEnt
RL methods robust, and prior work characterizes the disturbances against which these methods

---

Published as a conference paper at ICLR 2022
robust. Showing how to obtain robust policies from existing MaxEnt RL methods, which already constitute a significant portion of the RL methods in use today (Abdolmaleki et al. (0018) Haarnoja ltt al. 2018a Vieillard al. 2020], would be useful because it would enable practitioners to leverage existing, tried-and-true methods to solve robust RI. problems.
Stochastic policies, of the sort learned with MaxEnt RL., inject noise into the actions during training, thereby preparing themselves for deployment in environments with disturbances. For example, in the robot pushing task shown in Fig. ! the policy learned by MaxEnt RL pushes the white puck to the goal using many possible routes. In contrast, (standard) RL learns a deterministic policy, al always using the same route to get to the goal. Now, imagine that this environment is perturbed by adding the red barrier in Fig. I While the policy learned by (standard) RL always collides with this obstacle, the policy learned by MaxEnt RI. uses many routes to solve the task, and some fraction of these routes continue to solve the task even when the obstacle is present. While number of prior works have articulated the intuition that the stochastic policies learned via MaxEnt RI. should be robust to disturbances (Levine 2018 Abdolmaleki al., 2018 Lee et al. 2019), no prior work has actually shown that MaxEnt RL policies are provably robust, nor characterized the set of disturbances to which they are robust. Applications of MaxEnt RL methods to problems that demand robustness likely hampered by a lack of understanding of when such methods are robust, what kinds of reward functions should be used obtain the desired type of robustness, and how the task should be set up. The goal in our work is to make this notion precise, proving that MaxEnt RI. is already a robust RI. algorithm, and deriving the robust set for these policies.
The main contribution of this paper is theoretical characterization of the robustness of existing MaxEnt RL methods. Specifically, we show that the MaxEnt RL objective is a nontrivial lower bound on the robust RL objective, for a particular robust set. Importantly, these two objectives use different reward functions: learn a policy that maximizes a reward function on wide range of dynamics functions, one should apply MaxEnt RL to pessimistic version of that reward function. MaxEnt RI. not robust with respect to the reward function it is trained on. As we will show, the robust set for these methods is non-trivial, and in some important special cases can be quite intuitive. To our knowledge, our results are the first to formally show robustness of standard MaxEnt RL methods to dynamics perturbations and characterize their robust set. Importantly, our analysis and experiments highlight that robustness is only achieved for relatively large entropy coefficients. We validate our theoretical results set of illustrative empirical experiments.
2 RELATED WORK
The study of robustness in controls has long history, with robust or H control methods proposing provably robust solutions under various assumptions on the true dynamics (Zhou et al. 1996 Doyle
et 2013). RL offers an appealing alternative, since robust RL methods can principle learn robust policies without knowing the true dynamics. However, policies learned by standard RL algorithms often flounder in the face of environmental disturbances (Rajeswaran et al., 2016 Peng et al. 2018). The problem of learning robust policies that achieve high reward in the face of adversarial environmental disturbances, often called robust RL, has been well studied in the literature (Bagnell
al. | Nilim & Ghaoui| 2003 Morimoto & Doya 2005 Pinto et al. 2017: Tessler et al. 2019
Russel & Petrik][20 | Kamalaruban et al.] 2020; Russel et al.) 2020 |021; Derman et al.) 2021).
Prior robust RL methods are widely-applicabl but often require many additional hyperparameters
or components. For example, [Banelll et al. (2001) modify the Bellman backup in a Q-learning by
solving convex optimization problem in an inner loop, and Tessler et al. (2019) trains an additional
adversary policy via RL.
Robust RL different from robust control in the traditional sense, which focuses on stability independent of any reward function (Zhou et al. 1996; Doyle et al. (013). Robust RL problem involves estimating a policy's 's returns under "similar" " MDPs, problem that has been studied in terms of distance metrics for MDPs (Lecarpentier et al., 2020). Robust RL is different from maximizing the average reward across many environments, as done by methods such as domain randomization (Sadeghi & Levine 2016 Rajeswaran et al. 2016 Peng et al. 2018). Closely related to robust RL are prior methods that are robust to disturbances in the reward function (Hadfield- MladielldMMenll et al. 2017 Bobu et al. 2020 Michaud et al. 2020), , or aim to minimize cost function in addition
to maximizing reward CCow et al. [0017] Achiam et al. 2017: Chow et al. 2019; Carrara et al.
2019 Tang et al. 2020
l'hananjeyan al. 2021 Robust RL. also different from the problem of
2

---

Published as a conference paper at ICLR 2022
learning transferable generalizable RL agents (Lazaric 2012: Zhang et al. 2018 Justesen et al. 2018; Cobbe et al. 2019), which focuses on the average- case performance on new environments, rather than the worst-case performance.
PRELIMINARIES
We assume that an agent observes states St. takes actions at T 8t), and obtains re- wards r(tt, az). The initial state is sampled 81 (s1), and subsequent states sampled St+1 ~ p((+1 St. az). We will use p" (r) to denote the probability distribution over trajectories for policy æ. Episodes have T steps, which we summarize as a trajectory Without loss of generality, we can assume that rewards are undiscounted, as any discount can be
($1, a1, ST, ary)
addressed by modifying the dynamics to transition to an absorbing state with probability 1 - y. The standard RI. objective is:
T
r(st at) where pi II p(s++1 | Se: a A Σ
arg max E B (T)
1s ss)
is the distribution over trajectories when using policy π. In fully observed MDPs, there always exists a deterministic policy as a solution (Puterman 2014). The MaxEnt RL objective is to maximize the sum of expected reward and conditional action entropy. Formally, we define this objective in terms of a policy æ, the dynamics and the reward function
Max (T:; Bag tor (ar
Σ r(st, at) aHo St]
where Ho (at Se] - SA (att Se) log v(a e.) da, denotes the entropy of the action distribution. The entropy coefficient balances the reward term and the entropy term; use in our analysis. As noted above, the discount factor omitted because it can subsumed into the dynamics.
Our main result will be that maximizing the MaxEnt RL objective (Eq. I results in robust policies. We quantify robustness by measuring the reward of policy when evaluated on new reward function f or dynamics function p, which is chosen adversarially from some set:
max min Epor a sune)a si) T PEP,
Σ F(SE, at)
This robust RL objective defined terms of the sets of dynamics P and reward functions R. Our goal is to characterize these sets. The robust RI. objective can be interpreted as two-player, zero-sum game. The aim is to find Nash equilibrium. Our goal is to prove that MaxEnt RL (with a different reward function) optimizes lower bound this robust objective, and to characterize the robust sets P and R. for which this bound holds.
4 MAXENT RL AND ROBUST CONTROL
In this section, we prove the conjecture that MaxEnt RL is robust to disturbances the environment. Our main result is that MaxEnt RL can be used maximize lower bound on certain robust RL objective. Importantly, doing this requires that MaxEnt RI. be applied to a different, pessimistic, version of the target reward function. Before presenting our main result, we prove that MaxEnt RL is robust against disturbances to the reward function. This result is a simple extension of prior work, and we will use this result for proving our main result about dynamics robustness
4.1 ROBUSTNESS TO ADVERSARIAL REWARD FUNCTIONS
We first show that MaxEnt RL is robust to some degree of misspecification in the reward function. This result may be useful in practical settings with learned reward functions (Fu et al.) 2018 Xu
Denil 2019 Michaud et al. 2020) or misspecified reward function (Amodei et al. |0 Clark & Amodei 2016). Precisely, the following result will show that applying MaxEnt RL to one reward function, r(st. a1), results in policy that is guaranteed to also achieve high return range of other reward functions, f(st, at) R:
3

---

Published as a conference paper at ICLR 2022
Theorem 4.1. Let dynamics p(s
(s. at). policy T(a st), and reward function r(s,, ax) be given. Assume that the reward function is finite and that the policy has support everywhere (i.e., (a, st) for all states and actions). Then there exists a positive constant >> 0 such that the MaxEnt RL objective MaxBa is equivalent the robust RL objective defined by the robust set R R(m):
min E (ER(()
ae)
VT,
where the adversary chooses a reward function from the set
Σ log /x exp(r(se, ai') - F(s,, )d))
R(T) -
((sL.a)) ar)
E
] S $g).
(2)
Thus, when use MaxEnt RL with some reward function r. the policy obtained is guaranteed to also obtain high reward on all similar reward functions r that satisfy Eq. We discuss how MaxEnt RL can be used learn policies robust to arbitrary sets of reward functions Appendix [0 The proof can be found in Appendix [..2] and is simple extension of prior work (Grünwald et al. 2004] Ziebart et al., 201T). Our proof does not assume that the reward function is bounded nor that the policy is convex. While the proof is straightforward, it will be useful as an intermediate step when proving robustness to dynamics in the next subsection.
The robust set R corresponds to reward functions that are not too much smaller than the original reward function: the adversary cannot decrease the reward for any state action too much. The robust set R depends on the policy, so the adversary has the capability of looking at which states the policy visits before choosing the adversarial reward function F. For example, the adversary may choose to apply larger perturbations at states and actions that the agent frequently visits.
4.2 ROBUSTNESS TO ADVERSARIAL DYNAMICS
We now show that MaxEnt RL learns policies that are robust to perturbations to the dynamics. Importantly, to have MaxEnt RL learn policies that robustly maximize one a reward function, will apply MaxEnt RL to different, pessimistic reward function:
1
F(st, ag, 1)+1)
&
T log r(tt ar) + H(ss-1 st, ar].
(3)
The log(-) transformation is common in prior work on learning risk-averse policies (Mihatsch & Neuneier 2002). The entropy term rewards the policy for vising states that have stochastic dynamics, which should make the policy harder for the adversary to exploit. In environments that have the same stochasticity at every state (e.g.. LG dynamics), this entropy term becomes a constant and can be ignored. In more general settings, computing this pessimistic reward function would require some knowledge of the dynamics. Despite this limitation, we believe that our results may be of theoretical interest, taking step towards explaining the empirically -bbservee robustness properties of MaxEnt RL.
To formally state our main result, we must define the range of "similar" dynamics functions against which the policy will be robust. We use the following divergence between two dynamics functions:
Σ log //
p( St, at,at') |JAKS
d(p, T)
da, 'dst11..
((Stt1' Se, at")
SLET
This divergence is large when the adversary's 's dynamics f(st+1 St, at) assign low probability to transition with high probability in the training environment p. Our main result shows that applying MaxEnt RL to the pessimistic reward function results in a policy that is robust to these similar dynamics functions:
Theorem 4.2. Let an MDP with dynamics p(Se41 St. ax) and reward function r(st, at) be given. Assume that the dynamics have finite entropy (i.e., H|St41 St, at] ] is finite for all states and actions). Then there exists constant e Hy. [a, st] such that the MaxEnt RL objective with dynamics p and reward function F is a lower bound on the robust RL objective
min J JMasBa( T: p,r) 2 exp(JManEnd p. + log T), PEP(T)
4

---

Published as a conference paper at ICLR 2022
where the robust set is defined
_ a pls' s,
E m(aelss) p(se4alse.me)
[d(p, ((), EJ.
(5)
In defining the robust set, the adversary chooses a dynamics function from the robust set P inde- pendently at each time step; next result (Lemma 44.) will describe the value of in more detail. The proof, presented in Appendix [A.4] first shows the robustness to reward perturbations implies robustness to dynamics perturbations and then invokes Theorem 4.1 to show that MaxEnt RL learns policies that are robust to reward perturbations.
This result can be interpreted in two ways. First, if user wants to acquire policy that optimizes a reward under many possible dynamics, we should run MaxEnt RL with a specific, pessimistic reward function F(st. ag; St+1). This pessimistic reward function depends on the environment dynamics, so it may be hard to compute without prior knowledge or good model of the dynamics. However, some settings (e.g., dynamics with constant additive noise), we might assume that H[Se+1 Sr, an] is approximately constant, in which case we simply set the MaxEnt RL reward to F(s,, at, 8t+1) log r(s,, a.). Second, this result says that every time user applies MaxEnt RI,, they are (implicitly) solving robust RL problem, one defined in terms of a different reward function. This connection may help explain why prior work has found that the policies learned by MaxEnt RL tend to be robust against disturbances to the environment (Haarnoja al. 2019).
Theorem 4.2 relates one MaxEnt RL problem to another (robust) MaxEnt RL problem. We can also show that MaxEnt RL maximizes lower bound on unregularized RL problem.
Corollary 4.2.1. Under the same assumptions Theorem lower bound on the robust RL objective:
4.1 and 4.2
the MaxEnt RL problem is
Σ F( an)
min Ep(res 1 (se (e ).). (aee (ss)) (EP(R))FER(()
2 exp(JMannn(T:pp) + log T))
where the robust sets P(T) and R(xx) as defined as in Theorem 4.1 and 4.2
Our next result analyzes the size of the robust set by providing a lower bound on 6. Doing so proves that Theorem 4.2 is non-vacuous and will also tell us how to increase the size of the robust set. Lemma 4.3, Assume that the action space is discrete. Let a reward function r(st, ar) and dynamics p(s++1 St, at) be given. Let (aa 8t) be the corresponding policy learned by MaxEnt RL. Then the size of (in Eq. [ satisfies the following:
T aess(l [Hp]st+1] SL (11
St. ac] + H. [at Ss]] 2
T E aessr()))se).
[H [ag st]]
Ft Re)1mp(Se /11mm((s/1/55.n)
This result provides an exact expression for c. Perhaps more interesting is the inequality, which says that the size of the robust set is at least large as the policy's entropy. For example, if MaxEnt RL learns a policy with an entropy of 10 bits, then 2 10. This result immediately tells us how to increase the size of the robust set: run MaxEnt RL with larger entropy coefficient a. Many popular implementations of MaxEnt RL automatically tune the entropy coefficient so that the policy satisfies an entropy constraint (Haarnoja et al.) 20188). Our derivation here suggests that the entropy constraint on the policy corresponds to lower bound on the size of the robust set.
The constraint in the definition of the robust set holds in expectation. Thus, the adversary can make large perturbations to some transitions and smaller perturbations to other states, as long as the average perturbation is not too large. Intuitively, the constraint value e the adversary's "budget," which can use to make large changes to the dynamics in just a few states, or to make smaller changes to the dynamics in many states.
4.3 WORKED EXAMPLES
This section provides worked examples of our robustness results. The aim to build intuition for what results show and to show that, in simple problems, the robust set has intuitive interpretation.
5

---

Published as a conference paper at ICLR 2022
robust RL
MaxEnt
reward centropy
robust set, R-
original reward,
2.5
2.5
reward
2.0
2.0
1.5
1.5
0.4
0.6
0.4
0.5
F(a 1)
It(a 1)
nla = 1)
Figure 2: MaxEnt RL and Robustness Adversarial Reward Functions: (Left) Applying MaxEnt RL to one reward function (red dot) yields a policy that is guaranteed to get high reward on many other reward functions (blue curve). (Center) For each reward function (r(a 1), r(a 2)) on that blue curve, we evaluate the expected return of stochastic policy. The robust RL problem (for rewards) is to choose the policy whose worst-case reward (dark blue line) is largest. (Right) Plotting the MaxEnt RL objective (Eq. IT) for those same policies, we observe that the MaxEnt RL objective is identical to the robust RL objective.
Reward robustness. We present two worked examples of reward robustness. To simplify analysis, we consider the following, smaller robust set, which effectively corresponds to a weaker adversary:
r(st. at) log / exp(r(se, at) F(se, at dda,' S T CR.
(6)
For the first example, define 2-armed bandit with the following reward function and corresponding robust set:
ra)) [1 a= R R= =2(F] log exp(r(a) F(a))da sej.
(2 a ==1
a red dot, and the collection of reward functions, R, as blue line. In the center plot we plot the expected reward for each reward in R as function of the action a. The robust RL problem in this setting is to choose the policy whose worst-case reward (dark blue line) is largest. The right plot shows the MaxEnt RL objective. We observe that the robust RL objective and the MaxEnt RL objectives are equivalent.
Fig. (left) plots the original reward function,
For the second example, function composed of a task-specific reward Fisk and a penalty for deviating from some desired action a": rss, a) _ (s, a) (a - )²) The adversary will perturb this desired action by an amount Aa and decrease the weight the control penalty by 50%, resulting in the following reward function: F(s, a) A Frack (s, a) 1 (a (a* + Aa))², In this example, the subset of the robust set in Eq. 66 corresponds to perturbations Aa that satisfy
task with ID, bounded action space A = [-10, 10) and reward
1 log(2T) + log(20) (20) S T
Aa²
2
Thus, MaxEnt RL with reward function yields a policy that is robust against adversaries that perturb a" by most Aa (Vv)). See Appendix A.3 for the full derivation.
Dynamics robustness. The set of dynamics we are robust against, P. has an intuitive interpretation as those that are sufficiently close the original dynamics p(st-1 at). This section shows that, the case of to bounded L2 perturbation of the next state.
dynamics (described at the end of this subsection), this set corresponds
Because the robust set in Theorem 4.2 is defined terms of the policy, the adversary can intelligently choose where to perturb the dynamics based the policy's 's behavior. Robustness against this adversary also guarantees robustness against an adversary with a smaller robust set, that does not depend on the policy:
log //
clog ( ae) p( da, 'dstt'' < 6 c SP.
p(s St. ax)
T s
AxS
We will use this subset of the robust set in the following worked example.
Consider an MDP with ID, bounded, states and actions St, a, E 10, 10). The dynamics p(st+1 St. a,) = (s As, + Bag o = 1) the reward function is r(s,, a.) ||st112 and episodes have finite length T.. Note that the dynamics entropy H[st+1 St, a.] is constant. We assume

---

Published as a conference paper at ICLR 2022
HellCheetah-w2
Walker2d-22
Hopper-v2
12323
5300
Reward
6337
1333
2000
2300
0323
2000
5000
Reletive
Reletive
Re Relative
Maxtint RL to 0.0 0.00013
MexEnt RL (a 0.31
DOPG
PR:MDP
NR-MED
Figure 3: MaxEnt RL competitive with prior robust RL methods.
the adversary modifies the dynamics by increasing the standard deviation to v2 and shifts the bias by an amount 3, resulting in the dynamics p(St+1 St. a,) = (s As + Ba, + B, a = =2). choose any value of 3 that satisfies 5 + log(8v/]) (T) + (2g( S e. To apply MaxEnt RL to learn a policy that is robust any of these adversarial dynamics, we would use the pessimistic reward function specified by Theorem 4.2 F(8t, Ar St+1) = log |sel|2. See Appendix A.7 for the full derivation.
The robust set defined in Theorem 4.2 specifies that the adversary
4.4 LIMITATIONS OF ANALYSIS
We identify a few limitations of our analysis that may provide directions for future work. First, our definition of the robust set different from the more standard Ho and KI. divergence constraint sets used in prior work. Determining a relationship between these two different sets would allow future work to claim that MaxEnt RL is also robust to these more standard constraint sets (see) Lecarpentier al. (2020)) On the other hand, MaxEnt RL may not be robust to these more conventional constraint sets. Showing this may inform practitioners about what sorts of robustness they should not except to reap from MaxEnt RL. A second limitation is the construction of the augmented reward: to learn policy that will maximize reward function r' under a range of possible dynamics functions, our theory says to apply MaxEnt RL different reward function, V, which includes term depending on the dynamics entropy. While this term can be ignored in special MDPs that have the same stochasticity at every state, in more general MDPs it will be challenging estimate this augmented reward function. Determining more tractable ways to estimate this augmented reward function is an important direction for future work.
NUMERICAL SIMULATIONS
This section will present numerical simulations verifying our theoretical result that MaxEnt RL is robust to disturbances in the reward function and dynamics function. We describe our precise implementation of MaxEnt RI, standard RI., and all environments in Appendix
Comparison with prior robust RL methods. We compare MaxEnt RL against two recent robust RL methods, PR-MDP and NR-MDP Tessler al. 2019). These methods construct two-player game between a player that chooses actions and a player that perturbs the dynamics or actions. This recipe for robust RL is common in prior work (Pinto al. 2017), and choose to compare to Tessler et al. (219)) because it is a recent, high-performing instantiation of this recipe. We also include the DDPG baseline from Tessler al. (2019). We evaluate all methods on the benchmark tasks used in Tessler et al. (2019), which involve evaluating in an environment where the masses are different from the training environment. We ran MaxEnt RL with both small and large entropy coefficients, evaluating the final policy for 30 episodes and taking the average performance. We repeated this for 5 random seeds to obtain the standard deviation. The results shown in Fig. 3 suggest that MaxEnt RI. with large entropy coefficient is at least competitive, if not better, than prior purpose-designt robust RL methods. Note that MaxEnt RL is substantially simpler than PR-MDP and NR-MDP.
Intuition for why MaxEnt RL is robust. build intuition into why MaxEnt RI. should yield robust policies, compared MaxEnt RL (SAC (Haarnoja et al. 2018a)) with standard RL (TD3 (Fu- jimoto et al. 2018)) on two tasks. On the pusher task, shown in Fig. I a new obstacle was added during evaluation. On the button task, shown in Fig. 4b the box holding the button moved closer or further away from the robot during evaluation. Note that the robust RL objective corresponds to an adversary choosing the worst-case disturbances these environments.
Fig. (right) and Fig. (center)) show that MaxEnt RL has learned many ways of solving these tasks, using different routes push the puck to the goal and using different poses to press the button. Thus,
7

---

Published as a conference paper at ICLR 2022
Original Dynamiss
Dynamics
165
Stancard RL.
reward
1.5
Maxim R
0.0
Bd
-1.0
-0.5
1.0
Mavent
Staodard
disturbance button (Ay)
(b) Button:
Moved Goal
Pusher: New Obstacle
Figure 4: Robustness changes in the dynamics: MaxEnt RL policies learn many ways of solving a task, making them robust to perturbations such (Left) new obstacles and (Right) changes in the goal location.
when we evaluate the MaxEnt RL policy on perturbed environments, it is not surprising that some of these strategies continue to solve the task. contrast, the policy learned by standard RL always uses the same strategy to solve these tasks, so the agent fails when perturbation to the environment makes this strategy fail. Quantitatively, Fig. 44 and 4b (right) show that the MaxEnt RL policy is more robust than policy trained with standard RL.
In many situations, simply adding noise to the deterministic policy found by standard RI. can make that policy robust to some disturbances. MaxEnt RL does something more complex, dy-
Nadtit HL
Standland Rt
TO
namically adjusting the amount of noise depend- ing on the current state. This capability allows MaxEnt RL policies to have lower entropy in some states needed ensure high reward. We study this capability in the 2D navigation task shown in Fig. The agent starts near the top left corner and gets reward for navigating to the bottom right hand corner, but incurs large cost for entering the red regions. The policy learned by MaxEnt RL has low entropy initially to avoid colliding with the red obstacles, and then increases its entropy in the second half of the episode. To study robustness, we introduce new "L"-shaped ob- stacle for evaluation. The policy learned by MaxEnt RL often navigates around this obstacle, whereas the policy from standard RL always collides with this obstacle. Adding independent Gaussian noise to the actions from the standard RI, policy can enable that policy to navigate around the obstacle, but only at the cost of entering the costly red states.
return
Standard
retumn
0.8
go4l roturn
65.3
Figure 5: MaxEnt RL not standard RL noise.
Testing for different types of robustness. Most prior work on robust RL focuses changing static attributes of the environment, such as the mass or position of objects (Tessler et al., 2019 Kamalaruban al. 2020). However, analysis suggests that MaxEnt RL is robust against wider range of perturbations, which we probe in our next set of experiments.
First, introduced perturbations in the middle of an episode. We took the pushing task shown in Fig. and, instead of adding obstacle, per- turbed the XY position the puck after 20 time steps. By evaluating the reward of policy while varying the size of this perturbation, we can study the range of disturbances which Max- Ent RL is robust. We measured the performance of MaxEnt RL policies trained with different entropy coefficients
-30
reward
-40
-50
0
-60
1e-
0
le
The results shown in Fig. / indicate all methods perform well on the environment without any disturbances, but only the MaxEnt RL trained with the largest entropy coefficient is robust larger disturbances. This experiment supports our theory that the entropy coefficient determines the size of the robust set.
0.02
0.04
0.08
0.10
disturbance to puck
Figure 6: Robustness to dynamic perturbations: MaxEnt robust random external forces applied to the environment dynamics.
Our analysis suggests that MaxEnt RL not only robust to random perturbations, but is actually robust against adversarial perturbations. We next compare MaxEnt RI. and standard RL in this
8

---

Published as a conference paper at ICLR 2022
Malt
1.0
Success rate
Dag 7pod
MoDe EL
-darddddd
8000
Pog position
perturbation size
Figure 7: Robustness to adversarial perturbations of the environment dynamics.
minimax reward
1e. HalfChheethhhhhha v2
leJ
H2pper
leJ Walker2d-v2 v2
leJ
Ant-v2
-5
-2
0.0
0.0
1.0
1.0
train steps
train steps
train steps
train steps
MaxEnt RL
RL
Fictitious Play
Figure 8: MaxEnt RI. policies are robust disturbances the reward function.
setting. We trained both algorithms on a peg insertion task shown in Fig. (left). Visualizing the learned policy, we observe that the standard RL policy always takes the same route to the goal, whereas the MaxEnt RL policy uses many different routes to get to the goal. For each policy we found the worst-case perturbation to the hole location using CMA-ES (Hansen 2016). For small perturbations, both standard RL and MaxEnt RL achieve a success rate near 100%. However, for perturbations that are 1.5em 2cm in size, only MaxEnt RL continues solve the task. For larger perturbations neither method can solve the task. In summary, this experiment highlights that MaxEnt RL robust adversarial disturbances, as predicted by the theory.
Finally, our analysis suggests that MaxEnt RL is also robust to perturbations to the reward function. To test this theoretical result, we apply MaxEnt RL on four continuous control tasks from the standard OpenAl Gym (Brockman et al.) 2016) benchmark. We compare to SVG-0 (Heess et al.) 2015) (which uses stochastic policy gradients) and to fictitious play (Brown) 1951). the setting, fictitious play corresponds to modifying standard RL to use adversarially--cosen reward function for each Bellman update. We evaluate policy on an adversarially chosen reward function, chosen from the set defined in Equation
2 The analytic solution for this worst-case reward function ñ(st, at) = r(5t, a²) - log π aa; St... Both MaxEnt RL and standard RL can maximize the cumulative reward (see Fig. 11 1 in Appendix (B), but only MaxEnt RL succeeds maximizing the worst-case reward, as shown in Fig. [8. In summary. this experiment supports our proof that MaxEnt
RL solves robust RL problem for the set of rewards specified in Theorem 4.1
6 DISCUSSION
In this paper, we formally showed that MaxEnt RL algorithms optimize a bound on a robust RL objective. This robust RL objective uses a different reward function than the MaxEnt RL objective. Our analysis characterizes the robust sets for both reward and dynamics perturbations, and provides intuition for how such algorithms should be used for robust RL problems. To our knowledge, our work is the first to formally characterize the robustness of MaxEnt RL algorithms, despite the fact that numerous papers have conjectured that such robustness results may be possible. Our experimental evaluation shows that, in line with our theoretical findings, simple MaxEnt RI. algorithms perform competitively with (and sometimes better than) recently proposed adversarial robust RL methods on benchmarks proposed by those works.
Of course, MaxEnt RL methods are not necessarily the ideal approach to robustness: applying such methods still requires choosing hyperparameter (the entropy coefficient), and the robust set for MaxEnt RL not always simple. Nonetheless, we believe that the analysis in this paper, represents important step towards deeper theoretical understanding of the connections between robustness and entropy regularization in RL. We hope that this analysis will open the door for the development of new, simple algorithms for robust RI.

---

Published as a conference paper at ICLR 2022
Acknowledgments We thank Ofir Nachum, Brendan O' D'Donnguuu, Brian Ziebart, and anonymous reviewers for their feedback on an early drafts. BE is supported by the Fannie and John Hertz Foundation and the National Science Foundation GFRP (DGE 1745016).
REFERENCES
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. In International Conference on Learning Representations, 2018.
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbecl. Constrained policy optimization. In International Conference on Machine Learning, pp. 22-31. PMLR, 2017.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems Al safety, arXiv preprint arXiv:1606.06665 2016.
J Andrew Bagnell, Andrew Y Ng. and Jeff G Schneider. Solving uncertain Markov decision processes. 2001.
Andreea Bobu, Dexter RR Scobee, Jaime F Fisac, S Shankar Sastry, and Anca D Dragan. Less is more: Rethinking probabilistic models of human behavior. In International Conference Human- Robot Interaction, pp. 429 437, 2020.
Stephen Boyd, Stephen P Boyd, and Lieven Vandenherghe. Convex optinizatiion Cambridge University Press, 2004.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym. arXiv preprint 1606v:6606.136 2016.
George W Brown. Iterative solution of games by fictitious play. Activity analysis of production and allocation, 13(1):374-376, 1951.
Nicolas Carrara, Edouard I Leurent, Romain Laroche, Tanguy Urvoy, Odalric-Ambrym Maillard, and Olivier Pietquin. Budgeted reinforcement learning in continuous state space. In Advances in Neural Information Processing Systems, pp. 9299-9309, 2019.
Yinlam Chow, Mohammad Ghavanzadeh, Lucas Janson, and Marco Pavone. constrannd reinforcement learning with percentile risk criteria. The Journal of Machine Learning Research, 18(1):6070 6120, 2017.
Yinlam Chow, Ofir Nachum, Aleksandra Faust, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. Lyapunov-baset safe policy optimization for continuous control. arXiv preprint 1901 2019.
Jack Clark and Dario Amodei. Faulty reward functions in the wild, 2016. URL comm Rewardd---waad--uneeeee Suncition
Karl Cobbe, Oleg Klimov, Chris Hesse, Tachoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. In International Conference on Machine Learning, pp. 1282- 1289. PMLR, 2019.
Esther Derman, Matthicu Geist, and Shic Mannor. Twice regularized MDPs and the equivalence between robustness and regularization. Advances in Neural Information Processing Systems, 34, 2021.
John Doyle, Bruce A Francis, and Allen Tannenbaum. Feedback control theory. Courier Corporation, 2013.
Benjamin Eysenbach, Shixiang Gu, Julian Ibarz, and Sergey Levine. Leave no trace: Learning reset for safe and autonomous reinforcement learning. In International Conference on Learning Representations, 2018.
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. Search on the replay buffer: Bridging planning and reinforcement leamingg Advances Neural Information Processing Systems, 32, 2019.
Roy Fox, Ari Pakman, and Nañaal Tishby. Taming the noise in reinforcement learning via soft updates. In Uncertainty Artificial Intelligence, pp. 202-211, 2016.
Justin Fu, Avi Singh, Dibya Ghosh, Larry Yang, and Sergey Levine. Variational inverse control with events: A general framework for data-driven reward definition. Advances in Neural Information Processing Systems, 31, 2018.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, pp. 1587-1596. PMLR, 2018.
Peter D Grünwald, A Philip Dawid, et al. Game theory, maximum entropy, minimum discrepancy and robust bayesian decision theory. the Annals of Statistics, 127433333 2014.
10

---

Published as a conference paper at ICLR 2022
Sergio Guadarrama, Anoop Korattikara, Oscar Ramirez, Pablo Castro, Ethan Holly, Sam Fishman, Ke Wang, Ekaterina Gonina, Chris Harris, Vincent Vanhoucke, et al. TF-Agents: library for reinforcement learning tensorflow, 2018.
Tuomas Haarnoja, Haoran Tang, Pieter Abheel, and Sergey Levine. Reinforcement learning with deep energy- based policies. In International Conference on Machine Learning, pp. 1352-1361. PMIR, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference Machine Learning. pp. 1861 18700. PMLR, 2018a.
Tuomas Haarnoja. Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Picter Abbeel, al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 1812.0595, 2018b.
Tuomas Haarnoja, Sehoon Ha, Aurick Zhou, Tan, George Tucker, and Sergey Levine. Learning to walk via deep reinforcement learning. In Robotics: Science and Systems, 2019.
Dylan Galfield-Menell Smitha Milli, Pieter Abbeel, Stuart J Russell, and Anca Dragan. Inverse reward design. Advances in Neurul Information Processing Systems, 30. 2017.
Nikolaus Hansen. The CMA evolution strategy: A tutorial. arXiv preprint
2016.
Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrap, Tom Frez, and Yuval Tassa. Learning continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems, pp. 2944-2952, 2015.
Shiyu Huang, Hang Su, Jun Zhu, and Ting Chen. Svqn: Sequential variational soft q-learning networks. In International Conference on Learning Representations, 2019.
Andrew Ilyas, Shibani Santurkar. Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. Advances in Neural Information Processing Systems, 32, 2019.
Niels Justesen, Ruben Rodriguez Torrado, Philip Bontrager, Ahmed Khalifa, Julian Togelius, and Sebastian Risi. Illuminating generalization in deep reinforcement learning through procedural level generation. arXiv preprint arXiv: 10729, 2018.
Parameswaran Kamalaruban, Yu-Ting Huang, Ya-Ping Hsieh, Paul Rolland, Cheng Shi, and Volkan Cevher. Ro- bust reinforcement learning via adversarial training with langevin dynamics. Advances Neural Information Processing Systems, 33:8127-8138, 2020.
Hilbert Kappen. Path integrals and symmetry breaking for optimal control theory. Journal of statistical mechanics: theory and experiment, 2005(1 1011, 2005.
Alessandro Lazaric. Transfer in reinforcement learning: framework and survey, In Reinforcement Learning, pp. 143-173. Springer, 2012.
Erwan Lecarpentier, David Abel, Kavosh Asadi, Yuu Jinnai, Emmanuel Rachelson, and Michael I. Littman. Lipschitz lifelong reinforcement learning. arXiv preprint wrXiv:2001.05411. 2020.
Kyungjae Lee, Sungyub Kim, Sungbin Lim, Sungjoon Choi, and Songhwai Oh. Tsallis reinforcement learning: unified framework for maximum entropy reinforcement learning. arXiv preprint arXiv:1902.00137 2019.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint 1805 00909, 2018.
Erie Michaud, Adam Gleave, and Stuart Russell, Understanding learned reward functions. arXiv preprint arXiv:2012.05862 2020.
Oliver Mihatsch and Ralph Neuneier. Risk-sensitive reinforcement learning. Machine learning, 49(2-3))267-290, 2002.
Jun Morimoto and Kenji Doya. Robust reinforcement learning. Newral computation, 17(2):335-53) -39-335553999-3 2005.
Ofir Nachum, Mobammad Norouzi, Kelvin Xu, and Dale Schuurmans, Bridging the gap between value and policy based reinforcement learning In Advances in Neural Information Processing Systems, pp. 2775-2785, 2017.
11

---

Published as a conference paper at ICLR 2022
Arnab Nilim and Laurent Ghaoui. Robustness in markov decision problems with uncertain transition matrices. Advances Neural Information Processing Systems, 16:839 846, 2003.
Xue Bin Peng, Marcin Andrychowiez, Wojeiech Zaremba, and Pieter Abbecl. Sim-to-real transfer of robotic control with dynamics randomization. In International Conference on Robotics and Automation, pp. 3803- 3810. IEEE, 2018.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta, Robust adversarial reinforcement learning. In International Conference on Machine Learning, pp. 2817-2826. JMLR. org, 2017.
Martin L Puterman. Markov Decision Processes.: Discrete Stochastic Dynamic Programming. John Wiley
Sons, 2014.
Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, and Sergey Levine. Epopt: Learning robust neural
network policies using model ensembles. arXiv preprint arXiv:1610.0/283, 2016.
R. Tyrelll Rockafellar. Convex Analysis. Princeton University Press, 1970. ISBN 9780691015866. URL
http:// //wp:///www.// jstor wttp:///ttlee..et....oo.eo
Reazul Hasan Russel and Marck Petrik. Beyond confidence regions: Tight bayesian ambiguity sets for robust
MDPs. Advances in Neural Information Processing Syxtems, 2019.
Reazul Hasan Russel, Bahram Behzadian, and Marek Petrik. Entropic risk constrained soft-robust policy optimization. arXiv preprint urXiv:2006. 2020.
Reazul Hasan Russel, Mouhacine Benosman, Jeroen Van Baar, and Radu Corcodel. Lyapunov robust constrained- MDPs: Soft-constrained robustly stable policy optimization under model uncertainty. arXiv preprint Ttti:2/08.0270// 2021.
Fereshteh Sadeghi and Sergey Levine. CAD2RL: Real single-image flight without single real image. arXiv preprint 161 1.0200, 2016.
Yichuan Charlic Tang. Jian Zhang, and Ruslan Salakhutdinov. Worst cases policy gradients. Conference on Robot Learning, pp. 1078-10880 1098. PMLR, 2020.
Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforeement learning and applications in continuous control. In International Conference Machine Learning, pp. 6215-6224. PMLR, 2019.
Brijen Thananjeyan, Ashwin Balakrishna. Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang. Joseph E Gonzalez, Julian Ibarz, Chelsea Finn, and Ken Goldberg. Recovery rl: Safe reinforcement learning with learned recovery zones. Robotics and Automation Letters, 6(3):4915-499 2021.
Evangelos Theodorou, Jonas Buchli, and Stefan Schaal. A generalized path integral control approach to reinforcement learning. journal of machine learning research, 11(Nov)31177-3111, 2010.
Emanuel Todorov. Linearly-solvable Markov decision problems. Advances in Neural Information Processing Systems, pp. 1369-1376, 2007.
Mare Toussaint, Robot trajectory optimization using approximate inference. In International Conference on Machine Learning, pp. 1049-1056. ACM, 2009.
Nino Vieillard, Olivier Pictquin, and Matthieu Geist. Munchausen reinforcement learning. Advances Neural Information Processing Systems, 33:4235 2020.
Danfei Xu and Misha Denil. Positive-unlabeled reward learning. arXiv preprint arXiv:1911.00459/ 2019.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A henchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, pp. 1094-1 PMLR, 2020.
Chiyuan Zhang. Oriol Vinyals, Remi Munos, and Samy Bengio. study on overfitting in deep reinforcement learning. arXiv preprint 884,06934, 2018.
Kemin Zhou, John Comstock Doyle, Keith Glover, et al. Robust and optimal control, volume 40. Prentice hall New Jersey, 1996.
Brian D. Zichart. Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy PhD thesis, Carnegie Mellon University. 2010.
Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In AAAI, volume 8, pp. 1433-1433, Chicago, IL, USA, 2008
Brian D Ziebart, J Andrew Bagnell, and Anind K Dey. Maximum causal entropy correlated equilibria for markov games. In International Conference Autonomous Agents and Multiagent Systems, pp. 207-214, 201
12

---

Published as a conference paper at ICLR 2022
A PROOFS
A.1
USEFUL LEMMAS
Before stating the proofs, recall and prove (known) identities. Lemma A.I.
Epta.vl--og log v)]
min Bp(a.)) f(x.y)
ff(x, log E cfte'ln)
This identity says that the negative entropy function is the Fenchel dual of the log-sum-exp function.
This identity can be proven using calculus of variations:
Proof. We start by finding the function f(z, y) that optimizes the RHS of Eq. 7 Noting that the RHS is a convex function of (x,,)) we can find the optimal f(, y) by taking the derivative and setting it equal zero:
ef
Eptt.yyl-f(x y)1 + log Σ e
-p(z, v) + Ex ef(2'
(f v)
Setting this derivative equal to zero, we see that solution f* v) satisfies
(x.V)
Ex ef p(x,,))) v).
Now, we observe that p(v) = 1:
p(u) = Σ p(z,y)
Σ ef(().)
Eme
Exe((a)))
Ex =1.
We then have
p( 1) p(z,1) v)
(x.)))
Ex (1)
ply)
Taking the log of both sides, have
logp(z v) (x. log Σ (2'a)))
Multiplying both sides by and taking the expectation w.r.t. p(r, y). we obtain the desired result:
Ep(rn) log p(x v)] -=pp log Σ ef (a'
min (- v)1 log
> e
f(ma))
This lemma also holds for functions with more variables, result that will be useful when proving dynamics robustness. Lemma A.2.
Eptn.y.)) log.(p, -oog.,=)lloopppr,...
min Eptrmse)) f(x.).))
f(x,8,2 + log Σ elte'sa
Proof. Simply apply Lemma A.1 the random variables (xr, y) and
13

---

Published as a conference paper at ICLR 2022
A.2 PROOF OF THEOREM 4.1
This section will provide a proof of Theorem 4.1
Proof. We start by restating the objective for robust control with rewards:
min E Be ap (sclee) (E)(
Σ (St., at)
(8)
We now employ the KKT conditions (Boyd et al. 2004 Chpt. 5.5.3). If the robust set R strictly feasibly, then there exists a constraint value e 2 O (the dual solution) such that the constrained problem is equivalent the relaxed problem with Lagrange multiplier X == 1:
Eisse, i(se, ae) + tog / exp(r(st, a.) F(se, dd)
min E
Ry (a)s)( P (e))
To clarify exposition, we parameterize F by its deviation from the original reward, Ar(se, at) - r(st, at) - f(s,, We also define P (st, at) as the marginal distribution over states and actions visited by policy at time t. We now rewrite the relaxed objective in terms of Ar(st, at):
Σ (st, an) Ar(st, nc) log /
min T
Ag (me (ac)
exp(Ar((s a,) )da,
Pes1mp(Pe:1 (e))
[ r(st, at)
-E (asls)))
nen Me poppppe mn)
min E
(aella))
Σ
Ar(ss, ar) log /( exp(Ar(se, are )
so 2111pp(a) me)
Σ r(Se:
(asl((ss
4s pmpp(t a(sr.ae)
Ar(st, ae) + log / exp(Ar(se. at))
+ min
EpT (sn.ar)
Σ
r(s, ar) - Taa st)
(s.)
The last line follows from applying Lemma A.1
One important note about this proof is that the KKT conditions are applied to the problem of optimizing the reward function F, which is convex optimization problem. We do not require that the policy ग (a, st) or the dynamics p(S++1 St: ax) be convex.
A.3 WORKED EXAMPLE OF REWARD ROBUSTNESS
We compute the penalty for Example I in Section 4.3 using Gaussian integral:
10
PENALTY (f, log
exp(r(s, a) F(ss, a))da
-
= logg
exp( (-aa--aa - a²)²
aa²²² + ,(a - (a" + daa
10
10
1
log
exp(
(a (a" Aa))² + Aa? da
2
Aa²
+
log(2 + log(20).
2
A.4 PROOF OF THBOREM 4.2
The first step will be to convert the variation in dynamics into sort of variation in the reward function. The second step will convert the constrained robust control objective into an unconstrained (penalized) objective. The third step will show that the penalty is equivalent to action entropy.
Proof. We now provide the proof of Theorem 4.2
14

---

Published as a conference paper at ICLR 2022
Step 1: Dynamies variation reward variation.
Our aim is to obtain lower bound the
following objective:
Σ r(st: a+)
min E pEP(m) P
We start by taking log-transform of this objective, noting that this transformation does not change the optimal policy as the log function is strictly monotone increasing.
Σ r(st. at)
log E
aue aarr (ar/
S1+1 P(M
Note that we assumed that the reward was positive, so the logarithm well defined. We can write this objective in terms of the adversarial dynamics using importance weights.
log
I2
(mela()
p( St, at.) p(se+1
Σ r(se, ac)
se, at))
St /111pp(t (
logE
log Σ r(See at)
+ +Σ log p(st+1 St., at) losp(s St. at)
ae m4)
Ive
E
aeoemtne (n))
log
Σ r(se, at)
+ Σ
log ((se(
St, at) poge( Se, ac)
St A1l1 (111mp(s1111
log
1
Σ r(5e, ac)
l'' Σ
log p(s Stc, at) - loge4 at)
T
*t (asinppaui/a (ala
IVe
ae agoor(ae (x))
Σ log r(t, at) + log St. ac) - log St, as)
+ log T.
(9)
Both inequalities are applications of Jensen's inequality. As before, our assumption that the rewards are positive ensures that the logarithms remain well defined. While the adversary choosing the dynamics under which we will evaluate the policy, we are optimizing lower bound which depends on a different dynamics function. This step allows us to analyze adversarialll-choser dynamics as perturbations to the reward.
Step 2: Relaxing the constrained objective To clarify exposition, we will parameterize the adversarial dynamics deviation from the true dynamics:
Ar(S4+1, St: a,) = log p($ St, a,) - log p( St, a.).
(10)
The constraint that the adversarial dynamics integrate can be expressed
p(s++1 St. at)e (1(
ds =
Using this notation, we can write the lower bound on the robust control problem (Eq, 9 as follows:
min E (auon(a
Σ - log r(st ar) Ar(S++1 1 St, at)
+ +log?
(11)
Ar *t=rr (te)
Σ log /I
J arreeus' " day dse+i
s.t. E (.)
Sc
(12)
and
p(st+1 St, at)e Ar(sst 1.8.ae)))) ds = Vst. at.
(13)
((
The constraint in Eq. 12 the definition of the of adversarial dynamics P, and the constraint in Eq. ensuress that St+1 St, ar) represents valid probability density function.
Note that (r(S++1, at) 0 is strictly feasible solution to this constrained optimization problem for O. Note also that the problem of optimizing the function Ar(st, a,) convex optimization
15

---

Published as a conference paper at ICLR 2022
Chpt. 5.5.3). If the robust set P is strictly feasibly, then there exists > 0 (the dual solution) such that the set of solutions to the constrained optimization problem Eq. 1 equivalent to the set of solutions to the following
problem. We can therefore employ the KKT conditions (Boyd al. 2004
relaxed objective with Lagrange multiplier
min E (
Σ log (st, at) St,
St,
(14)
Ar 1440mp9e41180
log H1 Axs
crlann meiee" day dssti log
[ [((Suu St,, at)e
Artect1 an.")
d8 Vse, ac.
(15)
This step does not require that the policy x(ae 8t) the dynamics p(st+1 St, a+) be convex.
Our next step to show that the constraint does not affect the solution of this optimization problem. For any function Ar(s++1, St, at)), value but now satisfy the constraint. We construct c(st. a²) as
add constant c(st, a and obtain the same objective
c(se, at) = log p(st+1 St, ag)e Ar(e au ds
Vst. at.
First, we observe that adding c(St. at) to Ar does not change the objective:
Σ - log r(s,, an) - (Ar(se+ ar) + c(se, at))
E (()er(m)()e); See seezn=((Se=all (tc.
log If
+e(ss, day dse+1' + log T
AXS
E
as nu agpor(a
Σ T log fogr at) Ar(se+ ar)) - c( ac)
(o(oera)))
II. AXS
carlan+i e "n 'n dar dae dd'''dt dost
dist
log
log T
Σ log Sogrr(s. ar) Ar(stt1, aistraiJ + c(serai)
E (ncl)))
fe-rrp($t:1 ac)
(J
Artani1 a )a de da
log
log
AxS
Second, we observe that the new reward function Ar(s St, ar) + c(st. ar) satisfies the constraint in Eq. 15
/ p(st+1 St, ae
(Ar(olt1' (a m dsst1
s
-((ua))))) p(sa41 | St. at)e Ar am. ds
Ss P(St+1 St, a Ar(ec+1 (r "d8t+1
Ss p(((++1 St. a Ar(se (sn ( 1.
Thus, constraining Ar represent probability distribution does not affect the solution (value) to the optimization problem, so can ignore this constraint without loss of generality. The new, unconstrained objective is
Σ
_ log r(s,, ar) Sr, ax)
1
min E Ar
a
of s'mp((s (ls
//
+ +og
A edrlecri 8e day dhtsi + log
(16)
16

---

Published as a conference paper at ICLR 2022
Step 3: The penalty is the Fenchel dual of action entropy. We define pi (St. aq. S as the
marginal distribution of transitions visited by policy time L. We now apply Lemma A.2 Eq. 16
Σ log r(se, ac) St, at)
min E
anstlas (s)),
Ar
a ppppte (8):8e)
+ log !/
carieetiie me'' day "sa'dssi'' + log T
(17)
Axs
E ams st), d'mp(et al)
Σ - log r(t,, at) + log
min E or
und (as (ec).
Σ 1,, + log
AXS
a day dse+1
d'muptte mi.me)
Σ log r(st, at)
E aas sc). d'mplet 1
+ Tog
min Ar
Σ EpT (B.
Ar. ae) + log If
Arleetiin mel day dss+1
= E awai(ar st).
Σ T log ((sss,, - Se41 (s)) + log T
d' d' (R.))))
= E awel(n sch. d'
Σ log (St, at) - T(a
m(a (t)) - plggp(st+1 Stt, at)
+ +log" T
Summary. We have thus shown the follow:
min log JunnFa ( (r: r) (T: log
$EP
Taking the exponential transform of both sides, obtain the desired result.
A.5 ROBUSTNESS TO BOTH REWARDS AND DYNAMICS
While Theorem 4.2 is phrased terms of perturbations to the dynamics function, the reward function, we now discuss how this result can be used to show that MaxEnt RL is simultancously robust to perturbations in the dynamics and the reward function. Define a modified MDP where the reward is appended to the observation. Then the reward function is the last coordinate of the observation. this scenario, robustness to dynamics is equivalent robustness to rewards.
A.6 How BIG IS THE ROBUST SET (c)?
Our proof of Theorem 4.2 used duality to argue that there exists for which MaxEnt RL maximizes a lower bound on the robust RL objective. However, we not specify the size of this e. This raises the concern that might be arbitrarily small, even zero, in which the result would be vacuous. In this section we provide a proof Lemma 4.3 which provides a lower bound on the size of the robust set.
Proof. Our proof proceeds in three steps. We first argue that, at optimality, the constraint on the adversarial dynamics tight. This will allow us treat the constraint as an equality constraint, rather than an inequality constraint. Second, since this constraint holds with equality, then we can rearrange the constraint and solve for e terms of the optimal adversarial dynamics. The third step is to simplify the expression for
Step 1: The constraint on Ar holds with equality. Our aim here to show that the constraint on Ar in Eq. 12 holds with equality for the optimal Ar (i.e., that which optimizes Eq. 11 The objective
In the case where there multiple optimal Ar, we require that the constraint hold with equality for least of the optimal Ar.
17

---

Published as a conference paper at ICLR 2022
in Eq. 1970 Chapter 32).
11 is linear in Ar, there must exist optimal Ar at the boundary of the constraint (Rockafellar
Step 2: Solving for Since the solution to the constrained optimization problem in Eq. 11 occurs at the boundary, the constraint in Eq. holds with equality, immediately telling us the value of e:
Σ log !/ @ an (a.ae') day dds+1'
E = E ie (se
at 1mpp(ss (sc
/f Axs
ear(sct1' (rr(ect1'at,ai")) day do
= T E p(ss)
log
(18)
where Ar is the solution to Eq. This identity holds for all states St. The second line above expresses the expectation over trajectories as an expectation over states, which will simplify the analysis in the rest of this proof. The factor of T is introduced because we have removed the inner summation.
Step 3: Simplifying the expression for To better understand this value of c, recall that the following identity (Lemma A.2) holds for this optimal Ar:
Ar 1)
p(a Se+1 St)
VSe. at St+1
AAx Ar st 1141" dae', ds
We next take the log(-) of both sides and rearrange terms:
[/ AxS e ix dar'. dse+1' Ar(st, a St-1) log p(at, St+1 st).
log
Next, we substitute the definition of Ar (Eq. 10h and factor log p(a,, St+1 1st):
//
log
e f (1112) ds
logp($++1 Se, ac) log p( St+1 St, at)
log St. at) log π (at St)
log 1 (St41 Se: at)
log (at St).
Substituting this expression into Eq. we obtain more intuitive expression for e:
T
( S Des1np(st-11 (e)
[- loog St. (... ax) loog ((a st]]
T E (aele)
(Hpp [1p(st+1] Se: at] + Hzlaa se]]
Pe=rnp(st-11 (c)
T ((sm(a
[Hrr [a st.].
$t41np(stt11l (s))
The last line follows from our assumption that the state space discrete, the entropy Mp(se+1 St, ar] is non-negative. This same result will hold in environments with continuous state spaces as long as the (differential) entropy Hzfse41 St, ar] is non-negative.
A.7 WORKED EXAMPLE OF DYNAMICS ROBUSTNESS
We calculate the penalty using a Gaussian integral:
//A
p(
Be. at)
log
Axs p( [
da,ds
St, at)
= log // AAA
1 2exp( - (st+1 - (Ast + +Ba)))² (se+1 - (Ase + Bar 9)) dan
1
log //
1 2exp(-
- (SL+1 - (Ast + Bai 8)) 8 daa
22 10
1 (2)
log
00 +og((8 (20).
18

---

Published as a conference paper at ICLR 2022
a= 0.2
a=
3.0
2.5
1.0
0.5
0.5
1.0
1.5
2.0
2.5
3.0
0.5
1.0
1.5
2.0
2.5
3.0
r(a 1)
r(a
Figure 9: Effect of Temperature: (Left) For given reward function (blue dot), we plot the robust sets for various values of the temperature. Somewhat surprisingly, it appears that increasing the temperature decreases the of reward functions that MaxEnt is robust against. (Right) We examine the opposite: for given reward function, which other robust sets might contain this reward function. We observe that robust sets corresponding to larger temperatures (i.e., the red curve) can be simultaneously robust against more reward functions than robust sets at lower temperatures.
We make two observations about this example, First, the penalty would be infinite if the adversary dynamics ş had the same variance as the true dynamics, p: the adversary must choose dynamics with higher variance. Second, the penalty depends on the size of the state space. More precisely, the penalty depends on the region over which the adversary applies their perturbation. The adversary can incur smaller penalty by applying the perturbation over smaller range of states.
A.8 TEMPERATURES
Many algorithms for MaxEnt RL (Haarnoja et al. 2018a Fox et al., 2016, Nachum et al. 2017) include temperature > 0 to balance the reward and entropy terms:
MaxFor (T,, E Σ r(s at)
amH Ja, stl.
We can gain some intuition into the effect of this temperature the set of reward functions to which we are robust. In particular, including a temperature a results in the following robust set:
Us. (a:) 2 0 Vse. ae and f Mun (me) S Vist
R
r(st, at) + cruss (at)
/ " (mi)/a da S 1 Ass
r(ss, at) + Us. (at)
Us. (at) 2 0 Vst, ag and
(19)
In the second line, we simply moved the temperature from the objective to the constraint by redefining Ms (at) + Ux, (at).
We visualize the effect of the temperature in Fig. First, we fix a reward function r, and plot the robust set R for varying values of a. Fig. 9 (left) shows the somewhat surprising result that increasing the temperature (i.e., putting more weight on the entropy term) makes the policy less robust. In fact, the robust set for higher temperatures is strict subset of the robust set for lower temperatures:
a1 <2
R C R
This statement can be proven by simply noting that the function e 5 an increasing function of in Equation 19 It is important to recognize that being robust against more reward functions is not always desirable. In many cases, be robust everything, an optimal policy must do nothing.
We now analyze the temperature in terms of the converse question: if a reward function r' is included in robust set, what other reward functions are included in that robust set? To do this, we take reward function r', and find robust sets R that include r', for varying values of a. As shown in Fig. (right), if we must robust to and use high temperature, the only other reward functions to which we robust are those that are similar, pointwise weakly better, than r'. In contrast, when using a small temperature, we are robust against a wide range of reward functions, including those that are highly dissimilar from original reward function (i.e.. have higher reward for some actions, lower reward for other actions). Intuitively, increasing the temperature allows us to simultancously be robust larger set of reward functions.
19

---

Published as a conference paper at ICLR 2022
A.9 MAXENT SOLVES ROBUST CONTROL FOR REWARDS
In Sec. 4.1 we showed that MaxEnt RL is equivalent to some robust-reward problem. The aim of this section go backwards: given of reward functions, can we formulate MaxEnt RL problem such that the robust-reward problem and the MaxEnt RL problem have the same solution?
Lemma A.3. For any collection of reward functions there exists another reward function r such that the MaxEnt RL policy w.r.t. r is an optimal robust-rewan policy for R:
T rst, a.)
arg max ES
Σ r ax)
+ Hxla
S
arg max min E, P'RR
Σ
We use set containment, rather than equality, because there may be multiple solutions to the robust- reward control problem.
Proof. Let nt be a solution to the robust-reward control problem:
arg min Ex nE
(s at)
Define the MaxEnt RL reward function as follows:
r(e a,)
log (a, ss)).
Substituting this reward function Equation
I
we see that the unique solution is π = n².
Intuitively, this theorem states that problem that requires robustness with respect to any arbitrary set of rewards, if we can find the right corresponding reward function for MaxEnt RL. One way of viewing this theorem is providing an avenue to sidestep the challenges of robust-reward optimization. Unfortunately, we will still have to perform robust optimization learn this magical reward function, but at least the cost of robust optimization might be amortized. In some sense, this result is similar to Ilyas et al. (2019).
MaxEnt RL to solve any robust-reward control
A.10 FINDING THE ROBUST REWARD FUNCTION
the previous section, we showed that policy robust against any set reward functions R can be obtained by solving MaxEnt RL problem. However, this requires calculating reward function r* for MaxEnt RL, which not in general an element in R. In this section, we aim to find the MaxEnt reward function that results in the optimal policy for the robust-reward control problem. Our main idea is to find reward function r* such that its robust set, Ry,, contains the set of reward functions want to be robust against, R. That is, for each ra E R, we want
r, (St a2) r" (St, a+) + Uaa (ax) for some Us (ax) satisfying JA
Ue, (ee) day I 1 VSS.- (0
Replacing u with - r", we see that the MaxEnt reward function r must satisfy the following constraints:
[ e" (ss.aa)- da. S Vst E S,r e R.
We define R2 (R) the of reward functions satisfying this constraint w.r.t. reward functions in R:
(R) A p* [1 e" (e da < As, E S, E E
Note that satisfy the constraint by making r* arbitrarily negative, the set 12" (R) is non- empty. We now argue that all any applying MaxEnt RL to any reward function r" E R* (R) lower bounds the robust-reward control objective.
Lemma A.4. Let set of reward functions Rt be given, and r" E Rd (R) be an arbitrary reward function belonging the feasible set of MaxEnt reward functionn. Then
T
J (T, (T,T") < min Eg
Σ
rl (St, at)
VT E II.
yER
20

---

Published as a conference paper at ICLR 2022
normalized minimax reward
6.6
644
LaudrBourd MaxErt
Pecil
unform palicy
62-
steitte palicy
optimizaaiio steps
Figure 10: Approximately solving an arbitrary robust-reward control problem. In this experi- ment, we aim to solve the robust-reward control problem for an arbitrary set of reward functions. While know that MaxEnt RL can be used to solve arbitrary robust-reward control problems exactly, doing so requires that we already know the optimal policy ($ A.9). Instead, we use the approach outlined Sec. A0, which allows us to approximalely solve an arbitrary robust-reward control problem without knowing the solution apriori. This approach ("LowerBound + MaxEnt") achieves near-optimal minimax reward.
Note that this bound holds for all feasible reward functions and all policies, so it also holds for the maximum
max JMastim (x, r²) S min Ex ER))((
Σ
(st ac)
Vn E II.
r'ER
:==1
Defining nt = arg max, JMaxFRn r"), we get the following inequality:
Σ (st: an)
T Σ
(s an)
max (R)
MaxxEm (T, r*) S min Egg
S max min E. well YER
(20)
PeR
Thus, we can find the tightest lower bound by finding the policy a and feasibly reward r* that maximize Equation 20
T r(st, ar) + +Hxlae [s]]
Σ
max
(21)
1=1
/ e (r (se, S SIV ES, e
S.L.
It is useful note that the constraints simply LOGSUMEXP functions, which are convex. For continuous action spaces, we might approximate the constraint via sampling. Given a particular policy, the optimization problem w.r.t. has linear objective and convex constraint, so it can be solved extremely quickly using a convex optimization toolbox. Moreover, note that the problem can be solved independently for every state. The optimization problem is not necessarily convex in
A.11 ANOTHER COMPUTATIONAL EXPERIMENT
This section presents an experiment to study the approach outlined above. Of particular interest is whether the lower bound (Eq 20) comes close the optimal minimax policy.
We will solve robust-reward control problems on 5-armed bandits, where the robust set is collection of 5 reward functions, each drawn from pero-mean, mart-aaiancce Gaussian. For each reward function, we add constant to all of the rewards to make them all positive. Doing so guarantees that the optimal minimax reward positive. Since different bandit problems have different optimal minimax rewards, will normalize the minimax reward the maximum possible value is 1.
Our approach, which we refer to as "LowerBound MaxEnt", solves the optimization problem in Equation by alternating between (1) solving convex optimization problem find the optimal reward function, and (2) computing the optimal MaxEnt RL policy for this reward function. Step is done using CVXPY, while step 2 is done by exponentiating the reward function, and normalizing to sum to one. Note that this approach is actually solving a harder problem: solving the rowust-reward control problem for much larger of reward functions that contains the original set
21

---

Published as a conference paper at ICLR 2022
of reward functions. Because this approach is solving more challenging problem, do not expect that it will achieve the optimal minimax reward. However, we emphasize that this approach may be easier to implement than fictitious play, which we compare against. Different from experiments in Section [.1] the "LowerBound + MaxEnt" approach assumes access to the full reward function, not just the rewards for the actions taken. For fair comparison, fictitious play will also player that has access to the reward function. Fictitious play is guaranteed to converge to the optimal minimax policy, so we assume that the minimax reward converges to is optimal. We compare against two baselines. The "pointwise minimum policy" finds the optimal policy for new reward function formed by taking the pointwise minimum of all reward functions:
policy
7(a,) = min R r(a)). This strategy is quite simple and intuitive. The other baseline is a "uniform policy" that chooses actions uniformly at random.
We ran each method on the same set of 10 robust-reward control bandit problems. In Fig. T0 we plot the (normalized) minimax reward obtained by each method on each problem, as well as the average performance across all 10 problems. The "LowerBound + MaxEnt" approach converges to a normalized minimax reward of 0.91, close to the optimal value of In contrast, the "pointwise minimum policy" and the "uniform policy" perform poorly, obtaining normalized minimax rewards of 0.56 and 0.60, respectively. In summary, while the method proposed for converting robust- reward control problems to MaxEnt RL problems does not converge the optimal minimax policy, empirically it performs well.
B EXPERIMENTAL DETAILS
B. DYNAMICS ROBUSTNESS EXPERIMENTS (FIG.
We trained the MaxEnt RL policy using the SAC implementation from TF Agents (Guddaraama et al. 2018) with most of the default parameters (unless noted below).
We used the standard Pusher-v2 task from OpenAl Gym (Brockman et al. 2016). We used a fixed entropy coefficient of le - 2 for the MaxEnt RL results. For the standard RL results, we used the exact same codebase to avoid introducing any confounding factors, simply setting the entropy coefficient to a very small value 1e 5. The obstacle is a series three axis-aligned blocks with width 3em, centered at (0.32, -0.2), (0.35, -0.23), and (0.38, -0.26). We chose these positions to be roughly along the perpendicular bisector of the line between the puck's
Fig. 4a
's initial position and the goal. We used 100 episodes of length 100 for evaluating each method. To decrease the variance in the results, we fixed the initial state of each episode:
qpos = [0., 0., 0., 0., 0., 0., 0., 0.3, 1..3,-0.22 0.2, 0., 0.],
qvel = [0., 0., 0. 0., 0., 0., 0., 0., 0.,
We used the SuwyerBut www.eessenttttrtttnneeen environment from Metaworld (Yu et al. 2020), using maximum episode length of 151. For this experiment, perturbed the environment by modifying the observations such that the button appeared to be offset along the Y axis. We recorded the average performance over 10 episodes. For this environment we used an entropy coefficient of lel for MaxEnt RL and le - 100 for standard R
Fig. 4b
We used the standard Pusher-uu 22 task from OpenAl Gym (Brockman al. 2016). We modified the environment to perturb the XY position of the puck at time t = 20. We randomly sampled an angle 0 ~ Onif(0, 2T and displaced the puck in that direction by an amount given by the disturbance size. For evaluating the average reward of each policy on each disturbance size, we used 100 rollouts of length 151.
Fig. 6
We used modified version of the 2D navigation task from Eysenbach et al. 2019 with the following reward function:
Fig. 5
r(st, at) = ||se (8, 4) - 10 1 (st Satstaet)..
Episodes were 48 steps long.
We used larger value for the entropy coefficient for standard RI. in the previous experiment to avoid numerical stability problems.
22

---

Published as a conference paper at ICLR 2022
Fig. 7 We used the peg insertion environment from Eysenbach et al. (2018) Episodes most 200 steps long, but terminated as soon as the peg was in the hole. The agent received a reward of +100 once the peg was in the hole, in addition to the reward shaping terms described in Eysenbach et al. 2018).
B.2 REWARD ROBUSTNESS EXPERIMENTS (FIG.
Following Haamoja al. 82018a,, we use an entropy coefficient of 1/5. Fig. the thick line is the average over five random seeds (thin lines). Fig. F shows the evaluation of all methods on both the expected reward and the minimax reward objectives. Note that the minimax reward can be computed analytically as
F(st, ac) = r(se; a²) log x(a; st).
B.3 BANDITS (FIG.
The mean for arm i, μ4 is drawn from zero mean, unit-variance Gaussian distribution, μ4 (O, 1). When the agent pulls arm i, it observes noisy reward F3 ~ (N4,, 1). The thick line is the average over 10 random seeds (thin lines).
HalfCheetah-v2
Hopper-v2
Walker2d-v2
Ant-v2
3000
cumulaatve reward
2000
2000
5000
1000
-2000
minimax reward
2000
2000
-20000
-5000
-2000
500000
1000000
500000 1000000
500000 1000003
500000 1000000
train steps
train steps
train steps
train steps
MaxEnt RL
RL
Fictitious Play
Figure 11: (Top) Both RL (SVG) and MaxEnt RL (SAC) effectively maximize expected reward.
(Bottom) Only MaxEnt RL succeeds in maximizing the minimax reward.
C ROBUST RL ABLATION EXPERIMENTS
We additional ablation experiments study whether simple modifications the robust RL baseline from Section (Tessler et al.] (019)) would improve results. These experiments will help discern whether the good performance of MaxEnt RL. relative the PR-MDP and NR-MDP baselines, comes from using more recent RL algorithm (SAC instead of DDPG), or from the entropy regularization. We used the following ablations:
1. larger network: We increased the width of all neural networks to 256 units, so that the network size the exactly the same for the MaxEnt RL method.
2. dual critic: Following TD3 (Fujimoto al. 2018). added second Q function and used the minimum over two (target) Q functions. This change affects not only the actor and critic updates, but also the updates to the adversary learned by PR/NR-MDP,
3. more exploration: We increased the exploration noise from 0.2 to 1.0.
implemented these ablations by modifying the open source code released by Tessler et al. (2019).
The results, shown in Fig. 12 show that these changes do not significantly improve the performance of the NR-MDP or PR-MDP. Perhaps the one exception is on the Hopper-v2 task, where PR-MDP with the larger networks is now the best method for large relative masses. The dual critic ablation generally performs worse than the baseline. We hypothesize that this poor performance is caused by using the (pessimistic) minimum two functions for updating the adversary, which may result
23

---

Published as a conference paper at ICLR 2022
in weaker adversary. This experiment suggests that incorporating the dual critic trick into the action robustness framework may require some non-trivial design decisions.
HalfCheetehwe2
Walleerdd122
Hopperiu2
3003
Reward
3.30
-3000
500
12
1.3
2.4
3.0
Relative mads
Solative mass
Ro aElv mass
NI-MDP
(langer network)
NR-MDP (dual crick)
(mare exproration)
(a) NR-MDP ablations.
HellCheetah-v22
Walkeer2d-2
Hopper-22
11:22
3605
Rewradd
2002
1000
542
2.6
Re reletiiee mas5
Relative 17855
Reletive m955
PR-MDP
PR-MOP ( arger networki
PR-MOP (dual critic)
(b) PR-MDP ablations.
Ho MCheetah v2
Whiler2d v2
Hooper
Rewerdd
16800
1000
3097
5020
1000
1002
Selative mass
Ralarive m2s5
Relatie ma55
MaxEnt RL (a- 3.00011
DOPG
NR MDP
NR HDP (dual crtic)
PR MDP (larger network) PR-MDP (dual critic)
MaxEnt RL 0.1)
PS.MDP
NR NCP taanee network)
NR HDP (more exploration)
(c) Ablation experiments above overlaid on Fig.
Figure 12: Robust RL Ablation Experiments: Ablations of the action robustness frame- work [Tssseer et al. 2019) that use larger networks, multiple Q functions, or perform more exploration do not perform significantly better.
24


Reviews:
[{"review_id": "lutJARbSGCh", "reviewer": "Reviewer_VSYD", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is generally well written. The motivation of the work is clear, and the results seem technically correct (I did not have time to check the proofs in details, my apologies).\n\nMain comments:\n\n- Some recent references in the field of robust RL with guarantees, are missing:\n  1- Reazul Hasan Russel and Marek Petrik. Beyond con\fdence regions: Tight Bayesian ambiguity sets for robust MDPs. Advances in Neural Information Processing Systems, 2019.\n  2- Reazul Hasan Russel, Mouhacine Benosman, Jeroen Van Baar, Radu Corcodel, Lyapunov Robust Constrained-MDPs: Soft-Constrained Robustly Stable Policy Optimization under Model Uncertainty, arXiv:2108.02701, 2021.\n  3- Reazul Hasan Russel, Bahram Behzadian, Marek Petrik, Entropic Risk Constrained Soft-Robust Policy Optimization, arXiv:2006.11679, 2020.\n whereas other references are not properly cited: Berenkamp et al. 2017 simply uses Lyapunov theory (from control theory) to impose stability of the RL policy, no robustness is analyzed in that work. Similarly for Chow et al. 2018; please either make your statement related to these papers about stability, or remove the references since your work is not about Lyapunov stability of the RL policy.\n\n- Mathematically, the open integrals are rather unnecessary in this context. I assume all your integrations are done over the finite support of action space ? please explain, or amend the paper accordingly.\n\n- The statement about LQG dynamics having same stochasticity at every state, is not clear. Do you mean a dynamical system controlled by an LQG controller ?\n\n- In general the results proposed here in Theorems 4.2 and 4.3 are existence results, i.e., there exists and epsilon, etc. How can these bounds be used to design a policy such that the robustness is guaranteed for a desired epsilon ?\n\n- I found 'reward robustness' examples rather simple. Indeed, with the simple linear dynamics one can compute closed forms of the target reward \\bar{r}, etc.but what happens when the dynamics are nonlinear and a closed from solution is not easily obtainable ? do you plan to rely on a numerical search for \\bar{r} ? \n", "summary_of_the_paper": "The authors work on the important problem of robustness guarantees in RL algorithms. They analyze the robustness of max entropy RL w.r.t. dynamics uncertainties, and reward function uncertainties.", "summary_of_the_review": "In summary, the paper is a nice step forward in obtaining robustness guarantees for RL algorithms. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WeUx4IYD9q9", "reviewer": "Reviewer_r4e7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Maximum entropy is a form of regularization that prioritizes higher entropy over optimality. Previous works have shown that such regularization may lead to faster learning and in general that there is a connection between generalization <-> regularization <-> robustness.\n\nHence, while these results are not very surprising the clear connection is important and interesting.\n\nI find the theoretical results and the explanation very good. Maxent will be robust to small pertrubations in the reward/transitions which is similar to what we'd expect in real world robustness where uncertainty often stems from manufacturing defects and sensory-motor degredation.\n\nI do find the empirical results lacking and specifically the focus on NR/PR-MDP (build ontop of DDPG). Action robustness as a framework isn't necessarily limited to the DDPG agent, which raises a question as to whether the performance improvement observed by MaxEnt is due to the learning procedure (larger network capacity, improved stability via dual critics, better exploration via the entropy regularization of maxent) or does the maxent objective itself lead (in these sets of tasks) to improved generalization.\n\nfinally, I suggest the authors refer to prior work [1] that analyzes the connection between regularization and robustness in general MDPs (as maximum entropy is a special case of regularization).\n\n[1] Esther Derman, Matthieu Geist, Shie Mannor - Twice regularized MDPs and the equivalence between robustness and regularization\n", "summary_of_the_paper": "This work analyzes the implicit robustness of maximum entropy RL. They show, theoretically, that there exists some robust MDP for which the optimal solution is obtained by maximizing the max-ent objective. Empirically they show that indeed SAC learns behavior which is in a sense robust.", "summary_of_the_review": "interesting paper connecting robustness and maximum entropy.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NzBDAA1K2U4", "reviewer": "Reviewer_bxqc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors investigate an interesting problem, which consists in asking whether maximum entropy reinforcement learning can be thought as a robust problem. The authors correctly describes the related works, which shows the relationship between MaxEnt RL and some kind of robustness w.r.t. the reward. \n\nWhile the article is indicative of a great effort by authors on the topic, the work still suffers, in my opinion from some major weaknessess. The first one is related the clarity of the paper and the soundness of its claims. After having stated Theorem 4.2, the authors say that it \"provides a recipe for converting between MaxEnt RL and robust control problems\" and make two claims about that.\n1) \"On the one hand, if we want to acquire a policy that optimizes a reward under many possible dynamics, we can run MaxEnt RL with a pessimistic reward function\". The robust problem that they are lower bounding, though, is a MaxEnt problem too, hence, the obtained policies are not simply optimizing the actual reward under many possible dynamics, since ntropy regularization is involved also in the robust problem.\n2) \"On the other hand, this result says that every time a user applies MaxEnt RL, they are (implicitly) solving a robust RL problem, one defined in terms of a different reward function.\". Also this claim is ambiguous: it is not clear whether the authors are suggesting that the lower bound applies also using any reward in place of $\\bar{r}$ and substituting $r$ in the robust problem accordingly, or something else.\nTherefore, it is not clear whether the issue is about clarity (in which case the authors should improve their exposition) or whether instead the claims are not sound. It seems that the result presented by the authors connects only two MaxEnt problems, and one of them is a robust one, while the second one has a modified reward. In other words, the theorem gives some insights on the relationship between two regularized problems, but does not seem to relate the regularized problem in any way with some instance of the not-regularized one.\n\nThe second issue is related to relevance: to which extent the bound provided by the authors helps in shedding light to MaxEnt optimization? If MaxEnt is related to robust MaxEnt, is it possible to draw a parallel also to standard robustness?\n\nThe third weakness is in the experimental analysis, which does not help in clarifying the meaning of Theorem 4.2.\nSince results from Theorem 4.1 are known, I would like to suggest the authors to focus more on developing the intuition on Theorem 4.2, removing empirical results supporting the former one. ", "summary_of_the_paper": "The main goal of the paper is to show that optimizing a MaxEnt objective is equivalent to solve a robust problem with uncertainty on the dynamics. The authors provide a theorem which illustrate that solving MaxEnt with a modified reward is equivalent to solve a robust version of MaxEnt w.r.t. an uncertain dynamics. The authors provide some simple examples to show the soundness of their results, and they provide a numerical simulations on more complex problems to experimentally validate their claims.", "summary_of_the_review": "The authors claim to show a link between robust RL and MaxEnt RL, however, their main result is only about robust MaxEnt RL and MaxEnt RL(with modified reward). The relevance of the paper is, thus, difficult to state, since it is not clear how the result relates with any un-regularized formulation. The clarity of the exposition can also be improved, together with the experimental analysis which struggles to effectively shedding light on the theoretical results. For these reasons I recommend a weak reject.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0xPvIvEa8k", "reviewer": "Reviewer_qrWz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I find the paper very interesting and in general well-written. Proving that policies learned with entropy regularization are robust in some well-defined sense would be an important result in my opinion. Unfortunately, in the current form, some points are unclear to me (see. The entropy coefficient \\alpha in (1) seems to have disappeared in the proofs of Theorems 4.1 and 4.2. Is it related to the KKT multipliers?\n\nMoreover, I believe that the experimental results would have been stronger if the authors had included some experiments with deep RL algorithms based on stochastic policy gradients in contrast to TD3, which yields a deterministic policy. I guess that a stochastic policy would be more robust. How would it fare against the proposed approach?\n\nDetailed comments: \nThe paper should be checked for typos. \nIn the appendix, some references to equations or sections are incorrect, e.g.,\n\n-  page 14: Eq. A.2\n- Appendix A.3: Section 4.1 -> Section 4.3 \n\nIn page 13, it would be more rigorous to say that the negative entropy is the Fenchel dual of the log-sum-exp function. \n\nIn pages 14 and 16, the authors write \"there exists \\epislon \\ge 0 (the dual solution\")… \nCould the authors expand this explanation? Why the KKT multipliers don't appear in the relaxed objectives? \n\nIn page 16, \\Delta(s_{t+1}, s_t, a_t) -> \\Delta r(s_{t+1}, s_t, a_t) \n\nIn page 17, as time t -> at time t\n\n", "summary_of_the_paper": "The paper shows that maximum entropy RL can yield robust policy with respect to certain disturbances on the reward function and/or transition function.", "summary_of_the_review": "Well-written paper with interesting results related max entropy RL and robust RL. Some technical results should be clarified and the experimental results could be strengthened.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers thought this paper tackles an interesting question around whether MaxEnt RL already provides an important form of robustness. Such work helps us better understand the intersection between generalization, regularization and robustness. The reviewers had a number of comments, questions and clarifications and were generally satisfied with the detailed responses provided by the authors. There was some concern over the strength of the experiments and the authors also ran additional experiments. These addressed one reviewer’s concerns, though the other still thought the existing experiments were a bit too simple.
Decision: Accept (Poster)