If MaxEnt RL is the Answer, What is the Question?

1 INTRODUCTION . Reinforcement learning ( RL ) searches for a policy that maximizes the expected , cumulative reward . In fully observed Markov decision processes ( MDPs ) , this maximization always has a deterministic policy as a solution . Maximum entropy reinforcement learning ( MaxEnt RL ) is a modification of the RL objective that further adds an entropy term to the objective . This additional entropy term causes MaxEnt RL to seek policies that ( 1 ) are stochastic , and ( 2 ) have non-zero probability of sampling every action . MaxEnt RL can equivalently be viewed as probability matching between trajectories visited by the policy and a distribution defined by exponentiating the reward ( See Section 2 ) . MaxEnt RL has appealing connections to probabilistic inference ( Dayan & Hinton , 1997 ; Neumann et al. , 2011 ; Todorov , 2007 ; Kappen , 2005 ; Toussaint , 2009 ; Rawlik et al. , 2013 ; Theodorou et al. , 2010 ; Ziebart , 2010 ) , prompting a renewed interest in recent years ( Haarnoja et al. , 2018b ; Abdolmaleki et al. , 2018 ; Levine , 2018 ) . MaxEnt RL can also be viewed as using Thompson sampling ( Thompson , 1933 ) to collect trajectories , where the posterior belief is given by the exponentiated return . Empirically , MaxEnt RL algorithms achieve good performance on a number of simulated ( Haarnoja et al. , 2018b ) and real-world ( Haarnoja et al. , 2018a ; Singh et al. , 2019 ) control tasks , and can be more robust to perturbations ( Haarnoja et al. , 2018c ) . There is empirical evidence that behavior similar MaxEnt RL is used by animals in the natural world . While standard reinforcement learning is often used as a model for decision making ( Scott , 2004 ; Liu & Todorov , 2007 ; Todorov & Jordan , 2002 ) , many animals , including humans , do not consistently make decisions that maximize expected utility . Rather , they engage in probability matching , choosing actions with probability proportional to how much utility that action will provide . Examples include ants ( Lamb & Ollason , 1993 ) , bees ( Greggers & Menzel , 1993 ) , fish ( Bitterman et al. , 1958 ) , ducks ( Harper , 1982 ) , pigeons ( Bullock & Bitterman , 1962 ; Graf et al. , 1964 ) , and humans , where it has been documented so extensively that Vulkan ( 2000 ) wrote a survey of surveys of the field . This effect has been observed not just in individuals , but also in the collective behavior of groups of animals ( see Stephens & Krebs ( 1986 ) ) , where it is often described as obtaining the ideal free distribution . Probability matching is not merely a reflection of youth or ignorance . Empirically , more intelligent creatures are more likely to engage in probability matching . For example , in a comparison of Yale students and rats , Gallistel ( 1990 ) found that the students nearly always performed probability matching , while rats almost always chose the maximizing strategy . Similarly , older children and adults engage in probability matching more frequently than young children ( Stevenson & Odom , 1964 ; Weir , 1964 ) . While prior work has offered a number of explanations of probability matching ( Vulkan , 2000 ; Gaissmaier & Schooler , 2008 ; Wozny et al. , 2010 ; Sakai & Fukai , 2008 ) , its root cause remains an open problem . The empirical success of MaxEnt RL algorithms on RL problems is surprising , as MaxEnt RL optimizes a different objective than standard RL . The solution to every MaxEnt RL problem is stochastic , while deterministic policies can always be used to solve standard RL problems ( Puterman , 2014 ) . While RL can be motivated from the axioms of utility theory ( Russell & Norvig , 2016 ) , MaxEnt RL has no such fundamental motivation . It remains an open question as to whether the standard MaxEnt RL objective actually optimizes some well-defined notion of risk or regret that would account for its observed empirical benefits . This paper studies this problem , and aims to answer the following question : if MaxEnt RL is the solution , then what is the problem ? Answering this question is a first step towards understanding the empirical success of MaxEnt RL algorithms , and our analysis will suggest that MaxEnt RL might be applicable to problems typically considered to be much more complex than standard RL . In this paper , we show that MaxEnt RL provides the optimal control solution in settings with uncertainty and variability in the reward function . More precisely , we show that MaxEnt RL is equivalent to two more challenging problems : ( 1 ) regret minimization in a meta-POMDP , and ( 2 ) robust-reward control . The first setting , the meta-POMDP , is a partially observed MDP where the reward depends on an unobserved portion of the state , and where multiple episodes in the original MDP correspond to a single extended trial in the meta-POMDP . While seemingly Byzantine , this type of problem setting arises in a number of real-world settings discussed in Section 3 . Optimal policies for the meta-POMDP must explore at test-time , behavior that can not result from maximizing expected utility . In the second setting , robust-reward control , we consider an adversary that chooses some aspects of the reward function . Intuitively , we expect stochastic policies to be most robust because they are harder to exploit , as we formalize in Section 5 . Even if the agent will eventually be deployed in a setting without adversaries , the adversarial objective bounds the worst-case performance of that agent . Our result in this setting can be viewed as an extension of prior work connecting the principle of maximum entropy to two-player games ( Ziebart et al. , 2011 ; Grünwald et al. , 2004 ) . While both robust-reward control and regret minimization in a meta-POMDP are natural problems that arise in many real-world scenarios , neither is an expected utility maximization problem , so we can not expected optimal control to solve these problems . In contrast , we show that MaxEnt RL provides solutions to both . In summary , our analysis suggests that the empirical benefits of MaxEnt RL arise by implicitly solving control problems with variability in the reward . 2 PRELIMINARIES . We begin by defining notation and discussing some previous motivations for MaxEnt RL . An agent observes states st , takes actions at ∼ π ( at | st ) , and obtains rewards r ( st , at ) . The initial state is sampled s1 ∼ p1 ( s1 ) , and subsequent states are sampled s′ ∼ p ( s′ | s , a ) . Episodes have T steps , which we summarize as a trajectory τ , ( s1 , a1 , · · · , sT , aT ) . Without loss of generality , we can assume that rewards are undiscounted , as any discount can be addressed by modifying the dynamics to transition to an absorbing state with probability 1− γ . The RL objective is : arg max π Eπ [ T∑ t=1 r ( st , at ) ] = ∫ ( T∑ t=1 r ( st , at ) ) p1 ( s1 ) T∏ t=1 p ( st+1 | st , at ) π ( at | st ) dτ . In fully observed MDPs , there always exists a deterministic policy as a solution ( Puterman , 2014 ) . The MaxEnt RL problem , also known as the entropy-regularized control problem , is to maximize the sum of expected reward and conditional action entropy , Hπ [ a | s ] : arg max π Eπ [ T∑ t=1 r ( st , at ) ] +Hπ [ a | s ] = Eπ [ T∑ t=1 r ( st , at ) − log π ( at | st ) ] The MaxEnt RL objective results in policies that are stochastic , with higher-entropy action distributions in states where many different actions lead to similarly optimal rewards , and lower-entropy distributions in states where a single action is much better than the rest . Moreover , MaxEnt RL results in policies that have non-zero probability of sampling any action . MaxEnt RL can equivalently be defined as a form of probability matching , minimizing a reverse Kullback Leibler ( KL ) divergence ( Rawlik et al. , 2013 ) : max π Eπ [ T∑ t=1 r ( st , at ) ] +Hπ [ a | s ] = −min π DKL ( π ( τ ) ‖ pr ( τ ) ) , where the policy distribution π ( τ ) and the target distribution pr ( τ ) are defined as pr ( τ ) ∝ p1 ( s1 ) T∏ t=1 p ( st+1 | st , at ) e ∑T t=1 r ( st , at ) , π ( τ ) , p1 ( s1 ) T∏ t=1 p ( st+1 | st , at ) π ( at | st ) . Prior work on MaxEnt RL offers a slew of intuitive explanations for why one might prefer MaxEnt RL . We will summarize three common explanations and highlights problems with each . Exploration : MaxEnt RL is often motivated as performing good exploration . Unlike many other RL algorithms , such as DQN ( Mnih et al. , 2015 ) and DDPG ( Lillicrap et al. , 2015 ) , MaxEnt RL performs exploration and policy improvement with the same ( stochastic ) policy . One problem with this motivation is that stochastic policies can be obtained directly from standard RL , without adding an entropy term ( Heess et al. , 2015 ) . More troubling , while MaxEnt RL learns a stochastic policy , many MaxEnt RL papers evaluate the corresponding deterministic policy ( Haarnoja et al. , 2018b ) , suggesting that the stochastic policy is not what should be optimized . While improved exploration may be an ancillary benefit of MaxEnt RL , it remains unclear why we should expect MaxEnt RL to explore better than standard RL algorithms that learn stochastic policies . Probabilistic inference : Connections with probabilistic inference offer a second motivation for MaxEnt RL ( Abdolmaleki et al. , 2018 ; Haarnoja et al. , 2018b ; Todorov , 2007 ; Levine , 2018 ; Toussaint , 2009 ) . These approaches cast optimal control as an inference problem by defining additional optimality binary random variables Ot , equal one with probability proportional to exponentiated reward . These methods then maximize the following log-likelihood : log p ( Ot ) = log ∫ p1 ( s1 ) T∏ t=1 p ( Ot = 1 | st , at ) p ( st+1 | st , at ) π ( at | st ) dτ = log ∫ p1 ( s1 ) e ∑T t=1 r ( st , at ) T∏ t=1 p ( st+1 | st , at ) π ( at | st ) dτ = logEπ [ e ∑T t=1 r ( st , at ) ] ≈ Eπ [ T∑ t=1 r ( st , at ) ] + Varπ [ T∑ t=1 r ( st , at ) ] ( 1 ) The last term is a cumulant generating function ( i.e. , the logarithm of a moment generating function ( Gut , 2013 , Chpt . 6 ) ) , which can be approximated as the sum of expected reward and variance of returns ( Mihatsch & Neuneier , 2002 ) . Thus , directly maximizing likelihood leads to risk-seeking behavior , not optimal control . Equation 1 can also be directly obtained by considering an agent with a risk-seeking utility function ( O ’ Donoghue , 2018 ) . While risk seeking behavior can be avoided by maximizing a certain lower bound on Equation 1 ( Levine , 2018 ) , artificially constraining algorithms to maximize a lower bound suggests that likelihood is not what we actually want to maximize . Easier optimization : Finally , some prior work ( Ahmed et al. , 2018 ; Williams & Peng , 1991 ) argues that the entropy bonus added by MaxEnt RL makes the optimization landscape smoother . However , it does not suggest why optimizing the wrong but smooth problem yields a good solution to the original optimization problem .

Summary: This paper aims to theoretically understand the reason that MaxEnt RL (RL with an entropy bonus) works so well. It suggests that MaxEnt RL works well in the setting where there is uncertainty about the reward function. It proves two main theorems to support this. First, for any instance of a class of “meta-POMDPs” where the agent only has a *belief* over the goal trajectory, there exists a reward function for which MaxEnt RL on that reward function is the optimal solution to the meta-POMDP. Second, for any reward and MDP, there exists a set of reward functions such that MaxEnt RL maximizes the worst-case return for a reward chosen from that set.