Heuristic-Guided Reinforcement Learning

1 Introduction . Many recent empirical successes of reinforcement learning ( RL ) require solving problems with very long decision-making horizons . OpenAI Five [ 1 ] used episodes that were 20000 timesteps on average , while AlphaStar [ 2 ] used roughly 5000 timesteps . Long-term credit assignment is a very challenging statistical problem , with the sample complexity growing quadratically ( or worse ) with the horizon [ 3 ] . Long horizons ( or , equivalently , large discount factors ) also increase RL ’ s computational burden , leading to slow optimization convergence [ 4 ] . This makes RL algorithms require prohibitively large amounts of interactions and compute : even with tuned hyperparameters , AlphaStar needed over 108 samples and OpenAI Five needed over 107 PFLOPS of compute . A popular approach to mitigate the statistical and computational issues of tabula rasa RL methods is to warm-start or regularize learning with prior knowledge [ 1 , 2 , 5–10 ] . For instance , AlphaStar learned a policy and value function from human demonstrations and regularized the RL agent using imitation learning ( IL ) . AWAC [ 9 ] warm-started a policy using batch policy optimization on exploratory datasets . While these approaches have been effective in different domains , none of them explicitly address RL ’ s complexity dependence on horizon . In this paper , we propose a complementary regularization technique that relies on heuristic value functions , or heuristics1 for short , to effectively shorten the problem horizon faced by an online RL agent for fast learning . We call this approach Heuristic-Guided Reinforcement Learning ( HuRL ) . The core idea is simple : given a Markov decision process ( MDP ) M = ( S , A , P , r , γ ) and a heuristic h : S → R , we select a mixing coefficient λ ∈ [ 0 , 1 ] and have the agent solve a new MDP M̃ = ( S , A , P , r̃ , γ̃ ) with a reshaped reward and a smaller discount ( i.e . a shorter horizon ) : r̃ ( s , a ) : = r ( s , a ) + ( 1− λ ) γEs′∼P ( ·|s , a ) [ h ( s′ ) ] and γ̃ : = λγ . ( 1 ) 1We borrow this terminology from the planning literature to refer to guesses of V ∗ in an MDP [ 11 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . HuRL effectively introduces horizon-based regularization that determines whether long-term value information should come from collected experiences or the heuristic . By modulating the effective horizon via λ , we trade off the bias and the complexity of solving the reshaped MDP . HuRL with λ = 1 recovers the original problem and with λ = 0 creates an easier contextual bandit problem [ 12 ] . A heuristic h in HuRL represents a prior guess of the desired long-term return of states , which ideally is the optimal value function V ∗ of the unknown MDPM . When the heuristic h captures the state ordering of V ∗ well , conceptually , it becomes possible to make good long-term decisions by short-horizon planning or even acting greedily . How do we construct a good heuristic ? In the planning literature , this is typically achieved by solving a relaxation of the original problem [ 13– 15 ] . Alternatively , one can learn it from batch data collected by exploratory behavioral policies ( as in offline RL [ 16 ] ) or from expert policies ( as in IL [ 17 ] ) .2 For some dense reward problems , a zero heuristic can be effective in reducing RL complexity , as exploited by the guidance discount framework [ 18–23 ] . In this paper , we view heuristics as a unified representation of various forms of prior knowledge , such as expert demonstrations , exploratory datasets , and engineered guidance . Although the use of heuristics to accelerate search has been popular in planning and control algorithms , e.g. , A * [ 24 ] , MCTS [ 25 ] , and MPC [ 7 , 26–28 ] , its theory is less developed for settings where the MDP is unknown . The closest work in RL is potential-based reward shaping ( PBRS ) [ 29 ] , which reshapes the reward into r̄ ( s , a ) = r ( s , a ) +γEs′|s , a [ h ( s′ ) ] −h ( s ) while keeping the original discount . PBRS can use any heuristic to reshape the reward while preserving the ordering of policies . However , giving PBRS rewards to an RL algorithm does not necessarily lead to faster learning , because the base RL algorithm would still seek to explore to resolve long-term credit assignment . HuRL allows common RL algorithms to leverage the short-horizon potential provided by a heuristic to learn faster . In this work , we provide a theoretical foundation of HuRL to enable adopting heuristics and horizon reduction for accelerating RL , combining advances from the PBRS and the guidance discount literatures . On the theoretical side , we derive a bias-variance decomposition of HuRL ’ s horizon-based regularization in order to characterize the solution quality as a function of λ and h. Using this insight , we provide sufficient conditions for achieving an effective trade-off , including properties required of a base RL algorithm that solves the reshaped MDP M̃λ . Furthermore , we define the novel concept of an improvable heuristic and prove that good heuristics for HuRL can be constructed from data using existing pessimistic offline RL algorithms ( such as pessimistic value iteration [ 30 , 31 ] ) . The effectiveness of HuRL depends on the heuristic quality , so we design HuRL to employ a sequence of mixing coefficients ( i.e . λs ) that increases as the agent gathers more data from the environment . Such a strategy induces a learning curriculum that enables HuRL to remain robust to non-ideal heuristics . HuRL starts off by guiding the agent ’ s search direction with a heuristic . As the agent becomes more experienced , it gradually removes the guidance and lets the agent directly optimize the true long-term return . We empirically validate HuRL in MuJoCo [ 32 ] robotics control problems and Procgen games [ 33 ] with various heuristics and base RL algorithms . The experimental results demonstrate the versatility and effectiveness of HuRL in accelerating RL algorithms . 2 Preliminaries . 2.1 Notation . We focus on discounted infinite-horizon Markov Decision Processes ( MDPs ) for ease of exposition . The technique proposed here can be extended to other MDP settings.3 A discounted infinite-horizon MDP is denoted as a 5-tupleM = ( S , A , P , r , γ ) , where S is the state space , A is the action space , P ( s′|s , a ) is the transition dynamics , r ( s , a ) is the reward function , and γ ∈ [ 0 , 1 ) is the discount factor . Without loss of generality , we assume r : S×A → [ 0 , 1 ] . We allow the state and action spaces S and A to be either discrete or continuous . Let ∆ ( · ) denote the space of probability distributions . A decision-making policy π is a conditional distribution π : S → ∆ ( A ) , which can be deterministic . We define some shorthand for writing expectations : For a state distribution d ∈ ∆ ( S ) and a function V : S → R , we define V ( d ) : = Es∼d [ V ( s ) ] ; similarly , for a policy π and a function Q : S ×A → R , we define Q ( s , π ) : = Ea∼π ( ·|s ) [ Q ( s , a ) ] . Lastly , we define Es′|s , a : = Es′∼P ( ·|s , a ) . 2We consider the RL setting for imitation where we suppose the rewards of expert trajectories are available . 3The results here can be readily applied to finite-horizon MDPs ; for other infinite-horizon MDPs , we need further , e.g. , mixing assumptions for limits to exist . Central to solving MDPs are the concepts of value functions and average distributions . For a policy π , we define its state value function V π as V π ( s ) : = Eρπs [ ∑∞ t=0 γ tr ( st , at ) ] , where ρπs denotes the trajectory distribution of s0 , a0 , s1 , . . . induced by running π starting from s0 = s. We define the state-action value function ( or the Q-function ) as Qπ ( s , a ) : = r ( s , a ) + γEs′|s , a [ V π ( s′ ) ] . We denote the optimal policy as π∗ and its state value function as V ∗ : = V π ∗ . Under the assumption that rewards are in [ 0 , 1 ] , we have V π ( s ) , Qπ ( s , a ) ∈ [ 0 , 11−γ ] for all π , s ∈ S , and a ∈ A . We denote the initial state distribution of interest as d0 ∈ ∆ ( S ) and the state distribution of policy π at time t as dπt , with d π 0 = d0 . Given d0 , we define the average state distribution of a policy π as dπ : = ( 1− γ ) ∑∞ t=0 γ tdπt . With a slight abuse of notation , we also write d π ( s , a ) : = dπ ( s ) π ( a|s ) . 2.2 Setup : Reinforcement Learning with Heuristics . We consider RL with prior knowledge expressed in the form of a heuristic value function . The goal is to find a policy π that has high return through interactions with an unknown MDPM , i.e. , maxπ V π ( d0 ) . While the agent here does not fully knowM , we suppose that , before interactions start the agent is provided with a heuristic h : S → R which the agent can query throughout learning . The heuristic h represents a prior guess of the optimal value function V ∗ ofM . Common sources of heuristics are domain knowledge as typically employed in planning , and logged data collected by exploratory or by expert behavioral policies . In the latter , a heuristic guess of V ∗ can be computed from the data by offline RL algorithms . For instance , when we have trajectories of an expert behavioral policy , Monte-Carlo regression estimate of the observed returns may be a good guess of V ∗ . Using heuristics to solve MDP problems has been popular in planning and control , but its usage is rather limited in RL . The closest provable technique in RL is PBRS [ 29 ] , where the reward is modified into r ( s , a ) : = r ( s , a ) + γEs′|s , a [ h ( s′ ) ] − h ( s ) . It can be shown that this transformation does not introduce bias into the policy ordering , and therefore solving the new MDPM : = ( S , A , P , r , γ ) would yield the same optimal policy π∗ ofM . Conceptually when the heuristic is the optimal value function h = V ∗ , the agent should be able to find the optimal policy π∗ ofM by acting myopically , as V ∗ already contains all necessary long-term information for good decision making . However , running an RL algorithm with the PBRS reward ( i.e . solvingM : = ( S , A , P , r , γ ) ) does not take advantage of this shortcut . To make learning efficient , we need to also let the base RL algorithm know that acting greedily ( i.e. , using a smaller discount ) with the shaped reward can yield good policies . An intuitive idea is to run the RL algorithm to maximize V π λ ( d0 ) , where V π λ denotes the value function of π in an MDPMλ : = ( S , A , P , r , λγ ) for some λ ∈ [ 0 , 1 ] . However this does not always work . For example , when λ = 0 , maxπ V π λ ( d0 ) only optimizes for the initial states d0 , but obviously the agent is going to encounter other states in M. We next propose a provably correct version , HuRL , to leverage this short-horizon insight .

Summary: The paper aims to solve/mitigate the long term credit assignment problem in reinforcement learning. It does so by pre-training heuristic value functions and using them to shorten the horizon. The claim is that the proposed approach could potentially improve sample complexity, like other warm-start or regularization methods in RL.