Heuristic-Guided Reinforcement Learning

1 Introduction . Many recent empirical successes of reinforcement learning ( RL ) require solving problems with very long decision-making horizons . OpenAI Five [ 1 ] used episodes that were 20000 timesteps on average , while AlphaStar [ 2 ] used roughly 5000 timesteps . Long-term credit assignment is a very challenging statistical problem , with the sample complexity growing quadratically ( or worse ) with the horizon [ 3 ] . Long horizons ( or , equivalently , large discount factors ) also increase RL ’ s computational burden , leading to slow optimization convergence [ 4 ] . This makes RL algorithms require prohibitively large amounts of interactions and compute : even with tuned hyperparameters , AlphaStar needed over 108 samples and OpenAI Five needed over 107 PFLOPS of compute . A popular approach to mitigate the statistical and computational issues of tabula rasa RL methods is to warm-start or regularize learning with prior knowledge [ 1 , 2 , 5–10 ] . For instance , AlphaStar learned a policy and value function from human demonstrations and regularized the RL agent using imitation learning ( IL ) . AWAC [ 9 ] warm-started a policy using batch policy optimization on exploratory datasets . While these approaches have been effective in different domains , none of them explicitly address RL ’ s complexity dependence on horizon . In this paper , we propose a complementary regularization technique that relies on heuristic value functions , or heuristics1 for short , to effectively shorten the problem horizon faced by an online RL agent for fast learning . We call this approach Heuristic-Guided Reinforcement Learning ( HuRL ) . The core idea is simple : given a Markov decision process ( MDP ) M = ( S , A , P , r , γ ) and a heuristic h : S → R , we select a mixing coefficient λ ∈ [ 0 , 1 ] and have the agent solve a new MDP M̃ = ( S , A , P , r̃ , γ̃ ) with a reshaped reward and a smaller discount ( i.e . a shorter horizon ) : r̃ ( s , a ) : = r ( s , a ) + ( 1− λ ) γEs′∼P ( ·|s , a ) [ h ( s′ ) ] and γ̃ : = λγ . ( 1 ) 1We borrow this terminology from the planning literature to refer to guesses of V ∗ in an MDP [ 11 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . HuRL effectively introduces horizon-based regularization that determines whether long-term value information should come from collected experiences or the heuristic . By modulating the effective horizon via λ , we trade off the bias and the complexity of solving the reshaped MDP . HuRL with λ = 1 recovers the original problem and with λ = 0 creates an easier contextual bandit problem [ 12 ] . A heuristic h in HuRL represents a prior guess of the desired long-term return of states , which ideally is the optimal value function V ∗ of the unknown MDPM . When the heuristic h captures the state ordering of V ∗ well , conceptually , it becomes possible to make good long-term decisions by short-horizon planning or even acting greedily . How do we construct a good heuristic ? In the planning literature , this is typically achieved by solving a relaxation of the original problem [ 13– 15 ] . Alternatively , one can learn it from batch data collected by exploratory behavioral policies ( as in offline RL [ 16 ] ) or from expert policies ( as in IL [ 17 ] ) .2 For some dense reward problems , a zero heuristic can be effective in reducing RL complexity , as exploited by the guidance discount framework [ 18–23 ] . In this paper , we view heuristics as a unified representation of various forms of prior knowledge , such as expert demonstrations , exploratory datasets , and engineered guidance . Although the use of heuristics to accelerate search has been popular in planning and control algorithms , e.g. , A * [ 24 ] , MCTS [ 25 ] , and MPC [ 7 , 26–28 ] , its theory is less developed for settings where the MDP is unknown . The closest work in RL is potential-based reward shaping ( PBRS ) [ 29 ] , which reshapes the reward into r̄ ( s , a ) = r ( s , a ) +γEs′|s , a [ h ( s′ ) ] −h ( s ) while keeping the original discount . PBRS can use any heuristic to reshape the reward while preserving the ordering of policies . However , giving PBRS rewards to an RL algorithm does not necessarily lead to faster learning , because the base RL algorithm would still seek to explore to resolve long-term credit assignment . HuRL allows common RL algorithms to leverage the short-horizon potential provided by a heuristic to learn faster . In this work , we provide a theoretical foundation of HuRL to enable adopting heuristics and horizon reduction for accelerating RL , combining advances from the PBRS and the guidance discount literatures . On the theoretical side , we derive a bias-variance decomposition of HuRL ’ s horizon-based regularization in order to characterize the solution quality as a function of λ and h. Using this insight , we provide sufficient conditions for achieving an effective trade-off , including properties required of a base RL algorithm that solves the reshaped MDP M̃λ . Furthermore , we define the novel concept of an improvable heuristic and prove that good heuristics for HuRL can be constructed from data using existing pessimistic offline RL algorithms ( such as pessimistic value iteration [ 30 , 31 ] ) . The effectiveness of HuRL depends on the heuristic quality , so we design HuRL to employ a sequence of mixing coefficients ( i.e . λs ) that increases as the agent gathers more data from the environment . Such a strategy induces a learning curriculum that enables HuRL to remain robust to non-ideal heuristics . HuRL starts off by guiding the agent ’ s search direction with a heuristic . As the agent becomes more experienced , it gradually removes the guidance and lets the agent directly optimize the true long-term return . We empirically validate HuRL in MuJoCo [ 32 ] robotics control problems and Procgen games [ 33 ] with various heuristics and base RL algorithms . The experimental results demonstrate the versatility and effectiveness of HuRL in accelerating RL algorithms . 2 Preliminaries . 2.1 Notation . We focus on discounted infinite-horizon Markov Decision Processes ( MDPs ) for ease of exposition . The technique proposed here can be extended to other MDP settings.3 A discounted infinite-horizon MDP is denoted as a 5-tupleM = ( S , A , P , r , γ ) , where S is the state space , A is the action space , P ( s′|s , a ) is the transition dynamics , r ( s , a ) is the reward function , and γ ∈ [ 0 , 1 ) is the discount factor . Without loss of generality , we assume r : S×A → [ 0 , 1 ] . We allow the state and action spaces S and A to be either discrete or continuous . Let ∆ ( · ) denote the space of probability distributions . A decision-making policy π is a conditional distribution π : S → ∆ ( A ) , which can be deterministic . We define some shorthand for writing expectations : For a state distribution d ∈ ∆ ( S ) and a function V : S → R , we define V ( d ) : = Es∼d [ V ( s ) ] ; similarly , for a policy π and a function Q : S ×A → R , we define Q ( s , π ) : = Ea∼π ( ·|s ) [ Q ( s , a ) ] . Lastly , we define Es′|s , a : = Es′∼P ( ·|s , a ) . 2We consider the RL setting for imitation where we suppose the rewards of expert trajectories are available . 3The results here can be readily applied to finite-horizon MDPs ; for other infinite-horizon MDPs , we need further , e.g. , mixing assumptions for limits to exist . Central to solving MDPs are the concepts of value functions and average distributions . For a policy π , we define its state value function V π as V π ( s ) : = Eρπs [ ∑∞ t=0 γ tr ( st , at ) ] , where ρπs denotes the trajectory distribution of s0 , a0 , s1 , . . . induced by running π starting from s0 = s. We define the state-action value function ( or the Q-function ) as Qπ ( s , a ) : = r ( s , a ) + γEs′|s , a [ V π ( s′ ) ] . We denote the optimal policy as π∗ and its state value function as V ∗ : = V π ∗ . Under the assumption that rewards are in [ 0 , 1 ] , we have V π ( s ) , Qπ ( s , a ) ∈ [ 0 , 11−γ ] for all π , s ∈ S , and a ∈ A . We denote the initial state distribution of interest as d0 ∈ ∆ ( S ) and the state distribution of policy π at time t as dπt , with d π 0 = d0 . Given d0 , we define the average state distribution of a policy π as dπ : = ( 1− γ ) ∑∞ t=0 γ tdπt . With a slight abuse of notation , we also write d π ( s , a ) : = dπ ( s ) π ( a|s ) . 2.2 Setup : Reinforcement Learning with Heuristics . We consider RL with prior knowledge expressed in the form of a heuristic value function . The goal is to find a policy π that has high return through interactions with an unknown MDPM , i.e. , maxπ V π ( d0 ) . While the agent here does not fully knowM , we suppose that , before interactions start the agent is provided with a heuristic h : S → R which the agent can query throughout learning . The heuristic h represents a prior guess of the optimal value function V ∗ ofM . Common sources of heuristics are domain knowledge as typically employed in planning , and logged data collected by exploratory or by expert behavioral policies . In the latter , a heuristic guess of V ∗ can be computed from the data by offline RL algorithms . For instance , when we have trajectories of an expert behavioral policy , Monte-Carlo regression estimate of the observed returns may be a good guess of V ∗ . Using heuristics to solve MDP problems has been popular in planning and control , but its usage is rather limited in RL . The closest provable technique in RL is PBRS [ 29 ] , where the reward is modified into r ( s , a ) : = r ( s , a ) + γEs′|s , a [ h ( s′ ) ] − h ( s ) . It can be shown that this transformation does not introduce bias into the policy ordering , and therefore solving the new MDPM : = ( S , A , P , r , γ ) would yield the same optimal policy π∗ ofM . Conceptually when the heuristic is the optimal value function h = V ∗ , the agent should be able to find the optimal policy π∗ ofM by acting myopically , as V ∗ already contains all necessary long-term information for good decision making . However , running an RL algorithm with the PBRS reward ( i.e . solvingM : = ( S , A , P , r , γ ) ) does not take advantage of this shortcut . To make learning efficient , we need to also let the base RL algorithm know that acting greedily ( i.e. , using a smaller discount ) with the shaped reward can yield good policies . An intuitive idea is to run the RL algorithm to maximize V π λ ( d0 ) , where V π λ denotes the value function of π in an MDPMλ : = ( S , A , P , r , λγ ) for some λ ∈ [ 0 , 1 ] . However this does not always work . For example , when λ = 0 , maxπ V π λ ( d0 ) only optimizes for the initial states d0 , but obviously the agent is going to encounter other states in M. We next propose a provably correct version , HuRL , to leverage this short-horizon insight .

Summary: This work proposes a framework for incorporating heuristics into reinforcement learning. The main idea is to construct a new MDP where the new reward function is offset by a scaled heuristic and the new temporal discount factor is scaled down. The scaling of the heuristic and temporal discount factor are controlled by a mixing coefficient $\lambda \in [0, 1]$. The proposal is to run an RL algorithm in this new MDP with the mixing coefficient adjusted over time (HuRL). The main theoretical result relates performance in the new MDP to performance in the original MDP (Theorem 4.1). Empirical results in MuJoCo and Procgen with PPO and SAC show that HuRL improves on several baselines.