On Inductive Biases in Deep Reinforcement Learning

Many deep reinforcement learning algorithms contain inductive biases that sculpt the agent ’ s objective and its interface to the environment . These inductive biases can take many forms , including domain knowledge and pretuned hyperparameters . In general , there is a trade-off between generality and performance when we use such biases . Stronger biases can lead to faster learning , but weaker biases can potentially lead to more general algorithms that work on a wider class of problems . This trade-off is relevant because these inductive biases are not free ; substantial effort may be required to obtain relevant domain knowledge or to tune hyper-parameters effectively . In this paper , we re-examine several domain-specific components that modify the agent ’ s objective and environmental interface . We investigated whether the performance deteriorates when all these fixed components are replaced with adaptive solutions from the literature . In our experiments , performance sometimes decreased with the adaptive components , as one might expect when comparing to components crafted for the domain , but sometimes the adaptive components performed better . We then investigated the main benefit of having fewer domain-specific components , by comparing the learning performance of the two systems on a different set of continuous control problems , without additional tuning of either system . As hypothesized , the system with adaptive components performed better on many of the tasks . The deep reinforcement learning ( RL ) community has demonstrated that well-tuned deep RL algorithms can master a wide range of challenging tasks . Human-level performance has been approached or surpassed on board-games such as Go and Chess ( Silver et al. , 2017 ) , video-games such as Atari ( Mnih et al. , 2015 ; Hessel et al. , 2018a ; Horgan et al. , 2018 ) , and custom 3D navigation tasks ( Johnson et al. , 2016 ; Kempka et al. , 2016 ; Beattie et al. , 2016 ; Mnih et al. , 2016 ; Jaderberg et al. , 2016 ; Espeholt et al. , 2018 ) . These results are a testament to the generality of the overall approach . At times , however , the excitement for the constant stream of new domains being mastered by suitable RL algorithms may have over-shadowed the dependency on inductive biases of these agents , and the amount of tuning that is often required for these to perform effectively in new domains . A clear example of the benefits of generality is the AlphaZero algorithm ( Silver et al. , 2017 ) . AlphaZero differs from the earlier AlphaGo algorithm ( Silver et al. , 2016 ) by removing all dependencies on Go-specific inductive biases and human data . After removing these biases , AlphaZero could not just surpass human performance on Go but also learn to play Chess and Shogi . In general , there is a trade off between generality and performance when we inject inductive biases into our algorithms . Inductive biases take many forms , including domain knowledge and pretuned learning parameters . If applied carefully , such biases can lead to faster and better learning . On the other hand , fewer biases can potentially lead to more general algorithms that work out of the box on a wider class of problems . Crucially , most inductive biases are not free : for instance , substantial effort can be required to attain the relevant domain knowledge or pretune parameters . This cost is often hidden—for instance , one might use hyperparameters established as good in prior work on the same domain , without knowing how much data or time was spent optimising these , nor how specific the settings are to the given domain . Systematic studies about the impact of different inductive biases are rare and the generality of these different biases is often unclear . Another consideration is that inductive biases may mask the generality of other parts of the system as a whole ; if a learning algorithm tuned for a specific domain does not generalize out of the box to a new domain , it can be unclear whether it is due to the having the wrong inductive biases or whether the underpinning learning algorithm is lacking something important . In this paper , we consider several commonly used domain heuristics , and investigate if ( and by how much ) performance deteriorates when we replace these with more general adaptive components , and we assess the impact of such replacements on the generality of the agent . We consider two broad ways of injecting inductive biases in RL agents : 1 ) sculpting the agent ’ s objective ( e.g. , clipping and discounting rewards ) , 2 ) sculpting the agent-environment interface ( e.g. , repeating each selected action a hard-coded fixed number of times , or crafting of the learning agent ’ s observation ) . Our contributions show that all the carefully crafted heuristics commonly used in Atari , and often considered essential for good performance on this domain , can be safely replaced with adaptive components , while preserving competitive performance across the benchmark . Furthermore , we show that this results in increased generality for an actor critic agent ; the resulting fully adaptive system can be applied with no additional tuning on a separate suite of continuous control tasks that have different environmental interfaces . Here the adaptive system achieved much higher performance than a comparable system using the Atari tuned heuristics , and also higher performance than an actor-critic agent that was tuned for this benchmark ( Tassa et al. , 2018 ) . 1 BACKGROUND . Reinforcement learning ( Sutton & Barto , 1998 ) is a framework for learning and decision making under uncertainty , where an agent interacts with its environment , receiving an observationOt , selecting an action At , and then receiving the next reward Rt+1 , for each discrete time t. Problem setting : The behaviour of an agent is formally specified by a policy π ( At|Ht ) : a probability distribution over actions conditional on all previous observations , i.e . on the agent ’ s history Ht = O1 : t. The agent ’ s objective is to find a policy that collects as much reward as possible , in each episode of experience . Crucially , it must learn such a policy without direct supervision , by trial and error . The amount of reward collected from time t onwards - the return - is a random variable Gt = Tend∑ k=0 γkRt+k+1 , ( 1 ) where Tend is the number of steps until episode termination and γ ∈ [ 0 , 1 ] is a constant discount factor ( e.g. , modelling monetary inflation , or a fixed additional probability of termination ) . The agent seeks an optimal policy , that maximizes values v ( Ht ) = Eπ [ Gt|Ht ] . In fully observable environments the optimal policy depends on the last observation alone : π∗ ( At|Ht ) = π∗ ( At|Ot ) . Otherwise , the history may be summarized in an agent state St = f ( Ht ) . The agent ’ s objective is then to jointly learn the state representation f and policy π ( At|St ) , so as to maximize values . The fully observable case is formalized as a Markov Decision Process ( Bellman , 1957 ) . Actor-critic algorithms : In the literature , two families of RL algorithms have particular prominence . Value-based algorithms learn to estimate values vw ( s ) that approximate to the true values , vw ( s ) ≈ vπ ( s ) ≡ Eπ [ Gt|St = s ] , under some policy π . Values can be learned efficiently by exploiting a recursive decomposition vπ ( s ) = E [ Rt+1 + γvπ ( St+1 ) |St = s ] , known as Bellman equation , used in in temporal difference learning ( Sutton , 1988 ) by sampling and making incremental updates : ∆wt = ( Rt+1 + γvw ( St+1 ) − vw ( St ) ) ∇wvw ( St ) . ( 2 ) In policy-based reinforcement learning algorithms , the policy πθ ( At|St ) is parameterized directly , and a stochastic gradient estimate can be derived to update the parameters in the direction of steepest ascent of the value ( Williams , 1992 ; Sutton et al. , 2000 ) : for instance , according to the update ∆θt = Gt∇ log πθ ( At|St ) . ( 3 ) Value-based and policy-based methods are combined in actor-critic algorithms . If a state value estimate is available , the policy updates can be computed from incomplete episodes by using the truncated returns G ( n ) t = ∑n−1 k=0 γ kRt+k+1 +γ nvw ( St ) that bootstrap on the value estimate at state St+n according to vw . This can reduce the variance of the updates . The variance can be further reduced by using state values as a baseline in policy updates , as in the advantage actor-critic update : ∆θt = ( G ( n ) t − vw ( St ) ) ∇θ log πθ ( At|St ) . ( 4 ) 2 INDUCTIVE BIASES AND LEARNED SOLUTIONS . We now describe a few commonly used heuristics within the Atari domain , together with the adaptive replacements that we investigated in our experiments . 2.1 SCULPTING THE AGENT ’ S OBJECTIVE . Many current deep RL agents do not directly optimize the true objective that they are evaluated against . In particular , RL agents in the Atari environments are typically tasked with optimizing a different handcrafted objective that incorporates biases to make learning simpler . We consider two additional ways of sculpting the agent ’ s objective that are still commonly used : reward clipping , and the use of fixed discounting of future rewards by a factor different from the one used for evaluation . In many deep RL algorithms , the magnitude of the updates scales linearly with the returns . This makes it difficult to train the same RL agent , with the same hyper-parameters , on multiple domains , because good settings for hyper-parameters such as the learning rate vary across tasks . One common solution is to clip the rewards to a fixed range ( Mnih et al. , 2015 ) , for instance [ −1 , 1 ] . This clipping makes the magnitude of returns and updates more comparable across domains . However , this also radically changes the agent objective , e.g. , if all non-zero rewards are larger than one , then this amounts to maximizing the frequency of positive rewards rather than their cumulative sums . This can simplify the learning problem , and , when it is a good proxy for the true objective , can result in good performance . In other tasks , however , clipping can result in sub-optimal policies because the objective that is optimized is ill-aligned with the true objective . PopArt ( van Hasselt et al. , 2016 ; Hessel et al. , 2018b ) was introduced as a principled solution to learn effectively irrespective of the magnitude of returns . PopArt works by tracking the mean µ and standard deviation σ of bootstrapped returns G ( n ) t . Temporal difference errors on value estimates can then be computed in a normalized space , with nw ( s ) denoting the normalized value , while the unnormalized values ( needed , for instance , for bootstrapping ) are recovered by a linear transformation vw ( s ) = µ+ σ ∗ nw ( s ) . Doing this naively increases the non-stationarity of learning since the unnormalized predictions for all states change every time we change the statistics . PopArt therefore combines the adaptive rescaling with an inverse transformation of the weights at the last layer of nw ( s ) , thereby preserving outputs precisely under any change in statistics µ → µ′ and σ → σ′ . This is done exactly by updating weights and biases as w′ = wσ/σ′ and b′ = ( σb+ µ− µ′ ) /σ′ . Discounting is part of the traditional MDP formulation of RL . As such , it is often considered a property of the problem rather than a tunable parameter on the agent side . Indeed , sometimes , the environment does define a natural discounting of future rewards ( e.g. , inflation in a financial setting ) . However , even in episodic settings where the agent should maximize the undiscounted return , a constant discount factor is often used to simplify the learning problem ( by having the agent focus on a relatively short time horizon ) . Optimizing this proxy of the true return often results in the agent achieving superior performance even in terms of the undiscounted return ( Machado et al. , 2017 ) . This benefit comes with the cost of adding a hyperparameter , and performance can be quite sensitive to it—i.e. , learning might be fast if the discount is small , but the solution may be too myopic . Instead of tuning the discount manually , we use meta-learning ( cf . Sutton , 1992 ; Bengio , 2000 ; Finn et al. , 2017 ; Xu et al. , 2018 ) to adapt the discount factor . The meta-gradient algorithm by Xu et al . ( 2018 ) uses the insight that the updates in Equations ( 2 ) and ( 4 ) are differentiable functions of hyper-parameters such as the discount . On the next sample or rollout of experience , using updated parameters w + ∆w ( γ ) , written here as an explicit function of the discount , the agent then applies a gradient based actor-critic update , not to parameters w , but to the parameter θ that defines the discount γ which is used in a standard learning update . Xu et al . ( 2018 ) demonstrated that this improved performance on Atari , while using a separate hand-tuned discount factor for the metaupdate . We instead use the undiscounted returns ( γm = 1 ) to define the meta-gradient updates , to understand if this technique can fully replace the need to reason about timescales and discounts . A related heuristic , quite specific to Atari , is to track the number of lives that the agent has available ( in several Atari games the agent is allowed to die a fixed number of times before the game is over ) , and hard code an episode termination ( γ = 0 ) when this happens . We ignore the number of lives channel exposed by the Arcade Learning Environment in all our experiments .

Summary: This paper contains various numerical experiments to see the effects of some heuristics in reinforcement learning. Those heuristics include reward clipping, discounting for effective learning, repeating actions, and different network structures. However, since the training algorithms also greatly affect the performance of RL agents, it seems hard to draw any quantitive conclusions from this paper.