MDP Playground: Controlling Orthogonal Dimensions of Hardness in Toy Environments

1 INTRODUCTION . RL has succeeded at many disparate tasks , such as helicopter aerobatics , game-playing and continuous control ( Abbeel et al. , 2010 ; Mnih et al. , 2015 ; Silver et al. , 2016 ; Chua et al. , 2018 ; Fujimoto et al. , 2018 ; Haarnoja et al. , 2018 ) , and yet a lot of the low-level workings of RL algorithms are not well understood . This is exacerbated by the absence of a unifying benchmark for all of RL . There are many different types of benchmarks , as many as there are kinds of tasks in RL ( e.g . Todorov et al. , 2012 ; Bellemare et al. , 2013 ; Cobbe et al. , 2019 ; Osband et al. , 2019 ) . They specialise in specific kinds of tasks . And yet the underlying assumptions are nearly always those of a Markov Decision Process ( MDP ) . We propose a benchmark which distills difficulties for MDPs that can be generalised across RL problems and allows to control these difficulties for more precise experiments . RL algorithms face a variety of challenges in environments . For example , when the underlying environment is an MDP , however the information state , i.e. , the state representation used by the agent is not Markov , we have partial observability , ( see , e.g. , Mnih et al. , 2015 ; Jaakkola et al. , 1995 ) . There are additional aspects of environments such as having irrelevant features , having multiple representations for the same state , and the action range , that significantly affect agent performance . We aim to study what kind of failure modes can occur when an agent is faced with such environments and to allow other researchers the benefit of the same platform to be able to create their own experiments and gain high-level insights . We identify dimensions of MDPs which help characterise environments , and we provide a platform with different instantiations of these dimensions in order to understand the workings of different RL algorithms better . To this end , we implemented a Python package , MDP Playground , that gives us complete control over these dimensions to generate flexible environments . Furthermore , commonly used environments , that RL algorithms are tested on , tend to take a long time to run . For example , a DQN run on Atari took us 4 CPU days and 64GB of memory to run . Our platform can be used as a low-cost testbed early in the RL agent development pipeline to develop and unit test algorithms and gain quick and coarse insights into algorithms . The main contributions of this paper are : • We identify dimensions of MDPs that have a significant effect on agent performance , both for discrete and continuous environments ; • We open-source a platform with fine-grained control over the dimensions ; baseline RL algorithms can be run on these in as little as 30 seconds on a single core of a laptop ; • We study the impact of the dimensions on baseline agents in our environments ; • We evaluate transfer of some of these dimensions to more complex environments . 2 DIMENSIONS OF HARDNESS . We begin by defining basic deterministic version of MDP , followed by a POMDP and then motivate the dimensions of hardness . We define an MDP as a 7-tuple ( S , A , P , R , ρo , γ , T ) , where S is the set of states , A is the set of actions , P : S×A×S → S describes the transition dynamics , R : S×A→ R describes the reward dynamics , ρo : S → R+ is the initial state distribution , γ is the discount factor and T is the set of terminal states . We define a POMDP as an 9-tuple ( S , O , A , P , Ω , R , ρo , γ , T ) , where O represents the set of observations , Ω : S × A × O → R describes the probability of an observation given a state and action and the rest of the terms have the same meaning as for the MDP above . To identify the dimensions of hardness , we went over the components of MDPs and POMDPs and tried to exhaustively list dimensions that could make an environment harder . This has resulted in many dimensions and a highly parameterisable platform . To aid in understanding we categorise the dimensions according to the component of ( PO ) MDPs they affect . To clarify terminology , we will use information state to mean the state representation used by the agent and belief state to be equivalent to the full observation history . If the belief state were to be used as the information state by an agent , it would be sufficient to compute an optimal policy . However , since the full observation history isn ’ t tractable to store for many environments , agents in practice stack the last few observations to use as their information state which renders it non-Markov . An implicit assumption for many agents is that we receive immediate reward depending on only the current information state and action . However , this is not true even for many simple environments . In many situations , agents receive delayed rewards ( see e.g . Arjona-Medina et al. , 2019 ) . For example , shooting at an enemy ship in Space Invaders leads to rewards much later than the action of shooting . Any action taken after that is inconsequential to obtaining the reward for destroying that enemy ship . In many environments , we obtain a reward for a sequence of actions taken and not just the information state and action . A simple example is executing a tennis serve , where we need a sequence of actions which would result in a point if we served an ace . In contrast to delayed rewards , rewarding a sequence of actions addresses the actions taken which are consequential to obtaining a reward . Sutton et al . ( 1999 ) present a framework for temporal abstraction in RL to deal with such sequences . Environments can also be characterised by their reward sparsity ( Gaina et al. , 2019 ) , i.e. , the supervisory reward signal is 0 throughout the trajectory and then a single non-zero reward is received at its end . This also holds true for our example of the tennis serve above . Another characteristic of environments that can significantly impact performance of algorithms is stochasticity . The environment , i.e. , dynamics P and R , may be stochastic or may seem stochastic to the agent due to partial observability or sensor noise . A robot equipped with a rangefinder , for example , has to deal with various sources of noise in its sensors ( Thrun et al. , 2005 ) . Environments also tend to have a lot of irrelevant features ( Rajendran et al. , 2018 ) that one need not focus on . For example , in certain racing car games , though one can see the whole screen , we only need to concentrate on the road and would be more memory efficient if we did . This holds for table-based learners and approximators like Neural Networks ( NNs ) . NNs additionally can even fit random noise ( Zhang et al. , 2017 ) and having irrelevant features is likely to degrade performance . Another aspect we want to motivate is that of representations . The same underlying state may have many different external representations/observations , for example , feature space vs pixel space . Mujoco tasks may be learnt in feature space vs directly from pixels , and Atari games can use the underlying RAM state or images . For images , various image transformations ( shift , scale , rotate , flip and others ; Hendrycks & Dietterich , 2019 ) may manifest as observations of the same underlying state and can pose a challenge to learning . Further , we identify several additional dimensions for continuous control problems . For instance , for the task of reaching a target , we have target radius ( see , e.g. , Klink et al. , 2019 ) , a measure of the distance from the target within which we consider the target was successfully reached , action range , the range of actions that may be taken , action loss weight , a weight penalising actions , time unit , the discretisation of time , and transition dynamics order , the order of P that is considered . Dimensions of hardness of environments that we identify from the above discussion are ( with the ( PO ) MDP component they impact in brackets ) : • Reward Delay ( R ) • Sequence Length ( R ) • Reward Sparsity ( R ) • Stochasticity ( P and R ) • Irrelevant Features ( O ) • Representations ( O ) • Action Range ( A ) • Time Unit ( P ) • Target Radius ( T ) • Dynamics Order ( P ) While we include only these for the main paper , Appendix A lists all dimensions that are controllable in MDP Playground . While many dimensions may seem a curse at first , it is also the nature of RL that different dimensions tend to be important in different specific applications . For instance , terminalstatecosts are very important for many environments , however for the continuous environments we considered here , they were absent and hence unimportant . So , we allow setting terminal state costs ( in addition to goal state rewards to distinguish good and bad terminal states ) to be able to define toy environments where this dimension will be important . Another example is of rewardscale . The agents we tested here rescale or clip rewards already and the effects of this dimension are not as important as they would be otherwise . The experiments here are only a glimpse into the power and flexibility of MDP Playground . Users can even upload custom P s and Rs and custom images for representations O and our platform will take care of injecting the other difficulties for them ( wherever possible ) . This allows users to control different dimensions in the same base environment and even gain insights we haven ’ t had so far in our extensive experiments . We now mathematically highlight some of our dimensions of hardness to aid understanding . The information state of an agent to compute an optimal policy would need to stack the previous n+ d observation and action pairs from the environment where n denotes a sequence length and d denotes a delay , i.e. , a sequence of actions needs to be followed to obtain a reward which may be delayed by a certain number of steps . Sparsity controls the number of elements in S that are rewardable . Additionally , the continuous control dimensions can mathematically be described as follows . The target radius sets T = { s | ‖s− st‖2 < target radius } , where st is the target point . The action range sets A to be equal to the Cartesian product of the ranges of the action dimensions where A ⊂ Rn . The action loss weight , λ , shapes rewards to be R ( s , a ) shaped = R ( s , a ) − λ ∗ ‖a‖2 . The time unit , t , sets P ( s , a ) = s+ ∫ t Pcont ( s , a ) dt where Pcont is the underlying continuous dynamics function . The transition dynamics order , n , sets P to be in Cn , the set of functions differentiable n times .

Summary: The paper describes a new benchmark for evaluating reinforcement learning techniques (MDP-playground). It can be seen as a toolbox allowing to generate different MDPs with different characteristics. Each MDP will then be used to probe a particular ability of learning algorithms, resulting in a comparison of methods over multiple dimensions. Proposed dimensions are reward sparsity, stochasticity, delayed reward, etc....  In addition to this toolbox, the authors also evaluate some of the classical algorithms in the domain.