MDP Playground: Controlling Orthogonal Dimensions of Hardness in Toy Environments
Authors: Raghu Rajan
Jessica Lizeth Borja Diaz
Suresh Guttikonda
Fabio Ferreira
André Biedenkapp
Frank Hutter
Venue:  (2021)
Keywords: Reinforcement learning
Benchmarks
Efficiency
Reproducibility
Core issues
Algorithm analysis
Dimensions of hardness
OpenAI Gym

Abstract: We present MDP Playground, an efficient benchmark for Reinforcement Learning (RL) algorithms with various dimensions of hardness that can be controlled independently to challenge algorithms in different ways and to obtain varying degrees of hardness in generated environments. We consider and allow control over a wide variety of key hardness dimensions, including delayed rewards, rewardable sequences, sparsity of rewards, stochasticity, image representations, irrelevant features, time unit, and action max. While it is very time consuming to run RL algorithms on standard benchmarks, we define a parameterised collection of fast-to-run toy benchmarks in OpenAI Gym by varying these dimensions. Despite their toy nature and low compute requirements, we show that these benchmarks present substantial challenges to current RL algorithms. Furthermore, since we can generate environments with a desired value for each of the dimensions, in addition to having fine-grained control over the environments' hardness, we also have the ground truth available for evaluating algorithms. Finally, we evaluate the kinds of transfer for these dimensions that may be expected from our benchmarks to more complex benchmarks. We believe that MDP Playground is a valuable testbed for researchers designing new, adaptive and intelligent RL algorithms and those wanting to unit test their algorithms.


Under review as a conference paper at ICLR 2021
MDP PLAYGROUND: CONTROLLING ORTHOGONAL DIMENSIONS OF HARDNESS IN TOY ENVIRONMENTS
Anonymous authors Paper under double-bling review
ABSTRACT
We present MDP Playground, efficient benchmark for Reinforcement Learning (RI.) algorithms with various dimensions of hardness that can be controlled inde- pendently to challenge algorithms in different ways and obtain varying degrees of hardness in generated environments. We consider and allow control over a wide variety of key hardness dimensions, including delayed rewards, rewardable sequences, sparsity of rewards, stochasticity, image representations, irrelevant features, time unit, and action range. While it is very time consuming to run RL algorithms standard benchmarks, we define parameterised collection of fast-to-run toy benchmarks OpenAl Gym by varying these dimensions. Despite their toy nature and low compute requirements, present substantial challenges to current RL algorithms. Furthermore, since we can generate environments with desired value for each of the dimensions, in addition to having fine- aieeggrined control over the environments' hardness, we also have the ground truth available for evaluating algorithms. Finally, evaluate the kinds of transfer for these dimensions that may be expected from our benchmarks to more complex benchmarks. We believe that MDP Playground is a valuable testbed for researchers designing new, adaptive and intelligent RL algorithms and those wanting to unit test their algorithms.
show that these benchmarks
INTRODUCTION
RL has succeeded at many disparate tasks, such as helicopter aerobatics, game- play-ng and continuous
control (Abbeel et al. 2010 Mnih al. (0155] Silver et al. 2016 Chua et al. 2018 Fujimoto et al., 2018 Haarnoja et al., 2018), , and yet a lot of the low-level workings of RL algorithms are not well understood. This is exacerbated by the absence of unifying benchmark for all of RL. There many different types of benchmarks, as many as there are kinds tasks in RL (e.g. Todorov et al.,
2012 Bellemare et al., 2013 Cobbe et al., 2019 Osband al. 2019). They specialise in specific kinds of tasks. And yet the underlying assumptions are nearly always those of a Markov Decision Process (MDP). We propose benchmark which distills difficulties for MDPs that can be generalised across RL problems and allows to control these difficulties for more precise experiments.
RL algorithms face a variety of challenges in environments. For example, when the underlying environment is an MDP, however the information state, i.c., the state representation used by the agent is not Markov, we have partial observability, (see, e-g.. Mnih et al., 2015] Jaakkola al. 1995). There are additional aspects of environments such as having irrelevant features, having multiple representations for the same state, and the action range, that significantly affect agent performance. We aim to study what kind of failure modes can occur when an agent is faced with such environments and allow other researchers the benefit of the same platform to be able to create their own experiments and gain high-level insights.
We identify dimensions of MDPs which help characterise environments, and we provide a platform with different instantiations of these dimensions in order to understand the workings of different RL algorithms better. To this end, we implemented Python package, MDP Playground, that gives us complete control over these dimensions generate flexible environments.
Furthermore, commonly used environments, that RL algorithms are tested on, tend to take a long time to For example, DQN on Atari took CPU days and 64GB of memory run. Our

---

Under review as a conference paper at ICLR 2021
platform can be used as low-cost testbed early in the RL agent development pipeline to develop and unit test algorithms and gain quick and coarse insights into algorithms.
The main contributions of this paper
We identify dimensions of MDPs that have a significant effect on agent performance, both for discrete and continuous environments:
We open-source platform with fine-grained control over the dimensions; baseline RL algorithms can be run on these in as little as 30 seconds on a single core of laptop: baseline agents in our environments; We evaluate transfer of some of these dimensions to more complex environments.
We study the impact of the dimensions
DIMENSIONS OF HARDNESS
We begin by defining basic deterministic version of MDP, followed by POMDP and then motivate the dimensions of hardness. We define an MDP as a 7-tuple (S, A, P, R. Po. T), where S is the set of states, A is the set of actions, P: 8 A > S describes the transition dynamics, R S' A - TR describes the reward dynamics, Po : S - R + is the initial state distribution, 7 is the discount factor and 7' is the set of terminal states. We define POMDP an 9-tuple (S, O, A, P. S2 R. Po: 7, T), where O represents the set of observations, Ω S x A x O + R describes the probability of an observation given state and action and the rest of the terms have the same meaning as for the MDP above.
To identify the dimensions of hardness, we went over the components of MDPs and POMDPs and tried to exhaustively list dimensions that could make an environment harder. This has resulted in many dimensions and a highly parameterisable platform. To aid in understanding we categorise the dimensions according to the component of (PO)MDPs they affect.
To clarify terminology, we will use information state to mean the state representation used by the agent and belief state to be equivalent to the full observation history. If the belief state were to be used as the information state by an agent, it would be sufficient to compute an optimal policy. However, since the full observation history isn't tt tractable to store for many environments, agents in practice stack the last few observations to use as their information state which renders it non-Markov.
An implicit assumption for many agents is that receive immediate reward depending only the current information state and action. However, this not true even for many simple environments. In many situations, agents receive delayed rewardy (see e.g. Arjona-Medina et al. 2019). For example, shooting at an enemy ship in Space Invaders leads to rewards much later than the action of shooting. Any action taken after that is inconsequential to obtaining the reward for destroying that enemy ship.
In many environments, we obtain a reward for sequence of actions taken and not just the information state and action. A simple example is executing a tennis serve, where we need a sequence of actions which would result in point if we served an ace. In contrast to delayed rewards, rewarding a sequence of actions addresses the actions taken which are consequential to obtaining reward. [Sutton] et al. (9999) present framework for temporal abstraction in RL to deal with such sequences.
Environments can also be characterised by their reward sparsity (Gaina al. 2019). i.c., the supervisory reward signal is 0 throughout the trajectory and then single non-zero reward is received at its end. This also holds true for our example of the tennis serve above.
Another characteristic of environments that can significantly impact performance of algorithms is stochasticity. The environment, i.e., dynamics P and Rt may be stochastic or may seem stochastic to the agent due to partial observability sensor noise. A robot equipped with rangefinder, for example, has to deal with various sources of noise in its sensors (Thrun et al. 2005).
Environments also tend to have lot of irrelevant features (Rajendran et al., 2018) that one need not focus on. For example, in certain racing car games, though one can see the whole screen, we only need to concentrate table-based learners and approximators like Neural Networks (NNs). NNs additionally can even fit random noise (Zhang et al. 2017) and having irrelevant features is likely to degrade performance.
the road and would be more memory efficient if we did. This holds for
Another aspect we want to motivate is that of representattons. The same underlying state may have many different external representations/oosrraaaiii for xxamleeffattue space vs pixel space.
2

---

Under review as a conference paper at ICLR 2021
Mujoco tasks may be learnt in feature space vs directly from pixels, and Atari games the underlying RAM state or images. For images, various image transformations (shift, scale, rotate. flip and others; Hendrycks & Dietterich 2019) may manifest as observations of the same underlying state and can pose a challenge to learning.
Further, we identify several additional dimensions for continuous control problems. For instance, for the task of reaching a target, we have target radius (see, e.g.. Klink et al. 2019), measure of the distance from the target within which we consider the target was successfully reached, action range, the range of actions that may be taken, action loss weight, a weight penalising actions, time unit, the discretisation of time, and transition dynamics order, the order of P that considered.
Dimensions of hardness of
environments that we identify from the above are (with the (PO)MDP component they impact in brackets): Representations (O)
discussion
Reward Delay (R)
Sequence Length (R)
Action Range (AA)
Reward Sparsity (R)
Time Unit (P)
Stochasticity (P and Rt)
Target Radius (T)
Irrelevant Features (0)
Dynamics Order (P)
While we include only these for the main paper, Appendix A lists all dimensions that are controllable in MDP Playground. While many dimensions may seem curse first, it is also the nature of RL that different dimensions tend to be important in different specific applications. For instance, perminalstalecos are very important for many environments, however for the continuous environ- ments we considered here, they were absent and hence unimportant. So, allow setting terminal state costs (in addition to goal state rewards to distinguish good and bad terminal states) to be able to define toy environments where this dimension will be important. Another example is of cwwar.scae.. The agents we tested here rescale or clip rewards already and the effects this dimension are not as important as they would be otherwise. The experiments here only glimpse into the power and flexibility of MDP Playground. Users can even upload custom Ps and Rss and custom images for representations O and our platform will take care of injecting the other difficulties for them (wherever possible). This allows users to control different dimensions in the same base environment and even gain insights we haven't 't had so far in our extensive experiments.
We now mathematically highlight some of our dimensions of hardness to aid understanding. The information state of an agent to compute an optimal policy would need stack the previous n n- observation and action pairs from the environment where n denotes a sequence length and d denotes delay, i.e., sequence of actions needs to be followed to obtain a reward which may be delayed by certain number of steps. Sparsity controls the number of elements in S that are rewardable.
Additionally, the continuous control dimensions can mathematically be described as follows. The target radius sets 7 = {s ||s-=ell2 < target radius). where 8, is the target point. The action range sets to be equal to the Cartesian product of the ranges the action dimensions where C IR.". The action loss weight, X, shapes rewards to be R[(s, a) shopedd
R[s a) À * ||a]|2. The time unit, t, sets P(s. a) = s J Peont (s, a) dt where Peont is the underlying continuous dynamics function. The transition dynamics order, sets P to be in C". the set of functions differentiable times.
3 MDP PLAYGROUND
MDP Playground generates parameterised OpenAl Gym (Brockman et al.) 2016) environments to facilitate researchers to benchmark algorithms across dimensions hardness. also easily configurable, so that the environment difficulty can be controlled in fine-grained manner. We now briefly describe the discrete and continuous environments followed by implementation details of selected dimensions.
Discrete Environments In the discrete case, and A contain categorical elements, and we generate random instantiations of P and R after the remaining dimensions have been set. The generated P and R are deterministic and held fixed for the environment. We keep Po to uniform over the non-terminal states, and T is fixed be subset of S based user-chosen terminal state densiry.
Continuous Environments In the continuous case, environments correspond to the simplest real world task we could think of that would be worth solving, moving rigid body to target point,
3

---

Under review as a conference paper at ICLR 2021
similar to (Haarnoj et al. 2017 Klink et al. 2019). P is parameterised so that each action dimension affects the corresponding space dimension. P is designed such that when the transition dynamics order is n, the n² derivative of is set be equal to the action applied for time unit seconds body with configurable inertia. This is integrated over time to yield the next state. R is designed such that the reward for the current time step is the distance travelled towards the target since the last step.
Reward Delay We delay the reward for a state-action pair by a non-negative integer number of timesteps, which we call the delay, d.
Rewardable Sequence Length For discrete environments, we reward only specific sequences of states of positive integer length [ We consider states to be distinct along the sequences allowing for (S1- [ (8--7]- -J] sequences. For the continuous environment of moving to a target, n is variable since already corresponds to a real world task.
Reward Sparsity For discrete environments, we define the reward density rd of sequences in terms of the fraction of possible sequences of length n that are actually rewarded by the environment, given that n is constant. If num,, sequences are rewarded, we define the reward density to be ([))-IT)) and the sparsity as 1 - rd. For continuous environments, we do not define sparsity, but rather allow having a sparse or dense environment using a make make_denser configuration option.
rd = num,// ([S]-[T(-m)
Stochasticity For discrete environments, we implement a transition noise I_n E [0, 1]. With probability i_n, environment transitions uniformly random state that is not the true next state given by the generated P. We further implement a reward_ _oiiooiiee ør_ E R and add a normal random variable distributed according to (0, o2 to the true reward. For continuous environments both these noises are normally distributed and directly added to the states and rewards.
Irrelevant Features For discrete environments, we introduce new discrete dimensions. Each di- mension i has its own transition function P. which is independent of all other transition functions. However, only one of these is relevant to calculate the reward function. Similarly, in continuous environments we label existing dimensions of S and A as irrelevant and do not consider them in the reward calculation.
Representations For discrete environments, when this aspect is enabled, each categorical state associated with an image of regular polygon which becomes the externalyy visible state 8 to the agent. This image can further be transformed by shifting, scaling, rotating flipping, which are applied at random to the polygon whenever an observation is generated. The transforms are only applied within range that the polygon is always present in the image. Examples of some generated states can be seen in Figure g in Appendix [ This dimension is currently not implemented for continuous environments because it would be expensive.
Very low cost of execution Experiments with MDP Playground are cheap, allowing academics without special hardware perform insightful experiments. Wall-clock times depend lot on the algorithm, network size and dimensions of hardness. Nevertheless, to give the reader an idea of the runtimes involved, DQN experiments (with network with hidden layers of 256 units each) took on average 35s for complete run of DQN for 20 000 enviromment steps. In this setting we restricted Ray RLLib (Liang et al.) 2018) and the underlying Tensorflow (Abadi al., (013)) run on one core of laptop
2 This equates to roughly 30 minutes for the entire delay experiment shown in Figure settings for delay). Even when using the more expensive continuous or representation learning environments, training was only about 3-5 times slower. When parallelised on cluster, complete experiments can finish in a few minutes.
la which was plotted using 50 runs (10 seeds
4 EXPERIMENTS AND RESULTS
To demonstrate the usefulness of our benchmark, we evaluated Rllib implementations (Liang et al.
2018) of DQN (Mnih et al.) 2015), Rainbow DQN (Hessel et al. 2018 A3C (Mnih et al. 2016) on discrete environments and DDPG (Lillicrap et al. 2016). TD3 (Fujimoto et al. 2018) and SAC
In general, and n would not constants in real world environments, but for our toy environments, we use fixed n and d. ccoee-77-8800H CPU The full CPU specifications for single core can be found in Appendix
4

---

Under review as a conference paper at ICLR 2021
(Haarnoja et al. 2018) continuous environments over grids of values for the dimensions of hardness discussed above. We used fully connected networks except for pixel-based representations where we used Convolutional Neural Networks (CNNs) (LeCun et al. 1989). We first describe the discrete environment experiments, followed by those on continuous environments. All hyperparameters and the tuning procedure used are available in Appendix M
4.1 DISCRETE ENVIRONMENTS
We set [S] and |A| to 8 and the density of terminal states to 0.25 for the experiments. The reward scale is set to 1.0 whenever a reward is given by the environment. We generated random Ps that completely connected. This is done by selecting for each state for each action a, random successor state s', so that from each state there a transition possible to every state in state space through one of the available actions. We do this keep regular structure for the P. This leaves the environments in our experiments with a high bias nature versus the high variance nature seen in RL benchmarks in practice. Having a very complex or itself can introduce "noise" into the evaluation of algorithms and require many iterations of training before we can see if the agent is learning. We leave this for complex benchmarks to capture as they are closer real world use cases. For unit testing, especially, we need quick insights and given that most agents are agnostic to the choice of P and R, it is beneficial to have what we term high bias environments to test whether agents are learning. We discuss some more aspects of such design in Appendix
Varying reward delay Figures
depict the mean and standard deviation over 10 runs
la c
Rewars
Roward
0
for various delays. One
deley
delay
consists of 10 random
(a) DQN delay
(b) Rainbow delay
(c) A3C delay
seeds for the algorithm but uses fixed seed for We
Reward
50.
Reward
Reward
50
the environment.
plot the Area Under the Curve (AUC) which takes the mean over previous training rewards. As can be seen from the figure, all algorithms perform very well in the vanilla environment where the MDP fully observable as the information state of the agent is equal to the MDP's state. For all algorithms, performance degrades environments where the information state is non-Markov, Performance clearly degrades more as the information state needed to compute the optimal policy requires observations be stacked. interesting (and expected) that Rainbow DQN is somewhat more robust than DQN. The plots also show that DQN variants are more robust to delay as compared to A3C variants.
sequence length
2
sequence__engee
sequence
2
Jength
(d) DQN seq len
(c) Rainbow seq len
(f) A3C seq len
Figure 1: AUC of episodic reward at the end of training for different agents for varying delays (top) and sequence lengths (bottom). Error bars represent standard deviation. Note the reward scales.
Varying rewardable sequence length Results here are qualitatively similar to the ones for delay. However, we observe Figures Id that sequence length has more drastic effect terms of degra- dation of performance. The improvements of Rainbow DQN over DQN are also more pronounced for these harder problems.
The standard deviation in the preceding plots is high in many of the environments with non-Markov information state. Sometimes, however, the algorithms still managed perform well, which empha- sises that algorithms can perform well even when their assumption of having Markov information state is violated; this one of the possible explanatory factors for the fact that tuning seeds can lead to good results (refer Figure 5, Henderson typically associated with RL as can be seen in Appendix
al. 2018). This also leads to noisy learning curves
Varying representations For representations,
used image representations and applied various transforms (shift, scale, rotate and flip) one at a time and also all at once. We observed that the more transforms that are applied the images, the harder it is for algorithms to learn, can be seen in
5

---

Under review as a conference paper at ICLR 2021
Rewward
Reward
Raward
100
Rewpdd
50
$5rf
P07
image
Imape 2uount
(a) DQN
(b) Rainbow
(c) A3C
(d) DQN shift
Figure 2: AUC of episodic reward at the end of training for the different algorithms when varying representation. 's' denotes shift (quantisation of 1), "S' scale, 'f" flip and 'r' rotate in the labels in the first three subfigures and image_ste faae_sshh_aa_ttt______eeaaa_aea_ represents quantisation of the shifts in the DQN experiment for this. Error bars represent 1 standard deviation. Note the different reward scales.
Figures 2a This was to be expected since there are many more combinations to generalise over for the agent. What was unexpected was that the most problematic transform for the algorithms to deal with was shift. Despite the spatial invariance learned in CNNs (LoCun 2012), our results imply that that seems to be the hardest one to adapt to. As these trends were strongest in DQN, we evaluated further ranges for the individual transforms for DQN. shifting had the most possible different combinations that could be applied to the images. Therefore, have fewer possible values. Figure [2d shows that DQN's performance improved with increasing quantisation of shift (=> fewer possible values). We noticed similar trends for the other transforms as well, although not as strong as they do not have as many different values as shift (see Appendix (G) This indicates that shift and other types of invariance do not come for free and that one needs have sufficient amount of samples for the algorithm to become invariant to the transforms we desire. This also a good sanity check that our toy environments allow such insights and other potential insights we may not yet be aware of.
quantised the shifts to
Results for further dimensions and baselines can be found in the appendix. See Appendices G and that show additional interesting results for varying fransition noise, reward noise, reward sparsity. Further, even though our own focus with the benchmark designing and unit testing deep RL, we believe it is also valuable tool for theoreticians and evaluated tabular baselines Q-learning (Sutton & Barto 2018). Double Q-learning (van Hasselt] 2010) and SARSA (Sutton & Barto 2018) on the discrete non non-image based environments with similar qualitative results to those for deep agents.
These can be found Appendix H
4.2 CONTINUOUS ENVIRONMENTS
We set the state and action space dimensionalities to The state space range for each dimension was [-10, 10] while the default action space range was [-1, 1,11] 1]. The task would terminate when an algorithm would reach the target point, or after most 100 timesteps. We focus on results for DDPG
as results for TD3 and SAC are qualitatively similar (see Appendix G
(a) target radius rew.
(b) action range rew.
(c) time unit rew.
(d) irr. dims. rew.
199000.0 An mesn
kode det near
S0
0.1
525
055
1.0
255
4.
8.5
1.0
14..0n 20404 mix
Uim unk.
state dm
(e) target radius len.
(f) action range len.
(g) time unit len.
(h) imr. dims. len.
Figure 3: AUC of episodic reward (top) and lengths (bottom) for DDPG the end of training. Error bars represent standard deviation. Note the different y-axis scales.
Varying action range We observed that the total reward gets worse for action max 1. Up until
the value of 1, the episode lengths decreased as we would desire (see Figures 3b & 3f This can be attributed the fact that the exploration schedules for the studied agents take the max range available and explore based on that. But, as can be seen from these results, tuning these ranges or adapting exploration mechanisms can produce substantially better results.

---

Under review as a conference paper at ICLR 2021
Varying time unit We observed that increasing the time unit helps up to a point, until which the episode lengths also go down (see Figures Be & 3g). Thus, tuning this value to learn how long we must play action will also impact performance of algorithms significantly.
Varying target radius The target radius is a value which is generally set to a small enough value to be able to say that the algorithm has reached the target. However, we noticed that, for small values, all the continuous control agents oscillated around the target to reach it exactly. This can be observed in Figure 3aj and where we note that even though the task was learnt for different target radii, the episode lengths were shorter for larger radii as the agents kept oscillating outside the radius. Even for such a simple task all evaluated algorithms failed to adapt to performing ame-grained control near the target. We hypothesise that the agents did not learn to slow down close to the goal. Given more experience close to the goal, we expect the agents to be able to learn this behaviour.
Varying irrelevant features We observed that introducing irrelevant dimensions to the control prob- lem, while keeping the number of relevant dimensions fixed to 2. decreased an agent's performance (see Figures 3d & 3h). We believe this is because irrelevant features interfere with the learning process.
Additional interesting results, including results for action loss weight can be found in Appendix
USING MDP PLAYGROUND
We believe MDP Playground offers so much power and flexibility that can only list few sample use cases here and let the user choose to explore further what even we might not have thought of.
5.1 INSIGHTS INTO EXISTING AGENTS
Design and Analyse Experiments We allow the user the power inject dimensions into a base environment in a fine-grained manner and analyse results using the 1-D plots from before or radar plots with user defined weights for the plots. Since, different users might be interested in different dimensions, these are loaded dynamically from the data. For instance, radar plots for the dimensions we varied in our experiments can be seen as in Figures 4a and 4b
00001 sp0c
0.5
1.0
250
500
150
1063
optaodo
(a) discrete envs.
(b) cont. envs.
(c) reward density
(d) action range and time unit
Figure 4: Analysing and Debugging
Varying Multiple Dimensions It is possible to vary multiple dimensions at the same time in the same base environment. For instance, Figure shows the diagonal relationship between varying the action range and time unit together. The platform allows this for all of dimensions that can be controlled together. Even more such experiments can be found in Appendix I including varying both P and At noises together in discrete environments and interactions between P noise and the target radius, and interaction of the order of the transition dynamics with both time unit and action range.
Transfer complex environments We have designed wrappers for Atari and Mujoco which can be used to inject some of the dimensions we have for toy environments as well. For instance, for time unit, there is an optimal value of the time unit with decreasing performances on either side of this optimum. For all three agents (Figure 5c). we see peak performance for a time unit 0.4 smaller than that of the tuned vanilla environment. The same holds for SAC on Pusher whereas DDPG and TD3 failed to solve Pusher (see Appendix K The three agents also peaked on the toy environment at similar values (DDPG and TD3 at 0.2 and SAC at 0.5 closely followed by 0.2). All three agents peaked on Reacher at time unit 1.0. We attribute this to the frame skip of Reacher being while that of HalfCheetah and Pusher is 5. What does not transfer from the toy environments is that the underlying optimal time unit is much smaller for the complex environments (0.02s 0.2s). This is
7

---

Under review as a conference paper at ICLR 2021
likely because we require much more fine-gre tine-grained control for high -ihh-iimensionll environments. At the same time, this also indicates time unit should not be infinitesimallyy small to achieve too fine-grained control since there is optimal time unit for which should repeat the same action (Biedenkapp) ett al. 2020). This also has important consequences in environments where acquiring observations is expensive (Huang al. 2019)
Here we have described just one of the dimensions and environments. How- ever, we performed exten- sive experiments for dif- ferent environments and dimensions in Appendix D They all supported our argument that the high- level trends for the dimen- the
RORD ROND-0
10000
RonD
1000
92
0.4
3.0
2.0
9.2
4.0
0.2
0.4
1.0
2.0
4.0
(a) SAC time unit
(b) DDPG time unit
(c) TD3 time unit
Figure 5: AUC of episodic reward at the end of training on HalfCheetah varying time unit (bottom). Error bars represent standard deviation. Note the different y-axis scales.
sions similar trends on the toy environments. This might seem obvious to some readers after the fact, but the fact that these insights could have been gained directly on the toy environments without trying them on complex environments shows the potential insights that may be gained on toy environments.
5.2 DESIGNING NEW AGENTS
We hope our benchmark will help identify the inductive biases needed for designing new RL algorithms without getting bogged down by other sources of "noise" in the evaluation just as MNIST helped to identify the inductive bias of convolutions. MNIST by means represents the true distribution of image data in the real world but it retains the key properties needed to identify inductive biases needed to perform well image data. In the same way we believe retain key dimensions of the problems which need inductive biases to be identified for them. The fact that CNNs can classify and learn even random noise (see, e.g.. -hang al., 2017 Arpit et al. (017) shows that even random data should be sufficient to identify inductive biases as long as the key dimension is present in the data.
5.3 DEBUGGING EXISTING AGENTS
Analysing how an agent performs under the effect of various dimensions can reveal unexpected aspects of an agent. For instance, when using bsuite agents, we noticed that when we varied our environment's reward density, the performance of the bsuite Sonnet DQN agent would go up in proportion to the density (see Figure This did not occur for other bsuite agents. This seemed to suggest something different for the DQN agent and when we looked at DQN's hyperparameters we realised that it had a fixed schedule while the other agents had decaying schedules. Such insights can casily go unnoticed if the environments used are too complex. We know of many researchers who have had frustrating experiences debugging in complex environments, where it's 's unclear whether hyprrparameter was the cause or the difficulty of the environment itself and such toy environments aid debugging by removing the "noise" of complex, real environments.
6 DISCUSSION AND RELATED WORK
To the best of our knowledge, we are the first to perform a principled study of how significant aspects such as non-Markov information states, irrelevant features, representations and low-level dimensions, like time discretisation, affect agent performance. MDP Playground facilitates casy study of the presented dimensions, giving us an abstraction bridging MDPs and the real world.
Our framework is very well suited to designing, developing and unit-testing new algorithms. No other benchmark that we know of offers fine-grained control over orthogonal dimensions of difficulty for agents all of which can be applied in the same base environment.
In the case of Rainbow and vanilla DQN we demonstrated how our environments can be used to compare extensions of algorithms and how mitigates issues of the other. We show how learning
8

---

Under review as a conference paper at ICLR 2021
curve graphs with hardness inserted look reminiscent of typical RL curves. For example, for sequence lengths (and other dimensions in Appendix(] P) when the information state is non-Markov, we see the typical noise associated with RL algorithms (see, e.g., Henderson et al. 2018).
The concurrently developed behaviour Suite for RL (bsuite: Osband et al. 2019) is the closest related work to MDP Playground. [Osaand et al. 2019) collect known (toy) environments from the literature and use these to try and characterise agents based on their performance on these environments. Instead of providing individually controllable dimensions of hardness, bsuite provides environments that exhibit some of our dimensions of hardness that can not always be controlled by a user. Further, most environments in bsuite can be seen as an intermediate step between our MDPs and even more complex environments, One notable distinction between bsuite and MDP Playground is that bsuite provides no evidence of relevance of the contained environments to more complex problems, whereas we demonstrate that the identified trends on dimensions of hardness are not only relevant on toy environments but also transfer to more complex environments. Finally, bsuite offers no continuous benchmarks, whereas MDP Playground provides both discrete and continuous environments.
Maillard et al. (014) defines a novel theoretical metric for defining hardness MDPs. It captures difficulties within MDPs when the true state of the MDP is known. However, large part of the hardness in our MDPs comes from the agent not knowing the optimal information state to use. It be interesting to design metric which captures this aspect of hardness as well.
Further benchmark environments include Procgen (Cobbe et al. 2019), Obstacle Tower (Juliani et al. 2019) and Atari (Bellemare et al.] 2013). Procgen adds various heterogeneous environments and tries to quantify generalisation in RL. In a similar vein. Obstacle Tower provides a generalization challenge for problems in vision, control, and planning. These benchmarks do not capture orthogonal dimensions of difficulty and as a result, they do not have the same type of fine-grained control over their environments' difficulty and neither can each dimension be controlled independently. We view this as crucial aspect when benchmarking new algorithms. Dulac-Armold et al. (2020) has some overlapping dimensions with our platform but it consists of much more specific environments, and only those that are continuous. This means, they are further down in the RI. development pipeline than MDP Playground or bsuite which target the toy environment domain and thus complementary to our approach. More details on related work can be found in Appendix D
We believe we need a curriculum of benchmarks for RL. In this hierarchy, our benchmark is low- level, high bias benchmark that should be the first test for newly implemented agents and bsuite comes at the secondary level and other more specialised benchmarks that capture more specific environments like (Cobbe al. 2019 Cobbe et al. 2019; Juliani et al. 2019) are even more specialised.
7 CONCLUSION AND FUTURE WORK
We introduced a low-cost benchmark to test RL algorithms in environments with varying and controllable dimensions of hardness. We demonstrated their effects on well-known RL. algorithms. The platform allows us disentangle various factors that make RL benchmarks hard by providing fine-grained control over various dimensions. We further demonstrated how the performance of the studied agents is adversely affected by some dimensions of hardness of environments which are under the control of researchers even in real world environments, such as the time-unit. We will open-source our code facilitate better, cheaper, more reproducible, and more directed benchmarking in the RL community. We also evaluated the transfer of the dimensions to more complex environments and showed how similar challenges are present there as well. While we tried to exhaustively identify dimensions of hardness, it is unlikely that we have captured all orthogonal dimensions of hardness in RL. We welcome more dimensions that readers think will help us encapsulate further challenges in RL.
We believe providing theoretical and practical researchers such a powerful tool is like providing a programming language. Not every part of the MDP search space may be interesting to user, but they have the power to define their own toy MDPs easily with high-level primitives and not be limited to the glimpse we have provided here. Future work should try to research a stronger link for toy MDPs with real environments. We are convinced that MDP Playground can have a great impact on RL research and is important ingredient towards an era with more reproducible, generalisable and better understood RL research.

---

Under review as a conference paper at ICLR 2021
REFERENCES
M. Abadi, A. Agarwal, P. Barham, H. Brevdo, 7. Chen, C. Citro, G. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, Goodfellow, A. Harp. G. Irving, M. Isard, Y. Jia, R. Jozefowiez. L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, Shlens, B. Steiner, Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg. M. Wicke, Y. Yu, and X. Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL Uttpttps: /ttp:://wwwww. www www.orfwwsoooooooooooooooooo.. org/
Abbeel, A. Coates, and A. Y Ng. Autonomous helicopter aerobatics through apprenticeship learning. The International Journal of Robotics Research,
2010. 2010.
J. A. Arjona-Medina, M. Gillhofer, M. Widrich, T. Unterthiner, J. Brandstetter, and S. Hochre- iter. RUDDER: return decomposition for delayed rewards. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d' A'Achh--Bc, E. B. Fox, and R. Gameet (eds.), Proceedings of the 32nd Inter- national Conference on Advances in Neural Information Processing Systems (NeurIPS 19), pp. 13544-13555, 2019.
D. Arpit, S. Jastrzebski, N. Ballas, D. Krueger, E. Bengio, M. S. Kanwal, T. Maharaj. A. Fischer, A. C. Courville, Y. Bengio, and S. Lacoste-Julien. A closer look at memorization in deep networks. In D. Precup and Y. W. Teh (eds.). Proceedings of the 34th International Conference on Machine Learning, (/CML'17) pp.
PMLR, 2017.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, Jun 2013.
A. Biedenkapp. R. Rajan, F. Hutter, and M. Lindauer. Towards TempoRL: Learning when to act. In Workshop on Inductive Biases, Invariances and Generalization RL (BIG@@ ICML '00), July 2020.
G. Brockman, V. Cheung, L. Pettersson,, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. OpenAI gym. 1rXiv:6606.. 540 [cs. June 2016
P. Chrabaszez, I. Loshchilov, and F. Hutter. Back to basics: Benchmarking canonical evolution strategies for playing atari. In J. Lang (ed.), Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, (IJCAI'88, pp. 1419-1426. ijcai.org, 2018.
K. Chua, R. Calandra, R. McAllister, and S. Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Proceedings of the 31st International Conference on Advances in Neural Information Processing Systems (NeurIPS' 18), pp. 4754-4765, 2018.
K. Cobbe, C. Hesse, J. Hilton, and J. Schulman. Leveraging Procedural Generation to Benchmark Reinforcement Learning. arXiv:1912.00 588 [cs. Dec 2019.
K. Cobbe, O. Klimov, C. Hesse, T. Kim, and Schulman. Quantifying generalization in reinforcement learning. In K. Chaudhuri and R. Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning (ICML' (9), pp. 1282-1289. PMLR, June 2019.
Gabriel Dulac-Amold, Nir Levine, Daniel Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. An empirical investigation of the challenges of woald reinforcement learning. CoRR, abs/2003. 1881, 2020. URL https:t/p//. http://axxivv.oo
abs /twpb///200000 2203 11881
M. Fortunato, M. G. Azar, B. Piot, Menick, M. Hessel, Osband, A. Graves, Mnih, R. Munos, D. Hassabis, O. Pietquin, C. Blundell, and S. Legg. Noisy networks for exploration. In Proceedings of the International Conference on Learning Representations (ICLR' 18), 2018. Published online: iclr.cc
S. Fujimoto, H. van Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In J. G. Dy and A. Krause (eds.), Proceedings of the 35th International Conference on Machine Learning (ICML' (1)) pp. 1582-1591. PMLR, 2018
R. D. Gaina, S. M. Lucas, and D. with statistical forward planning methods. In Proceedings of the 33rd Conference on Artificial Intelligence (AAAI' 19), pp. 1691-1699 AAAI Press, 2019.
Tackling sparse rewards in real-time games
10

---

Under review as a conference paper at ICLR 2021
T. Haarnoja, H. Tang, P. Abbeel, and S. Levine. Reinforcement learning with deep energy- based policies. In D. Precup and Y. W. Teh (eds.). Proceedings of the 34th International Conference on Machine Learning, (ICML 17)), pp. 1352-1361. PMLR, 2017.
T. Haarnoja, A. Zhou, P. Abbeel, and Sergey Levine. Soft Actor-Critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In G. Dy and A. Krause (eds.), Proceedings of the 35th International Conference on Machine Learning (ICML') (8), pp. 1856-1866 1865. PMLR, 2018.
P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger. Deep reinforcement learning that matters. S. A. Mcllraith and K. Q. Weinberger (eds.), Proceedings of the Conference on Artificial Intelligence (AAAI'I8), 18), pp. 3207-3214, AÁAI Press, 2018.
D. Hendrycks and T. G. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In Proceedings of the International Conference Learning Representations (ICLR' 19), 2019. Published online: iclr.co
M. Hessel, J. Modayil, H. van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver. Rainbow: Combining improvements in deep reinforcement learning. In S. A. Mcllraith and K. Q. Weinberger (eds.), Proceedings of the Conference on Artificial Intelligence (AAAI' 18), pp. 3215-3222. AAAI Press, 2018.
Y. Huang, V. Kavitha, and Q Zhu. Continuous-time markov decision processes with controlled observations. In Proc. of Allerton 19, pp. 32-39. IEEE, 2019.
T. Jaakkola, S. P. Singh, and M. I. Jordan. Reinforcement learning algorithm for partially observable markov decision problems. In G. Tesauro, D. S. Touretzky, and T. K. Leen (eds.), Proceedings of the 7th International Conference on Advances in Neural Information Processing Systems (NeurIPS'955 pp. 345-352, 1995.
A. Juliani, A. Khalifa, V.P. Berges, J. Harper, E. Teng, H. Henry, A. Crespi. J. Togelius, and D. Lange. Obstacle Tower: A Generalization Challenge in Vision, Control, and Plamning. In S. Kraus (ed.), Proceedings of the Eiggtt International Joint Conference Artificial Intelligence (IJCAI), pp. 2684-2691. ijcai.org, Feb 2019.
P. Klink, H. Abdulsamad, B. Belousov, and J. Peters. Self-paced contextual reinforcement learning- In L. P. Kaelbling, D. Kragic, and K. Sugiura (eds.), 3rd Annual Conference on Robot Learning, (CoRL'I9) 19), pp. 513-529. PMLR, 2019.
Y. LeCun. Learning invariant feature hierarchies. In A. Fusiello, V. Murino, and R. Cucchiara (eds.), Computer Vision ECCV 2012, pp. 496-505. Springer, 2012.
Y. LeCun, B. Boser, J. Denker, D. Henderson, R. Howard, W. Hubbard, and L. Jackel. Backpropaga- tion applied to handwritten zip code recognition. Neural Comput., 1(4):511-551 1989.
E. Liang, R. Liaw, R. Nishihara, P. Moritz, R. Fox, K. Goldberg, J. E. Gonzalez, M. I. Jordan, and Stoica. RLlib: Abstractions for distributed reinforcement learning. J. Dy and A. Krause (eds.), Proceedings of the 35th International Conference Machine Learning (ICML '18, volume 80, pp. 3059-3068. Proceedings Machine Learning Research, 2018.
P. Lillicrap, Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, Silver, and D. Wierstra. Continuous control with deep reinforcement learning. In Y. Bengio and Y. LeCun (eds.), Proceedings of the International Conference on Learning Representations (ICLR' 2016. Published online: iclr.cc
Odalric Ambrrc Maillard, Timothy A. Mann, and Shie Mannor. How hard is my mdp?" the distribution-norm to the rescue". In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger (eds.), Advances in Neural Information Processing Systems Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pp. 1833-1843,,,, 2014. URL httptttttttppppeeettttt www.. httpp:///ppaaeesnn....cccceee/ p st www.:ttrwwoooitooooooooooooooooo
27: Annual Conference
5441 -how- wweo--hhrd-ii

---

Under review as a conference paper at ICLR 2021
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. A. Riedmiller, A. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforee- ment learning. Nature, 518(7540):529-533 2015.
V. Mnih, A. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In M. Balcan and K. Weinberger (eds.), Proceedings of the 33rd International Conference on Machine Learning (ICML' 16), volume 48, pp. 1928-1937, 2016.
D. Nau. Pathology on game trees revisited, and alternative to minimaxing, Artif. Intell., 21(1-2): 221-244, 1983.
I. Osband, Y. Doron, M. Hessel, J. Aslanides, E. Sezener, A. Saraiva, K. McKinney, T. Lattimore, C. Szepezvari, S. Singh, B. Van Roy, R. Sutton, Silver, and H. Van Hasselt. Behaviour suite for reinforcement learning. In Proceedings of the International Conference on Learning Representations (ICLR' 19), 2019. Published online: iclr.ce.
J. Pearl. Theoretical impediments to machine learning with seven sparks from the causal revolution. In Y. Chang, C. Zhai, Y. Liu, and Y. Maarek (eds.), Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, (WSDM' 18), pp. 3. ACM, February 2018.
Rajendran, J. Ganhotra, S. Singh, and L. Polymenakos. Learning end-to-end goal-oriented dialog with multiple answers. In E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujui (eds.), Proceedings of the Conference on Empirical Methods Natural Language Processing (EMNLP' 19), pp- 3834 3843. Association for Computational Linguistics, 2018.
R. Ramanujan, A. Sabharwal, and B. Selman. On adversarial search spaces and sampling- based planning. R. I. Brafman, Geffner, Hoffmann, and A. Kautz (eds.), Proceedings of the 20th International Conference on Automated Planning and Scheduling, (ICAPS" pp. 242-245. AAAI, 2010.
Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv: 1707.0634 Lcs.LG]. 2017.
D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. P. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484 489, 2016.
Elizabeth S Spelke and Katherine D Kinzler. Core knowledge. Developmental science, 10(1):89-96, 2007.
R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. The MIT Press, second edition, 2018 URL URRLttppp//inoommeeeeeeeee www.iidaas.eet/tooooooooooooo www./e/e///ttt.ee wtw.okk/hheebbookoooooo 2nd. html
R. Sutton, D. Precup, and Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 1
1999.
C. Thornton, F. Hutter, H. Hoos, and K. Leyton-Brown. Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms. In I. Dhillon, Y. Koren, R. Ghani, T. Senator, P. Bradley, R. Parekh, J. He, R. Grossman, and R. Uthurusamy (eds.), The 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD' 13), pp. 847-855. ACM Press, 2013.
S. Thrun, W. Burgard, and D. Fox. Probabilistic robotics. Intelligent robotics and autonomous agents. MIT Press, 2005.
E. Todorov, T. Erez, and Y. Tassa. MuJoCo: A physics engine for model-based control. In Interna- tional Conference on Intelligent Robots and Systems (IROS' 12), pp. 5026-5033. IEEE, 2012.
H. van Hasselt. Double Ielaannng. In J. Lafferty, C. Williams, J. Shawe-Taylor R. Zemel, and A. Culotta (eds.), Proceedings of the 24th International Conference on Advances in Neural Information Processing Systems (NeurlPS' 10), pp. 2613-2621, 2010.
12

---

Under review as a conference paper at ICLR 2021
T. Wang, X. Bao, I. Clavera, J. Hoang, Y. Wen, E. Langlois, S. Zhang, G. Zhang, P. Abbeel, and J. Ba. Benchmarking model-based reinforcement learning. arXiv:1907.02000 [cs.LG], 2019.
C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning requires rethinking generalization. In 5th International Conference on Learning Representations, (ICLR'I7). OpenReview.net, 2017.
13

---

Under review as a conference paper at ICLR 2021
A DIMENSIONS OF HARDNESS IN MDP PLAYGROUND
We list here the hardness dimensions for MDP Playground. Details on each meta-feature can be found in the documentation for the class casp__paayrouudd envs -LTooyEnv in the accompanying code.
Reward Delay
Rewardable Sequence Length
Reward Sparsity
Noise
R. Noise
Irrelevant Features
Transforms for Representation Learning
Reward Shift
Reward Scale
State space size/dimensionality
Action space size/dimensionality
Terminal State Density
Terminal State Reward
Relevant Dimensions (for both state and action spaces)
Only for discrete environments:
Image Representations
Only for Image Representations:
Shift Quantisation
Scale Range
Rotation Quantisation
Only for continuous environments:
Target Point
Target Radius
Time Unit
Inertia
State Space Max
Action Space Max
Transition Dynamics Order
Reward Function
Currently fixed hardness dimensions:
Initial State Distribution
A.1 ADDITIONAL SPARSITY OPTION FOR SEQUENCES
With regard to sparsity, recall the tennis serve again. The point received by serving an ace would be sparse reward. We as humans know reward ourselves for executing only part of the sequence correctly. Rewards in continuous control tasks to reach a target point (e.g. in Mujoco, Todoroy al. 2012). are usually dense (such as the negative squared distance from the target). This lets the algorithm obtain a dense signal in space to guide learning, and it is well known (Sutton & Barto 2018) that it would be much harder for the algorithm to learn it only received a single reward the target point. The environments in MDP Playground have configuration option, maake_enneee to allow this kind of reward shaping to make the reward denser and observe the effects on algorithms. To achieve this, when make_ denser is True, the environment gives fractional reward if a fraction of a rewardable sequence is achieved discrete environments. For continuous environments, for the move to target point reward function, this option toggles between giving dense reward as described in the main paper and giving a sparse reward when the agent is within the target radius.
14

---

Under review as a conference paper at ICLR 2021
B ALGORITHM FOR GENERATING MDPS
Algorithm Generating MDPs with MDP Playground
1: Input:
2: number of states |S],
3: number of actions |A],
4: reward delay d.
5: length of rewardable sequences n,
6: density of rewardable sequences rd,
7: transition noise I_n or dtm.
8: reward noise dr
9: soward_scald,
10: reward_shift,
11: term_state_rewan
12: make_denser,
13: terminall_state_densttt
relesion__dimersionn
15: Hardness dimensions specific to continuous environments:
target_poing
17: target_radius
tramsitioo__@yramlcss.oo
19: time_unit
20: inertia
21:
22: function INIT INII_ITTAAAAT TRANSITION CEEESTIIONFUUNCTIINOOO FUNCTION():
23:
if discrete environment then
24:
for each state do
For generating a completely connected
25:
Set possible successor states: S
26:
for each action do
Set P(s, a) = s' sampled uniformly from S' and remove s' from S'
28:
irrelevant features then Generate dynamics Parr of irrelevant part of state space was done for P
29:
else
31:
Do nothing as continuous environments have a fixed parameterisation
32:
33:
function INIT_ _ET_REWARD____ REWARDDFUNCTIONON FUNCTION(n):
34:
if discrete environment then Randomly sample rd*
35:
[ 18-17- -)] sequences and store in rewardaable_sqqeeees
36:
else
37:
Do nothing as continuous environments have fixed options for the reward function
38:
15

---

Under review as a conference paper at ICLR 2021
39:
function TRANSITION BACOOINFUNTTTOO,OO a): if discrete environment then
40:
41:
P(s.a) a)
42:
1) <t_n then s' = random state in S \ (P(ss
43:
>> Inject noise
44:
if irrelevant features then
45:
Execute dynamics P.r of irrelevant part of state space and concatenate with
46:
if representation learning then
47:
image of corresponding polygon(s) with applied selected transforms
48:
else
49:
Set transitionn aanaanssiiiodddamiiiiiiia dwwadiccsoordee cs_order
50:
b Superscript n represents n derivative Set 8" a" /inertia & Each state dimension is controlled by each action dimension for in reversed(range(()) do
Set a" = a
51:
52:
53:
Set s = =
3
time time_unit?
Di f current time step.
)=
54:
8441+ 00
55:
8141
56:
return
58:
function REWARD COIIARDDFUNCTTOONS.. a):
0
60:
if irrelevant features then
Sub-select the part of state space relevant to state sequence n states ending d steps in the past in rewardable_seeueneeee then
reward
62:
discrete environment then if not make _enserdeeerrreeeeeee then
63:
64:
65:
== reward_scale
66:
else
67:
for in range(n) do if sequence states ending d steps in the past is a prefix sub-sequence of sequence seeearcablesseeueccee then
68:
69:
r+ i/n
70:
else
71:
Distance moved towards the target_point
72:
N \O, o²r
73:
r* rewa rd_scale
74:
r+ reward_shiff
75:
if reached terminal state then
76:
r+ setrn__sateereeeeee
77:
return r
78:
79:
function MAINQ:
80:
INIT TERMINAL_ STATES()
Set T according to terminal haalaaaate_eeeeeeeeeaa ity
81:
INIT INIT _NTT_STATE_____ (INTE_
Set Pa to uniform distribution over non-terminal states
82:
INIT TRA_STRINTT "RTTTASTTIIOOOUUNCTIOOOOOO
83:
INIT _NIT_ REWARD NHEWARD_FUNCTIONE
16

---

Under review as a conference paper at ICLR 2021
C DESIGN DECISIONS
C.I REGULAR STRUCTURE OF TRANSITION FUNCTION FOR EXPERIMENTS
is always possible design adversarial Ps (Nau 1983 Ramanujan al. 2010 which can be made arbitrarily hard to solve. So, we imposed a regular structure on P. If we have environment, say the Earth, where a big reward (e.g. hidden treasure) placed in an unknown and deliberately unexpected location. Then evaluating the intelligence of an agent on such an environment clearly does not give us a true measure of agent intelligence. This also problem with many benchmark environments, e.g., gbert has bug which allows the agent to achieve very large number of points (Chrabaszez et al.] (2188) and breakout has a scenario where, if an agent creates hole through the bricks, it can achieve very large number of points. Even though the latter is sign of intelligent behaviour, skews the distribution rewards and introduces variance in the evaluation. In keeping with the low-level and high bias nature of toy environments, we leave these more complex benchmarks with more variant environments and instead impose a regular structure on P. We do allow completely random generation ofPs which do not have regular structure and have much more variance in their connections, but as we noticed in initial experiments this produces variant environments and leads noisy evaluations. We also plan let users also generate or provide their own P so that they might play with toy versions of environments they are interested in.
17

---

Under review as a conference paper at ICLR 2021
D EFFECT OF DIMENSIONS ON MORE COMPLEX BENCHMARKS
We tested the trends of the dimensions more complex Atari and Mujoco tasks. For Atari, we ran the agents on seam_rider, breakout, gbert and space invaders when varying the dimensions delay and transition noise. For Mujoco, we ran the agents on HalfCheetah, Pusher and Reacher using mujoco-py when varying the dimensions time unit and action runge. We evaluated 5 seeds for 500k: steps for Pusher and Reacher, 3M for HalfCheetah and 10M (40M frames) for Atari. The values shown for action range and time unit are relative to the ones used in Mujoco.
Varying transition noise We observe similar trends for injecting transition noise into Atari envi- ronments for all three agents as for the toy environments. We also observe that for some of the environments, transition noise actually helps improve performance. This has also been observed in prior work (Wang et al., 2019). This happens when the exploration policy was not tuned optimally since inserting transition noise is almost equivalent to e-greedy exploration for low values of noise. We also observed a similar effect for the toy environments in Figure [3 in Appendix G However, we also observe that performance drop is different for different environments. This be expected as there are other dimensions of hardness which we cannot control or measure for these environments.
Varying reward delay We see that performance drops for the delay experiments when more delay is inserted. For qbert (Figure (6c), these drops are greater on average across the agents. However, for breakout (Figure(6b) (6)) in many instances, we don't even see performance drops. In beam _eneer_rrrrreee (Figure 6ab and space_ _nteederrssssss (Figure (dd). the magnitude of these effects are intermediate to breakout and gbert. This trend becomes clearer when we also look at Figures 82b in Appendix K We believe this is because large delays from played action to reward are already present in breakout, which means that inserting more delays does not have as large an effect as in qbert (Figures (6c). Agents strongest affected in qbert which, upon looking at gameplay, from rewarding action to reward compared to the other games. The trends for delay however were noisier. Many considered environments tend to also have repetitive sequences which would dilute the effect of injecting delays. Many of the learning curves in Appendix L with delays inserted, indistinguishable from normal learning curves. We believe that, in addition to the motivating examples, this is empirical evidence that delays are already present in these environments and so inserting them does not cause the curves to look vastly different. In contrast, when we see learning curves for transition noise, we observe that, as we inject more and more noise, training tends to smoother curve as the agent tends towards becoming a completely random agent.
believe has the least delays
To analyse transfer of dimensions between toy and complex benchmarks, we use the Spearman rank correlation coefficient between corresponding toy and complex experiments for performance across different values of the dimension of hardness. The Spearman correlation was >= 0.7 for 19 out of 24 experiments and a positive correlation for four of the remaining five. DQN with delays added on breakout was the only experiment with correlation O.
Reward
500
Rowerr
Resword
1,000
250
2 delay
2
4
delay
dalay
delay
(a) DQN beam rider
(b) DQN breakout
(c) DQN qbert
(d) DQN space invaders
Reward
500
Reward
Howeed
1.000
2:00
0.01 0.02
0.25
0.0
0.03
0.02
0.1
0.25
0.52 0.02 0.1 transition naise
transiton noise
transition, nolie
transkion nobe
(c) DQN beam_rides
( DQN breakout
DQN qbert
(h) DQN sace_invaders
Figure 6: AUC of episodic reward for DQN on various environments at the end of training. Error bars represent 1 standard deviation. Note the different y-axis scales.
Varying action range We observed similar trends as discussed prior, in that there was an optimal value of action range with decreasing performances on either side of this optimum. Figure ?? shows this for all considered agents HalfCheetah (for SAC and DDPG, runs for action range values >= and >= 4 crashed and are absent from the plot). Qualitatively similar results on the other environments are given in Appendix K This supports the insight gained our simpler environment
18

---

Under review as a conference paper at ICLR 2021
that tuning this value may lead to significant gains for agent. For already tuned environments, such as the ones in Gym. this dimension is easily overlooked but when faced with new environments setting it appropriately can lead to substantial gains. In fact, even in the tuned environment setting of Gym, found that all three algorithms performed best for an action range 0.25 times the value found Gym for Reacher (Figures 833 83k 83g in Appendix [Kp. This observation is representative for the types of insight our benchmark can yield for RL algorithm design, as ideally an agent would adaptively set these bounds since these are under its control. Moreover, the learning curves in Appendix [ further show that for increasing action range the training gets more variant. The difference in performances across the different values of action range is much greater in the complex environments. We believe this is due to correlations within the multiple degrees of freedom as opposed to a rigid object in the toy environment. Due to the high bias nature of the toy environments, the high level trend of an optimal action range transfers to more complex ones.
10000
-00000
mcton_spoc_maa
(a) SAC action range (b) DDPG action range
(c) TD3 action range
Figure 7: AUC of episodic reward at the end of training on HalfCheetah varying action range. Error bars represent standard deviation. Note the different y-axis scales.
19

---

Under review as a conference paper at ICLR 2021
E MORE ON RELATED WORK
Our benchmark is also designed with long term AGI in mind. Dimensions like identifying delays and sequences will be essential to solving AGL. Most current algorithms lack such capability (Pearl 2018).
Many of the other benchmarks mentioned in the main paper are largely vision-based, which means that a large part of their problem solving receives benefits from advances in the vision community while our benchmarks try to tackle pure RL problems in their most toy form. This also means that our experiments are extremely cheap, making them a good platform to test out new algorithms' robustness to different challenges in RL.
A parallel and independent work along similar lines as the MDP Playground, which was released a month before ours on arXiv, is the Behaviour Suite for RL (bsuite, Osband al. (2019)). In contrast to our generated benchmarks, that suite collects simple RL benchmarks from the literature that are representative of various types of problems which occur in RL and tries to characterise RL algorithms. Unlike their framework, where currently there is no toy environment for Hierarchical RL (HRL) algorithms, the rewardable sequences that we describe would also fit very well with HRL. Additionally, we also have a toy continuous environment whereas bsuite currently only has discrete environments. An important distinction between the two platforms could be summed up by saying that they try to characterise algorithms while we try to characterise environments with the aim that new adaptable algorithms can be developed that can tackle environments of desired difficulty. They also do not generate completely random P and R for their environments like we do, which would help avoid algorithms overfitting to certain benchmarks.
For some readers, it might feel obvious that injecting many of these dimensions causes difficulties for agents. But to the best of our knowledge no other work has tried to collect all orthogonal dimensions in one place and study them comprehensively and what aspects transfer from toy to more complex environments.
The nature of the toy environments is one of high bias. We believe that the transfer of the hardness dimensions from toy to complex environments occurs because the algorithms we have tested are environment agnostic and usually do not take aspects of the environment into account. Q-learning for instance is based on TD-errors and the Bellman equation. The equation is agnostic to the environment and while adding deep learning may help agents learn representations better, it does not remove the problems inherent in deep learning. While it's nice to have general algorithms that may be applied in a black box fashion, by studying the dimensions we have listed and their effects on environments, we will gain deeper insights into what is needed to design better agents.
An additional comment can be made comparing the continuous and discrete complex environments comparisons to the toy benchmarks. The noise in comparing the discrete environments was higher and we believe this is due to the discrete environments being much more sparse and having many more lucky areas that can be exploited with the qbert bug and breakout strategy mentioned. In comparison, continuous environments usually employ a dense reward formulation in which case the value functions are likely to be continuous.
Algorithms like DQN (Mnih et al.) 2015) were applied to many varied environments and produce very variable performance across these. In some simple environments, DQN's performance exceeds human performance by large amounts, but in other environments, such as Montezuma's 's revenge, performance is very poor. For some of these environments, e.g. Montezuma's 's revenge, we need very specific sequence of actions to get a reward. For others, there are different delays in rewards. A problem with evaluating on these environments is that we have either no control over their difficulty or little control such as having different difficulty levels. But even these difficulty levels, do not isolate the confounding factors that are present at the same time and do not allow us to control the confounding factors individually. We make that possible with our Dimensions.
When designing the platform, we went over the components of an MDP and tried to exhaustively add as many parameterisable dimensions possible with the condition that they are all orthogonal and can be applied independently of each other. In a sense, this is an attempt to capture fundamental dimensions of hardness in the same way that human cognition is founded, in part, on four different systems and endow humans with abstract reasoning abilities (Spelke & Kinzler 2007). We don't t try capture, say credit assignment or generalisation as dimensions. These to be dealt with
20

---

Under review as a conference paper at ICLR 2021
higher level the same way that intelligent behaviour and reasoning arise from the interplay of different underlying cognitive systems but which process objects or space at a lower sensory level.
E.1 DEBUGGING WITH MDP PLAYGROUND
We discuss here further 2 more examples of how the toy environments helped debug RL algorithms in practice.
When merging some of our environments into bsuite, we noticed that when we varied sparsity, the performance of their DQN agent would down in proportion to the environment's 's sparsity, This did not oceur for other agents. This seemed to suggest something different for the DQN agent and when we looked at DQN's hyperparameters we realised that it had fixed schedule. While that may be desirable in some situations, we felt hurt DQN's performance because it was not allowed to explore enough early nor exploit what it learnt fully later. When we use regular structured environments, the agent performances are freed the "noise" that is present due to irregular transition functions and this makes it easy to see high-level trends.
In the Ray version we used, we observed that the noisy nets (Fortunato et al. 2018 implementation was broken the toy environments and then observed the same complex benchmarks. This makes it easy to debug if something major is broken.
21

---

Under review as a conference paper at ICLR 2021
F
SAMPLE STATES USED FOR REPRESENTATION LEARNING
(a) No transforms
(b) Shift
(c) Scale
(d) Rotate
(I) No transforms
(g) Flip
(h) All transforms
(e) All transforms
Figure When using the meta-feature representation learning in discrete environments, each categorical state corresponds O, each state n corresponds to a polygon with n + 3 sides). Various transforms can be applied to the polygons randomly each time step. Samples shown correspond states and
image of polygon (if the states were numbered beginning from
22

---

Under review as a conference paper at ICLR 2021
G MORE EXPERIMENTS AND ADDITIONAL REWARD PLOTS
We continue with the experiments and results from the main paper here.
G.1 1
RESULTS FOR VARYING TRANSITION AND REWARD NOISES
Newerd
Roward
Hlwward
50
50
0.02
0.01
0.25
0.0
0.02
transttion
(a) DQN
(b) Rainbow
(c) A3C
(d) A3C LSTM
Figure 9: Mean episodic reward at the end of training for the different algorithms when varying
transition noise. Error bars represent standard deviation. Note the different reward scales.
Reward
Buwwadd
50
25
10
30
25
10
25
reward noise
nolse
reward
DQN
(b) Rainbow
(c) A3C
(d) A3C LSTM
Figure 10: Mean episodic reward at the end of training for the different algorithms when varying reward noise. Error bars represent standard deviation. Note the different reward scales.
We see a similar trend during training. as for delays and sequences, when we vary the transition noise Figure the reward noise in Figure TO] Performance degrades gradually as more and more noise is injected. It is interesting that, during training, all the algorithms seem to be more sensitive to noise in the transition dynamics compared to the reward dynamics: transition noise values as low as 0.02 lead to a clear handicap in learning while for the reward dynamics (with the reward scale being 1.0) reward noise standard deviation of or
still resulted in learning progress.
Rewnrd
100
50
0.0
0.01
0.02
0.25
0.0
0.01
0.02
0.1
0.25
0.0
0.01
0.1 brnsition molse
0.02
0.25
0.0
0.01 0.02
0.1 0.25
Danition neise
banskion
brankKinn nosse
(a) DQN
(b) Rainbow
(c) A3C
(d) A3C LSTM
Figure 11: Mean episodic reward for evaluation rollouts (max 100 timesteps) at the end of training for the different algorithms when varying transition noise. Error bars represent standard deviation.
Interestingly, when we plot the evaluation performances Figures and 12 we see, comparing with the training plots, that the training performance of the algorithms is more sensitive to noise in the transition dynamics (Figure 9 than the eventual evaluation performance (Figure TT]. While is obvious that the mean episodic reward during training would be perturbed when noise is injected into the reward function, it non-trivial that injecting noise into the transition function still leads to good learning (as displayed in the evaluation rollout plots). An additional seeming anomaly is that the evaluation rollouts for A3C variants especially (and DQN to a small extent), suggest that it performs better in the presence of transition noise. This might indicate that A3C in the presence of no transition noise does not explore enough (as was also conjectured in the unexpected results for varying the sparsity meta-feature) and is actually helped when transition noise is present during training.
As mentioned earlier, one of the advantages of platform is that it allows to introduce all the hardness dimensions on the same base environment at the same time. This is helpful to understand interaction effects between them. We plot the most interesting interaction effects in Figure 13 where we varied both transition and reward noise over respective grid values. This plot shows that our
Here, for evaluation, and not for training hecause training is in the noisy environment, evaluated in the corresponding environment without noise assess how well the true learning proceeding.
23

---

Under review as a conference paper at ICLR 2021
Revard
50
15
Reward
50
50
reward_nolse
(a) DQN
(b) Rainbow
(c) A3C
(d) A3C LSTM
Figure 12: Mean episodic reward for evaluation rollouts (max 100 timesteps) at the end of training for the different algorithms when varying reward noise. Error bars represent standard deviation. Note the different reward scales.
observation, that transition noise helps A3C out during evaluation, is only clearly valid when the reward noise not so high (Ør_, <= 1) as disrupt training. The corresponding heatmap plot for training when varying the noises and additional ones for jointly varying delay and rewardable sequence length are present in the Appendix (Figures 37
0.01
0.01
trandibin, nelee
0.02
0.02
0.02
0.02
tranviton:
tranettin,
6.1
tranditinn
0.1
0.25
0.25
0.25
6.25
20
10
15
rwward nosss
rewand pube
Iwarard JISS
reward cobe
(a) DQN
(b) Rainbow
(c) A3C
(d) A3C LSTM
Figure 13: Mean episodic reward for evaluation rollouts (max 100 timesteps) the end of training for the different algorithms when varying transition and reward noise.
G.2 RESULTS FOR VARYING REWARD SPARSITY
Reward
50
Reward
50
Reward
Reward
50
0.25
0.5
0.75
0.25 0.5 0.75
0.25
0.5
0.75
0.25
0.5
0.75
reward_ denastd
reward reward_density
reward reward_density
reward reward_dense density
DQN
(b) Rainbow
(c) A3C
A3C LSTM
Figure 14: Mean episodic reward at the end of training for the algorithms when varying reward sparsity. Error bars represent standard deviation. Note the different reward scales.
Figure 14 shows the results of controlling the meta-feature sparsity in the environment. The DQN variants able to learn the important rewarding states in the environment even when these sparse while the behaviour of A3C was unexpected. One explanation could be that A3C's exploration was not very good, in which case increasing reward density would help as in Figure [4c] But adding in an LSTM to the A3C agent seems to show the opposite trend (Figure [1d] as increasing reward density leads to worsening performance. This could indicate that having greater density of rewarding states makes it harder for the LSTM to remember one state to stick to. This behaviour of A3C warrants more investigation in the future.
We have observed A3C is more variant in general than its DQN counterparts and this should be expected as it launches and collects data from several instances of the same environment which induces more variance.
The make_ make_denser configuration option, makes learning smoother and less variant across different runs of an algorithm, as can be seen in Figure 15a for DQN when compared to Figure 17 for corre- sponding sequence lengths. To evaluate the true learning of algorithms, we turn off the make denser option in the evaluation rollouts. The learning curves for these be for DQN in Figure [5b] The agent still does not perform as well as might be expected when making the reward signal denser during training. This probably due the sequence lengths still violating the complete observability
24

---

Under review as a conference paper at ICLR 2021
assumption made by the algorithm. The plots for learning curves for the remaining algorithms
present in Figures 75] 75 The plots for final mean reward during training and evaluation are given in
Figures 14 and 21
SONASTAC Smalt
SSHAM0IOO Jengh
(a) Training Learning Curves
EEQUEncE lagg
MEQLEnce langth
SECUMITC largth
(b) Evaluation Learning Curves
Figure 15: Learning curves for DQN when make denser is True for rewardable sequences. Please note the different Y-axis scales and the fact that with longer rewardable sequences, greater number of seeds do not learn anything for the evaluation rollouts (reward R 0).
G.3 FURTHER RESULTS FOR VARYING REWARD DELAYS AND SEQUENCES
Reward
Reward
daay
Sequence Jength
(a) A3C + LSTM delay (b) A3C + LSTM seq len
Figure 16: Mean episodic reward the end of training for different agents for varying delays (top) and sequence lengths (bottom). Error bars represent standard deviation. Note the reward scales.
sequence length
sequence length 2
sequence length 3
sequence length 4
60
20
Reward
Roward
Reward
Reward
5000 10000 15800 20000 Train Timesteps
5000 10000 15000 20000 Train Timestaps
5000 10000 15000 20000 Train Timestags
5000 10000 15000 20000 Train Timesteps
Figure 17: Train Learning Curves for 10 runs with different seeds for DQN when varying sequence lengths. Please note that each different colour corresponds to one of seeds each subplot.
Note that we varied delay on a logarithmic scale and sequence length on a linear one which means that this effect is more pronounced than may first appear when looking the figures. We additionally also plot the learning curves, when varying sequence lengths, in Figure 17 We see how training proceeds much more smoothly and is less variant across different seeds for the vanilla environment (where the sequence length 1) and that the variance across seeds is very large for sequence length 3.
G.4 SELECTING TOTAL TIMESTEPS FOR RUNS
We ran the experiments and plot the results for DQN variants up to 20 000 environment timesteps and the ones for A3C variants up to 150 000 time steps since A3C took longer to learn as can be seen Figure 18 We refrain from fixing a single number of timesteps for our environments (as, e.g., bsuite does), since the study of different trends for different families of algorithms will require different numbers of timesteps. Policy gradient methods such as A3C are slower in general compared to value-based approaches such as DQN. Throughout, we always run 10 seeds of all algorithms to
'In terms of environment steps. Wallclock time used still similar.
25

---

Under review as a conference paper at ICLR 2021
DON ABC
Reward
50
10
10 Train timesteps
Figure 18:
Evaluation rollouts (limited to 100 timesteps per episode) for DQN and A3C in the vanilla environment which shows that DQN learns faster than A3C in terms of the number of timesteps.
obtain reliable results. We repeated many of our experiments with an independent set of 10 seeds and obtained the same qualitative results.
Roward
Roward
Reward
Reward
50
delay
delay
(a) DQN
(b) Rainbow
A3C
(d) A3C LSTM
Figure 19: Mean episodic reward for evaluation rollouts (limited to 100 timesteps) at the end of training for the different algorithms when varying delay. Error bars represent standard deviation.
Reward
Reward
50
Reward
50
Reward
50
sequence_lenge
sequence_length
sequence_
(a) DQN
(b) Rainbow
(c) A3C
(d) A3C LSTM
Figure 20: Mean episodic reward for evaluation rollouts (limited to 100 timesteps) at the end of
training for the different algorithms when varying sequence lengths. Error bars represent 1 standard deviation.
Reward
Reward
Reward
Reward
100
50
50
0.25 0.5 0.75
0.25
0.5
0.75
0.25
0.5
0.75
reward_odd reward_density
reward density
reward -eward_density
reward reward_density
(a) DQN
(b) Rainbow
(c) A3C
(d) A3C LSTM
Figure 21: Mean episodic reward for evaluation rollouts (limited to 100 timesteps) at the end of training for the different algorithms when varying reward sparsity. Error bars represent 1 standard deviation.
Reward
Reward
Reward
Reward
200
250
250
200
0
0
0
0
sequence length
sequence length
sequence Jength
sequence length
(a) DQN
(b) Rainbow
(c) A3C
(d) A3C L.TMM
Figure 22: Mean episodic reward the end of training for the different algorithms when make denser is True for rewardable sequences. Error bars represent standard deviation.
26

---

Under review as a conference paper at ICLR 2021
Reward
Reward
Reward
Reward
250
250
200
0
0
0
0
sequence _lenth
sequence length
sequence Jength
sequence length
(a) DQN
(b) Rainbow
(c) A3C
(d) A3C LSTM
Figure 23: Mean episodic reward for evaluation rollouts (limited to 100 timesteps) at the end of training for the different algorithms when meake_dense is True for rewardable sequences. Error bars represent standard deviation.
100
Reward
Reward
Reward
50
50
none
(0.
08 3833990
22 F1
image_transfore transforms
A3C LSTM
(b) DQN scale
DQN rotate
Figure 24: Mean episodic reward at the end of training for the different algorithms when varying representation learning. 's' represents shift, 'S' represents scale, 'f" represents flip and 'r' represents rotate in the labels in the first subfigure. scale_range represents scaling ranges in the second subfigure. image_ re__ro_oor ro_quant is represents quantisation of the rotations in the third subfigure. Error bars represent 1 standard deviation.
0.05
0.1
0.25
0.5
1.0
2.0
4.0
8.0
0.1
0.2
0.5
1.0
2.0
4.0
8.0
100
Sarper
medius
urst
sant JONCE cim
(a) target radius rew.
action max rew.
(c) time unit rew.
(d) irr. dims, rew.
apxxdb man
300
0.05
U.1
5:25
0.1 525
1.5
0.5
10
(e) target radius len.
(f) action max len.
(g) time unit len.
(h) irr. dims. len.
Figure 25: Mean episodic reward (above) and lengths (below) for TD3 at the end training. Error bars represent 1 standard deviation.
0.55
01
0.55
3.5
0.1
0.25
0.5
1.0
2.0
4.0
8.0
011
0.2
0.5
1.0
4.0
8.0
10
Cangerool
30800
5p302
max
3m0
(a) target radius rew.
(b) action max rew.
(c) time unit rew.
(d) irr. dims. rew.
Kode Ae Team
19200010 Jon mncon
50
0.25
0.5
1.0
2.0
4.0
8.0
24.0n 20000
maa
Um9cum
unt
$1202 100000000 dm
(e) target radius len.
(f) action max len.
(g) time unit len.
(h) irr dims. len.
Figure 26: Mean episodic reward (above) and lengths (below) for SAC at the end of training. Error bars represent standard deviation.
27

---

Under review as a conference paper at ICLR 2021
0 0
Reward
Reward
0.01
0.1
1.0
5.0
10.0
100.0
0.01
0.1
1.0
5.0
10.0
100.0
0.01
0.1
1.0
5.0
10.0
100.0
action leson_ssss_____ weight
action
weight
aCt on wwwoonnoosoooooooooooooooooooooe jess_
(a) DDPG episode
(b) TD3 episode rew.
SAC episode rew.
episode Jen mean
100
1000
100
episoce len
episode,
0.01
0.1
1.0
5.0
10.0
100.0
0.01
0.1
1.0
5.0
100.0
0.01
0.1
100
5.0
10.0 100.0
action, loss weight
action lass weight
action loss, jeigtt
(d) DDPG episode len.
(c) TD3 episode len.
(f) SAC episode len.
Figure 27: Mean episodic reward (above) and lengths (below) at the end of training for evaluation rollouts for DDPG, TD3 and SAC when varying action_ Joss_lossssee Error bars represent standard deviation.
Roward
10-
Reward
0.01
0.1
5.0
10.0 100.0
0.01 0.1
5.0 10.0 100.0
action_ loss, weight
action scdoo_loss_Weeee weight
act on Seciin_los_____o_ _on_loss___ihhhl
(a) DDPG episode rew.
(b) TD3 episode rew.
(c) SAC episode rew.
Seodde__lee episccu len mean
100
100
Seroooo len
action_ Joss, weight
action lass weight
action action_loss bos__wiight
(d) DDPG episode len.
(c) TD3 episode len.
(f) SAC episode len.
Figure 28: Mean episodic reward (above) and lengths (below) the end of training for evaluation rollouts for DDPG, TD3 and SAC when varying action loss weight.. Error bars represent 1 standard deviation.
Roward
10
Reward
0.01
1.0
5.0
100.0
0.01 0.1
1.0
5.0
10.0
0.01
0.1
1.0
5.0
10.0
100.0
less weight
action_loss weight
actior_los_______
(a) DDPG episode rew.
(b) TD3 episode rew.
(c) SAC episode rew.
walsode__eee. Jen. mnean
episode_ len
sploode len,
0.01
5.0
1.0
5.0
5.0 10.0 100.0 weight
action weight
weight
(d) DDPG episode len.
(e) TD3 episode len.
(1) SAC episode len.
Figure 29: Mean episodic reward (above) and lengths (below) at the end of training for evaluation rollouts for DDPG, TD3 and SAC when varying action loss_lossseee weight, Error bars represent standard deviation.
28

---

Under review as a conference paper at ICLR 2021
0 0
Reward
Reward
0.01
0.1
1.0
5.0
10.0
100.0
0.01
0.1
1.0
5.0
10.0
100.0
0.01
0.1
1.0
5.0
10.0
100.0
action leson_ssss_____ weight
action
weight
aCt on wwwoonnoosoooooooooooooooooooeoe jess_
(a) DDPG episode
(b) TD3 episode rew.
SAC episode rew.
episode Jen mean
100
1000
100
episoce len
episode,
0.01
0.1
1.0
5.0
10.0
100.0
0.01
0.1
1.0
5.0
100.0
0.01
0.1
100
5.0
10.0 100.0
action, loss weight
action lass weight
action loss, joigtt
(d) DDPG episode len.
(c) TD3 episode len.
(f) SAC episode len.
Figure 30: Mean episodic reward (above) and lengths (below) at the end of training for evaluation rollouts for DDPG, TD3 and SAC when varying action_ Joss_loiisee Error bars represent standard deviation.
Roward
10-
Reward
0.01
0.1
5.0
10.0 100.0
0.01 0.1
5.0 10.0 100.0
action_ loss, weight
action scdoo_loss_Weeee weight
act on Seciin_los_____o_ _on_loss___whhhl
(a) DDPG episode rew.
(b) TD3 episode rew.
(c) SAC episode rew.
Seodde__lee episccu len mean
100
100
Seroooo len
action Joss, weight
action lass weight
action action_loss bos__wiight
(d) DDPG episode len.
(c) TD3 episode len.
(f) SAC episode len.
Figure 31: Mean episodic reward (above) and lengths (below) the end of training for evaluation rollouts for DDPG, TD3 and SAC when varying action loss weight.. Error bars represent 1 standard deviation.
Roward
10
Reward
0.01
1.0
5.0
100.0
0.01 0.1
1.0
5.0
10.0
0.01
0.1
1.0
5.0
10.0
100.0
less weight
action_loss weight
actior_los_______
(a) DDPG episode rew.
(b) TD3 episode rew.
(c) SAC episode rew.
walsode__eee. Jen. mnean
episode_ len
sploode len,
0.01
5.0
1.0
5.0
5.0 10.0 100.0 wei_ht
action weight
weight
(d) DDPG episode len.
(e) TD3 episode len.
(1) SAC episode len.
Figure 32: Mean episodic reward (above) and lengths (below) at the end of training for evaluation rollouts for DDPG, TD3 and SAC when varying action loss_lossseee weight, Error bars represent standard deviation.
29

---

Under review as a conference paper at ICLR 2021
H
PLOTS FOR TABULAR BASELINES
Reward
50
Reward
50
Reward
50
0
2 delay
4
8
0
1
4
8
2 delay
8
delay
(a) Q-L.armnig
(b) SARSA
(c) Double Q-Learning
Figure 33: Mean episodic reward (limited 100 timesteps) at the end of training for three different
tabular baseline algorithms when varying reward delay. Error bars represent standard deviation.
50
50
50
Reward
Reward
Reward
2
2
4
sequence length
sequence length
sequence _enctthh
(a) Q-Leaming
(b) SARSA
(c) Double Q-Learning
Figure 34: Mean episodic reward (limited to 100 timesteps) at the end of training for three different
tabular baseline algorithms when varying sequence length. Error bars represent standard deviation.
50
50
50
Reward
Reward
Reward
0
10
25
0
1
10
25
5 10
25
reward noise
reward noise
reward _ooise
(a) Q-Learning
(b) SARSA
(c) Double Q-Learning
Figure 35: Mean episodic reward (limited to 100 timesteps) at the end of training three different tabular baseline algorithms when varying reward noise. Error bars represent standard deviation.
50
50
50
Reward
Reward
Reward
0.0
0.01 transition_noiia
0.02
0.1
0.25
0.0
0.01
0.02 0.1
0.25
0.0
0.01
0.02
0.1
0.25
transition_noise
transition_
(a) Q-I.carnin
(b) SARSA
(c) Double Q-Learning
Figure 36: Mean episodic reward (limited 100 timesteps) at the end of training for three different tabular baseline algorithms when varying transition noise. Error bars represent standard deviation.
30

---

Under review as a conference paper at ICLR 2021
1
PLOTS FOR VARYING 2 HARDNESS DIMENSIONS TOGETHER
80
60
60
paniwoy
60
60
palway
40
8
2
2
sequence_ Jenghh
sequence sequence_length
sequence_ LLnnghh
sequence_lengee
(a) DQN
(b) Rainbow
(c) A3C
(d) A3C LSTM
Figure 37: Mean episodic reward at the end of training for the different algorithms when varying delay and sequence lengths. Please note the different colorbar scales.
80
80
2-
-60
60
delay
delay
60
8-
20
20
2
2
2
sequence_jenge Jenggh
sequence sequence_length
sequence_iene Jength
sequence_lengee
(a) DQN
(b) Rainbow
(c) A3C
(d) A3C LSTM
Figure 38: Mean episodic reward for evaluation rollouts (limited to 100 timesteps) the end of training for the different algorithms when varying delay and rewardable sequence lengths. Please note the different colorbar scales.
0.01
tranaiion pode
bateille puise
tranertionn pode
0.02
0.02
0.02
9.02
preweng
0.1
40
0.1
0.1
0.3
20
20
20
0
-0
reward notse
reward noise
reward nosee
reward noise
(a) DQN
(b) Rainbow
(c) A3C
(d) A3C + LSTM
Figure 39: Mean episodic reward the end of training for the different algorithms when varying transition noise and reward noise. Please note the different colorbar scales.
31

---

Under review as a conference paper at ICLR 2021
1
20
M90 pas piewas
1
100 Saban
delay
4
10
delay
10
Dev Pr
1
pag AR
2
sequence_ lenghh
sequence Jength
sequence Jength
sequence_lnnge
(a) DQN
(b) Rainbow
(c) A3C
(d) A3C LSTM
Figure 40: Standard deviation of mean episodic reward at the end of training for the different algorithms when varying delay and sequence lengths. Please note the different colorbar scales.
1
25
20
"ARO piewas
15
PS Popoa
15
4-
15
1A0 delsy
delay
10
2
sequence_lenge length
sequence length
sequence length
sequence Jenuth
(a) DQN
(h) Rainbow
(c) A3C
(d) A3C LSTM
Figure 41: Standard deviation of mean episodic reward for evaluation rollouts (limited to 100 timesteps) at the end of training for the differrnt algorithms when varying delay and rewardable sequence lengths. Please note the different colorbar scales.
0.01
36
transtton, natee
tranrttionnn nolde
'100 Piš piewiig
tranation octie
NnO PS Dreann
0.02
0.02
0.02
0.02
Po
0.1
0.1
16
925
10
10
mmnrrd nolee
rewand reward_roise noise
reward noie
rewannd rolse
(a) DQN
(b) Rainbow
(c) A3C
(d) A3C LSTM
Figure 42: Standard deviation of mean episodic reward at the end of training for the different algorithms when varying transition noise and reward noise. Please note the different colorbar scales.
0.02
36
tonsition rels
0.02
dag pis Saway
trandtinn se00
0.01
0.02
10.03
traehtiee relee
0.01
0.02
wng PIS Slooa
tranatio
8.1
2o
0.1
7.5
163
0.25
0.55
0.25
2.5
10
10
10
-000
10
reward_netre
roward roward_nolse noise
roward nolse
roward_rot wward_rolsc
(a) DQN
(b) Rainbow
(c) A3C
(d) A3C LSTM
Figure 43: Standard deviation of mean episodic reward the end of training for evaluation rollouts (limited to 100 timesteps) at the end of training for the different algorithms when varying transition noise and reward noise. Please note the different colorbar scales.
32

---

Under review as a conference paper at ICLR 2021
0.02
0.02
0.02
transition _oois
transition noise
80
Uesu Ua] appssss
transition noise
80
Reeead
0.1
60
р.eeay
transition
0.25
0.25
40
0.25
0.25
40
2
2
Ueaw
0.5
0.5
20
0.5
20
0.25
0.5
0.25
0.25 0.5
Bargee_radiuu
target radius
target_radius
target radius
(a) TD3 episode rew.
(b) TD3 episode len.
(c) SAC episode rew.
(d) SAC episode len.
Figure 44: Mean episodic reward and lengths at the end of training for the different algorithms when
varying P noise and target radius.
action 2000 mod
adbit www.e.
0.5
3.0
2.
0.05
0.02
2,05
0.1
(a) DDPG P Order 2
(b) DDPG P Order 3
(c) TD3 P Order 2
(d) SAC P Order 2
Figure 45: Mean episodic reward the end of training for the different algorithms when varying action space max and time unit for given P order.
33

---

Under review as a conference paper at ICLR 2021
J ADDITIONAL LEARNING CURVES
delay sequence length
delay 0, seguenceeeeeeee length
delay sequence length
delay sequence length
60
Rewaard
Roward
Roward
60
50
10
2
5000 10000 15000 20000
5000
10000 15000 20000
5000 10000 15000 20000
5000 10600 15000 20000
Train Timesteps
Train Timosteeps
Train Timestaps
Train Timesteps
delay sequence length
delay sequence length
delay sequence length
delay sequence length
40-
Rewaadd
Reward
40
Reward
50
20
5000 10000 15000 20000
5000 10000 15000 20000
5000 10000 15000 20000
5000 10000 15000 20000
Train Timesteps
Train Timesteps
Train Timesteps
Train Timesteps
delay sequence length
delay 2, sequence length
delay
75
40
Rewardd
Roward
80
Reward
50
50-
20
5000 10000 15000 20000
5000 10000 15000 20000
5000 10000 15000 20000
5000 10000 15000 20000
Troi Timestops
Train Timestops
Train Timestops
Train Timestops
delay
length
delay sequence length
delay length
delay length
75
50
Roward
50
Reward
Reward
60
25
20
5000 10000 15000 20000
5000 10000 15000 20000
5000 10000 15000 20000
5000 10000 15000 20000
Troin Timestops
Train Timestops
Train Timestops
Train Timesteps
delay 8, sequence length
delay 8, sequence length
delay sequence length
delay sequence length
75
16
Rowwrrd
Reward
Reward
20
25
25
5000 10000 15000 20000 Trai Timestops
5000 10000 15000 20000
5000 10000 15000 20000 Train Tlinestops
5000 10000 15000 20000 Train Timesteps
Train Timestops
Figure 46: Training Learning Curves for DQN when varying delay and sequence lengths. Please note the different colorbar scales.
34

---

Under review as a conference paper at ICLR 2021
delay sequence length
delay sequence, length 2
delay O, sequence length 3
delay O, sequence length
100
60
Rewward
Reward 40
Roward
20
20000
Train Timesteps
Train Timesteps
Train Timestaps
Train Timesteps
delay sequence length
delay sequence length 2
delay 1, sequence length
delay 1, sequence length
40
Weoward
Reward
40
20
20
10000
20000
20000
10000
20000
10000
20000
Train Timesteps
Train Timesteps
Train Timesteps
Train Timesteps
delay sequence length
elay elgelay
sequence length
delay 2, sequence length
delay 2. sequence length
100
Roward
Roward
50
20000
10000
20000
10000
20000
10000
20000
Train Timestaps
Train Timestops
Train Timestops
Train Timestops
delay sequence length
delay sequence, length
delay
length
delay 4, sequence length
60
60
Rewarrd
Reward
Rewerd
50
80
20
10000
20000
10000
20000
10000
20000
10000
20000
Train Timestegs
Train Timesteps
Train Timestops
Trai Timestops
delay
length
delay sequence, length
delay sequence, length
delay 8, sequence length
60
50
Rowerr
50
Roward
R0
Roward
20
20
10000 20000
10000 Train Timesteps
20000
10000
20000
10000
20000
Train Timesteps
Train Timesteps
Train Timesteps
Figure 47: Evaluation Learning Curves for DQN when varying delay and sequence lengths. Please note the different colorbar scales.
35

---

Under review as a conference paper at ICLR 2021
trandtion, noise D rewart moteetnneenn nolse revant miserfnnstion, polse rewant postedrstioo, poke pwwrd_rohvrihyle poke reaard nals
20000
20303
10030
1003
20030
23650
30100
22000
trameliann roise 0.01, reward nolewbsllan nolve 01. rewand noitmbsiion noise 61, revard polrebiiion, mise c1, rodestkiiie make 0.01, ruive 25
23600
10300 Than Tireenss
Tral Tmmetgee
Thai Timetege
Than Timeshorr
Tran Timeewor
Trangten_note 0.02, reward nolime wwwnneettnnnnteeee noise 02. rewant noltmistiion notee 6 02 rewant nommriitton polse reant rolmmbitirr prke 0.02, rewere 25
Nowsrd
20000
20303
10030
20030
20030
23600
10300
22000
tranetionn noie reward moisetamet noise rwoadd moserianell neise 0.1, rwaand àrsiiion noks swward rohserbbsiiien pase temerd nolv
30
sel
23860
UP3E3
Tral Timerage
Thai Eimetege
Thai Timeshoor
Than Timeewor
The Tiirensss
transtion noisuesiiinn
noisuisition 25, nobrrSitionn
codematition nose
1000
20003
10:30
20355
10336
200te
10300
23100
10320
25006
Figure 48: Training Learning Curves for DQN when varying transition noise and reward noise.
bion, noise reward moliderraa nolse reeord nristeerstiion roise rewerd poidedtutlion
remerd
neive reward poise
10
[c0)
120
Roourd
Runord
Honard
Nowe
1030
34000
10436
20000
1000 Timelago
22000
manag
tronstion_nnee 6.01,
brttctionnn 01. rewars_ natm tetion_notse 01, reward noter notse 150
totttttttnnnntte noise 01, rewans_ndee
Hunurd
50
#2
1990
20000
1001
20000
10130 Timettos
23000
10000
23000
1807 Train
20060
Tran
Trair
tronstion hoice 02. reword zo robitttttonn noise 02, reward rultalitioon noice 0. 02, reward notso feation noise & 02 reward ro brotttion noise & 02, rewars noise 25
Ronard
Hemurd
Tas mecrps
Timeorpe
Timedrpe
neistransition
appttoo
100
120
:00
Rhoaad
10303
22003
10002
200co
20000
20000
10000
20000
10000
22000
tronstion nolge 0.25, reword, boatsttonn noice 0.25, reward, nobra toonnattoooooooo noise 25,5 reward notoortation, noice 6.25 rewers no boagogoonn noise 0.25, rewars nelie 25
Rawerd
rowurd
30
TranT 20
Trai mesuu
Tran meeduu
Tru i
Timerage
Thann Timutago
Figure 49: Evaluation Learning Curves for DQN when varying transition noise and reward noise. Please note the different Y-axis scales.
36

---

Under review as a conference paper at ICLR 2021
delay O, sequence length
delay O, sequence length
delay O, sequence length
delay sequence length
75-
75-
60
60
Reward
Reward
Reward
60
50
23
25
5000 10000 15000 20000
5000 10000 15000 20000
10000 15000 20000 Train Timestaps
5000 10000 20000
Train Timesteps
Train Timestops
Train Timestaps
delay 1, sequence length
delay 1, sequence length
delay 1, sequence length
delay sequence length
60
50
Ruwward
50
Reward
40
Reward
20
20
5000 10000 15000 20000 Train limesteps
5000 10000 15000 20000 Timestopps
10000 15000 20000 Train Timesteps
5000 10000 20000
Train Timesteps
delay 2, sequence length
delay 2. sequence length
delay sequence length
delay sequence length
75-
75
60
60
Roward
50
Reward
40
Reward
25
20
20
5000 10000 15000 20000
5000 10000 15000 20000
20000 15000 20000
5000 10000 20000
Train Timestops
Train Timestops
Train Timestops
Train Timestaps
delay 4, sequence length
delay 4, sequence length
delay sequence length
delay sequence length
60
60
50-
Reward
80
Roward
40
Reward
25
20
20
5000 10000 15000 20000
5000 10000 15000 20000
10000 15000 20000
5000 10000 20000
Train Timesteps
Train Timestops
Train Timestups
Train Timesteps
delay 8, sequence length
delay 8, sequence length
delay sequence length
delay sequence length
75
50
Rewaard
Reward
40
Reward
40
20
5000 10000 15000 20000 Train Timestops
5000 10000 15600 20000
5000 10000 15000 20000
5000 10600 15000 20000 Train Timesteps
Train Timestops
Train Timestops
Figure 50: Training Leamingg Curves for Rainbow when varying delay and sequence lengths. Please note the different Y-axis scales.
37

---

Under review as a conference paper at ICLR 2021
delay sequence length
delay sequence length 2
delay O, sequence length 3
delay o, sequence length
75
Rowward
Reward 25
50
Roward
25
20000
Train Timesteps
Train Timesteps
Train Timestaps
Train Timesteps
delay sequence length
delay 1, sequence length 2
delay 1, sequence length 3
delay 1, sequence length
100
100
Ruwwardd
Roward
Reward
40
40
10000
20000
10000
20000
10000
20000
10000
20000
Train Timesteps
Train Timesteps
Train Timesteps
Train Timesteps
delay
sequence length
delay sequence length
delay 2, sequence length
delay 2. sequence length
100
100
Roward
Reward
Roward
10000
20000
10000
20000
10000
20000
10000
20000
Train Timestaps
Train Timestops
Train Timestops
Trai Timestops
delay
sequence length
delay sequence, length
delay sequence length
delay 4, sequence length
60
60
50
Rewardd
Reward
Rewerd
40
20
10000
20000
10000
20000
10000
20000
10000
20000
Train Timestegs
Train Timesteps
Train Timesteps
Train Timestops
delay sequence length
delay sequence, length
delay sequence length
delay 8, sequence length
60
60
Reward
40
Roward
Roward
8g.
20
20
10000 Train Timesteps
20000
10000
20000
10000
20000
10000 Train Timesteps
20000
Train Timesteps
Train Timesteps
Figure 51: Evaluation Learning Curves for Rainbow when varying delay and sequence lengths. Please note the different Y-axis scales.
38

---

Under review as a conference paper at ICLR 2021
traretiion noie 9.0 reward mossetimeetnn nois U.O. reland rossranslen, niise C.O, renand nosk postersiiion noss D.G, rwward reiseibbsiion mose D.C, teward rolve
Renard
Rewand
St
Reaard
Timatage
Timebagge
Timemwo
nissuustion
rosmmtttion
53
1003
1001s
20100
13710 Timatouss
13118
23000
LEGO
20000
Train
Thai
Timestoas
The
tronsten_noise 0.02. reward noltoisttonn noise 02. reward, noltmisttion noice 0 02. reward notrodiition rose 0.02, rewars wwwnss_rottttttt redmmttttion nose 0.02, rewars_rebs:
1003
X030
1003
13635
3310
13066 Tinzees
33000
70000
Tmmlage
Timekaa
transtion, noise rewant_ not noise 1.1. mosernnstenn
restrirstioonnen
33
Rewart
1000 Iran Timsttgs
12002 Tran mimettoss
12006 tran Timestos
23000
12000 irreseess
Tran Timestoss
tronsten_roise: 0.25, reward, nolsmesttonn noise 25, reward, nolmmisttion noice 0 25, reward notrodsition nose 0.25, rewars rodtmitttionn nosse 0.25, reward rolse 25
12
1903 Tran Emalog
X030
1003E
30300
13638 Timadous
33260
13026
33000
13620 Tinassss
20000
Tran Timolag
Figure 52: Training Learning Curves for Rainbow when varying noises. Please note the different Y-axis scales.
reword_nobtree stonnnotse rewaard_noistrrdiooeo www.sontoostttto..o...o reward www.oroogoooooooooooooo reward_notster www.roottttttttn..eooo reward_nnose 120)
200
Raward
haward
10:00
20003
20003
13356
20000
1000
20000
10:97
25400
tronstion nolse 01. reword no brattttoon noice 0.0L rewars, nolira to noise o ol reward notso rerrrtation noise 6.01 rewers bradttionn noise 0.01 rewers neice 25
10300 Tran mevengs
Trai mieepa
Trui merees
Tra A Timeewar
Trish Tinestag:
transtion nentte rewerd tobemttttonnnn noise 02. reward, nobe to toonunotee te o oz, reward mnterffttionnneiie 02. reward wwwooronnnnnnn monooonneeeeeeeeeeoooeeoo & 02, rewant poiie 25
Ranurd
Ramurd
20003
19035 mastppa
2000
20000
25000
Tiemrsss
Timettos
Timatass
Timestops
tronsttoon
nobtrensttinn 0.1. reward polstrinnction, noise 1. reward no ardiiittoon
rolstratbttion reward noise
300
hawer
10300
30003
Trai
30535
20002
12000
Timewos
20000
30020
20000
10303
25000
transiion noise 0.25, reward no tokubiition noise 25,5 reward re de kion neise O 25, reward nobsrtsition neise 2.25, reward ro boddoon neise 6 reward noise 25
Dortrd
Ronard
Tran messUa
Trai mestoss
Tro Timessss
Trah Timostap
Figure 53: Evaluation Learning Curves for Rainbow when varying noises. Please note the different Y-axis scales.
39

---

Under review as a conference paper at ICLR 2021
delay 0, sequence length
delay 0, sequence length
delay sequence length 3
delay O, sequence length
100
75
Reward
30
Rewann
8
8
Train Timesteps
Train Timestaps
Train Timesteps
Train Timesteps
delay 1, sequence length
delay 1, sequence length
delay 1, sequence length 3
delay 1, sequence length
Reward
Reward
20
Reward
Roward
80
10
100000 150000
50000 100000 150000
50000 100000 150000
50000 100000 150000
Train Timesteps
Train Timestaps
Train Timesteps
Train limesteps
delay 2, sequence length
delay sequence langth
delay sequence length
delay 2. sequence length
40
Roward
Roward
Roward
Roward
10
6
50000 100000 150000
100000 150000
50000 100000 150000
50000 100000 150000
Train Timestops
Train Timestaps
Train Timestops
Trai Timestops
delay 4, sequence length
delay
sequence length
delay sequence length
delay 4, sequence length
20
Reward
Reward
55
2
50000 100000 150000
100000 150000
50000 100000 150000
50000 100000 150000 Trai Timestops
Train Timesteps
Train Timestaps
Train Timestego
delay 8, sequence length
delay sequence length
delay sequence, length
delay 8, sequence length
0.04
Roward
Rewaard
Roward
0.03
10
20
0.02
0.01
50000 100000
150000
50000 190000 150000 Train Timestaps
50000 100000
150000
50000
100000 150000
Train Timestops
Train Timestops
Train Timestops
Figure 54: Training Learning Curves for A3C when varying delay and sequence lengths. Please the different Y-axis scales.
40

---

Under review as a conference paper at ICLR 2021
delay O, sequence length
delay sequence length
delay sequence length
delay sequence length
40
20
Roward
Reward
Rewwand
20
100000 250000
100000 150000
100000 150000
50000 100000 150000
Train Timesteps
Train Timesteps
Train Timestops
Train Timestaps
delay 1, sequence length
delay
sequence length
delay sequence length
delay sequence length
30
7.5
Reward
20
Reward
Weward
1.
2.5
100000
250000
50000 100000 150000
100000 150000
100000 150000
Train Timesteps
Train T'ieestpss
Train Timesteps
Train Timesteps
delay sequence length
delay sequence length
delay sequence length
delay sequence length
100
15
e0
Roward
Roward
10
Reward
5.0
50
2.5
50000 100000 150000
100000
150000
100000 150000
50000 100000 150000
Train Timestips
Train Timestops
Train Timesteps
Train Timesteps
delay sequence length
delay sequence length
delay sequence length
delay sequence length
20
10
Reward
Roward
Newer
50000 100000
150000
100000
150000
50000 100000 150000
50000 100000 150000
Train Timestops
Train Timesteps
Train Timestaps
Train Timesteps
delay sequence length
delay sequence length
delay sequence length
delay sequence length
20
60
20
Rowerd
10
Roward
Roward
Neward
10
50000
100000
150000
50000
100000
150000
50000 100000
150000
50000
100000
150000
Train Timestops
Train Timesteps
Train Timestaps
Train Timesteps
Figure 55: Evaluation Learning Curves for A3C when varying delay and sequence lengths. Please note the different Y-axis scales.
41

---

Under review as a conference paper at ICLR 2021
trop notee rewars, rww.rrseeesteeeeeeeerreeee storreesttoooooeoo nolbe reward www.ernnttonnnnseeee.. reword derdiittonnnoooo perdiittooonnreeeeeeeeeee nolse reward_ costrabationnno nolee www.rssnoiss.....
100)
2011
Rewand
nowerd
1.0
8
50075 200807 139000
5033 J0009 15970
50393 1000300 150995
5009SS 200000 153920
SOUTS 200007 330000
160
neise 0.01 reward relsatttion nokee 6.01 reward nolom tion noise 0 6.0 reward nobserBition noise 6.01, rewars rosm noise rewars raise
100
100
Ronurd
8
30000E
50002 Timesooss
Tann reesings
Trar me
Than meetens
Tra Thmessigs
tranailion ion, swmard nuteelle neluw reward nollee tion moise 02, reward molae
rweard cedeebiii nobon ruise
200
20
Rowars
500GE 160003 1500c0 mexoaao
50030 20000E 150000
50363 120000 250002
50003 200000 153000
50OGS 200000 120000
Than TeEsAA
Timesess
tranetion, nakee reward mrinte noten roward moistrinst noise roward no notee reward woiseranation noioe C.
20
8 AAN
GOITE TO00S 1903co
50933 TrEWE
CoVa33 Teekan yoobae
CANSS 160000 15300e
KAWVE 100000
tronstton, nolze 2.25 reward robrmsttionnnnoo nolse 0.25, reward nolzo www.iitttooonniooeoooeooeeee noice 0. 25, reward, cwtprddooottrttte notsertstioon noice 0.25, TewOrS robiiitionn notre O 2.25, reword paise 25
20)
Reaurd
Reaard
Reamn
Renant
Rossed
-20
Tan meserip
Trar miltip
TYST
Tra Timaaoo
Trs
Figure 56: Training learning Curves for A3C when varying noises. Please note the different Y-axis scales.
nelse reward neistránsition
230
120
100
honad
50005 200003
350000
50000 200000 250000
50003 100030 150000
50003 100030 150000
50000 :00000 153000
tronst on, nolde 0.01. reword ro tiann noice 0.01, rww.re.rooooooooo no bon. nolse 0 01. www.rr/eeetoororrreeeeeee notsortttionn nolie 6.01 rewars robttttiion nolse 0.01. rww.rs.wwdeec. 25
20
TranT Teeeress
Thei meehips
Thin Tememoss
Tra n Timeewar
Tran Timesdaps
reHord
noize
noke
noise 02, reward notseragtion
robetdtion noise
220
1:0
200
Nonard
Reaare
50020 200383 550000
20030E 250000
100030 150000
50002 200030 153000
50000 100000 153000
trarsdicr Kien noise C.I. rwaar zoikee Bn
neise ewoard noisteansitinn maise rwward po Brabion neks reword coiste adhitin neiue C.1. rewardd nolse
120
150
Anwer
2
5003E 16032 250000
1003SE
1000E
20003 100030 193360
10000 100000
Than
Than
transtton, polse 0.25, rewarS Bommiettinnn nolee 0.25, revars no tion nolee 0. 25, reward pobsuitttion, noise 0.25, rewars colmmtriin, noise O. recand prise 25
1007E 35000
1000SE Than mmstiaaa
T50000
153000
Their Timestoss
Tra TimARaas
Than Timnslags
Figure 57: Evaluation Leariing Curves for A3C when varying noises. Please note the different Y-axis scales.
42

---

Under review as a conference paper at ICLR 2021
delay sequence length
delay 0, sequence length 2
gelay O, sequence length
delay sequence length
Reward
Roward
40
20
50
20
150000
100000 150000
50000 150000
50000 100000 150000
Train Timesteps
Train Timestaps
Train Timestops
Train Timesteps
delay sequence length
delay 1, sequence length 2
delay 1, sequence length
delay sequence length
15 to
50
Roward
20
Reward
1o
25.
150000
50000 100000 150000
50000 100000 150000 Timesteps
50000 100000 150000
Train Timesteps
Train Timesteps
Train Timesteps
delay requence length
delay 2, sequence length
delay 2. sequence length
delay sequence length
50.
Roward
Rnward
20
50000
150000
50000 100000 150000
50000 100000 150000
50000 150000
Train Timesteps
Train Timestops
Train Timestops
Train Timestaps
delay sequence length
delay sequence length
gelay sequence length
delay sequence length
20
Reward
Reward
Roverd
20
50000 100000 150000
50000 100000 150000
50000 100000 150000
50000 100000 150000
Train Timestaps
Train Timesteps
Train Timestops
Train Timestaps
delay sequence length
delay sequence length
delay sequence length
delay sequence length
Roward 6 tt
7.5
Rowaard
5.0
20
2.5
50000 100000 150000 Train Timestops
50000 100000 150000
150000 Train Timestops
50000
100000
50000 100000 150000
Train Timestops
Train Timestaps
Figure 58: Training Learning Curves for A3C with LSTM when varying delay and sequence lengths. Please note the different Y-axis scales.
43

---

Under review as a conference paper at ICLR 2021
delay O, sequence length
delay sequence length
delay sequence length
delay sequence length
40
40
20
20
20
150000
150000
50000 200000 150000
50000 150000
Train Timesteps
Train Timestops
Train Timesteps
Train Timeestpps
delay 1, sequence length
delay sequence length
delay sequence length
delay sequence length
Reward
Reward
10
20
10
150000
100000 150000
50000
100000 150000
50000
150000
Train Timesteps
Train Timeestpss
Train Timesteps
Train Timeettps
delay sequence length
delay
sequence length
delay sequence length
delay sequence length
40
10
Rewerd
10
20
50000 100000 150000
50000 100000 150000
0
50000
150000
50000 150000
Train Timesteps
Train Timesteps
Train Timestaps
Train Timestops
delay sequence length
delay sequence length
delay sequence length
delay sequence length
Rowerd
20
20
10
100000
150000
50000 100000 150000
50000 100000 150000
50000 150000
Train Timestaps
Train Timestaps
Train Timestaps
Train Tiiessteps
delay sequence. length
delay sequence length
delay sequence length
delay sequence length
60
20
Rewerd
50
Reward
Reward
Roward
10
50000 100000
150000
50000 100000 150000
50000 100000
150000
50000
100000 150000
Train Timestaps
Train Timestaps
Train Timestaps
Train Timesteps
Figure 59: Evaluation Learning Curves for A3C with LSTM when varying delay and sequence lengths. Please note the different Y-axis scales.
44

---

Under review as a conference paper at ICLR 2021
nolse 6.0. revars nesterrttttttttnoo notse 0.0. www.wa/ www.rrrritttenneeeeeeoo 0.0. reward dordissttonnn
nolse nolse reward, restraaattttaatennn holee 0.0. reward nolse
Ramer
50035 16000: 1300c0
50033 T000E 150360
50003 1:0030 153002
50R03 200000 153000
2000E 160003 150020 Timerraa
Tran maxuato
Than messlg
Timesess
6.01 robsastiion
nolsa
reward notsersitionn
rosuatton
5
&
50033 TOWAEE
S0001 100000 153920 Timastose
SANTEE T000S
tranel inn ncive 02. reward pulletsilaa kim neise 0.02, reward nolse kion metive 0 02, reward moleetb ion nolse 6.02, rMAand wwwaanoooossseeeee poserdller aIr reiee 0.02, ERWMES paise
8 8
4
Tran meeepp
Thai medios
Tra TImeOKKK
transhion
20
Reard
5000S 200005 150020
50023 200003 150000
50003 310000 150000
50003 200000 153000
50002 200000 150000
tronsition_ notse 0.25, reward_ oibaatttoonnn noice 0.25, rewars www.:oooommssoooooo bonnnosee 0 25. rew.rss intsortattoonnn ionn notce 6.25, reward ro noise 0.25, rewons_ndies 25
teond
Sawen 5
-20
Tran TmaXAgg
Tran mESAAA
Tran THEAAA
Tru TImeAss
Tre TunnTimaxaaa
Figure 60: Training Learning Curves for A3C with LSTM when varying noises. Please note the different
Y-axis scales.
transkion neise C.O.
neise C.O neward
reward ionddinttion noise reward noistr ardirditiion neise reward noise
120
100
50020 2003:0 250000
50000 200000 250000
50003 100000 150000
50002 200030 153000 Timexess
50000 100000
153000
tramet ion neise 0.01, twward roleetation nolse 0.01, recard nadew roise 01. recardd potsertition, pedcarddneewrbttiii
noive 5.01, reAAnn coleedotion, noise 0.01, rwccars 25
120
20
50
1003E 160326 550000
50000 T003SE
10003E
20003 100030 195360
50000 1000000
transtion noise 0.02, reword zo koacction noise 0.02. reward rokr Kion neise 0 02, reward notsersition noise O 0202. reward po colotttiin neise & 6.02. reward noise 25
ZED
Trant mescess
Trai mest
Trom meetoos
Tra Timessess
Hram Timestapo
330
120
120
lowad
5003E 100383 352000
50000 10030E 752000
50603 100OLE 150060
50003 200030 153000
70000 100000 153000
transhion ion noise 5.25, reward _n tinn noire 0.25, reward note toon, honnnolse 6 25, rew.ard_potwwttteor pobenttttinn nolse 6.25, reward po noise 0.25 revans, ndise 25
nowurd
TranTmensps
TraT TimeRaas
Ther Timettos
Tra Timassas
Tran Timustopo
Figure 61: Evaluation Learning Curves for A3C with LSTM when varying noises. Please note the different Y-xxis scales.
45

---

Under review as a conference paper at ICLR 2021
reward reward_density 0.25
reward reward_density 0.5
reward density 0.75
75
Reward
Reward
Reward
50
50
50
25
0
5000 10000 15000 20000 Train Timesteps
5000 10000 15000 20000
5000 10000 15000 20000
Train Timesteps
Train Timesteps
Figure 62: Training Learning Curves DQN when varying reward sparsity. Please note different Y-axis scales.
reward density 0.25
reward density 0.5
reward density 0.75
100
100
100
Reward
Reward
Reward
50
50
50
0
10000
20000
10000
20000
10000
20000
Train Timesteps
Train Timesteps
Train Timesteps
Figure 63: Evaluation Learning Curves for DQN when varying reward sparsity. Please note the different
Y-axis scales.
reward density 0.25
reward density 0.5
reward density 0.75
75
Reward
Reward
Reward
50
50
50
25
01
5000 10000 15000 20000 Timesteps
5000 10000 15000 20000 Timesteps
5000 10000 15000 20000 Timesteps
Figure 64: Training Learning Curves for Rainbow when varying reward sparsity. Please note the different Y-axis scales.
reward density 0.25
reward density
reward density 0.75
100
100
100
Reward
Reward
Reward
50
0
10000
20000
20000
20000
Train Timesteps
Train Timesteps
Train Timesteps
Figure 65: Evaluation Learning Curves for Rainbow when varying reward sparsity. Please note different
Y-axis scales.
reward density 0.25
reward density 0.5
reward density 0.75
100
100
100
Reward
Reward
Reward
50
50
50
0
0
0.
50000 100000 150000
50000 100000 150000
50000 100000 150000
Train Timesteps
Train Timesteps
Train Timesteps
Figure 66: Training Learning Curves for A3C. when varying reward sparsity. Please note the different Y-axis scales.
46

---

Under review as a conference paper at ICLR 2021
reward eeward_density 0.25
reward density 0.5
reward density 0.75
100
100
100:
Reward
Reward
Reward
0
0
50000 100000 150000
50000 100000 150000
50000 100000 150000 Train Timesteps
Train Timesteps
Train Timesteps
Figure 67: Evaluation Learning Curves for A3C when varying reward sparsity. Please note the different Y-axis scales.
reward density 0.25
reward density 0.5
reward density 0.75
100
100
100
Reward
Reward
Reward
50
50
50
50000 100000 150000 Train Timesteps
50000 100000 150000
50000 100000 150000 Train Timesteps
Train Timesteps
Figure 68: Training Learning Curves for A3C LSTM when varying reward sparsity. Please note the
different Y-axis scales.
reward density 0.25
reward density 0.5
reward density 0.75
100
100
Reward
Reward
Reward
50
50
50
0
0
0.
50000 100000 150000 Timesteps
50000 100000 150000
50000 100000 150000 Timesteps
Train Timesteps
Figure 69; Evaluation Learning Curves for A3C + I.STM when varying reward spansity. Please note the different Y-axis scales.
sequence length
sequence length
sequence length
300
Reward
Reward
200
Reward
400
50
100
200
0
0
10000
20000
10000
20000
10000
20000
Train Timesteps
Train Timesteps
Train Timesteps
Figure 70: Training Learning Curves for Rainbow when make denser is True for rewardable sequences. Please note the different Y-axis scales.
sequence length 2
sequence length
sequence length 4
300
Reward
Reward
Reward
200
400
50
100
200
0
0
0
10000
20000
10000
20000
10000 Train Timesteps
20000
Train Timesteps
Train Timesteps
Figure 71: Evaluation Learning Curves for Rainbow when make make_denser is True for rewardable sequences.
Please note the different Y-axis scales.
47

---

Under review as a conference paper at ICLR 2021
sequence length 2
sequence length
sequence length
Reward
Reward
200
400
50
100
200
50000 100000 150000 Train Timesteps
50000
100000 150000
0
50000
100000 150000
Train Timesteps
Train Timesteps
Figure 72:
Training Leaming Curves for A3C when make_densee denser is True for rewardable sequences. Please note the different Y-axis scales.
sequence length 2
sequence length 3
sequence length 4
100
Reward
Reward
200
Reward
400
50
100
200
50000 100000 150000 Train Timesteps
50000
100000 150000
50000 100000 150000
Train Timesteps
Train Timesteps
Figure 73:
Evaluation Learning Curves for A3C when make denser True for rewardable sequences. Please note the different Y-axis scales.
sequence length 2
sequence length
sequence length 4
75
Reward
Reward
200
Reward
400
100
200
25
0
0
0
50000 100000 150000 Timesteps
50000
100000 150000
0
50000 100000 150000
Train Timesteps
Timesteps
Figure 74: Training Learning Curves for A3C + I.STM when make denser is True for rewardable sequences.
Please note the different Y-axis scales.
sequence length 2
sequence length
sequence length
Reward
Reward
200
Reward
400
50
100
200
0
0
50000 100000 150000 Train Timesteps
50000
100000 150000
0
50000 100000 150000
Train Timesteps
Train Timesteps
Figure 75: Evaluation Learning Curves for A3C L.STM when make denser True for rewardable se- quences. Please note the different Y-axis scales.
sequence length 2
sequence length
sequence length 4
75
Reward
Reward
200
Reward
400
50
200
0
0
50000 100000 150000
50000 100000 150000
0
50000
100000 150000
Train Timesteps
Train Timesteps
Train Timesteps
Figure 76: Training Learning Curves for A3C + LSTM when make denser is True for rewardable sequences.
Please note the different Y-axis scales.
48

---

Under review as a conference paper at ICLR 2021
sequence length 2
sequence length
sequence length
Reward
Reward
200
400
50
100
200
25
0
0
50000 100000 150000 Train Timesteps
50000
100000
150000
0
50000 100000 150000
Train Timesteps
Train Timesteps
Figure 77: Training Learning Curves for + when make denser is True for rewardable sequences. Please note the different Y-axis scales.
sequence length 2
sequence length
sequence length 4
75
Reward
Reward
200
Reward
400
50
25
100
200
0
50000 100000 150000 Train Timesteps
100000 150000 Train Timesteps
50000
50000 100000 150000
Train Timesteps
Figure 78: Training Learning Curves for A3C + LSTM when make denser is True for rewardable sequences. Please note the different Y-axis scales.
sequence length 2
sequence length
sequence length 4
75
Reward
Reward
200
Reward
400
100
200
25
0
0
0
0
50000 100000 150000 Timesteps
50000
100000 150000
0
50000 100000 150000
Train Timesteps
Timesteps
Figure 79: Training Learning Curves for A3C + I.STM when make denser is True for rewardable sequences. Please note the different Y-axis scales.
sequence length 2
sequence length
sequence length
75
Reward
Reward
200
Reward
400
50
25
100
200
50000 100000 150000
50000
100000 150000
50000 100000 150000
Train Timesteps
Train Timesteps
Train Timesteps
Figure 80: Training Learning Curves for A3C + LSTM when make denser is for rewardable sequences. Please note the different Y-axis scales.
sequence length 2
sequence length
sequence length 4
75
Reward
Reward
200
Reward
400
50
200
0
0
50000 100000 150000
50000 100000 150000
0
50000
100000 150000
Train Timesteps
Train Timesteps
Train Timesteps
Figure 81: Training Learning Curves for A3C + LSTM when make_denser is True for rewardable sequences.
Please note the different Y-axis scales.
49

---

Under review as a conference paper at ICLR 2021
K
PERFORMANCE ON COMPLEX ENVIRONMENTS
Roware
L000
Revert
Recore
2500
delay
delay
delay
(a) Rainbow beam rider
(b) Rainbow breakout
(c) Rainbow gbert
(d) Rain. space invaders
huward
1o00
Roward
:00
Itwward
2500
Rowaar
250
0.0
0.01
0.02
0.25
0.0
0.01
0.02
0.1
0.25
0.0
0.01
0.02
0.1
025
0.0
0.01
0.02
0.1
0.25
transkion nose
transition, nosse
transtion nose
transition, noise
Rainbow beam_rider
Rainbow breakout
Rainbow qbert
(h) Rain. space_invaders
Ruward
1000
:00
Nuward
Rowand
2000
2
dela
delay
deay
delsy
(i) A3C beam rider
(i) A3C breakout
(k) A3C qbert
A3C space invaders
Roward
1000
Reward
Reward
2000
300
0.0
0.51
0.62
0.1 0.25
0.03 0.02
0.0 0.01
0.09
0.1 025
0.0
0.02
0.02
0.1
0.25
Itans poke
transition, rolie
transkion, nobe
transitin, hoise
(m) A3C beam_rider
(n) A3C breakout
(0) A3C qbert
(p) A3C space space_invaders invaders
Figure 82: AUC of episodic reward for agents the end of training, Error bars represent 1 standard deviation. Note the different y-axis scales.
0.2
0.4
1.0
2.0
4.0
0.5
1.0
2.5
5.0
30.0
F250
action max
(b) time unit
action max
(d) time unit
Rewward
025
03
28
0.2
0.4
1.0 tme_unk
2.0
4.0
035
Emiuurk
(c) action max
(f) time unit
(g) action max
(h) time unit
100
355
63
025 65 1-5
2.5
1.0 tmo_unt
100 005
6.1
6.35
20
0.5
1.0
Unteunit
2.5
5.0
30.0
ii action max
(j) time unit
(k) action max
(I) time unit
Figure 83: AUC episodic reward at the end of training on HalfCheetah varying action max (top) and time unit (bottom). Error bars represent standard deviation. Note the different y-axis scales.
50

---

Under review as a conference paper at ICLR 2021
L LEARNING CURVES FOR COMPLEX ENVIRONMENTS
delay
delly
cally
delay
delly
$10
1000
Semen 250
603
250
400
2.5
1.0
55 Trais Timastops
5.6
25
0.2
3.0
Timsstaga
Tran Tmmstops
Tran Timessasa
Figure 84: Training Learning Curves DQN on beam rider when varying delay. Please note the different
Y-axis scales.
celay
delay
700
delay 2.
dalay 4.
delay
200
300
200
2,50
1.00
0.32
0.5
0.75
1.20
2.75
Timestass
307
Trab 127
Figure 85: Training Learning Curves for DQN on breakout when varying delay. Please note the different
Y-axis scales.
4020
2000
2000
2020
1000
1000
1920
Than Timeslopo
Train Timcatops
Tran Timanesss
167
Train Timestops
167
Train Thmessopp
Figure 86: Training Learning Curves for DQN on qbert when varying delay. Please note the different Y-axis scales.
51

---

Under review as a conference paper at ICLR 2021
delay
600
230
203
220
025
6.7
0.25
9.56
0.50
0.55
Tireetpps
Timestepe
187
Timestope
Tran Timestase
Figure 87: Training Learning Curves for DQN on space_ when varying delay. Please note the different Y-axis scales.
trnstoo
transition
transition
nosse
1000
2000
730
500-
500
600
Train Liresteps
Tran timeseas
1n7
Traa Timestpps
367
Train Timesteps
107
Trai: Timesteps
Figure 88: Training Learning Curves for DQN on beam_rider when varying transition noise. Please note the different Y-axis scales.
transition, noise 0.0
transition, nowe 0.01
transition, noise 0.02
transition, noise 0.3
transition, noke 0.25
132.
Roandd
0.00
035
0.75
100
625
3.50
2.75
167
300
0.35
173
600
0.0
0.75
1.00
0.5 3.50 575
167
Figure 89: Training Learning Curves for DQN on breakout when varying transition noise, Please note the different Y-axis scales.
transition noise a.e
transition noise 0.03
transition nose .2
transition naise 0.1
transition noise 0.25
Senved
5020
3020
2000
2000
2003
1030
1000
2000
35
3.0
Figure 90: Training Learning Curves DQN on qbert when varying transition noise. Please note different Y-axis scales.
transition roise
transition
transition 0.25
400
Saneed
600
360
200
Tran Timeetae
Trar Timestep
16)
Timeetppe
Tran Timestass
Figure 91: Training Learning Curves for DQN on space invaders when varying transition noise. Please note the different Y-axis scales.
lay
celly
1500
1500
1030
3000
of
1oco
1030
2003
590
sco
Soc
503
Traln Timislagp
167
Trasn misslsss
2-3
1P7
23 Tran Timerisss
192
Trabs Timealops
16]
Train Temcaloss
Figure 92: Training Learning Curves for Rainbow on beam rider when varying delay. Please note the different Y-axis scales.
de O
delay 1
delay
delay 4
delay g
303
200
200
300
0.00
0.55
050
0.5
1.00
5.00
6.35
3.50
2.35
3,30
3.00
035
0.50
3.75
1.03
0.00
0.75
050
350 2335 3,00 0.75
1.00
635
Figure 93: Training Learning Curves for Rainhow on breakout when varying delay. Please the different Y-xiis scales.
52

---

Under review as a conference paper at ICLR 2021
400
1050
601
2530
200
203
Trsh Timestage
Trah mimssipp
Tran Timettese
187
Trah Timestope
207
Tran Timeerase
Figure 94: Training Learning Curves for Rainbow on qbert when varying delay. Please note the different Y-axis scales.
d
delay
deay
delay
delay
430
200
Trat iinetops
147
Trai Timesteps
167
Train Timesteps
167
Timestop
167
Tran limeseas
167
Figure 95: Training Learning Curves for Rainbow space_ invaadrs when varying delay. Please note the different Y-axis scales.
transition, noise 0.o
transition noise 0.03
transition nose 0.02
transition, noise 0.1
transition, noise 0.25
15:00
1500
1500
2503
1020
1000
1000
1003
500
500
502
6.5
Toi
2.5
10
5.0
55
1.0
5.
Timessess
35
0.
Trahn
05
3.0
50
0.5
15
Figure 96: Training Learning Curves for Rainbow beam rider when varying transition noise. Please note the different Y-axis scales.
Transition noise 0.o
Thansit on maise 0.01
transition noise 0.02
transition paise 0.1
transition nase 0.25
150
Seneed
200
120
800
0.00
0.0
0.25
2:75
0.55
1,03
0.00
030
100
022
3.50
5:75
Figure 97: Training Learning Curves for Rainbow on breakout when varying transition noise. Please note the different Y-axis scales.
transition
transition 0.25
7530
80co
4000
tensed
5020
tenere
Abco
2002
8000
20c0
2000
2003
Trah Tirialaps
Traht Tinasops
Tran Timeness
167
Trais Timeslops
167
Train Timessaps
Figure 98: Training Learning Curves for Rainbow on qbert when varying transition noise. Please note the different Y-axis scales.
transition
transition,
transition
transitionn
400
530
200
200
0.0
030
0.75
1.00
Tra Timesksps
6.75
187
Thai Timaskaps
0.20
167
0.00
030
0.75
187
2.75 Tran Timestasa
167
Figure 99: Training Learning Curves for Rainbow space invaders when varying transition noise. Please note the different Y-axis scales.
delay
oelly
delay
delay 4
celay II
1590
1500
lownd
1030
1000
2003 sac
62
3.5
1.
6.0
85
1.0
35
1.0
0.5
05
3.0
E.0
0.5
15
Figure 100: Training Learning Curves for A3C Y-xiis scales.
when varying delay. Please note the different
53

---

Under review as a conference paper at ICLR 2021
delay
delay
202
100
0.25 6.7
0.25
0.50 Thmentepe
0.50 Timestops
0.25
6.75
Trrectpp
Train Timestepe
Tran Timestage
Figure 101: Training Learning Curves for A3C on breakout when varying delay. Please note the different Y-axis scales.
delay
celay
4010)
4000
4002
2002
Train Timesteps
187
Toin Timestaps
367
Tram Timestess
167
Train Timestaps
267
Train limessess
Figure 102: Training Learning Curves for A3C on qbert when varying delay. Please note the different Y-axis scales.
delay
delay
delay
delay 4
delay 8
600
Boors
100
4400
200
2000
200
600
0353
0.75
1.0
625
0.50
6.7
167
0.23
0.35
17
6.00
0.2
0.5 0.75
530
025
3,50 2US
167
Figure 103: Training Learning Curves for A3C ppace_invaders when varying delay. Please note the different Y-axis scales.
transition noise a.o
transition naise 0.01
transition nase 0.02
transition, naise 0.1
transition naise 0.25
1590
15:0
Sanurd
1020
tanura
boco
1000
5003
520
500
50z
3.5
1.0
60
55
35
3.0
0.5
Figure 104: Training Learning Curves for A3C beam rider when varying transition noise, Please note different Y-axis scales.
transition 0.01
transition, roise
transition
transition, 0.25
130
Tireetpp
Tran Timeetpe
Toar Timestpp
167
Titeetppe
Tran Timennss
Figure 105: Training Learning Curves for A3C on breakout when varying transition noise. Please note the different Y-axis scales.
transition
transition
transitian,
4nco
6000
4000
2020
1000
2000
2000
1.0
23
15
Thahh Timicalogo
Trasn Timcsloss
Train
10T
Trabs Timealaps
167
Train Tincaloss
Figure 106: Training Learning Curves for A3C gbert when varying transition noise. Please note the different Y-axis scales.
Pereition_roise 0.
transition noise 0.01
transition, noise 0.02
transition noise 0.1
transition, no 0.25
002
630
Bewnr
400
aco
3.0
0.55
0.5
0.7
1.00
5.00
635
030
2.35
3,35
500
035
0.50
3.75
1.03
0.00
0.55
050
0.75
1.00
2.35 3,00 %
635
3.50
Figure 107: Training leaming Curves for A3C space_ invaders when varying transition noise. Please note the different Y-axis scales.
54

---

Under review as a conference paper at ICLR 2021
adtion_5pacc_ma
Rower
40C0
Resses
30000
2000
5200
Toan Timedapp
Tireetpe
Thann Tinedaps
Thhn Timestepe
Thh TAnnTe
Figure 108: Training Learning Curves for SAC when varying action max. Please note the different Y-axis scales.
0.2
1.0
time unit
4.0.
10000
15020
10200
10300
10000
--000
5000
5020
1000
2200
01
71th Timestape
1.7
Thah Timestope
Tran Thaesppp
16c
Test Timesepps 185
Than Tireettppe
Figure 109: Training Learning Curves for SAC when varying time unit. Please note the different Y-axis scales.
Tan Treelae
XVBUU Taai TOESSOO
Figure 110: Training Learning Curves for SAC when varying action max. Please note the different Y-axis scales.
time
time uunitnntt000 0.4,
unit
time_ unit
4.0.
-100
-150
203200 400200
200
100260 2003C3
-203
5A503
100200
Thh Timestepe
Tran Timettage
Toun Timwdeess
Trai "Tnvetope
Figure 111: Training Learning Curves for SAC when varying time unit Please note the different Y-axis scales.
Figure 112: Training Learning Curves for SAC when varying action max. Please note the different Y-axis scales.
time, unit
time unit
unit
time,
Dewars
Reways
-200
202
Tell Thraoops 145
Train Timastppa
Tran Timesoosa
Trah T mossasa
Thil Timeslops
Figure 113: Training Learning Curves for SAC when varying time unit Please note the different Y-axis scales.
SEALS S.L..
actiet m5 95.
13000
2000
Loceo
10000
5ax0
Trah
Figure 114: Training Learning Curves for DDPG when varying action Please note the different Y-axis scales.
55

---

Under review as a conference paper at ICLR 2021
time
10200
4000
Raneed
Besser
2003
22503
Timettops
Tran TimeDoos
Tha Tinecippe 1et
TOO Terestrpe
Figure 115: Training Learning Curves for DDPG when varying time unit. Please note the different Y-axis scales.
Figure 116: Training Learning Curves for DDPG when varying action Please note the different Y-axis scales.
time unit 0.2.
Linw unit 0.4,
Lime unit 1.0.
Time unit 2.0.
time unit 4,0.
-50
-30
Saward
1100
-100
1100
-102
-200
220035 Timasosss
420000
130002 Trun
205002
20003 303000 That TImalopa
Trah Timsslopo
Tran Timesiops
Figure 117: Training Learning Curves for DDPG when varying unit. Please note different Y-axis scales.
Figure 118: Training Learning Curves for DDPG when varying action max. Please note the different Y-axis scales.
time unit O.S.
time unt 1.0.
time, unit 2.5
time, unit S.O,
time unit 10.0
Roord
% 4
50
0.5
203160 350030
100036
200320
50035 Timestegs
130003
23500
43000
Figure 119: Training Learning Curves for DDPG when varying time unit. Please note different Y-axis
scales.
750 11
Figure 120: Training Learning Curves for TD3 when varying action max. Please note the different Y-axis scales.
time un D.2,
time unit 6.4.
Fime unit 1.0
6300
time, unl 2.0.
time unit 4.0
Caze-
5200
2003
2020
2000
5.2
0.5
3.0
0.0
2.5
5.5
7.5
limestoss
55
0.5
3.0
2.5
250020 300020 750000
Figure 121: Training Learning Curves for TD3 when varying time unit. Please note the different Y-axis scales.
56

---

Under review as a conference paper at ICLR 2021
Loce
Figure 122: Training Learning Curves for TD3 when varying action scales.
Please the different Y-axis
time_unt
-50
-50
-1.00
100
200
-200
125
-200
150
-130
Troh Timeslaps
Tran Timesioss
200
Tuan Timosioss
Tran Timssgi
Than Timslag
Figure 123: Training Learning Curves for TD3 when varying time unit. Please note the different Y-axis scales.
1500
Figure 124: Training Learning Curves for TD3 when varying action Please the different Y-axis scales.
Senun
-00
Tran Tinsstass
17311 limesttep
Tras epr
Tran misttess
Taan Imosless
Figure 125: Training Learning Curves for TD3 when varying time unit. Please note the different Y-axis scales.
57

---

Under review as a conference paper at ICLR 2021
M HYPERPARAMETER TUNING
We gained some interesting insights into the significance of certain hyperparameters while tuning them for the different algorithms. Thus, our toy environments might in fact be good beds for researching hyperparameters in RL, too. For instance, target network update frequency turned out to be very significant for learning and sub-optimal values led very noisy and unreliable training and unexpected results such as networks with greater capacity not performing well. Once we tuned it, however, training was much more reliable and, as expected, networks with greater capacity did well. We now describe the tuning process and an example insight in more detail.
Hyperparameters were tuned for the vanilla environment; we did so manually in order to obtain good intuition about them before applying automated tools. We tuned the hyperparameters in sets, loosely order of their significance and did runs over each setting to get more robust performance estimate. We describe small part of our hyperparameter tuning for DQN next. All hyperparameter settings for tuned agents can found in Appendix N
We expected that quite small neural networks would already perform well for such toy environments and we initially grid searched over small network sizes (Figure (26a). However, the variance in performance quite high (Figure 126b). When tried tune DQN hyperparameters learning starts and target network update freeatterreueeee frequency, however, became clear that the target network update frequency was very significant (Figure T26c and [26d] and when we repeated the grid search network sizes with a better value of 800 for the target network update frequency (instead of the oldd 80) this led to both better performance and lower variance (Figure T26e and 1261).
We then changed the network number of neurons grid to [128, 256, 512) and changed target network update frequency grid [80, 800, 8000] and continued with further tuning using the grid values specified in Appendix[N N
(a) Reward
(b) Std dev.
(c) Reward
(d) Std. Dev.
(e) Reward
Std dev.
Figure 126: Mean episodic reward at the end of training for different hyperparameter sets for DQN. Please note the different colorbar scales.
58

---

Under review as a conference paper at ICLR 2021
N TUNED HYPERPARAMETERS
The code for corresponding experiments for both discrete and continuous environments can be found the accompanying code for the paper. The README describes how to run the experiments using config files and which config files correspond to which experiments. Experiments environments were rum with Ray 0.7.3, while for the continuous environments, they were run with Ray 0.9.0. We had to use Ray 0.7.3 for the discrete environments and Ray 0.9.0 for the continuous ones because we had run the discrete cases for a previous version of the paper on 0.7.3. DDPG was working and SAC was implemented in Ray that time. We tried to Ray 0.9.0 also for the discrete version but found for the 1st algorithms we tested that, for the same hyperparameters, the results did transfer even across implementations of the same library. This further makes our point about using our platform to unit test algorithms.
the discrete
Since did not save the hyperparameter grids for discrete environments separate files, they are provided here. The names of the hyperparameters for the algorithmss will match those used in Ray 0.7.3.
N.1 DQN num
num_ layerss [1, 2, 41
layer_widths [8, 32, 128] at first
layer_widths [128, 256, 512] # after setting
target_net_uplate__ffe
= 800 showed that 128 was the best
number of the old 3, changed search grid for number of neurons
["tanh", "relu", "sigmoid"]
learning_startss [500, 1000, 2000, 4000, 8000]
target_jettorr__pptee__eeeee [8, 80, 800] at first
carret__ntworr_uuuatteeeee freqs [80, 800, 8000] # after seeing
target_net__ppaat_ffess
800 is much better than 80, changed
the grid for it double_dqp = [False, True]
learning rates [le-2, 1e-3, 1e-4, 1e-5, 1e-6]
adam_epsilons = [1e-3, 1e-4, 1e-5, 1e-6] # also tried [le-1, 1e-4, 1e-7, 1e-10]
tune, run
"DON", stop=!
"timesteps_total"" 20000,
b config=(
"adam_ "adem_epsilon"" 1e-4, irtta_annnaaing__aactiooo 1.0,
"buffer_size": 1000000, "dcuble_g"; False,
"dueling": False,
"explooration_finaleess'' 0.01,
"explloration__racciion"" 0.1,
"final_prioritieee_rpplaa beta"; 1.0, "hiddens": None, "learning_starts" 1000, "Ir": 1e-4, "n_step": 1, "ncisy": False, "num_atoom" 1,
"prioritized replay": False,
apprioiitizedrepllas haample_batch_stie 4,
0.5,
59

---

Under review as a conference paper at ICLR 2021
"schedule max timesteps": 20000, detarget_netwaae _tl____dda_______e 800, "timesteps per iteration.: 100, "train_batchi" "traiin_bacch_size" 32,
"env": "RLIoy-vO", "env_config": dunny_seed': dunmy_seed, seed'; 0, state__spac__type' 'discrete',
'action_space_type's 'discrete',
staate_space_size': state_space_sizee
: action_space_siae size,
geenerate_random__ii True,
delay': delay,
ssequenceelennthh': pequunce__eegthee
rewara_density's reward_density,
terminal_state__ensittt" terminal_state_densig
sepeats_in_sequnness False,
reward__nme desward_unit:: 1.0, False, completeey_commettel True
)
"model":
" "fenet_hiddens": [256, 256],
"ohe", "custom_options""" (),
"fenet_activationn" "tanh",
"use_latm"; False,
"max_seq_ien": 20,
"Istm_celi_size": 256,
sussin_ss_prrv_ccciorreeeaaa False,
"callbacks";
"on_episode_end": tune. function (on_episode__ndd)
"on_train_resultt, tune. function (on_train_resulle
"evallaation__intervaa""" 1,
"evaluatiion_config""
"exppooratioo_fraccioo"". 0,
"expploration_ffina_eess""" 0, "batch_mode": "complete_episoter"" hhrrzonn: 100, "env senv_config;: { "dummy_eval"" True,
N.2 RAINBOW
num_layerss [1, 2, 3, 4]
layer_widths [128, 256, 512]
fcnet_activations = ["tanh", "relu", "sigmoid") tearning_rates [le-2, 1e-3, 1e-4, 1e-5, 1e-6]
learning_startss [500, 1000, 2000, 4000, 8000] target_nettork__pddte_f_eee [80, 800, 8000]
60

---

Under review as a conference paper at ICLR 2021
double_dap _qn
[False, True]
tune. run
"DQN",
stop=(
"timesteps_total": 20000,
config=(
"adam_epsilon" 1e-4,
"buffer_size: 1000000, "double_g": True, "dueling": True, "Ir": 1e-3,
[aaglloratioo_ffil_epppppp 0.01,
"exploratiion_mactiionn 0.1,
"gchebulle_mxx_tmmertpep""" 20000,
"leerningg_saarts": 500, ttargee_eetwokk_paaat__eeety 80, "n_step": 4, "neisy": True, "num_atoms": 10,
rpriooritiieed_pplla"" True,
"perboritiiaedreelll__lllp""" 0.75,
"prrooritiieedrreplaybbtt'''''" 0.4,
#Final_prrcrrtteeeereellaa=""d"" 1.0,
"beta__aneeiinng_faactttoot 1.0,
"sample_batch_sizee 4, " 1000, "traain_batchhsizee" 32, imin_iter_timm__"" 1,
"env": "RLToy-vO", "eny_config": {
duumyyssedd dummy_seed, seed' 0,
sstate_space_type' 'discrete',
"action_space_tyy type' 'discrete',
sstate_space_size" State_spaceesize
"action_space_iiie action_space_siae _ize,
_generate_random__md. True,
deela': delay,
seegeencelenggth': sequence_length,
reeara_density" reward dewaitdd
deerminal_state_deneee
comminal_statee stateddesitttttttttt...
r=ppats_ii__eequecee False,
reward_unit': 1.0,
make_denser's False, ccmmpeeteiyy_commetddd True
b
"model":
"fcnet_hiddens": [256, 256],
" "custonuurreroocsser""" "ohe",
"custtom_options": (),
"fcnet_activation": "tanh", "use_lstm"; False,
"max_seq_ien"; 20,
"1stm_celi_size" 256, "1 srt____us__ree_actiioi reward":: False,
61

---

Under review as a conference paper at ICLR 2021
"callbacks":
"on "on_ wwweeppsdde__eeee end" tune. (on wwwoeenisodeeeeeeeeeeeeeeeeeeeee end),
"on_train_resultt" tune. funct (on_train_resully,
"evaluaation_inttevaa."""" 1, "evallattior_coniiq"
"axpploratioo__fraatioo"" 0,
"explorationn_fial_pss 0, "batch_mode"; "complete_episoder" horizon': 100, "env_config": "dummy_eval"; True,
N.3 A3C
Grids of value for the hyperparameters over which they were tuned:
num_layerss [1, 2, 3, 4]
layer_widths [64, 128, 256]
learning rates [le-2, 1e-3, 1e-4, 1e-5, 1e-6] fcnet_activationn = ["tanh", "relu", "sigmoid"]
lambdas = [0, 0.5, 0.95, 1.0)
grad_clips [10, 30, 100]
of_loss_coeffe [0.1, 0.5, 2.5]
cntropy_coeffs [0.001, 0.01, 0.1, 1]
tune. run
"A3C", stop-!
"timesteps_total": 150000,
config=(
seample_batch_size 10,
Aarain_batch_size" 100, "use "use_pytorcch" False, "lambda": 0.0, "grad_clip"; 10.c, "Ir": 0.0001,
"Ir_schedule": None, "yf_loss_coeff"; 0.5, "entrropy_coeff" 0.1, "min_iter_time__ 0, "'sampie_async": True,
5000,
"num_workers": 3,
"rum_envs_per__orkee.""" 5,
"optimizer"; {
"grads_poae Bacil__e__sse___ step": 10
62

---

Under review as a conference paper at ICLR 2021
"env": "RLIoy-vO",
"env_ sww_configg: {
dummy_seed" dummy_seed.
seed': 0,
staate_spac__tpee' : 'discrete', spaloo__eppc_tyy'' 'discrete',
"state__pace_siie state state_space__s size,
'action_space_iiae action_space_siee size, generate_random__onn True,
'delay' delay,
ssequence_lenntthh: sequence_length,
reward_density': reward_density,
serminaal_state_densttty termina__state_ddnsil
repeats_ii__sequncee, False,
reward_unit': 1.0, baake_denser's False, comglettely_comeete"" True
"model":
" sicnet_hiddens, [128, 128, 128), "ohe",
" "custom_options"; (),
" "fcnet_activation"" "tanh",
"use_lstm"; False,
"max_seq_ien": 20,
blstt_celi_size": 256,
actionn rewaarardddr__aard"... reward"; False,
"cal :calkbacks": "on_episode_end"" tune function (on_eiiioo
"on_train_resultt; tune function (on_train resull)) rer
}
"evallattion_intervaa""" 1,
"evalluatioo_cooni""": {
"expplorattio__rraccoonn"" 0,
"expploration_ffinaeees"""" 0,
"batch mode": "bateh_mode": "complete_eppssoeee"
horizon': : 100,
"env "env_config": { "dummy_eval"; True,
b
1,
N.4 A3C LSTM
Grids of value for the hyperparameters over which they were tuned:
num lanersy [1, 2, 3, 4]
layer_widths [64, 128, 256]
learning_rates [1e-2, 1e-3, 3, 1e-4, 1e-5, []-6] fonet_activations = ["tanh", "relu", "sigmoid"]
lambdas [0, 0.5, 9.95, 1.0]
grad_clips [10, 30, 100]
63

---

Under review as a conference paper at ICLR 2021
vf foefff [0.1, 0.5, 2.5] entropy_coefts [0.001, 0.01, 0.1, 1J
lstm_cell_siiae size = [64, 256, 512] [False, True]
"A3C", tune,
stop={
"timesteps_total"" 150000,
} config=(
haample_batch_sizee 10,
srrain_batch_sizer 100, "use_pytorch" False, "lambda": 0.0, "grad_clip"; 10.0, " "Ir": 00001, "Ir_schedule"" None, "vf le__loss_coefffi 0.1, "entropy_coeef"; 0.1, "min__iter_timm__"": 0, "sample_async": True,
[t_imestos_per__teraatioo 5000, "nun_workers": 3, "aum_envs_per__orker 5,
"optimizer": {
10
,
"env": "RLToy-vO", "env_config": duumyy_sedd dummy_seed, seed' 0,
sstate_space_type's 'discrete',
'discrete',
sstate_spacessize" State_space_size
"action_space_iiie action_space_sinn
_generate_random__md. True,
deela': delay, seegeence_enngth': sequence_length,
reeara_density" reward dewaityd
deerminal_state_deneee
terminal_statee _etaatedeesity....
r=ppats__i__eqquecee False,
reward_unit': 1.0,
make_denser's False, ccmmpeeteiyyccommetddd True
b
"model":
"fcnet_hiddens"": [128, 128, 128], " "custoouprrerocessor"" "ohe",
"custtom_options":: (),
"Fonet_activation"" "tanh",
"use_lstm"; True,
"max_seq_ien": delay sequence_length,
"istm_celi_sizer 64,
_estm_use_pre___ee ptw.uue_rree_acioooreeeaera True,
64

---

Under review as a conference paper at ICLR 2021
"callbacks";
"on_episode_end": tune. function (on_episode_edd)
"on_train_resultt:: tune. function ton_train_resullll
b,
"eratlatiion_interval."" 1, "evallattion_coffi"::
"explorattion_fractioo""" 0,
"expllorattoo__finae_es"""""" 0,
"batch_mode"; "complete_epissdees" horrizon: 100,
"env_ "env_config": {
"dummy_eval": True,
b
65

---

Under review as a conference paper at ICLR 2021
O MORE ON CONCLUSION AND FUTURE WORK
Among the continuous environments, we have a toy task of moving along a line. Here, we hand out greater rewards the closer a point object is to moving along a line. This is also better task to test exploration than the completely random discrete environments. It already gave some interesting results and further work will follow. We are the process of implementing plug and play model- based metrics to evaluate version because analytical calculation would be intractable many cases) between the true dynamics models and the learnt one to keep track of how model learning is proceeding. Our Environments plan to allow using their transition and reward functions to perform imaginary rollouts without affecting the current state of the system.
algorithms, such as the Wasserstein metric (likely a sampled
Another significant meta-feature reachability in the transition graph. We believe a lot of insights can be gained from graph theory to model toy environments which try to mimic specific real life situations at a very high level. We plan that users can specify their own transition graphs and also plan to add random generation of specific types of transition graphs.
Even though we have a playground to generate environments where the dimensions such as sequence length constant, being able to solve environments with variable delay and sequence lengths and identifying them (i.e., segmentation events in the time domain) is another area we are currently working on with attention-based agents and various other ideas.
The fine-grained control of dimensions allows relating these to good hyperparameter choices. So, our playground could also be used learn mapping from hardness dimensions hyperparameters for different types of environments and even to warm-start hyperparameter optimisation for environments with similar hardness dimensions. This holds promise for future meta-learning algorithms. similar vein, it could also be used to perform Combined Algorithm Selection and Hyperparameter Optimisation (Thornton al,, 2013), since it's clear that currently different RL algorithms do well in different kinds of environments.
Further interesting toy experiments which are already possible with our platform are varying the terminal state densities to have environments for benchmarking safe RL.
The states and actions contained in rewardable sequence could just be a single compound state and compound action if we discretised time in suitable manner. This brings us to the idea of learning at multiple timescales. HRL algorithms with formulations like the options framework (Sutton et al.) 1999), could try to identify these rewardable sequences at the higher level and then carry out atomic actions at the lower level.
We also hope to benchmark other algorithms like PPC Schulman et al. 2017). Rudder (Arjona Medina et al. 2019). MCTS (Silver et al. 2016). DDPC (illicrap et al. 2016) on continuous tasks and bable-based algorithms and to show theoretical results match with practice toy benchmarks.
We also aim to promote reproducibility in RL as in (Henderson et al., 2018) and hope our benchmark helps with that goal. To this end, we have already improved the Gym Box and Discrete Spaces to allow their seeds to be controlled at initialization time as well.
We need different RL algorithms for different environments. Aside from some basic heuristics such as applying DDPG (Lillicrap et al., 2016) to continuous environments and DQN to discrete environments, it is not very clear when use which RL algorithms. We hope this will be first step to being able to identify from the environment what sort of algorithm to use and to help build adaptive algorithms which adapt the environment hand. Additionally, aside from being great benchmark for RL algorithms, it is also a great didactic tool for teaching how RL algorithms work in different environments.
'ee tried PPO hut could not get to learn 'We tried DDPG also but there seemed to be bug in the implementation and it crashed even tuned examples from Ray
66

---

Under review as a conference paper at ICLR 2021
P CPU SPECIFICATIONS
processor
0
vendor_id
GenuineIntel
cpu family
6
model
158
model name stepping
Intel (R)
Core (TM) 17-8850H CPU e 2. 60GHz
10
microcode
0xb4
cpu MHz
900. 055
cache size
9216 KB
physical id
0
siblings
12
core id cpu cores
0
apicid
0
initial apicid 0
fpu fpu_exceptice
yes
yes
cpuid level wp flags
22 yes
fpu vme de pse tsc pae mce cx8 apic sep mtrr pge nca cmov pat pse36 clflush dts acpi max fxsr sse sse2 ss ht tm pbe syscall pdpelgb rdtsep 1m constant tsc art arch Tach_perffon pebs bts rep_good nopl xtopology nonstop_te tsc epuid aperfmperf tsc_known freq pni pelmulgdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm peid sse4_1 sse4_2 x2apic movbe popent tsc_deadl ine_time avx fl6c rdrand lahf _1m abm 3dnowprefetch cpuid_fault epb invpeid_single pti sabd ibrs ibpb stibp tpr_shadon vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmil hle avx2 smep bmi2 erms invpeid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbvl xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_windes hwp_epp md_clear flush_lid cpu_meltdown spectre_vl spectre_v2 apec_store_bypase 1ltf mds swapgs
xsave
bugs
begomips
5184.00
clflush size 64
cache_ bache_alignment 64
address sizes
39 bits physical, 48 bits virtual
power management:
67


Reviews:
[{"review_id": "nhRR80pptcd", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes a new benchmark for evaluating reinforcement learning techniques (MDP-playground). It can be seen as a toolbox allowing to generate different MDPs with different characteristics. Each MDP will then be used to probe a particular ability of learning algorithms, resulting in a comparison of methods over multiple dimensions. Proposed dimensions are reward sparsity, stochasticity, delayed reward, etc....  In addition to this toolbox, the authors also evaluate some of the classical algorithms in the domain. \n\n== Comments\n\nFirst of all, the idea of evaluating RL techniques over multiple dimensions is very interesting, since right now the comparison of RL techniques is very weak, and providing simple tools in that direction is crucial to make  advances in the field. The MDP-playground approach is a good approach toward this goal and proposes a large number of different metrics on both continuous and discrete MDPs, making this platform the most complete in the domain. \n\nBut I identify two drawbacks in the proposed toolbox: first of all, if many metrics are proposed, it is very difficult to know which of these metrics are really relevant, and which are not. Said otherwise, the methodology would gain if these metrics could be connected to real use-cases e.g what are the relevant dimensions underlying atari environments, robotics ones, etc... Right now, imagine I evaluate my model over all the different metrics, and compare my model to other models, I still don't know which model I have to choose for solving a concrete use-case. A second drawback is readability: the approach is somehow proposing too many metrics such that being able to understand which model is good and which model is bad is very difficult. I would propose the authors to think about organizing these metrics in a way that they can be easily presented to users (e.g using spider plots ? by using a hierarchy ? ) At last, the paper is just comparing a few models over these metrics, while I would expect to have a more complete benchmark of existing models. \n\nTo conclude, if I really like the approach proposed in this paper, and if I think that it is a nice step toward a better evaluation of RL algorithms, I find that the paper is lacking some important characteristics to make MDP-playground really usable: i) a good way to summarize the performance of RL algorithms too many metrics allowing a good understanding of the methods ii) a comparison of more algorithms and iii) a  link between the proposed metrics and classical benchmarks.\n\n== \nConsidering the modifications made on the paper, I increase my score \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "VfKWA64CWAB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents \"MDP Playground\", a family of procedurally generated MDPs that can be used to benchmark certain dimension of difficulty considered by the authors to be challenging to current RL algorithms.  The paper presents the effects of the various perturbations to the MDP on state-of-the-art learning algorithms and discusses particular dimension of interest in the paper.  A full and exhaustive analysis of results is presented in the appendix.  The \"MDP Playground\" is slated to be open-sourced so that the community can benchmark against it.\n\n\nPros:\nI think that high-level difficulties of MDPs are under-studied and that the over-reliance on benchmarks such as Atari or Mujoco make people look more at per-environment/game performance instead of thinking about high-level issues with the MDPs (exploration, delays et.c). Although this is done in a hand-wavy manner, all attempts at formalising this seem essential to better understand what the actual degrees of difficulty are for particular tasks and whether novel proposed approaches are really tackling the challenge they claim to be tackling.\n\nI find the use of two procedurally generated MDP, one continuous and the other discreet, in a very simple task setup to be a good design choice, and also allows for quick iteration.\n\nCons:\nAlthough this is an opinionated position, I'm not super happy with the choice of perturbations. Some are very general, such as delays or stochasticity, while others are very specific such as target radius, action max, or action loss weight.  I feel the nomenclature around the dimensions of 'hardness' (nit, perhaps 'difficulty' would be a better word here) is not very clear.  The proposed dimensions seemed to be inspired by some tasks the authors are working with, but in that case it would make more sense to ground their choice by describing the tasks and arguing why these are particularly important.  For example, target radius seems to be completely arbitrary, I could define an infinite number of reward functions that describe a goal and use all sorts of topologies to window my reward and shape it as the agent nears the goal, is this really a general problem for RL though?  I remain unconvinced.\n\nWith relation to bsuite I would have also like to see more discussion on why the additional dimensions of difficulty make sense.  For example bsuite already proposes noise as an evaluation dimension, how does MDP Playground's noise differ?  \n\nMore generally, not all dimensions were clearly described, in particular stochasticity, it wasn't clear to me how this was defined.  I would have appreciated more time spent on describing the challenges rather than the analysis of the results on all sorts of environments, in the end the core contribution here is the framework and its structure, the analysis is slightly out of scope given the length of the paper.\n\nThere seems to also be very similar work in this space [https://arxiv.org/abs/2003.11881] that also proposes an open-source benchmark, it would be interesting to compare the choice of hardness dimensions to the ones used here.\n\nConclusion:\nOverall I think this is a good direction of work, but it is a bit too unprincipled, and the paper structure kind of confusing.  I would prefer perhaps less degrees of hardness, or perhaps a couple 'families' to make your thought process easier to understand.  Then, further discussion and grounding for each family of tasks would be great, to understand where these dimensions of hardness would manifest themselves.  Finally, spending more time on describing each hardness dimension clearly instead of compacting it all into the end of Section 2 would also make this an easier read.  \n\nI think this will be hard to achieve for the rebuttal phase, but I encourage continued work in this domain and look forward to seeing the authors' response.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "5wII6BnPxq", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a suite of benchmark tasks designed to test (and possibly debug) reinforcement learning algorithms. Deemed the MDP Playground, these environments are applicable to both discrete and continuous RL agents and allow tuning of various dimensions of complexity - reward delays, reward sparsity, stochasticity, etc. The authors demonstrate their framework by evaluating the performance of many well-known RL agents across a variety of these playground environments. Additionally they conduct similar experiments on Atari and Mujoco tasks and observe similar trends in agent performance when injecting noise, reward delays, and varying action max values. Finally, the MDP Playground is very quick to run and facilitates fast experimentation.\n\nIt's my view that the efficacy of a testing and debugging suite like MDP Playground is measured by the actionable insights that can be generated with it. To this end the authors describe some findings that may be applicable in the design of new environments (such as action max needing tuning in continuous action environments), but little is shown about new insights gained toward understanding shortcomings of existing algorithms or routes for building better RL agents.\n\nAdditionally, why do we believe that the structure of the MDPs generated by MDP Playground will resemble that of the problems that RL practitioners in the community are interested in solving? Specifically for discrete environments having a completely connected transition function consisting of 8 states and 8 actions seems like it may not resemble more complicated environments like Atari. Similarly, moving a pointmass in a 2D plane likely has many differences from learning how to locomote a multi-jointed robot. It's not clear that insights gained from Playground environments will transfer to more complex environments.\n\nTangentially, it might be interesting to have a tool that could analyze a particular complex environment and automatically generate a corresponding Playground MDP that would somehow capture the same measures of difficulty, such that agents could be debugged/optimized in this low-cost environment before being transferred back to the complex environment.\n\nI have read the author response and stand by my original score of the paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "40aRW9_FEe8", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "--------------------------------------\n\nPOST-REBUTTAL COMMENTS\n\nAs a result of the discussion the paper has improved, so I'm increasing my score. However, the core issue, that the proposed benchmarks don't seem to capture the difficulty structure of either real problems or more complex benchmarks, remains.\n\n--------------------------------------\n\n\nSUMMARY:\n\nThe paper introduces MDP Playground, a parameterized suite of MDPs with low computational requirements that nonetheless present significant challenges for existing RL algorithms. The authors argue that MDP Playground is a valuable testbed for developing and evaluating ne RL algorithms. The paper also assess how the identified dimensions of hardness that can be exercised in MDP Playground transfer to more complex benchmarks. The paper also uses MDP Playground to evaluate several rllib algorithms.\n\n\nHIGH-LEVEL COMMENTS:\n\nThis work has a lot of potential. It targets an important problem in RL research: studying the behavior of RL algorithms as an environment changes along various dimensions of hardness. It is valuable to have a benchmark suite of the sort this work aims to deliver, one that allows varying these dimensions in a controlled way and consists of problems that are simple enough for debugging. \n\nHowever, in its current form this work has non-trivial weaknesses in contribution and presentation that make publication at ICLR or similar conferences premature:\n\n\n1) MDP Playground's problems are completely unintuitive. They are randomly generated and aren't inspired by any real scenarios. This raises the question of how meaningful they are. In ML, it is understood from results such as the no-free-lunch theorem that no learner can do well across all possible tasks, and the sensible approach is to make ML algorithms work well on meaningful data distributions that are encountered in reality. Unfortunately, RL as a field hasn't been good at sticking to this principle: many of its existing benchmarks, such as videogames, look \"interesting\" and difficult, but in a very different way that real decision-making scenarios are. MDP Playground exacerbates this issue -- since its MDPs are randomly generated and don't have a natural interpretation, it's difficult to get even an intuition for a good behavior in them, and difficult to understand how to generate problems that that have a combination of hardness along different dimensions that is representative of realistic scenarios.\n\nA case in point is the paper's omission of MDP non-ergodicity (and presence of constraints on the desired policy as a common real factor that causes non-ergodicity) from its list of hardness dimensions. In reality, and even in some existing benchmarks, the learner can reach irrecoverable (absorbing) failure states, such as robots damaging themselves or objects they interact with, unless they are very careful. There are goal-directed MDP models with such states -- see, e.g., reference [a] at the end of the review -- and learning in their presence in realistic scenarios requires costly resets -- see, e.g., reference [b]. MDP Playground doesn't help with researching this aspect, e.g., by having knobs for probabilities of entering such states, penalties for doing so and the cost of recovery, despite it being ubiquitous in reality.\n\nAnother major omission is the fact that in reality agent actions have durations that can make it very hard to learn a policy within a given time budget. The existing benchmark doesn't allow studying the effects of action duration on learning time. \n\nGiven these gaps, I doubt that MDP Playground in its current form adds much value over the existing RL benchmarks, which have some drawbacks but have interpretability as a big asset for debugging.\n\n\n2) The paper's analysis of dimensions of hardness is quite unprincipled, contrary to the paper's claims. What makes many decision-making problems (both existing ones and those introduced in MDP Playground) hard is partial observability, but the paper never formally states what POMDPs are, nor even mentions this term. If it defined POMDPs formally, the incorrectness of some of the statements the paper makes about problems with partial observability (see the detailed comments below) would become obvious, and the connections between partial observability and hardness would become much clearer mathematically. See, e.g., reference [c] for a formal but accessible treatment of POMDPs.\n\n\n\nTECHNICAL ISSUES/RELATED WORK:\n\n-- In the intro, the paper claims that \"partial observability [is] when the underlying environment is assumed to be an MDP, however, the state formulation, i.e., the observation used by the agent is not Markovian\". This is an extremely inaccurate statement at best. In POMDPs, observations are Markovian -- their probability depends only on the current (hidden) state. \"State formulation\" is also Markovian, as it is fully observable MDPs. So are the belief states. What is non-Markovian in POMDPs is the optimal policy w.r.t. the observations: an optimal policy can depend on the entire observation history. However, again, each observation history maps to a belief state, and in the belief state space the optimal policies are Markovian.\n\n\n-- The same goes for several other statements, e.g. \"performance degrades in environments where the delayed reward induces partial observability and hence makes the state used by the algorithm non-Markovian\". How can delayed reward introduce partial observability? How can partial observability make \"the state used by the algorithm\" (an imprecise term in its own right) non-Markovian?\n\n\n-- Out of the dimensions of hardness the paper does identify, why is \"sequence length\" a distinct dimension, rather than being an instance of reward delay or reward sparsity?\n\n\n-- In the related work, a notable omission is reference [d], which looks at aspects that MDP hard and analyzes existing benchmarks w.r.t. those aspects.\n\n\n-- I wasn't sure what the benefit of the \"varying reprsentations\" experiment that tries to enforce various kinds of invariances was. First of all, the experiment is about applying various types of data augmentation to images, not about varying representations. Second, the conclusion that \"this indicates that shift and other types of invariance do not come for free and that one needs to have sufficient amount of samples for the algorithm to become invariant to the transforms we desire.\" is rather obvious and well-known.\n\n\n-- A similar remark goes for several other experiments: varying time units, target variance, irrelevant features. Their results are completely expected, and while they might be a useful sanity check, the results description can be condensed to 1 line each and placed in the figure captions.\n\n\n-- A similar comment goes for the entire content of Section 4.3 as well-- it is very verbose and can be greatly condensed and structured into a short list with bullet points summraizing the findings. \n\n\nTypos:\n\n\"Very low cost execution\" --> \"Very low cost of execution\"\n\n\n[a] Kolobov, Mausam, Weld, \"A Theory of Goal-Oriented MDPs with Dead Ends\", UAI-2012\n\n[b] Eysenbach, Gu, Ibarz, Levine,  \"Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning\", ICLR-2018\n\n[c] Kochenderfer, \"Decision Making Under Uncertainty: Theory and Application\", MIT Press, 2015\n\n[d] Maillard, Mann, Mannor \"“How hard is my MDP?” The distribution-norm to the rescue\", NIPS-2014", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I thank the authors for their submission and very active participation in the author response period. The reviewers and I acknowledge the importance of designing toy environments that allow the community to systematically investigate strengths and weaknesses of RL approaches. That said, the reviewers have criticized that it is unclear whether experiments on the proposed toy MDPs would transfer to more complex standard RL benchmarks (such as Atari) [R1 & R2], and that the proposed metrics and axes of variation seem not well motivated or systematic [R1,R2,R3 & R4], thus casting doubts regarding what insights the community will be able to gain from experiments on MDP Playground. In particular, I agree with the reviewers R1's and R4's assessment that proposing many dimensions of variation, even if they are orthogonal, without a well formulated motivation and grounding in actual tasks the community cares about is not particularly helpful for advancing our understanding of current challenges in RL. Post rebuttal, R2 and R4 stand by their strong stance against acceptance; and R1 has increased their score as a result of the improvements of the updated paper, but they still lean towards rejection. Thus, I recommend rejection.
Decision: Reject