SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$\mathcal{E}_t^j (s,a)$  denote the event that the  $j$ th episode has  $(s_t,a_t) = (s,a)$ . Let  $\mathcal{E}_t(s,a)$  be the event that  $\mathcal{E}_t^j (s,a)$  occurs in at least one one of those episodes, i.e.
$$
\mathcal {E} _ {t} (s, a) := \bigvee_ {j = 1} ^ {m} \mathcal {E} _ {t} ^ {j} (s, a).
$$
Finally, let  $\mathcal{C}$  be the event defined by
$$
\mathcal{C}:= \bigwedge_{(s,a)\in \mathcal{S}\times \mathcal{A}}\bigvee_{t\in [T]}\mathcal{E}_{t}(s,a).
$$
That is,  $\mathcal{C}$  is when every state-action pair has been seen at some timestep in at least one episode. We can thus equivalently define  $L = \min \{m\mid \mathbb{P}_m(\mathcal{C})\geq 1 / 2\}$ .
We will now start by showing the upper bound of  $L \leq \lceil \log (2SA) / p \rceil$ . Let  $m = \lceil \log (2SA) / p \rceil$ . We can write
$$
\begin{array}{l} \mathbb {P} _ {m} (\neg \mathcal {C}) = \mathbb {P} _ {m} \left(\exists (s, a) \in \mathcal {S} \times \mathcal {A}: \forall t \in [ T ], j \in [ m ] \quad \neg \mathcal {E} _ {t} ^ {j} (s, a)\right) \\ \leq \sum_ {(s, a) \in \mathcal {S} \times \mathcal {A}} \mathbb {P} _ {m} \left(\forall t \in [ T ], j \in [ m ] \quad \neg \mathcal {E} _ {t} ^ {j} (s, a)\right) \\ \leq \sum_ {(s, a) \in \mathcal {S} \times \mathcal {A}} \min  _ {t \in [ T ]} \mathbb {P} _ {m} \left(\forall j \in [ m ] \quad \neg \mathcal {E} _ {t} ^ {j} (s, a)\right) \\ = \sum_ {(s, a) \in \mathcal {S} \times \mathcal {A}} \left(1 - \max  _ {t \in [ T ]} \mathbb {P} _ {m} \left(\mathcal {E} _ {t} ^ {1} (s, a)\right)\right) ^ {m} \\ \leq \sum_ {(s, a) \in \mathcal {S} \times \mathcal {A}} \left(1 - \min  _ {(s ^ {\prime}, a ^ {\prime}) \in \mathcal {S} \times \mathcal {A}} \max  _ {t \in [ T ]} \mathbb {P} _ {m} (\mathcal {E} _ {t} ^ {1} (s ^ {\prime}, a ^ {\prime}))\right) ^ {m} \\ \leq \sum_ {(s, a) \in \mathcal {S} \times \mathcal {A}} \left(1 - \min  _ {(s ^ {\prime}, a ^ {\prime}) \in \mathcal {S} \times \mathcal {A}} \max  _ {t \in [ T ]} \mu_ {t} (s ^ {\prime}, a ^ {\prime})\right) ^ {m} \\ \leq \sum_ {(s, a) \in \mathcal {S} \times \mathcal {A}} \exp \left(- m \min  _ {(s ^ {\prime}, a ^ {\prime}) \in \mathcal {S} \times \mathcal {A}} \max  _ {t \in [ T ]} \mu_ {t} (s ^ {\prime}, a ^ {\prime})\right) \\ \leq \sum_ {(s, a) \in \mathcal {S} \times \mathcal {A}} \frac {1}{2 S A} \\ = \frac {1}{2}, \\ \end{array}
$$
which proves the upper bound.
To show the lower bound, take any  $(s,a)\in \arg \min_{(s,a)\in S\times A}\sum_{t\in [T]}\mu_t(s,a)$ . First, suppose that  $\sum_{t\in [T]}\mu_t(s,a)\geq \log (2)$ . Then the lower bound in the lemma must be at most  $1 / 2$ , which is clearly true. Thus, we may subsequently assume  $\sum_{t\in [T]}\mu_t(s,a) < \log (2)$ .
Suppose  $m < C / \sum_{t\in [T]}\mu_t(s,a)$  Then
$$
\begin{array}{l} \mathbb {P} _ {m} (\neg \mathcal {C}) = \mathbb {P} _ {m} \left(\exists (s ^ {\prime}, a ^ {\prime}) \in \mathcal {S} \times \mathcal {A}: \forall t \in [ T ], j \in [ m ] \quad \neg \mathcal {E} _ {t} ^ {j} (s ^ {\prime}, a ^ {\prime})\right) \\ \geq \mathbb {P} _ {m} \left(\forall t \in [ T ], j \in [ m ] \quad \neg \mathcal {E} _ {t} ^ {j} (s, a)\right) \\ = \left(1 - \mathbb {P} _ {m} \left(\exists t \in [ T ] \quad \mathcal {E} _ {t} ^ {1} (s, a)\right)\right) ^ {m} \\ \geq \left(1 - \sum_ {t \in [ T ]} \mathbb {P} _ {m} \left(\mathcal {E} _ {t} ^ {1} (s, a)\right)\right) ^ {m} \\ = \left(1 - \sum_ {t \in [ T ]} \mu_ {t} (s, a)\right) ^ {m} \\ \stackrel {\mathrm {(i)}} {\geq} \exp \left(- 2 m \sum_ {t \in [ T ]} \mu_ {t} (s, a)\right) \\ > \frac {1}{2}, \\ \end{array}
$$
where (i) uses the fact that  $1 - x \geq \exp(-2x)$  for  $x \in [0, \log(2)]$ . This shows that with probability greater than  $1/2$ , not all state-action pairs will be seen in  $m$  episodes, and thus establishes that  $L > m$ , which is the desired bound.
D.3 Effective planning window (EPW)
Perhaps the closest existing concepts to our effective horizon are various notions of "effective planning window." This generally refers to tree-based planning algorithms which only consider action sequences of some length  $W$  from the current state, rather than considering action sequences all the way until the end of the MDP. For instance, Kearns et al. [24] show that in discounted MDPs, one need only plan to some  $\epsilon$ -horizon to obtain an  $\epsilon$ -optimal policy. Jiang et al. [53] build on this and show that one may want to use a different discount factor for planning than the one that is used for evaluation. Malik et al. [8] also introduce a notion of effective planning window based on the number of timesteps one must look ahead in an MDP to avoid terminal states.
We do not directly apply any of these previous results to our setting. Since we are concerned primarily with finite-horizon undiscounted MDPs, it does not make much sense to apply a discount factor as in Kearns et al. [24] and Jiang et al. [53]. We find that the assumptions in Malik et al. [8] are quite unusual and do not really hold in any of the environments in BRIDGE. In particular, the analysis in Malik et al. [8] requires that a trajectory through an MDP is either optimal or ends early in a terminal state.
Instead of directly using these results, we define a notion of effective planning window based on the length of action sequences one must consider in an MDP while ignoring any rewards after the sequence.
Definition D.5 (Effective planning window). Define  $Q_{t}^{1}(s,a) = R(s,a)$  for all  $(t,s,a)\in [T]\times \mathcal{S}\times \mathcal{A}$  and let  $Q^i = QVI(Q^{i - 1})$  for  $i = 2,\dots ,T$ . The effective planning window of an MDP is the minimum  $W\in [T]$  such that all policies in  $\Pi (Q^{W})$  are optimal.
Note that the effective planning window bears significant similarity to the  $k$ -QVI-solvability property from Definition 5.1. However,  $Q^1$  is defined as equal to the reward function for the EPW, while in Definition 5.1 it is equal to  $Q^{\pi^{\mathrm{expl}}}$ .
The EPW also results in sample complexity bounds of  $T^2 A^W$  very similar to those of  $T^2 A^H$  for the effective horizon. However, we find empirically that  $H < W$  in 82% of the MDPs in BRIDGE, making the effective horizon-based bounds generally tighter.
Theorem D.6. For any MDP with effective planning window  $W$ , there is an RL algorithm whose sample complexity is at most  $T^2 A^W$ .
Proof. We will use the following algorithm:
1: procedure PLANOVERWINDOW(W)
2: for  $i = 1,\dots ,T$  do
3: for  $a_{i:i + W - 1}\in \mathcal{A}^k$  do
4: Sample an episode following  $\pi_1, \ldots, \pi_{i-1}$ , then actions  $a_{i:i+W-1}$ , and then arbitrary actions.
5:  $\hat{R}_i(s_i,a_{i:i + W - 1})\gets \sum_{t = i}^{i + W - 1}R(s_t,a_t).$
6: end for
7:  $\pi_i(s_i) \gets \arg \max_{a_i \in \mathcal{A}} \max_{a_{i+1:i+W-1} \in \mathcal{A}^{k-1}} \tilde{R}_i(s_i, a_{i:i+W-1})$ .
8: end for
9: return  $\pi$
10: end procedure
Again, this algorithm is quite similar to GORP (Algorithm 1) except that it only samples a single episode per action sequence and it ignores rewards beyond the planning window. Clearly, PLANOVER-WINDOW will take  $T^2 A^W$  steps in the environment. Thus, to bound the sample complexity, we only need to show it returns an optimal policy with probability at least  $1/2$ .
To prove this, we will show that
$$
\max  _ {a _ {i + 1: i + W - 1} \in \mathcal {A} ^ {k - 1}} \hat {R} _ {i} \left(s _ {i}, a _ {i: i + W - 1}\right) = Q _ {i} ^ {W} \left(s _ {i}, a _ {i}\right). \tag {24}
$$
Based on line 7 of the algorithm, this is enough to show that  $\pi \in \Pi(Q^W)$ , and thus that  $\pi$  must be optimal by Definition D.5.
To prove (24), we will first show by induction that  $Q_{t}^{j}(s_{t}, a_{t}) = \max_{a_{t+1:t+j-1} \in \mathcal{A}^{j-1}} \sum_{t'=t}^{t+j-1} R(s_{t'}, a_{t'})$ , where  $s_{t'+1} = f(s_{t'}, a_{t'})$  for  $t' = i, \ldots, i + W - 2$ . The base case when  $j = 1$  is by definition:  $Q_{t}^{1}(s_{t}, a_{t}) = R(s_{t}, a_{t})$ . For the inductive step, assume the formula holds for  $j$  and note that
$$
\begin{array}{l} Q _ {t} ^ {j + 1} \left(s _ {t}, a _ {t}\right) = \operatorname {Q V I} \left(Q _ {t} ^ {j}\right) \left(s _ {t}, a _ {t}\right) \\ = R \left(s _ {t}, a _ {t}\right) + \max  _ {a _ {t + 1} \in \mathcal {A}} Q _ {t + 1} ^ {j} \left(f \left(s _ {t}, a _ {t}\right), a _ {t + 1}\right) \\ = R \left(s _ {t}, a _ {t}\right) + \max  _ {a _ {t + 1} \in \mathcal {A}} \max  _ {a _ {t + 2: t + j} \in \mathcal {A} ^ {j - 1}} \sum_ {t ^ {\prime} = t + 1} ^ {t + j} R \left(s _ {t ^ {\prime}}, a _ {t ^ {\prime}}\right) \\ = \max  _ {a _ {t + 1: t + j} \in \mathcal {A} ^ {j}} \sum_ {t ^ {\prime} = t} ^ {t + j} R \left(s _ {t ^ {\prime}}, a _ {t ^ {\prime}}\right). \\ \end{array}
$$
Next, note that by the way  $\hat{R}$  is constructed,  $\hat{R}(s_i, a_{i:i+W-1}) = \sum_{t=i}^{i+W-1} R(s_t, a_t)$ , where  $s_{t+1} = f(s_t, a_t)$  for  $t = i, \ldots, i+W-2$ . Thus combining this with the formula proved by induction, (24) clearly holds and the proof is complete.
D.4 Other bounds
One other work that derives sample complexity bounds for RL with random exploration is Dann et al. [6]. They define an algorithm which maintains at all times a current best policy  $\pi$ , and acts according to this policy but with some exploration noise, e.g., via an  $\epsilon$ -greedy policy  $\mathrm{expl}_{\epsilon}(\pi)$ . They introduce the notion of a "myopic exploration gap," which is defined as
$$
\alpha = \sup  _ {\pi^ {\prime}, c \geq 1} \frac {1}{\sqrt {c}} \left(J \left(\pi^ {\prime}\right) - J (\pi)\right)
$$
such that for all  $(t,s,a)\in [T]\times S\times \mathcal{A}$
$$
\mu_ {t} ^ {\pi^ {\prime}} \leq c \mu_ {t} ^ {\exp_ {\epsilon} (\pi)} (s, a)
$$
$$
\mu_ {t} ^ {\pi} \leq c \mu_ {t} ^ {\exp_ {\epsilon} (\pi)} (s, a).
$$
This gap is shown to generalize the notion of covering length as well as various others from the literature. However, we find that it is not so useful in many environments in BRIDGE.
The problem we find is illustrated in the MDP below:
In this MDP, one need simply follow the actions which give rewards of 1 to achieve the optimal return of  $T$ . One can show  $H = W = 1$  in this MDP, which give identical sample complexities of  $T^2 A$ .
However, the difficulty with the myopic exploration gap is that the analysis in Dann et al. [6] cannot rule out the policy  $\pi$  which takes all actions to the right (achieving return  $T - 1$ ) from being chosen at some point while running their RL algorithm. If this happens, then the only way to find a better policy is to completely switch to the policy  $\pi'$  which takes all left actions (achieving return  $T$ ). This implies that  $\alpha$  is maximized when  $c = A^T$ , leading to  $\alpha = (1 / A)^{T/2}$ . Since the sample complexity bounds in Dann et al. [6] are  $O(1 / \alpha^2)$ , this gives a bound proportional to  $A^T$ , which is no better than the worst case.
Thus, whenever there are "distracting" rewards, no matter how distant, as in this case, the theory from Dann et al. [6] cannot give good sample complexity bounds. There is also no easy way to calculate  $\alpha$  directly for an arbitrary environment. For these reasons, we do not include their bounds in our experiments.
E Dataset details
In this appendix, we give a detailed explanation of how we chose the MDPs in BRIDGE and how we constructed their tabular representations. See Figure 5 for an overview of BRIDGE.
E.1 Environments
We limited the horizon of environments for BRIDGE to some  $T \in \{10, 15, 20, 30, 50, 70, 100, 200\}$ , depending on the environment, in order to avoid the state space becoming intractably large. We use subscripts to denote the horizon to which an environment is limited. For instance,  $\mathrm{PONG}_{50}$  refers to the Atari game Pong limited to 50 timesteps.
Frameskip We carefully used frameskip for each environment. Frameskip is a standard practice in Atari [54] in which each action taken in the environment is played for a certain number of frames; the agent only receives the next state after all these frames have completed. We use unusually high frameskips in order to capture episodes with longer wall-clock times in a small number of environment timesteps. The frameskip values we use are listed in
Table 4. For most Atari games, we use a frameskip of 30, corresponding to taking 2 actions per second. The frameskips for Procgen environments vary; we chose ones that tended to align with how long it took the agent to perform various low-level tasks in the environment like moving one space. We did not use frameskip for Minigrid.
The BRIDGE dataset
55 Progen levels
33 Minigrid gridworlds
67 Atari games
Num. states  $S$  Num. actions  $A$
Horizon  $T$
Figure 5: Our BRIDGE dataset consists of 155 deterministic MDPs with full tabular representations. We include MDPs from three popular RL benchmarks which cover a range of state space sizes, action state sizes, and horizons.
Environment	Frameskip
MONTEZUMAREVENGE	24
All other Atari games	30
BIGFISH	8
CHASER	2
CLIMBER	6
COINRUN	8
DODGEBALL	8
FRUITBOT	8
HEIST	2
JUMPER	8
LEAPER	6
MAZE	1
MINER	1
Ninja	8
PLUNDER	8
STARPILOT	8
All Minigrid gridworlds	1
Table 4: Frameskip values used for the MDPs in BRIDGE.
Atari games For each of the 57 Atari games in the Arcade Learning Environment (ALE) benchmark [43], we attempted to construct tabular representations for each horizon  $T \in \{10,20,30,50,70,100,200\}$ . However, we excluded environments once the state space exceeded 100 million states. We kept multiple horizon-limited versions of games, i.e., BRIDGE contains  $\mathrm{PONG}_{10}$ ,  $\mathrm{PONG}_{20}$ ,  $\mathrm{PONG}_{30}$ , etc. For some games, even  $T = 10$  produced too many states, so we did not include these at all. We use the minimal action sets for each MDP rather than all 18 possible Atari actions.
We made one exception to these procedures for Montezuma's Revenge, as it is an environment well-known for being difficult to explore, so we wanted to make sure to include it in BRIDGE. We found that with  $T = 10$ , there was not enough time to get any reward, and with  $T = 20$  there were too many states. We found that using  $T = 15$  and a frameskip of 24 did allow an agent to receive reward, so we used this version in BRIDGE.
We made a couple other modifications to the standard Atari setup. First, we limited agents to one life: as soon as a life is lost, the episode ends. Second, in  $\mathrm{SKIING}_{10}$ , we added an additional 200 frames of NOOP actions after the 10 timesteps ( $= 300$  frames) in each episode. This is necessary to correctly reflect the reward incentives in SKIING with longer horizons.
Finally, we scaled the rewards for many Atari games to make the reward scale more uniform across different games. Often, when deep RL is applied to Atari, rewards are clipped to  $[-1, 1]$  to avoid instability. However, the MDP with clipped rewards may have a different optimal policy than the unclipped MDP. Thus, instead of clipping, we use scaling. We generally choose the scale factor based on the multiples of points received in the game: for instance, in ATLANTIS, rewards are always received in multiples of 100 so we scale by  $1/100$ . Table 5 lists the reward scaling factors for all games where we apply scaling.
Game	Reward scaling factor
ALIEN	1/10
AMIDAR	1/10
ASSAULT	1/21
ASTERIX	1/50
ASTEROIDS	1/10
ATLANTIS	1/100
BANKHEIST	1/10
BATTLEZONE	1/1000
BEAMRIDER	1/44
CENTIPEDE	1/100
CHOPPERCOMMAND	1/100
CRAZYCLIMBER	1/100
DEMONATTACK	1/10
FROSTBITE	1/10
GOPHER	1/20
HERO	1/25
KANGAROO	1/100
MONTEZUMAREVENGE	1/100
MSPACMAN	1/10
NAMETHISGAME	1/10
PHOENIX	1/20
PRIVATEEYE	1/100
QBERT	1/25
ROADRUNNER	1/100
QEAQUEST	1/20
SKIING	1/100
SPACEINVADERS	1/5
TIMEPILOT	1/100
VIDEOPINBALL	1/100
WIZARDOFWOR	1/100
Table 5: Factors by which Atari games rewards are scaled by in BRIDGE, for those where we apply reward scaling.
Progen levels The Progen benchmark [44] consists of 16 games. For each game, one can generate an arbitrary number of random levels, each of which is identified by a seed. Furthermore, each game has an "easy" and "hard" difficulty, each with different levels, and some have an additional "exploration" level which presents a particularly difficult exploration challenge.
While the benchmark is designed to measure generalization of RL agents trained on some number of levels to unseen levels, we use each level as a separate MDP. For each game, we attempted to construct an MDP for the easy levels with seeds 0, 1, and 2, the hard level with seed 0, and the exploration level if it exists for that environment. We denote  $\mathbf{MAZE}_{30}^{\mathrm{E1}}$  to be the easy level with seed 1 for the MAZE game limited to  $T = 30$  timesteps;  $\mathbf{MAZE}_{30}^{\mathrm{H0}}$  is the analogous hard level with seed 0 and  $\mathbf{MAZE}_{30}^{\mathrm{EX}}$  is the exploration level.
We generally increased the horizon for each game to the highest value in  $\{10, 20, 30, 40, 50, 70, 100, 200\}$  before the number of states was greater than 100 million. The horizon values we ultimately chose can be seen in the table in Appendix G.3.
Minigrid gridworlds Minigrid [14] is an extensible framework for building gridworlds. We considered all the pre-built gridworlds included in Minigrid for inclusion in BRIDGE except for those requiring natural language observations for specifying the task to be completed. We also excluded gridworlds with more than 1 million states, since for technical reasons we were unable to parallelize the construction of tabular MDPs for Minigrid. For gridworlds with randomized start states, we chose the start state with seed 0. We use  $T = 100$  for all the gridworlds.
E.2 Constructing tabular representations
For each of the environments described above, we wrote a program to compute a full tabular representation of the transition function  $f$  and reward function  $R$ . Our program uses a search procedure to iteratively explore every state-action pair. We keep a queue of states that need to be explored, which at first is just the initial state. In parallel, a number of worker threads take states from this queue. After popping a state, a worker thread sets the environment to that state and then takes a previously unexplored action, storing the resulting next state and reward. If there are still unexplored actions in the current state, it adds it back to the queue. If the next state is not terminal and has not had all its actions, it can continue this process.
While the search procedure for exhaustively enumerating states is conceptually simple, we experienced difficulties implementing it efficiently due to the massive scale of some of the MDPs in BRIDGE. For instance, the full state of the Atari simulator used in the ALE is about 10-12 KB of data. Storing 100 million states by themselves would thus require over one terabyte of memory! We avoided this problem by aggressively compressing state data using dictionary compression. Other challenges included efficiently parallelizing the data structures we used to store the queue of states, the transition function, and the reward function. Our final implementation is able to explore more than 20,000 state-action pairs per second in PONG while running on 64 cores.
Once we have enumerated all states and actions that can be reached in the given horizon, we also apply a consolidation step to reduce the number of states. Often, the internal representation of states in the Atari and Progen environments includes extra or superfluous data, which leads to duplicate states in our tabular representation. We repeatedly consolidate states that (a) have the same screen, (b) have the same rewards for each action, and (c) lead to the same next states for each action. When no more states can be consolidated, we store the resulting transition and reward functions.
We excluded any MDPs for which every sequence of actions results in the same total reward, since these are uninteresting from an RL perspective.
E.3 Reward shaping
For each Minigrid environment, we constructed one or more versions with shaped rewards for our experiments on the effects of reward shaping. We used three potential functions for shaping:
1.  $\Phi_{\mathrm{dist}}(s)$ : the negative distance from the state to the nearest goal. Distance is measured as the minimum number of moves needed to reach the goal, assuming there are no obstacles in the way.
2.  $\Phi_{\mathrm{doors}}(s)$ : the number of doors that are open.
3.  $\Phi_{\mathrm{pickup}}(s)$ : the number of objects that have been picked up at least once.
For one or more potential functions  $\Phi$ , we augment each reward  $R(s, a)$  with  $\Phi(f(s, a)) - \Phi(s)$ . The potential functions are chosen to incentivize useful behavior in the environments: moving towards goals, picking up objects like keys that could be helpful, and opening doors to reach more parts of the gridworld.
For each Minigrid MDP, we use all potential functions that apply to that MDP. For instance, if an MDP does not have any doors, we do not use  $\Phi_{\mathrm{doors}}$ . We also apply the combination of  $\Phi_{\mathrm{doors}}$  and  $\Phi_{\mathrm{pickup}}$  if both are applicable to an MDP.
When analysing the reward shaping results, we only include MDPs for which PPO/DQN converged on both the unshaped and shaped versions.
E.4 Datasheet for BRIDGE
We provide a datasheet, as proposed by Gebru et al. [55], for the BRIDGE dataset.
E.4.1 Motivation
For what purpose was the dataset created? We have described the purpose extensively in the paper: we aim to bridge the theory-practice gap in RL. BRIDGE allows this by providing tabular representations of popular deep RL benchmarks such that instance-dependent bounds can be
calculated and compared to empirical RL performance.
Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)? Not specified for the double-blind reviewing process.
Who funded the creation of the dataset? Also not specified for the double-blind reviewing process.
Any other comments? No.
E.4.2 Composition
What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? The instances are Markov Decision Processes (MDPs).
How many instances are there in total (of each type, if appropriate)? There are 155 MDPs in BRIDGE. They include 67 MDPs based on Atari games from the Arcade Learning Environment [43], 55 MDPs based on Progen games [44], and 33 MDPs based on MiniGrid gridworlds [14].
Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? The MDPs in BRIDGE are based on a small subset of the many environments that are used for empirically evaluating RL algorithms. We aimed to cover a range of the most popular environments. To make our analysis possible, we excluded environments that were not deterministic or did not have discrete action spaces. We also reduced the horizon of many of the environments to make it tractable to compute their tabular representations.
What data does each instance consist of? For each MDP, we provide the following data:
- A transition function and a reward function, which are represented as a matrix with an entry for each state-action pair in the MDP.
- A corresponding gym environment [56] that can be used to train policies for the MDP with various RL algorithms.
- Properties of the MDP that are calculated from its tabular representation, including the effective planning window, bounds on the effective horizon, bounds on the covering length, etc.
- Results of running RL algorithms (PPO, DQN, and GORP) on the MDP. This includes the empirical sample complexity as well as various metrics logged during training.
- For MiniGrid MDPs, there are additional versions of the MDP with shaped reward functions (see Appendix E.3) which also include all of the above data.
- For Atari and Progen MDPs, there is additionally a non-uniform exploration policy (see Appendix F.2). For Atari games, this is trained via behavior cloning from the Atari-HEAD [57] dataset; for Progen games, it is trained on other Progen levels. We include the above data recalculated using the non-uniform exploration policy in place of the uniformly random exploration policy.