SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Is there a label or target associated with each instance? In this paper, we aim to bound and/or estimate the empirical sample complexity of RL algorithms, so these could be considered targets for each instance.
Is any information missing from individual instances? There is no information missing.
Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)? No.
Are there recommended data splits (e.g., training, development/validation, testing)? No.
Are there any errors, sources of noise, or redundancies in the dataset? We do not believe there are errors or sources of noise in the dataset. The tabular representations of the MDPs have been carefully tested for correspondence with the environments they are based on. There is some redundancy, as many Atari games are represented more than once with varying horizons.
Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? The dataset is mostly self-contained, except that the gym
environments rely on external libraries. There are archival versions of these available through package managers like PyPI.
Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor- patient confidentiality, data that includes the content of individuals' non-public communications)? No.
Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? No.
E.4.3 Collection process
How was the data associated with each instance acquired? The data was collected using open-source implementations of each environment.
What mechanisms or procedures were used to collect the data (e.g., hardware apparatuses or sensors, manual human curation, software programs, software APIs)? As described in Appendix E.2, we developed a software tool to construct the tabular representations of the MDPs in BRIDGE. We validated the correctness of the tabular MDPs through extensive testing to ensure they corresponded exactly with the gym implementations of the environments.
If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)? The MDPs in BRIDGE were selected from three collections of commonly used RL environments: the Arcade Learning Environment, ProcGen, and MiniGrid. We chose these three collections to represent a broad set of deterministic environments with discrete action spaces. Within each collection, the environments were further filtered based on the criteria described in Appendix E.1.
Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)? Only the authors were involved in the data collection process.
Over what timeframe was the data collected? The dataset was assembled between February 2022 and January 2023. The RL environments from which the MDPs in BRIDGE were constructed were created prior to this; see the cited works for each collection of environments for more details.
Were any ethical review processes conducted (e.g., by an institutional review board)? No.
E.4.4 Preprocessing/cleaning/labeling
Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? Yes, various preprocessing and analysis was done. See Appendix E.2 for details.
Was the "raw" data saved in addition to the preprocessed/cleaned/ labeled data (e.g., to support unanticipated future uses)? Yes, this is included with the dataset.
Is the software that was used to preprocess/clean/label the data available? Yes, this is available with the rest of our code.
E.4.5 Uses
Has the dataset been used for any tasks already? The dataset has thus far only been used to validate our theory of the effective horizon in this paper.
Is there a repository that links to any or all papers or systems that use the dataset? There is not. However, we will require that any uses of the dataset cite this paper, allowing one to use tools like Semantic Scholar or Google Scholar to find other papers which use the BRIDGE dataset.
What (other) tasks could the dataset be used for? We hope that the BRIDGE dataset is used for further efforts to bridge the theory-practice gap in RL. The dataset could be used to identify other properties or assumptions that hold in common environments, or to calculate instance-dependent sample complexity bounds and compare them to the empirical sample complexity of RL algorithms.
Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses? As we have already mentioned, BRIDGE is restricted to deterministic MDPs with discrete action spaces and relatively short horizons. This could mean that analyses of the dataset like ours do not generalize to the broader space of RL environments that may have continuous action spaces, stochastic transitions, and/or long horizons. We have included some experiments, like those in Appendix F.1, to show that our theory of the effective horizon generalizes beyond the MDPs in BRIDGE. We encourage others to do the same and we hope to address some of these limitations in the future with extensions to BRIDGE.
Are there tasks for which the dataset should not be used? We do not foresee any particular tasks for which the dataset should not be used.
E.4.6 Distribution
Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? Yes, we will distribute the dataset publicly.
How will the dataset will be distributed (e.g., tarball on website, API, GitHub)? We are still finalizing the method through which the dataset will be distributed.
When will the dataset be distributed? We plan to make the dataset public in May or June 2023.
Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? It will be distributed under CC-BY-4.0.
Have any third parties imposed IP-based or other restrictions on the data associated with the instances? The Atari ROMs used to construct the Atari MDPs in BRIDGE are copyrighted by the original creators of the games. However, they are widely used throughout the reinforcement learning literature and to our knowledge the copyright holders have not complained about this. Since we are not legal experts, we do not know if releasing our dataset violates their copyright, but we do not believe that we are harming them since the tabular representations in BRIDGE are only useful for research purposes and cannot be used to play the games in any meaningful way.
Do any export controls or other regulatory restrictions apply to the dataset or to individual instances? No.
E.4.7 Maintenance
Who will be supporting/hosting/maintaining the dataset? We (the authors) will support and maintain the dataset.
How can the owner/curator/manager of the dataset be contacted (e.g., email address)? Redacted for double-blind review.
Is there an erratum? We will record reports of any errors in the dataset and release new versions with descriptions of what was fixed as necessary.
Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)? We will release new versions of the dataset to correct any reported errors as described above. We may also expand the dataset in the future with more MDPs or new kinds of MDPs, such as stochastic or continuous-action-space MDPs. Any updates will be communicated through the service we use to host the dataset (TBD).
If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were the individuals in question told that their data would be retained for a fixed period of time and then deleted)? No.
Will older versions of the dataset continue to be supported/hosted/maintained? If so, please describe how. If not, please describe how its obsolescence will be communicated to dataset consumers. We hope to find a host for the dataset that will retain older versions of the dataset. We only plan to maintain the latest version of the dataset, however. We will note this policy in the
dataset's description.
If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so? There is no predefined mechanism to contribute to the dataset, but we will consider external contributions on a case-by-case basis. We encourage others to extend and build on the dataset.
F Experiment details
In this appendix, we describe details of the experiments from Section 6. In particular, we describe how we calculate the empirical sample complexity of PPO and DQN.
We use the implementations of PPO and DQN from Stable-Baselines3 (SB3) [58]. For the network architectures, we use convolutional neural nets (CNNs) for Atari and Progen and a fully-connected network for Minigrid. The network architectures are the default CNN and fully-connected architectures chosen by SB3. We used the below hyperparameters for PPO and DQN, which are mainly taken from the tuned Atari hyperparameters in the RL Baselines3 Zoo [59] repository.
We used a discount rate of  $\gamma = 1$  for all environments and algorithms except for PPO and DQN in MiniGrid environments, where we used  $\gamma = 0.99$ . We found that the performance of PPO and DQN significantly degraded when we used  $\gamma = 1$  for MiniGrid.
PPO We use the following hyperparameters for PPO:
Hyperparameter	Value
Training timesteps	5,000,000
Number of environments	8
Number of steps per rollout	{128,1280}
Clipping parameter (ε)	0.1
Value function coefficient	0.5
Entropy coefficient	0.01
Optimizer	Adam
Learning rate	2.5 × 10-4
Number of epochs per training batch	4
Minibatch size	256
GAE coefficient (λ)	0.95
Advantage normalization	Yes
Gradient clipping	0.5
Table 6: Hyperparameters we use for PPO.
For each environment, we try rollout lengths of 128 and 1,280, as we find this is the most sensitive hyperparameter to tune.
DQN We use the following hyperparameters for DQN:
Hyperparameter	Value
Training timesteps	5,000,000
Timesteps before learning starts	0
Replay buffer size	100,000
Target network update interval	1,000
Training frequency	4
Gradient steps per training step	1
Optimizer	Adam
Learning rate	10-4
Exploration fraction	{0.1, 1}
Final ε	0.01
Learning rate	10-3
Gradient clipping	10
Table 7: Hyperparameters we use for DQN.
We try decaying the  $\epsilon$  value for  $\epsilon$ -greedy over the course of either 500 thousand or 5 million timesteps, as we found this was the most sensitive hyperparameter to tune for DQN.
Calculating the empirical sample complexity In order to compute the empirical sample complexities of PPO and DQN, throughout training we run evaluation episodes and see if the algorithms have discovered an optimal policy yet. During the evaluation, PPO policies take actions according to the argmax over the probabilities they assign to each action, rather than sampling as during training episodes. DQN takes actions greedily with respect to its current Q-function (i.e., with  $\epsilon = 0$ ). If the total episode reward during the evaluation is the optimal return, then we terminate the training run and record the total number of timesteps interacted with the environment as the empirical sample complexity. We take the median sample complexity over 5 random seeds and then the minimum over all hyperparameter settings to get the final empirical sample complexity.
We found that in some environments PPO achieved optimal reward during almost all the training episodes but none of the evaluation episodes. This can happen if a policy does not assign the highest probability to an optimal action in some states but can make up for this by being very likely overall to obtain the highest possible total reward. Thus, if more than half of the training episodes during an iteration achieve the optimal return, we also count this as converging to an optimal policy for the purposes of calculating the empirical sample complexity.
F.1 GORP vs. deep RL algorithms over longer horizons
To show that GORP is not just effective over short horizons, we ran additional experiments comparing GORP, PPO, and DQN in more typical Atari benchmark environments with frameskip 4 and a horizon of  $T = 27,000$  (corresponding to a maximum of 30 minutes of gameplay). Similarly to the environments in BRIDGE, we limit agents to a single life and make the environments entirely deterministic. The hyperparameters for PPO and DQN are identical to those given in Tables 6 and 7 except for the following changes: we train for 50 million timesteps; we use a discount rate of  $\gamma = 0.99$ ; and, we set an entropy coefficient of 0.01 for PPO.
We use a training batch size of  $10^{4}$  for PPO and decay  $\epsilon$  for DQN over the course of the first 5 million timesteps. For GORP, we use  $k = 1$  and tune  $m$  for each environment.
F.2 Exploration policies
For the experiments in Section 6 we needed pre-trained policies to initialize PPO with; here, we describe the details of how we trained them. We used two different training methods: one for Atari environments and one for Progen environments.
In the Atari environments, we trained policies via behavior cloning (BC), i.e., supervised learning, from human data in the Atari-HEAD dataset [57]. We resampled the actions and processed the screen images from the dataset to align with the frameskip and observation preprocessing of our Atari environments. We trained a BC policy on each environment for 400 batches of 500 timesteps each. We used Adam with a learning rate of  $10^{-3}$ . We also added an entropy bonus to the loss function with
a weight of 0.1 to avoid the BC policy assigning very little weight to some actions. Our theoretical results in Section B.1 suggest that this should improve the sample complexity. Since not all Atari games are included in Atari-HEAD, we only used a subset for the experiments in Section 6.
In the Progen environments, we pre-trained policies on a set of levels not included in BRIDGE. In particular, we trained a policy with PPO on 500 easy levels for 25 million timesteps, which in very similar to the methodology in Cobbe et al. [44]. We also use an entropy bonus in PPO with weight 0.1 for the same reason as above.
Once we have the pre-trained policies, we compute tabular representations of them on the corresponding MDPs in BRIDGE. To do so, we feed the observations for every state in the MDP through the pre-trained policy network and record the resulting action distribution. This allows us to compute  $Q^{\pi^{\mathrm{expl}}}$  and thus obtain bounds on the effective horizon when using the pre-trained exploration policy.
F.3 Computational resources
For deep RL experiments, we used a mix of A100, A4000, and A6000 GPUs from Nvidia. We ran the algorithms either on separate GPUs or sometimes we ran multiple random seeds simultaneously on the same hardware. We used 1-8 CPU threads to run the RL environments. Using this setup, PPO and DQN generally took 2-8 hours to complete 5 million timesteps of training. We used early stopping when the algorithms found an optimal policy before 5 million timesteps, so the amount of compute per experiment was often less than this.
For constructing and analyzing the tabular MDPs in BRIDGE, we used up to 128 CPU threads and 500 GB of memory. The amount of time necessary to construct and analyze the MDPs ranged from less than a minute to around 5 days.
G Additional experiment results
Here, we present additional results from the experiments in Section 6.
G.1 Example of the effective horizon failing to predict generalization
As we described in the discussion, the effective horizon cannot model generalization across different states. For instance, in PONG-30 (Pong limited to 30 timesteps/15 seconds), the effective horizon gives a sample complexity of roughly 5 billion timesteps and empirically GORP takes over 80 million timesteps to converge to an optimal policy (Appendix G.3). However, both PPO and DQN converge in under 500,000 environment steps. We hypothesize this is because they are able to generalize the skill of hitting the ball across the multiple rounds of the Pong game, which the effective horizon cannot capture because it considers learning separately at every timestep.
G.2 Additional plots
Figure 6: The distribution of the minimum values of  $k$  for which the MDPs in BRIDGE are  $k$ -QVI solvable. About two thirds are 1-QVI-solvable, meaning they can be solved by simply acting greedily with respect to the Q-function of the random policy. The MDPs are split into those which PPO can and cannot solve in 5 million steps; among those that can be solved efficiently, the values of  $k$  are even lower.
Figure 7: A comparison of sample complexity bounds and the empirical sample complexities of PPO and DQN across the MDPs in BRIDGE. In each plot, every dot represents one MDP and its color indicates which benchmark it comes from. Our effective horizon-based bound most closely correlates with empirical sample complexity. See Table 2 for a quantitative comparison of the bounds.
Figure 8: A comparison between the empirical change in sample complexity and the change predicted by sample complexity bounds due to reward shaping. See Table 3a for a quantitative comparison.
Figure 9: A comparison between the empirical change in the sample complexity of PPO and the change predicted by sample complexity bounds due to initializing with a pre-trained policy. The initial policies for Atari are trained from human data and those for Progen are trained on other procedurally generated levels. See Table 3b for a quantitative comparison.
G.3 List of MDPs in BRIDGE with statistics
The following table lists all the MDPs in BRIDGE, along with various properties: the number of states  $S$ , number of actions  $A$ , horizon  $T$ , minimum  $k$  for which the MDP is  $k$ -QVI-solvable, a bound on the effective horizon using the techniques in Appendix C, a bound on the covering length  $L$  using Lemma D.4, and the effective planning window  $W$ .
G.4 Table of bounds and empirical sample complexities
This table lists all the sample complexity bounds we calculate for each MDP in BRIDGE along with the empirical sample complexities of PPO, DQN, and GORP.
G.5 Table of returns
This table lists the returns for the random and optimal policies in each MDP as well as the final returns achieved by PPO, DQN, and GORP. We ran PPO and DQN for 5 million timesteps in each environment, and GORP for both 5 million and 100 million timesteps.
G.6 Learning curves for BRIDGE MDPs
Figure 10: Learning curves for PPO, DQN, and GORP on all the MDPs in BRIDGE. Solid lines show the median return (over multiple random seeds) of the policies learned by each algorithm throughout training. We use 5 random seeds for PPO and DQN and 101 random seeds for GORP. The shaded region shows the range of returns over all random seeds for PPO and DQN, and shows the range from the 10th to 90th percentile over random seeds for GORP. The optimal return in each environment is shown as the dashed black line.