The Generalization Gap in Offline Reinforcement Learning
Authors: Ishita Mediratta
Qingfei You
Minqi Jiang
Roberta Raileanu
Venue: ICLR 2024 poster (2024)
Keywords: Offline RL
Dataset
Generalization
Procgen
Webshop
Primary area: reinforcement learning
TL;DR: We assess generalization capabilities of most widely used offline learning algorithms and introduce offline RL datasets from Procgen and Webshop environments to facilitate research on generalization in Offline RL algorithms

Abstract: Despite recent progress in offline learning, these methods are still trained and tested on the same environment. In this paper, we compare the generalization abilities of widely used online and offline learning methods such as online reinforcement learning (RL), offline RL, sequence modeling, and behavioral cloning. Our experiments show that offline learning algorithms perform worse on new environments than online learning ones. We also introduce the first benchmark for evaluating generalization in offline learning, collecting datasets of varying sizes and skill-levels from Procgen (2D video games) and WebShop (e-commerce websites). The datasets contain trajectories for a limited number of game levels or natural language instructions and at test time, the agent has to generalize to new levels or instructions. Our experiments reveal that existing offline learning algorithms struggle to match the performance of online RL on both train and test environments. Behavioral cloning is a strong baseline, outperforming state-of-the-art offline RL and sequence modeling approaches when trained on data from multiple environments and tested on new ones. Finally, we find that increasing the diversity of the data, rather than its size, improves performance on new environments for all offline learning algorithms. Our study demonstrates the limited generalization of current offline learning algorithms highlighting the need for more research in this area.

Published as a conference paper at ICLR 2024
THE GENERALIZATION GAP IN OFFLINE REINFORCEMENT LEARNING
Ishita Mediratta" Qingfei You*" Minqi Jiang Roberta Raileanu
Meta, @ University College London
ABSTRACT
Despite recent progress offline learning, these methods are still trained and tested the same environment. In this paper. compare the generalization abilities widely used online and offline learning methods such as online reinforcement learning (RL), offline RL, sequence modeling, and behavioral cloning. Our experi- ments show that offlin learning algorithms perform worse on new environments than online learning ones. We also introduce the first benchmark for evaluating generalization in offline learning, collecting datasets of varying sizes and skill- levels from Proegen (2D video games) and WebShop (e-commerce websites). The datasets contain trajectories for limited number of game levels or natural language instructions and at test time, the agent has to generalize to new levels or instructions. Our experiments reveal that existing offline learning algorithms struggle to match the performance of online RI on both train and test environments. Behavioral cloning is a strong baseline, outperforming state-of-the-ar offline RL and sequence modeling approaches when trained on data from multiple environments and tested new ones. Finally, we find that increasing the diversity of the data, rather than its size, improves performance on new environments for all offline learning algorithms. Our study demonstrates the limited generalization of current offline learning algorithms highlighting the need for more research in this area.
INTRODUCTION
Training agents offline from static datasets (Levine et al.) 2020 Reed ct al. 2022
Lee et al. 2022)
has demonstrated significant potential in application domains where online data collection can be expensive or dangerous, such as healthcare (Liu et al., 2020a), education (Singla et al.) 2021
robotics (Singh et al. (002)), or autonomous driving (Kiran et al. 2021 Prudeecio et al. 2023). ). The abillity generalize new scenarios is crucial for the safe deployment of these methods particularly in such high-stakes domains. However, the generalization abilities of offline learning algorithms to new environments (with different initial states, transition functions, reward functions) remains underexplored. A key reason for this is that existing offline learning datasets predominantly focus on singleton environments where all trajectories are from the same environment (such as playing an Atari game or making a humanoid walk), thereby limiting the evaluation of generalization. In contrast, there is a large body of work focused on evaluating and improving the generalization of online reinforcement learning (RL) methods Cobbe et al. (2018 8); Packer et al. (2019); Zhang et al. (2018a by: Cobbe et al. (2020); Küttler et al. (2020) Raileanu et al. 2021]) Raileanu & Fergus (2021); Jiang et al. (202 la). This paper aims bridge this gap by assessing the generalization performance
of offline learning algorithms (including hehavioral cloning, sequence modeling (Chen et al., 2021 and state-of-the-arr offline RL (Fujimoto et al. 2018 2019 Kumar et al., | Kostrikov al. 2020) approaches) in two different scenarios: (1) unseen levels in the case of Proegen (Cobbe et al. 2020 and (2) unseen instructions in the case of WebShop (Yao al. (022). Our results show that none of the benchmarked offline learning methods, (i.e. BCQ (Fujimoto et al. 2018), CQL (Kumar al. 2020a), IQL (Kostrikov et al. 2021). BCT, and DT (Chen et al. 20.1)) are able to generalize as well as behavioral cloning (BC), underscoring the need of developing offline learning methods with better generalization capabilities.
These authors contributed equally this work

---

Published as a conference paper at ICLR 2024
Proggen
Instructions
Instructions
TRAIN
TEST
Instruction 1:1 am looking for yuani pic stai miees dlear
Instruction 1: am looking for high power sound sutwooter uses bruetooth 660.00 dellara
maze
i
your longue this affective tongue sctaper for wcbulli kids, ea lower than 40.00 dollara
tongue
conted hyglene
system, price then
dodgeball
Instruction looking
electric
Instruction would
inch gold
toot grinde and price lower than doliers
throw pillow cover my living room, price lower then 00
ninja
Instruction nm looking for a 4-Eie sthiliing unit sepresso claseeic wlao, it stroudd have engineernt sotodd and price lower than 50 00 dallars
Instruction Am trying find bluetooth with diallers
caveflyer
lower than 50
(a)
(b)
Figure 1: (a) Sample screenshots from the train and test environments of four Procgen games. (b) Sample instructions (item descriptions) from the train and test set of human demonstrations from WebShop. Red and blue highlight the type and attributes of the desired item, respectively.
In this work, we first introduce a collection of offline learning datasets of differrent sizes and skill- levels from Procgen (Cobbe al., (020) and WebShop (Yao et al. 2022 to facilitate a comprehensive evaluation of the generalization abilities of offline learning algorithms. The Procgen benchmark consists of 16 procedurally generated 2D video games which differ in their visual appearances, layouts, dynamics, and reward functions. Since the levels are procedurally generated, generalization to new levels can be assessed in this benchmark. We create number of Procgen datasets that aim to test an agent's 's ability of solving new levels i.c., with the same reward function but different initial states and dynamics.
Webshop is a simulated e-commerce website environment with more than 1 million real-world products. Given a text instruction describing the desired product, the agent needs navigate multiple types of webpages and issue different actions to find, customize, and purchase an item. We create a number of WebShop datasets both by using the human demonstrations provided and by generating suboptimal trajectories, which aim to test an agent's ability of following new instructions i.e., with the same dynamics but different initial states and reward functions. Figure I shows some sample observations from Procgen and trajectories from WebShop.
We then benchmark a variety of widely-used offline learning algorithms,, allowing us to verify the generalization of these algorithms and establish baselines for future research. On the expert and suboptimal datasets from Procgen (Section 4.1 all offline learning methods underperform online RL at test time, with BC generally outperforming offline RL and sequence modeling approaches. On WebShop's human demonstrations dataset (Section 4.6), BC again outperforms the offline learning methods in terms of both final score and success rate. We also study the generalization of these algorithms as the diversity (Section (444) and size (Section of the training data increases, observing that an increase in data diversity significantly improves generalization while increasing the size of training data does not. These findings not only provide insights into the strengths and weaknesses existing algorithms but also emphasize the necessity for more research on understanding and improving generalization in offline learning. We hope our open- sourced datasets, baseline implementations, and proposed evaluation protocols can help lower the barrier for future research in this area.
BACKGROUND
This work studies the effectiveness of offline learning algorithms in contextual Markov decision processes (CMDPs; Hallak et al. 2015). A CMDP is defined as tuple M (C, A,A,M(c) where C is the set of contexts, is the set of states, .A is the set of actions and M is a function that maps a specific context
C Markov decision process (Sutton & Barto 2018 MDP) M(c) = AS, TT, TR, Rr, ,P²), where T : S x A x C - S is the contextual transition function, R S C + R is the contextual reward function, and pf is the initial state distribution conditioned on the context. Given a CMDP, reinforcement learning (RL) seeks to maximize the value of the agent's policy æ defined as E EE T
where re is the reward at time t and is the time horizon.
Note that C is not observable. The space of contexts C will be split into training set Cirasn validation set Crats and test set Crest- The training contexts are used for generating the datasets, the validation ones used for performing hyperparameter sweeps and model selection, and the
2

---

Published as a conference paper at ICLR 2024
test sets used to evaluate the agents via online interactions with the environment. In the case of Proegen, the context с corresponding to an instance of the environment (or level) an M(c) determines the initial state and transition function. In the case of Webshop, the context e corresponding to an instance of the environment (or instruction) M(c) determines the initial state and reward function.
In this paper, the agents learn from offline dataset 'D which contains trajectories generated by behavior policy B by interacting with the training contexts. This policy can have different degrees of expertise (e.g., expert or suboptimal), hence the offline learning algorithms must learn extract meaningful behaviors using this static dataset without access to any online interactions with the environment. Since the dataset is fixed, it typically does not cover the entire state-action distribution of the environment. Because of this, offline RL algorithms can suffer from distributional shift and hence, they must employ techniques that enable them to generalize to new states at test time (Levine et al. 2020) or prevent sampling actions which are out-of distribbtino (Kumar et al. 2020a Fujimoto et al.
2018
Kostrikov et al. 2021
EXPERIMENTAL SETUP
Cral 50, and [Crese = 100, in line with prior work (Cobbe et al. 2020 Raileanu & Fergus 2021 Jiang et al. 2021b), while for WehShop we use Ctrainl = 398,
For Procgen, we use [Crrain 200,
Coal 54, and Creatl 500 instructions, unless otherwise noted (Yao et al. 2022). We collect
the following datasets as part of our offline learning benchmark: (1) Procgen expert dataset with
1M transitions, (2) Procgen mixed expert-suboptimal dataset with 1M transitions, (3) Procgen expert dataset with 10M transitions, (4) Procgen suboptimal with 25M transitions, (5) WebShop human dataset with 452 trajectories, and (6) WebShop suboptimal datasets with 100, 1K, 1.5K, 5K, and 10K trajectories.
For the Procgen datasets, we evaluate 7 methods which are competitive on other offline learning benchmarks (Kurin et al., 2017; Fu et al., 2020 Agarwal et al.) 2020 Todorov et al.. 2012) and frequently used in the literature (Fu et al. 2020; Gulcehre et al. 2020, Qin et al. 2022): (1)
Behavioral Cloning (BC), (2) Batch Constrained O-Learning (BCQ) Fujimoto et al. (2019) (3)
Conservative Q-Learning (CQL) Kumar et al. (2020a), (4) Implicit Q- Q-Learnin (QQL) Kostrikoy
et al. (2021), (5) Behavioral Cloning Transformer (BCT) Chen et al. (2021), and (6) Decision Transformer (DT) [hen et al. (202I). For the WebShop datasets, we evaluate BC, CQL and BCQ. We cannot evaluate existing transformer-based approaches such DT or BCT due their limited context lengths of the underlying causal transformer. Many WebShop states have 512 tokens, so we typically cannot fit multiple (state, action) pairs in the transformer's context. Similarly for IQL, is not straightforward to implement the loss function since in WebShop, the action space differs for each state so the size of the action space not fixed. Since these algorithms cannot be applied to WebShop without significant changes, they are out-of-scope for this paper.
4 EXPERIMENTAL RESULTS
4.1 GENERALIZATION TO NEW ENVIRONMENTS USING EXPERT DATA
Figure 2 shows the IQM performance (Agarwal et al. 20216) of baselines averaged across all 16 Procgen games when trained using the IM expert dataset, normalized using the min-max scores provided in Cobbe et al. (2020). As we can see, BC outperforms all other sequence modeling or offline RL methods by significant margin on both train and test levels. This is in line with prior work which also shows that BC can outperform offline RL algorithms when trained on expert trajectories (Levine et al. 2020). Sequence modeling approaches like DT and BCT perform better than offline RL methods on the training environments, but similarly or slightly worse on the test environments. The gap between BC and DT or BCT is small for training. but large for test. This indicates that, relative to standard BC, transformer-base based policies like DT or BCT, may struggle more with generalization to new scenarios, even if they are just as good on the training environments. For per-game performance, refer to Figures 21 22 and Table 5 in Appendix N
3

---

Published as a conference paper at ICLR 2024
Procgen Expert Dataset
PPO Train Performance
Min-Max: Normalzzed IOM Reture
0.5
PPO Test Performance
Expert Daraset Average
Train
0.3
0.3
0.0
BCQ
IQL
Figure 2: Performance on Proegen 1M Expert Dataset. Train and test min-max normalized returns aggregated across all 16 Procgen games, when trained expert demonstrations. Each method was evaluated online across 100 episodes on levels sampled uniformly from the test The IQM aggregate metric is computed over 5 model sceds, with the error bars representing upper (75th) and lower (25th) interval estimates. BC outperforms all offline RL and sequence modelling approaches on both train and test environments. All offline learning methods lag behind online RL on both train and test.
Generalization New Environments (Expert Dataset)
environments, underperforming online test time even when trained on expert demonstrations (generated by fully- online RL agents). Behavioral cloning a competitive approach, outperforming state-of-the-art offline RI. and sequence modeling methods on both train and test environments.
Existing offline leamnnng methods struggle to generalize
4.2 GENERALIZATION TO NEW ENVIRONMENTS USING MIXED
DATA
envi- ronments when trained on expert demonstrations, Prior works (Bhargava et al., [023 Kumar et al. 2020a Kostrikov al., 2 singleton environments (where agents are trained and tested on the same environment) show that state-of-the-ar offline RL methods typically outperform BC when trained on suboptimal demonstrations. In this section, we investigate whether this finding holds when agents are trained and tested on different environments. For this, we create mixed, expert-subeptimal dataset by uniformly mixing data from the expert PPO checkpoint and another checkpoint whose performance was 50% that of the expert. Therefore, these datasets have average episodic returns of about 3/4th those of the expert datasets.
In the previous section, observed that offline RL methods struggle generalize
Contrary to prior results singleton environments, Figure 3 shows that even with suboptimal data, BC outperforms other offline learning baselines on test levels. However, all methods have a similar generalization gap average (Figure 3 from Appendix N). suggesting that their generalization abilities are similar. This result indicates that BC can train better on diverse datasets containing trajectories from different environments relative other offline learning approaches, even if these demonstrations are subotpimal. In Procgen and other CMDPs, it common for methods with better training performance to also have better test performance since learning solve all the training tasks is non-trivial and existing algorithms are typically underfitting rather than overfitting. Hence, in such settings the challenges lie both in optimization and generalization (Jiang al.).
IQL, which achieved state-of-the-arr on other singleton environments (Kostrikov et al. 202T> strug- gles to train well on data from multiple levels and also fails to generalize new levels at test time. Thus, it appears that training on more diverse datasets with demonstrations from different environ- ments and generalizing to new ones poses a significant challenge offline RI. and sequence modeling approaches, despite their effectiveness when trained on more homogenous datasets from a single environment. However, this finding is not necessarily surprising since these offline learning methods have been developed singleton environments and haven't been evaluated on unseen environments (different from the training ones). Hence, we believe the community should focus more on the setting propose here (testing agents in different environments than the used to collect training data) to improve the robustness of these algorithms and make them better suited for real-world applications where agents are likely to encounter scenarios test time.
4

---

Published as a conference paper at ICLR 2024
Procgen 1M Mixed Expert Suboptimal Dataget
Mixed Detaset Average
Min-Max Normalzed IOM Reture
0.5
Train
0.3
0.2
0.3
0.0
BCQ COL
Figure 3: Performance on Proegen 1M Mixed Expert-Suboptimal Dataset. Train and test min-max normal- ized returns aggregated across all 16 Procgen games, when trained on mixed expert-suboptimal demonstrations. Each method evaluated online across 100 episodes levels sampled uniformly from the test set. The IQM aggregate metric is computed over model seeds, with the error bars representing upper (75th) and lower (25th) interval estimates. Similar to Figure 2 BC outperforms all offline RL and sequence modelling approaches on both train and test environments. All offline learning methods lag behind online RL on both train and test.
Generalization to New Environments (Mixed
Dataset)
Behavioral cloning also outperforms bate-of-theeart offline RL and sequence modeling methods on both train and test environments when learning from suboptimal data from multiple environments.
4.3 TRAINING AND TESTING ON A SINGLE ENVIRONMENT
In the previous section, we found that, when trained on data form multiple environments and tested on new ones, BC outperforms offline RL algorithms. However, prior work showed that offline RL methods typically outperform BC when trained on suboptimal data from a single environment and tested the same one (Kumar et al. 2020a 2022 [hhargava et al. 2023). At the same time, BC has been shown to outperform offline RI. when trained on expert data from a single environment. Here, we aim verify whether these two observations hold in Procgen order confirm the correctness of our implementations.
Therefore, in this section, we show the results when training and testing agents on expert and suboptimal data from a single level. We conduct this experiment across two different datasets with either expert or suboptimal demonstrations, two different game levels with seeds 40 and 1, and all 16 Procgen games. We collect 100,000 expert trajectories in both of these levels by rolling out the final PPO checkpoint, and 100,000 suboptimal trajectories in both of these levels by uniformly sampling out the transitions from two checkpoints, the final one and another checkpoint whose performance is 50% that of the final checkpoint, similar to what we did in the previous section.
Figure shows the performance of these baselines when trained on the expert and suboptimal datasets from level 40. In Appendix | we report aggregate performance on the expert and suboptimal datasets, well per- game performance when training on the expert dataset, where on most games offline learning methods perform about as well PPO, with many of these algorithms achieving the maximum score possible (see Coinrun, Leaper, Ninja, Maze, Miner in Figure 10 from Appendix Tp. Figure [4] (bottom) shows the results when training on the suboptimal dataset, where offline RL methods either comparable or better than BC, which is in line with prior work. One exception is Chaser where most offline learning methods struggle to match the average dataset performance when learning from both expert and suboptimal demonstrations. This is not surprising since Chaser is highly stochastic environment containing multiple moving entities whose actions cannot be predicted. In such settings, one typically needs much more data to learn good policies offline (in order to cover larger fraction of the states that can potentially visited). The same applies for other games where the performance of offline learning lags behind that of online learning methods when trained and tested on the same environment. Prior work also shows that offline RL and sequence modeling approaches can struggle to learn robust policies in stochastic environments (Bhargava al. 2023 Brandfonbrener al. 2022 Ostrovski et al. 2021 Ghugare et al. 2024).
both datasets on all 16 games. Figure 4 (top) shows the results
We report the results on level 1 in Appendix | These results are consistent with the broader literature which shows that offline RL performs comparable to, or in some case, better than BC when trained and tested suboptimal demonstrations from the same environment. However, shown in previous
5

---

Published as a conference paper at ICLR 2024
Level Seed 40: Expert Dataset
Dataser perforance Level Seed
chaser
coinrun
jumper
Ieaper
Averngg Rebum
10
Level Seed 40; Mixed Expert- Expert-Subopae Suboptimal Dataset
Dataser Average
PPG parforance Level Soed 40
calorun
jumper
Iesper
Average Retur
Figure 4: Performance of each baseline across selected Procgen games when trained and tested on the same level using expert and suboptimal dataset. Here report performance selected levels: Chaser, Coinrun. Jumper, and Leaper. For all games, refer Figures and TO Appendix
sections, this finding does hold true when these algorithms are trained and tested on multiple different environments. In such settings, BC tends to outperform other offline learning methods as shown here.
Training and Testing on Single Environment
All offline learning algorithms perform well when trained and tested in the same environment but struggle learn and generalize when trained multiple environments and tested on ones. When trained and tested the environment using expert data, behavioral cloning performs best, expected. When trained on mixed expert-suboptinal data, offline RL performs comparable to or better than behavioral cloning on most games, which is in line with prior work.
4.4
THE EFFECT OF DATA DIVERSITY ON GENERALIZATION
Pan W Nemaates 124 tat.am
SEp Nometted IOe Ratuum
0.5
0..5
0.4
0.4
0..
0.3
0.3
0.2
0.1
400 800 Number training lovels
800 Number training levels
Number of training levels
(a) BC
(b) BCQ
(c) DT
Figure 5: The Effect of Data Diversity on Performance. Train and test performance of offline learning algorithms for varying number of training levels in the 1M expert datasets, aggregated across all Proegen games. The plot shows the IQM and bars represent the 75-th and 25th percentiles computed over model seeds. While the training performance doesn't change much with the number training levels, the test performance increases (and generalization gap decreases) with the diversity the dataset.
To investigate the role of data diversity on the generalization capabilities of offline learning algorithms, we conduct an experiment to analyze how the performance of each offline learning algorithm scales with the number of training levels while keeping the dataset size fixed to 1M transitions. We run these experiments on Procgen. We consider 200, 400, 800, 1k, 10k and 100k training levels. For each game, we train PPO policies for 25M steps on the corresponding number of levels. We then use the final PPO checkpoints (after 25M training steps) collect a total of 1M transitions (from the corresponding levels) and train each offline learning algorithm on these datasets (using the same hyperparameters for all datasets). To evaluate these policies, follow the procedure outlined in Section H.2.1 More specifically, for each dataset we randomly sample 100 test levels from [n, oo) where [250, 450, 850, 1050, 10050, 100050), respectively, and evaluate the models via online

---

Published as a conference paper at ICLR 2024
interactions with these levels. In each case, the levels from In - 51, used for evaluation and the remaining ones from [0, n - 51] are used for training. Since we use a fixed number of transitions for all datasets, the number of levels is proxy for the dataset diversity. Figure S shows that:
The Effect of Data Diversity on Generalization
Increasing the diversity the dataset by, for example, increasing the number of training environments while keeping the size of the dataset fixed, leads to significant performance improvements on new environments for all offline Icarning algorithms.
This finding suggests that in order to train better- generalizing agents using offline data, we should create diverse datasets covering a broad range of experiences. Note that the results presented here use 50 levels for validation for each run. We also experiment with proportional number of validation levels (with respect to the number of training levels) in Appendix K and find that it leads to the same conclusion.
4.5 THE EFFECT OF DATA SIZE ON GENERALIZATION
In other domains of machine learning, is well-established that dataset size can play piv- otal role in model performance and generaliza- tion (Kumar et al. 2022; Kaplan et al.) 2020). While the significance of the data diversity has been underscored in the previous section, the im- pact of the dataset size remains an open question. In this section, we investigate how generaliza- tion correlates with the dataset size when the diversity and quality of the dataset is fixed. For this, we scale the training datasets, both expert and suboptimal (in Appendix (). in Procgen by progressively increasing the dataset size from million to 5 million and subsequently to 10 million transitions. Throughout this scaling pro- cess, we keep all other hyperparameters same as well the number of training levels constant at 200. As can be seen in Figure O across four offline learning algorithms, there only a slight increase in both train and test performance as a consequence of increasing the dataset size. However, note that the generalization gap remains almost constant. This indicates that increasing the diversity of the dataset (while maintaining its total size) can lead to a bigger generalization improvement than increasing the size of the dataset (while maintaining its diversity) (see Figure 5).
Scaling Prosgen 1DN Export Dotaset
Tran
Vumber
Figure 6: The Effect of Data Size on Perfor- mance (Expert). Train and test min-max normal- ized IQM scores for BC. BCQ and DT the size of the dataset is increased from to 10 million tran- sitions aggregated across 5 random trials. Scaling the dataset size while keeping the number of train- ing levels fixed (to 200) leads to a slight increase in both train and test returns, but the generalization gap remains about the same. For remaining algo- rithms, see Figure 16 in Appendix.
The Effect of Data Size on Generalization
Increasing the dataset size alone without increasing its diversity by, for example, increasing the number of transitions without also increasing the number of training environments, does not lead to significant performance improvements new environments for any of the offline learning algorithms.
4.6 GENERALIZATION TO NEW INSTRUCTIONS USING HUMAN DEMONSTRATIONS
Here we aim to assess a different type of generalization, namely generalization to unseen initial state and reward functions. Moreover, we also want to test if our conclusions hold in more challenging and realistic domains, hence we benchmark three algorithms, BC, CQL and BCQ, on a challenging benchmark, WebShop. T the train and test scores (where average score average reward +10) and the success rate (i.e., % of rollouts which achieve the maximum reward 10). While the average score measures how closely an agent policy is able to follow the given instruction, the success rate determines how correct the actions taken are. In our experiments, BC obtains a higher score but lower success rate than BCQ. This indicates that BCQ only clicks "buy" products where has
7

---

Published as a conference paper at ICLR 2024
high confidence, while BC may click even when its confidence is lower, thus collecting additional points for partially correct selections but not having as high a success rate as BCQ.
WebSnop
Demonstratians
Following similar evaluation procedure as in Yao et al. (022), for each of our pre-trained models, roll out the policy the entire test set of goals (i.e. E [0, 500)) and the first 500 goals from the train set (i.e E [1500, 2000)). For each offine learning algorithm, we compute the mean and standard deviation of the average train and test scores and success rates using 3 model seeds.
Average
Average
60
50
20
40-
Success Rdta
30
10
10
As Figure 7 shows, on the human demonstration dataset, BC achieves a higher score than CQL and BCQ, on both train and test instructions from the human dataset. Note that the differ- ence between train and test scores in BC is not very large. However, if trained for longer, all of these methods obtain much better training performance but their test performance starts decreasing, suggesting that they are prone to overfitting. During training, BCQ has slightly higher success rate than BC and CQL. At test time, however, BC achieves the highest success rate, thus making BC a better-performing baseline in this domain. We report results on suboptimal demonstrations in Appendix L.I
178m T0St.
173mn
T0St
IRA 1est
1r9in 10ST
1n5m 355t
trim
0CQ
COL
0C
BCQ
COL
Figure 7: Performance on WebShop. Scores and success rates (%) for BC, BCQ. and CQL when trained on a dataset human demonstrations from WebShop. Results were calculated by taking an average across 500 instructions (item descriptions) for both train and test. The mean and standard deviation were computed across 3 model seeds.
5 RELATED WORK
Generalization in RL A large body of work has emphasized the challenges of training online RL
agents that can generalize to new transition and reward functions (Rajeswaran et al. 2017]] Machado
et al. 2018: Justesen et al. 2018 Packer et al. 2019; Zhang et al., 2018a b. Nichol et al.
2018
Cobbe et al. 2018: Julii0 Juliani et al. 2019
Küttter et al. 2020; Grigsby & Qi 2020 Chen 2020
Bengio et al.
[2020] Bertran et al. 2020
Ghosh et al. [021 Kirk et al. 2023
Ajay et al. 202 I
Ehrenberg et al. 2022; Lyle et al.
2022
A number of different RL environments have recently
been created to support research
the generalization abilities of RL agents (Juiiai et al.) 2019
Küttler et al., 2020: Samvelyan et al. [202]] Frans & Isola 2022 Albrecht et al. 2022). However, all
of these simulators focus on benchmarking online rather than offline learning algorithms and don't have associated datasets. A natural way to alleviate overfitting is to apply widely- regularization techniques such as implicit regularization (Song et al.) 2020)) dropout (Igl et al. 2019), batch normal- ization (Farebrother et al. 2018)), or data augmentation (Ye et al. 2020 Lee et al. 2020 Laskin et al.,
2020 Raileanu et al. [202 I| Wang et al., 2020 Yarats et al. 2021| Hansen & Wang 2021 Hansen
et al. 2021 Ko & Ok 2022). Another family of methods aims to learn better state representations
via bisimulation metrics (Zhang et al. 2021a; 2020 Agarwal et al. 2021a), information bottle-
necks (ggl et al. 2019 Fan & Li (022), attention mechanisms (Carvalho et al.) 202I). contrastive
learning Mazoure et al. 2022), adversarial learning (Roy & Konidaris 2020 Fu et al. [202]] Rahman
& Xue 2021), ) or decoupling representation learning from decision making (Stooke et al. 1
Sonar et al. 2020). Other approaches use uncertainty-drived exploration (Jiang et al.),, policy-value
decoupling (Raileanu & Fergus ), information-ttheortic approaches (Chen) 2020 Mazoure
et al.
2020), non-stationarity reduction (Igl et al. Nikishin et al., 2022), curriculum learning (Jiang
ctt al.
20216: Team et al. 2021| Jiagg et al., 202 laj Parker-Holder et al. 2022). planning (Anand
ct al.
2022), Bowward-backward representations (Touati & Ollivier 2021). or diverse policies (Kumar
et al.
2020b). Note that all these works consider the online rather than offline RL. setting. More
similar to our work, |Mzzurre et al. (2022) train an offline RL agent using contrastive learning based on generalized value functions, showing that it generalizes better than other baselines in some cases. However, they don't focus creating offline RL benchmarks for generalization across both transition and reward functions, and they don't compare offline RL algorithms with other competitive approaches based on sequence modeling or transformer policies. Similarly Yang al. (2023) studies the generalization capabilities of offline goal-conditioned RL (GCRL) methods and observes that existing GCRL methods do not generalize to unseen goals.
8

---

Published as a conference paper at ICLR 2024
Offline RL Benchmarks For many real-world applications such as education, healthcare, au- tonomous driving, or robotic manipulation. learning from offline datasets is essential due to safety
concerns and time constraints (Dasari et al. 2019 Cabi et al.
2019: Li ct al. 2010 Strehl et al. 2010
Thomas et al.; 2017: Henderson et al., 2008 Pietquin et al. 2011 Jaques et al.
2020). Recently,
there has been growing interest in developing better offline RL methods (Levine al.
2020
Prudencio et al.
2023; Kostrikoy et al. [202]] Fujimoto et al., [2018] 2019
Agarwal et al.
2020
Nair et al. 2020 Fujimoto & Gul [02] Liu et al. (4/00 Zanette et al. [022 Lambert al.
2022
Rashidinejad et al. 2001 Yarats et al. 2022] Brandfonbrener et al. [002]] Wu et 2019 Zhang
al.
2021b; Cheng et al. 2022) which aim to learn offline from fixed datasets without online
interactions with the environment. With it, a number of offline RL datasets have been created (Fu
ct al.
2020 Qin et al. 2022] Gulcehre et al., 2020; Brant & Stanley 2019; Zhou et al., 2022 Ramos et al. 2021 Kurin et al. 2017). However, all these datasets contain trajectories collected from
single environment instance. In contrast, our collected datasets aim to evaluate an agent's ability to generalize environment instances after being trained purely offline on a dataset of trajectories from similar yet distinct environment instances. V-D4RL (Lu et al.) 2023) also proposes a benchmark suite for assessing the robustmess of offline RL methods but in the continuous control domain. A
number of large-scale datasets of human replays have also been released for StarCraft (Vinyals et al.
2017). Dota (Berner al. 2019), MineRL (Guss et al., 2019), and MineDojo (Fan al. 2022).
However, training models on these datasets requires massive computational resources, which makes them unfeasible for academic or independent researchers. More similar ours, (Hambro et al. 2022)
introduces a large-scale offline RL dataset of trajectories from the popular game of NetHack (Küttler
al. 2020) consisting of billion soate-action-scon transitions from 100,000 bot trajectories. In NetHack, each instance of the game is procedurally generated, meaning that the agent needs to generalize in order to perform well both during training and at test time. Here too, BC outperforms offline RL, which is in line with our results. However, this dataset requires significant resources to train highly performant agents, only considers generalization across different transition functions and not across different reward functions, and does not allow for clear split between train and test scenarios. In contrast, one can train competitive agents on our datasets in just a few hours, making it more accessible benchmark for generalization in offline RI. that should enable fast iteration research ideas.
6 CONCLUSION
In this paper, we compare the generalization abilities of online and offline learning approaches. Our experiments show that existing offline learning algorithms (including state-of-the-art offline RL, behavioral cloning, and sequence modeling approaches) perform significantly worse on new environments than online RL methods (like PPO). Our paper is first to introduce benchmark for evaluating the generalization of offline learning algorithms. The absence of such a benchmark has historically limited our understanding of these algorithms'' real-world applicability, so work strives to bridge this gap. To achieve this, we release a collection of offline learning datasets containing trajectories from Procgen and WebShop.
Our results suggest that existing offline learning algorithms developed without generalization in mind are not enough tackle these challenges, which is crucial in order to make them feasible for real-world applications. We observe that increasing dataset diversity can lead to significant improvements in generalization even without increasing the size of the dataset. Contrary to prior work on offline RL in singleton environments, we find that their generalization does not significantly improve with the size of the dataset without also enhancing its diversity. Hence, we believe more work is needed to develop offline learning algorithms that can perform well on new scenarios at test time.
ACKNOWLEDGEMENTS
We would like to thank Sharath Raparthy for helping in open-sourcing the code. We would also like to express our sincere gratitude to (in alphabetical order) Amy Zhang, Eric Hambro, and Mikayel Semvelyan for providing valuable feedback on this project.

---

Published as a conference paper at ICLR 2024
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, pp. 104-114. PMLR, 2020.
19 22 28
Rishabh Agarwal, Marlos C. Machado, P. S. Castro, and Marc G. Bellemare. Contrastive behavioral similarity embeddings for generalization in reinforcement learning, International Conference On
Learning Representations, 2021a. g
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Deep reinforcement learning the edge of the statistical precipice. Advances in neural information processing systems, 01-29304-2932) 2021b. 3 22
Anurag Ajay. Ge Yang, Ofir Nachum, and Pulkit Agrawal. Understanding the generalization gap in
visual reinforcement learning. 2021.
Joshua Albrecht, Abraham Fetterman, Bryden Fogelman, Ellie Kitanidis, Bartosz Wróblewski, Nicole Seo, Michael Rosenthal, Maksis Knutins, Zack Polizzi, James Simon, et al. Avalon: A benchmark for generalization using procedurally generated worlds. Advances in Neural Information Processing Systems, 35:12813-1288 2022. 8
Ankesh Anand, Jacob C. Walker, Yazhe Li, Eszter Vértes, Julian Schrittwieser, Sherjil Ozair, Theo- phane Weber, and Jessica B. Hamrick. Procedural generalization by planning with self-supervised world models. In The Tenth International Conference on Learning Representations, 2022. g
Emmanuel Bengio, Joelle Pineau, and Doina Precup. Interference and generalization in temporal difference learning. International Conference On Machine Learning, 2020.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement learning. CoRR, Abs/1912.0068// 2019. g
Martin Bertran, Natalia Martinez, Mariano Phielipp, and Guillermo Sapiro. Instance-based gen- eralization in reinforcement learning. Advances in Neural Information Processing Systems, 33: 11333-11344, 2020. g
Prajjwal Bhargava, Rohan Chitnis, Alborz Geramifard, Shagun Sodhani, and Amy Zhang. Se- quence modeling is robust contender for offline reinforcement learning. arXiv preprint arXiv:
2300..1550,, 2023.
David Brandfonbrener, Alberto Bietti, Jacob Buckman, Romain Laroche, and Joan Bruna. When does return- conditioned supervised learning work for offline reinforcement learning? Advances in Neural Information Processing Systems, 35:1542- 11111-2255,, 2022. 5
Jonathan Brant and Kenneth O. Stanley. Benchmarking open-endedness in minimal criterion coevolution. In Proceedings of the Genetic and Evolutionary Computation Conference, GECCO '19, pp. 72-80, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450361 188.
doi: 10. 00.1144/3321777.3321257
Serkan Cabi, Sergio Gomez Colmenarejo. Alexander Novikov, Ksenia Konyushkova, Scott E. Reed, Rae Jeong, Konrad Zolna, Y. Aytar, D. Budden, Mel Vecerík, Oleg O. Sushkov, David Barker, Jonathan Scholz, Misha Denil, N. D. Freitas, and Ziyun Wang. Scaling data-driven robotics with reward sketching and batch reinforcement learning. Robotics: Science And Systems, 2019, 9
Wilka Carvalho, Andrew Lampinen, Kyriacos Nikiforou, Felix Hill, and Murray Shanahan. Feature-attending recurrent modules for generalization in reinforcement learning. arXiv preprint arxiv:2112.08866, 2021.
Jerry Zikun Chen. Reinforcement learning generalization with surprise minimization. ArXiv. abs/2004. 12399, 2020. g
10

---

Published as a conference paper at ICLR 2024
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-15097, 2021.
22
24
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv: 2306. 15595, 2023. T8
Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offline reinforcement learning. In International Conference on Machine Learning, pp. 3852-3878.
PMLR, 2022. g
Karl Cobbe, Oleg Klimov, Christopher Hesse, Tachoon Kim, and J. Schulman. Quantifying gen- eralization in reinforcement learning. International Conference On Machine Learning. 2018.
Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, pp. 2048-2056, 2020.
20 23
Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn. Robonet: Large-scale multi-robot learning. In 3rd Annual Conference on Robot Learning, CoRL. 2019, Osaka, Japan, October 30 November 2019, Proceedings, volume 100 of Proceedings of Machine Learning Research, pp. 885-897. PMLR, 2019. 9
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Compulational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, Jume 2-7, 2019, Volume I (Long and
Short Papers), pp. 4171-4186. Association for Computational Linguistics, 2019. 22
Ron Dorfman, Idan Shenfeld, and Aviv Tamar. Offline meta reinforcement leamingg-ideniffabiiity challenges and effective data collection strategies. Advances Neural Information Processing Systems, 34:4607-46188 2021. 19
Andy Ehrenberg, Robert Kirk, Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. A study of off-policy learning in environments with procedural content generation. In ICLR Workshop on Agent Learning in Open- Endee-eds,,,s, 2022. g
Jiameng Fan and Wenchao Li. Dribo: Robust deep reinforcement learning via multi-view information bottleneck. In International Conference on Machine Learning, pp. 6074-6102. PMLR, 2022. g
Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang. Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar, Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems, 35:
18343-18362, 2022. g
Jesse Farebrother, Marlos C. Machado, and Michael H. Bowling. Generalization and regularization in dgn, ArXiv, abs/1810.00123, 2018. 8
18
Kevin Frans and Phillip Isola. Powderworld: A platform for understanding generalization via rich task distributions. arXiv preprint arX1v.2211.1305,, 2022. g
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep
data-driven reinforcement learning. arXiv preprint arXiv:2004.072111 2020. 3 28
Xiang Fu, Ge Yang, Pulkit Agrawal, and Tommi Jaakkola. Learning task informed abstractions. In International Conference Machine Learning, pp. 3480-3491. PMLR, 2021.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.

---

Published as a conference paper at ICLR 2024
Scott Fujimoto, D. Meger, and Doina Precup. Off Off-policy deep reinforcement learning without exploration. International Conference On Machine Learning, 2018. B
Scott Fujimoto, Edoardo Conti, Mohammad Ghavamzadeh, and Joelle Pineau. Benchmarking batch deep reinforcement learning algorithms. arXiv preprint arXiv:1910.011000 2019.
Divyansh Garg, Joey Hejna, M. Geist, and S. Ermon. Extreme q-learning: Maxent rl without entropy. International Conference On Learning Representations, 2023, doi: 10.448550/rxivv2330 ..0328....
18
Dibya Ghosh, Jad Rahme, Aviral Kumar, Amy Zhang, Ryan Adams, and Sergey Levine. Why generalization in rl is difficult: Epistemic pomdps and implicit partial observability. Advances in
Neural Information Processing Systems, 34:25502-255155 2021. g
Raj Ghugare, Matthieu Geist, Glen Berseth, and Benjamin Eysenbach. Closing the gap between td learning and supervised learning-a generalisation point of view. arXiv preprint arXiv:2401.1123: 2024. 5
Jake Grigsby and Yanjun Qi. Measuring visual generalization in continuous control from pixels. ArXiv, abs/2010.06740, 2020. 8
Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Thomas Paine, Sergio Gómez, Konrad Zolna, Rishabh Agarwal, Josh S Merel, Daniel J Mankowitz, Cosmin Paduraru, et al. RI unplugged: A suite of benchmarks for offline reinforcement learning. Advances in Neural Information Processing
Systems, 33:7248-72533 2020.
9 28
William H. Guss, Brandon Houghton, Nicholay Topin. Phillip Wang, Cayden Codel, Manuela Vcloso, and Ruslan Salakhutdinov. MineRL: A large-scale dataset of Minecraft demonstrations. Twenty- Eightt International Joint Conference on Artificial Intelligence, 2019. URL http:
//minerl 10
Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual markov decision processes. arXiv preprint arXiv: 000.00022.022... 2015. 2
Eric Hambro, Roberta Raileanu, Danielle Rothermel, Vegard Mella, Tim Rocktäschel, Heinrich Kuttler, and Naila Murray. Dungeons and data: A large-scale nethack dataset. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
URL
wwt.:/iwww..eeeeeooo..eeeeeeee htt.:/orrrmmmiii/NN//aaa. OxNN
Nicklas Hansen and Xiaolong Wang. Generalization in reinforcement learning by soft data aug- mentation. In 2021 IEEE International Conference Robotics and Automation (ICRA), pp. 13611-13617. IEEE, 2021.
Nicklas Hansen, Hao Su, and Xiaolong Wang. Stabilizing deep q-learning with convnets and vision transformers under data augmentation. Advances Neural Information Processing Systems, 34: 3680-3693, 2021. 8
Philippe
Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit geleanning actor-critic method with diffusion policies. arXiv preprint ar0iv:2304. 10573, 2023. [8
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016. doi: 10.1 109/0CPR.20006.00
22
James Henderson, Oliver Lemon, and Kallirroi Georgila. Hybrid reinforceeeen//saprrviieed learning of dialogue policies from fixed data sets. Computational Linguistics, 34(4):487-511, 2008. g
Maximilian Igl, Gregory Farquhar, Jelena Luketina, Wendelin Bochmer, and Shimon Whiteson. Transient non-stationarity and generalisation in deep reinforcement learning. 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. g
12

---

Published as a conference paper at ICLR 2024
Maximilian Igl, Kamil Ciosek, Yingzhen Li, Sebastian Tschiatschek, Cheng Zhang, Sam Devlin, and Katja Hofmann. Generalization in reinforcement learning with selective noise injection and information bottleneck. In Advances in Neural Information Processing Systems. 2019. g 18
N. Jaques, J. H. Shen, A. Ghandeharioun, C. Ferguson, A. Lapedriza, N. Jones, S. Gu, and R. Picard. Human-centric dialog training via offline reinforcement learning. Empirical Methods in Natural Language Processing (EMNLP), 2020.
Minqi Jiang, Michael Dennis, Jack Parker-Holder, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Replay-guided adversarial environment design. Advances in Neural Information Processing Systems, 34: 18:11884-197 2021a.
Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In The International Conference Machine Learning. 2021
18
Yiding Jiang, J Zico Kolter, and Roherta Raileanu. Uncertainty-drive exploration for generalization in reinforcement learning. In Deep Reinforcement Learning Workshop NeurlPS 2022. 4 18
Arthur Juliani, Ahmed Khalifa, Vincent-Pierre Jonathann Harper, Ervin Teng, Hunter Henry, Adam Crespi. Julian Togelius, and Danny Lange. Obstacle Tower: A Generalization Challenge in Vision, Control, and Planning. In IJCAI, 2019.
Jonathan
Niels Justesen, Ruben Rodriguez Torrado, Philip Bontrager, Ahmed Khalifa, Julian Togelius, and Sebastian Risi. Illuminating generalization in deep reinforcement learning through procedural level generation. arXiv: Learning, 2018.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alee Radford. Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv: 2001 00066, 2020. 7
Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems, 23(6):4909 49266 2021.
Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. A survey of zero-shot generalisation in deep reinforcement learning. Journal of Artificial Intelligence Research, 76: 201-264, 2023, g
Byungchan Ko and Jungseul Ok. Efficient scheduling of data augmentation for deep reinforcement learning. Advances in Neural Information Processing Systems, 35:33289-33301, 2022 8
Ilya Kostrikov, Denis Yarats, and R. Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. International Conference On Learning Representations, 2020.
Ilya Kostrikov, Ashvin Nair, and S. Levine. Offline reinforcement learning with implicit q-learning. International Conference On Learning Representations, 2021.
22
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-leariin for offline
reinforcement learning. Advances in Neural Information Processing Systems, 33:1 179-1191, 2020a.
19
22
28
Aviral Kumar, Rishabh Agarwal, Xinyang Geng, G. Tucker, and S. Levine. Offline q-learning diverse multi-task data both scales and generalizes. International Conference Learning Representations, 2022. doi: 10.48550/arxiv.2211.. 1.1144444 5
Saurabh Kumar, Aviral Kumar, Sergey Levine. and Chelsea Finn. One solution is not all you need: Few-shot extrapolation via structured maxent rl. Advances in Neural Information Processing
Systems, 33:8198-822 8210,, 2020b. g
Vitaly Kurin, Sebastian Nowozin, Katja Hofmann, Lucas Beyer, and Bastian Leibe. The atari grand challenge dataset. arXiv preprint arXiv: 1705. 10998, 2017. 3 9
13

---

Published as a conference paper at ICLR 2024
Heinrich Küttler, Nantas Nardelli, Alexander H. Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The NetHack Learning Environment. In Proceedings of the Conference Neural Information Processing Systems (NeurIPS), 2020.
8
Nathan Lambert, Markus Wulfmeier, William Whitney, Arunkumar Byravan, Michael Bloesch, Vibhavari Dasagi. Tim Hertweck, and Martin Riedmiller. The challenges of exploration for offline reinforcement learning. arXiv preprint arXiv:2201. /861, 2022.
Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Rein- forcement learning with augmented data. Advances in neural information processing systems, 33:
19884-19895, 2020. g 18
Kimin Lee, Kibok Lee, Jinwoo Shin, and Honglak Lee. Network randomization: A simple technique for generalization in deep reinforcement learning. In International Conference on Learning Representations. https://opemreview net/forum, 2020.
8
Kuang-Huce Mueng Lee, Ofir Nachum, Mengjiao Sherry Yang, Lisa Lee, Daniel Freeman, Sergio Guadar- rama, Ian Fischer, Winnie Xu, Eric Jang, Henryk Michalewski, et al. Multi-gan galli decision transformers. Advances in Neural Information Processing Systems, 35:27921-27936 2022.
19
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXivv2005.01643, 2020.
18
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequene pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880, 2020. 21
Langing Li, Rui Yang, and Dijun Luo. Focal: Efficient fully-offline meta-reinforcement learning via distance metric learning and behavior regularization. International Conference On Learning Representations, 2021. 19
Lihong Li, Wei Chu, John Langford, and Robert E Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web, pp. 661 661-670, 2010.
Siqi Liu, Kay Choong See, Kee Yuan Ngiam, Leo Anthony Celi, Xingzhi Sun, and Mengling Feng. Reinforcement learning for clinical decision support in critical care: comprehensive review. Journal of medical Internet research, 22(7):e18477, 2020a.
Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Provably good batch off-policy reinforcement learning without great exploration. Advances in neural information processing systems, 33: 13:1264-1274, 2020b. 9
Cong Lu, Philip J Ball, Tim GJ Rudner, Jack Parker-Holder. Michael A Osborne, and Yee Whye Teh. Challenges and opportunities in offline reinforcement lcarning from visual observations.
Transactions Machine Learning Research, 2023. 9
Clare Lyle, Mark Rowland. Will Dabney, Marta Kwiatkowska, and Yarin Gal. Learning dynamics and generalization in deep reinforcement learning. In International Conference on Machine Learning. pp. 14560-14581. PMLR, 2022.
Marlos C. Machado, Mare G. Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael H. Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. In IJCAI, 2018.
Bogdan Mazoure, Remi Tachet des Combes, Thang Long Doan, Philip Bachman, and R Devon Hjelm. Deep reinforcement and infomax learning. Advances Neural Information Processing Systems, 33:3686-3698, 2020. g
Bogdan Mazoure, Ilya Kostrikov, Ofir Nachum, and Jonathan J Tompson. Improving zero-shot generalization in offline reinforcement learning using generalized similarity functions. Advances in Neural Information Processing Systems, 35:25088-2511 2022. g 18
14

---

Published as a conference paper at ICLR 2024
E. Mitchell, Rafael Rafailov, X. B. Peng, S. Levine, and Chelsea Finn. Offline meta-reinforcemeet learning with advantage weighting. International Conference On Machine Learning. 2020. I9
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. ArXiv,
abs/1312.5602, 2013. 23
Amirkeivan Mohtashami and Martin Jaggi. Landmark attention: Random- infinite context
length for transformers. arXiv preprint arXiv: 2305. 16300, 2023. [8
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets, arXiv preprint NXXv:22006.09359, 2020. 9
Alex Nichol, V. Pfau, Christopher Hesse, O. Klimov, and John Schulman. Gotta learn fast: A new benchmark for generalization rl. ArXiv, abs/1804.0372)) 2018, g
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International Conference Machine Learning, pp. 16828-16847 PMLR, 2022.
Georg Ostrovski, Pablo Samuel Castro, and Will Dabney. The difficulty of passive learning in deep reinforcement learning. CoRR, abs/21 140200 2021. URL httppp/////xxvv/.////a//
2110. 14000.14000 4020 5
Charles Packer, Katelyn Gao, Jernej Kos, Philipp Krähenbühl, Vladlen Koltun, and Dawn Song. Assessing generalization in deep reinforcement learning. ICI.R, 2019. 8
Jack Parker-Holder, Minqi Jiang, Michael Dennis, Mikayel Samvelyan, J. Foerster, Edward Grefen- stette, and Tim Rocktaschel. Evolving curricula with regret-based environment design. Interna- tional Conference On Machine Learning, 2022. doi: 10448500arXiw.2203.0333 g
Olivier Pietquin, Matthieu Geist, Senthilkumar Chandramohan, and Hervé Frezza- Buet. Sample- efficient batch reinforcement learning for dialogue management optimization. ACM Transactions on Speech and Language Processing (TSLP), 7(3):1-21, 2011. 9
Vitchyr H. Pong, Ashvin V. Nair, Laura M. Smith, Catherine Huang. and Sergey Levine. Offline meta- reinforcement learning with online self-supervision. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML. 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 1781 1-17829, PMLR.
19
Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini. A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems, 2023. 1
Rong-Jun Qin, Xingyuan Zhang, Songyi Gao, Xiong-Hui Chen, Zewen Li, Weinan Zhang, and Yang Yu. Neorl: A near real-world benchmark for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:24753-24765. 2022. 3
Md Masudur Rahman and Yexiang Xue. Adversarial style transfer for robust policy optimization in reinforcement learning. 2021.
Roberta Raileanu and Rob Fergus. Decoupling value and policy for generalization in reinforcement learning. In The International Conference on Machine Learning. 2021.
18
20
22
23
Roberta Raileanu, Maxwell Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Automatic data augmentation for generalization in reinforcement learning. Advances in Neural Information
Processing Systems, 34:5402-5415. 2021.
I
8 18
Aravind Rajeswaran, Kendall Lowrey, Emanuel Todorov, and Sham M. Kakade. Towards generaliza- tion and simplicity in continuous control. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 6550-6561, 2017.
15

---

Published as a conference paper at ICLR 2024
Sabela Ramos, Sertan Girgin, Léonard Hussenot, Damien Vincent, Hanna Yakubovich, Daniel Toyama, Anita Gergely, Piotr Stanczyk, Raphael Marinier, Jeremiah Harmsen, et al. Rids: ecosystem to generate, share and use datasets in reinforcement learning. arXiv preprint arXiv:2111.0272 2021.
Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline rein- forcement learning and imitation learning: tale of pessimism. Advances in Neural Information Processing Systems, 34:1 14:11772-11716, 2021. 9
Scott Reed, Konrad Zoina, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist
agent. arXiv preprint arXiv:2205.06175. 2022. [
Josh Roy and G. Konidaris. Visual transfer for reinforcement learning via wasserstein domain confusion. Aaai Conference On Artificial Intelligence, 2020. doi: 10.09//aai.v35511....
Mikayel Samvelyan, Robert Kirk, Vitaly Kurin, Jack Parker Parker-Holder, Minqi Jiang, Eric Hambro, Fabio Petroni, Heinrich Küttler, Edward Grefenstette, and Tim Rocktäschel. Minihack the planet: A sandbox for open-ended reinforcement learning research. arXiv preprint arXiv:2109, 13202, 2021.
g
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning (ICML), 2015. 20
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy
optimization algorithms. arXiv preprint arXiv: 2017.
22
Bharat Singh, Rajesh Kumar, and Vinay Pratap Singh. Reinforcement learning in robotic applications:
comprehensive survey. Artificial Intelligence Review, pp. 1 1-46, 2022.
Adish Singla, Anna N Rafferty, Goran Radanovic, and Neil Hefferman. Reinforcement learning for education: Opportunities and challenges. arXiv preprint arXivv2207.000::
2021.
[
Anoopkumar Sonar, Vincent Pacelli, and Anirudha Majumdar. Invariant policy optimization: Towards
stronger generalization in reinforcement learning. Conference On Learning For Dynamics & Control, 2020. g
Xingyou Song, Yiding Jiang, Stephen Tu, Yilun Du, and Behnam Neyshabur. Observational overfitting in reinforcement learning. In International Conference Learning Representations, 2020. URL
https::/openreviie: net fante//ooumiiii Holidd-HHHihhhe ott.://awwa.aa....eooooooooooaoa
18
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 9870-9879. PMLR. 2021.
Alex Strehl, John Langford, Lihong Li, and Sham M Kakade. Learning from logged implicit exploration data. Advances in neural information processing systems, 23, 2010.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018. 2
Adrien Ali Taiga, Rishabh Agarwal, Jesse Farebrother, Aaron Courville, and Mare G Bellemare. Investigating multi-task pretraining and generalization in reinforcement learning. The Eleventh International Conference Learning Representations, 2022. T9
Open Ended Learning Team, Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michael Mathieu, et al. Open-ended learning leads to generally capable agents. arXiv preprint arXiv:2107. 12007. 2021. 8
Philip Thomas, Georgios Theocharous, Mohammad Ghavamzadeh, Ishan Durugkar, and Emma Brunskill. Predictive off-policy policy evaluation for nonstationary decision problems, with applications to digital marketing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, pp. 4740 2017.
16

---

Published as a conference paper at ICLR 2024
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-basee control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 0026-5033.
IEEE, 2012. 3
Ahmed Touati and Yann Ollivier. Learning one representation to optimize all rewards. Advances in Neural Information Processing Systems, 34:13-23, 2021. g
Oriol Vinyals. Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle
Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, et al. Starcraft ii: A
new challenge for reinforcement learning. arXiv preprint 17X1i:17088047882. 2017.
Kaixin Wang, Bingyi Kang, Jie Shao, and Jiashi Feng, Improving generalization in reinforcement learning with mixture regularization. Neural Information Processing Systems, 2020. g
Yifan Wu, George Tucker, and Ofrr Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.1133000 2019. 9
Rui Yang. Lin Yong, Xiaoteng Ma, Hao Hu, Chongjie Zhang, and Tong Zhang. What is essential for unseen goal generalization of offline goal-conditioned rl? In International Conference on Machine Learning, pp. 39543-39571. PMLR, 2023. 8
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744-20757, 2022.
20
21
22
23
Denis Yarats, R. Fergus, A. Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved
data-augmented reinforcement learning. International Conference On Learning Representations,
2021. 8 18
Denis Yarats, David Brandfonbrener, Hao Liu, Michael Laskin, Pieter Abbeel, Alessandro Lazaric, and Lerrel Pinto. Don't t change the algorithm, change the data: Exploratory data for offline reinforcement learning. arXiv preprint orXiv:2201.13425 2022.
Chang Ye, Ahmed Khalifa, Philip Bontrager, and Julian Togelius. Rotation, translation, and cropping for zero-shot generalization. 2020 IEEE Conference on Games (CoG), pp. 57-64. IEFF, 2020.
Andrea Zanette, Martin Wainwright, and Emma Brunskill. Provable benefits of actor-critic methods for offline reinforcement learning. Advances neural information processing systems, 34:13626- 13640, 2021. 9
Amy Zhang, Nicolas Ballas, and Joelle Pineau. A dissection of overfitting and generalization in continuous reinforcement learning. ArXiv, abs/ 00111860007 906.07737, 201 Sa. I g
Amy Zhang, Clare Lyle. Shagun Sodhani, Angelos Filos, M. Kwiatkowska, Joelle Pineau, Y. Gal, and Doina Precup. Invariant causal prediction for block mdps. International Conference Machine
Learning, 2020. g 18
Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. In International Conference on Learning Representations, 2021a.
8 18
Chi Zhang, Sanmukh Kuppannagari, and Prasanna Viktor. Brac+: Improved behavior regularized actor critic for offline reinforcement learning. In Asian Conference on Machine Learning, pp.
PMLR, 2021b. g
Chiyuan Zhang, Oriol Vinyals, Rémi Munos, and Samy Bengio. A study overfitting in deep reinforcement learning. ArXiv, abs/1 804.06893, 2018b.
Tony Zhao, Jianlan Luo, Oleg O. Sushkov, Rugile Peyceviciute, N. Hoess, Jonathan Scholz, Schaal, and S. Levine. Offline meta-reinforcemennt learning for industrial insertion. leee International Conference On Robotics And Automation, 2021. doi: 10.1 10.1100///a446999000 0022.9812231 19
G. Zhou, Liyiming Ke, S. Srinivasa, Abhi Gupta, A. Rajeswaran, and Vikash Kumar. Real world offline reinforcement learning with realistic data source. leee International Conference On Robotics And Automation, 2022. 9
17

---

Published as a conference paper at ICLR 2024
A APPENDIX
We have open-sourced our codebase and datasets on GitHub" The code repository consists of lwo separate folders: Procgen and 2. WebShop. Each of these sub-repositories have well- documented README mdd with all the necessary steps needed to reproduce the experimental results in this paper to implement and train other offline learning models, or to generate other datasets and use them.
License
We have released the codebase and the datasets under CC-by So--by-Ne NC license.
B LIMITATIONS AND FUTURE WORK
In this section, we discuss some potential limitations of our work and suggest future research avenues. First, our main datasets in Proegen have million transitions. Contrasting this with many offline RL datasets like Atari (Agarwal et al., (020)) where the dataset consists of the entire training trajectory from the behaviour policy, the dataset size could have been inflated to 25M (since in Procgen, PPO uses 25M evironment steps to reach convergence). We believe this is good step. However, training offline learning methods on such large datasets requires multi-GPU parallelism and significant computational resources, especially when using sequence modelling approaches. Thus, opted for a more lightweight dataset which can enable a broad community of researchers to make progress on these problems without requiring extensive computational resources. Future work could explore the possibility of leveraging larger datasets, while seeking more efficient computational strategies.
We also note that all of our datasets have a discrete action space. Lately, numerous offline RL et al.) 20033 Garg et al.] (023), have been developed exclusively for
algorithms (Hansen-Esttee continuous action spaces. Adapting their loss function to discrete action space is non-trivial task, so we leave it to future work to address this conversion problem which should extend the applicability of these algorithms.
Our datasets from the WebShop environment did not explore cross- cross-product (i.e. training on selected instructions having some categories of products and testing on never-seen-befors categories) general- ization, an aspect that could be fascinating direction for subsequent studies. The ability of models to generalize across diverse products could prove extremely useful
web-navigation environments.
also worth considering the latest improvements like (Chen et al. 2023 Mohtashami & Jaggil 2023). that make transformer models better at handling longer context length in situations like WebShop. This could help deal with the problem of long inputs in the state space. This might allow to sequential decision-man making like the Decision Transformers that require multiple steps observations in one context.
Lastly, Offline RI. algorithms have a quadratic error bound on the horizon since there is no control over the data-generating policy and it may require evaluating diitr---iitiibttion states (Levine al.) 2020).
However, this theoretical observation has only been empirically validated on singleton environments. The goal of our study is to empirically evaluate offline RL algorithms on new environments rather than provide a theoretical explanation. Our results are in line with the theoretical results demonstrating that offline RL algorithms struggle with generalization but only in theory but also in practice. We hope this will inspire future work that aims to overcome these limitations, well as a theory of why existing offline RL algorithms struggle to generalize to new environments (which better models our setting and would be an extension of prior work on this topic).
One promising avenue for future research to combine offline RL methods with techniques that improve generalization in online RL such as data augmentation (Laskin et al. [2200] Raileanu et al.)
Yarats et al. 202T), regularization (Song et al. 2020 [g] et al. 2019 Farebrother et al. 2018).
representation learning (Zhang al. 2021a 2020 Agarwal et al. 2021a] Mazoure et al. (022), or
other approaches focused on data collection, sampling, or optimization (Jiang et al. 2021b Raileanu
& Fergus] 2021; Jiang et al.). is also possible that entirely new approaches that take into account the particularities of this problem setting will need to be developed in order to tackle the challenge of generalization in offline learning. As mentioned in the paper, sequence modeling approaches policies cannot be directly applied to more complex environments
that rely on transformer-
https:
com
18

---

Published as a conference paper at ICLR 2024
like WebShop due to their limited context lengths. Thus, we expect these methods will benefit from future advances in transformer architectures that can handle longer inputs. We hope our study and benchmark enable faster iteration research ideas towards developing more general agents that can learn robust behaviors from offline datasets.
C BROADER IMPACT
This paper proposes datasets for evaluating generalization of behaviors learned offline via behavioral cloning, offline reinforcement learning, and other approaches. We also evaluate ofte-of-the-ard methods on this benchmark, concluding that more work is needed to train agents that generalize to new scenarios after learning solely from offline data. On the one hand, improving the generalization of offline learning methods can be important for developing more robust and reliable autonomous agents that take reasonable actions even in states that they haven't encountered during training, which are likely to be common once such agents are deployed in real-world applications. On the other hand, deploying autonomous agents for high-stakes applications can have negative consequences, so additional safety precautions should be taken when considering deployment in the real-world. Since our results are based on simulated environments for video games and e-commerce like Proegen and WebShop, which are somewhat simplified compared to real-world settings, we do not foresee any direct negative impact on society.
D EXTENDED RELATED WORK
Multi-task RL Recently, significant attention has been directed towards the large-scale multi- task offline RL and meta RL settings. Notably, pioneering multi-task approaches such as those outlined in Kumar et al. (2022); Lee al. (2022); Taiga al. (2022) have showcased their efficacy in harnessing extensive multi- task datasets based on Atari Agarwal et al. (2020). However, these
settings are different from the problem we consider our this paper. To elaborate, [ee et al. (2022)
pre-trains the model on select Atari games, followed by subsequent fine-tuning on the remaining ones. Similarly, a lot of meta RL works (Li al. 2021 Pong et al., /Zhao et al. 2021 Dorfman et al. 2021 Mitchell et al. 2020) also try to tackle this problem but require some finetuning at test time. In contrast, our benchmark centers around the evaluation of zero-shot generalization which is a more challenging setting with wider applicability in the real-world. Similarly, the works presented in Kumar et al. (2020a) and Taiga et al. (022)) delve into the realm of inter-game generalization in Atari, employing training dataset comprising of approximately 40 games. Our focus, in contrast, resides in scrutinizing intra-game inter-level generalization within the Procgen framework. It is worth highlighting that our dataset architecture prioritizes memory efficiency, allowing for seamless execution by the academic community without necessitating access to a large number of GPUs, requirement which, regrettably, constrained the feasibility of the approaches proposed in (Kumar ct al. 2020a) and (Taiga et al., (2022) that heavily relies on many TPUs. This emphasis on accessible resources aligns with our intent to facilitate broader engagement and reproducibility in this research area.
E DISCUSSION
In this section, we discuss few points that we believe are the reasons why BC generalizes better than offline RL, even in the presence of suboptimal data. We believe the reason offline RL methods fall behind BC is that they adopt a risk-averse approach, avoiding actions not encountered during training. This becomes a limitation when agents are tested in new environments, as they are likely to default to suboptimal policies due to unfamiliar states. On the other hand, BC, unbounded by these constraints, utilizes its learned representations to select the best action by identifying the most similar training state to the current test state. If BC effectively learns state representations, it could generalize well in new environments. Regarding why offline learning methods are outperformed by online RL, we think that the advantage of online RL comes from its ability to gather and learn from a broader range of states through its own data collection, as opposed to the fixed dataset in BC and offline RL (Agarwal et al. 2020). This exposure representations and decision decision-making in new scenarios. Our experiments in demeonstratte that training with more varied data significantly enhances offline learning methods' generalization.
variety of states enables better learning of
19

---

Published as a conference paper at ICLR 2024
However, indicated in Section 4.2 merely using data from multiple PPO checkpoints (i.e. in the case of suboptimal dataset) is not sufficient. This data, being sparsely sampled, doesn't 't cover the entire state space. Understanding how training dynamics affect data diversity is an area worth exploring in future research.
F DATASET DETAILS
F.1 PROCGEN
Environment Procgen (Cobbe al. 2020) is online RI benchmark that serves assess gener- alization in 16 different 21 video games. Procgen makes use of procedural content generation in order to generate new level (corresponding to particular seed) when the episode is reset. This way an unlimited number of varying levels can be generated for each game, each level having different dynamics, layouts, and visual appearances (such as background colors and patterns, or number and location of various items and moving entities), but the same reward function. This makes Procgen good benchmark for testing an agent's ability to generalize to new environment instances (levels) with unseen initial states and Iransition functions but the same reward function.
single transition in Proegen comprises of an observation (represented by an RGB image of shape 64x64x3), a discrete action (the maximum action space is 15), scalar reward (which can be dense or sparse depending on the game), and a boolean value indicating whether the episode has ended.
Offline Data Collection Each level of a Proegen game is procedurally generated by specifying the level seed which is a non-negative integer. We use levels [0, 200) for collecting trajectories and offline training, levels [200, 250) for hyperparameter tuning and model selection, and levels [250, oo) for online evaluation of the agent's performance.
To generate the offline learning datasets based on Procgen, for each game, train PPO (Schulman et al. 2015) policies (with random seeds 0, 1, and 2) using the best hyperparameters found in (Raileanu & Fergus (02T) for 25M steps the easy version of the game. We save model checkpoints once every 50 epochs (from a total of 1525 epochs). We then use these checkpoints to collect trajectories for constructing an expert dataset and a suboptimal dataset.
To create the expert dataset, we roll out the final checkpoint from a single pretrained PPO model, also referred as the expert policy, in the training levels (i.e., allow interact online with the environment) and store 1 million {state, action, reward, terminated] transitions for each Procgen game. Figures 24 and show the number of episodes and transitions per level respectively. As can be seen, there can be variation in the number of transitions across the 16 games since some games have shorter episodes than others.
To create the mixed supott-suboptimal datasets, we evenly combine data from both the expert PPO checkpoint and another checkpoint that achieves 50% the expert's performance. Consequently. these datasets showcase average episodic returns that are approximately 75% of those seen in the expert datasets. Table D shows the average return per dataset collected by different PPO model seeds for 1M transition steps. Since ran our experiments the 1M expert dataset collected from Seed 1, we also collected this suboptimal dataset using PPO's checkpoints from this model seed only. Figures 26 and 27 show the number of episodes per level and total number of transitions per level for each game in Proegen from the 1 M mixed expert-suboptimal dataset (seed 1) which was used for all experiments in Section 4.2
Data Storage For each episode, we store all the corresponding (state, action, reward, terminal) transitions. Each trajectory is then stored as single .mpa file, with the name of timeslamp inder length level relurn. This naming convention allows for analyzing and working with trajectories filtered by level or return.
F.2 WEBSHOP
WebShop Yao et al.) 2022) is text-based veb-navvgatton environment, built to assess the natural language instruction following and sequential decision making capabilities of language-based agents. In this environment, there are two types of tasks: search and choice. For the
Environment
20

---

Published as a conference paper at ICLR 2024
Table Procgen 1M Dataset Average Return:
This table shows the average return of each game's dataset collected by different model seeds of the expert PPO policy, as well as the suboptimal dataset.
Game
Mixed Expert-Suboptimal
Expert
Seed
Mixed
Seed
Seed
Seed
Bigfish
10.61
9.57
12.03
14.01
6.69
Bossfight
6.05
8.31
8.32
8.10
8.62
Caveflyer
4.95
7.23
6.74
6.97
7.79
Chaser
5.32
6,50
5.95
6.68
6.93
Climber
6.37
8.58
8.30
8.53
8.59
Coinrun
7.16
9,42
9.23
9.64
9.33
Dodgeball
3.50
5,46
5.97
4.61
6.26
Fruitbot
22.36
29.21
29.97
29.81
29.15
Heist
5,96
7.85
7.84
7.97
7.26
Jumper
6.88
8.55
8.53
8.47
8.50
Leaper
2.03
2.68
2.68
2.71
2.72
Maze
6.83
9.35
9.44
9.22
9.33
Miner
9.40
12.68
12.78
12.46
12.74
Ninja
6.01
8.07
7.85
8.05
8.00
Plunder
4.15
5.26
5.34
5.53
5,05
Starpilot
20.93
27.23
26.75
27.69
29.34
search task, the agent has learn to generate relevant keywords based description of the desired product in order to increase the likelihood of getting a good product match within the search results. For the choice task, the agent needs to look through the search page and each item's individual page select, customize, and buy the item that matches all the attributes mentioned in the instruction. Since the scope of our study is assess the sequential decision making capabilities of offline agents, we limit our study to only the choice task. For the search task, we use pre-trained BART model (Lewis) jet al. 2020) used in (Yao et al. 2022) to generate a search query at test time and then continue rolling out our pre-trained policies.
Offline Data Collection For WebShop, we collect two types of offline learning datasets based on: (i) the human demonstration dataset provided by the authors which allows us to create fixed size dataset of high quality (since the human demonstrations can be considered a gold standard), and (ii) the imitation learning (IL) policy pre- pre-trained on these human demonstrations which allows us to create multiple datasets of varying sizes order study how performance scales with the dataset size and diversity. We also collect environment rewards for 452 out of 1571 human demonstrations provided by simulating the trajectories via the gym environment provided in WebShop's source code The initial state of this environment is determined by random selection of 10 English letters. So we call the reset function repeatedly until the environment generates an instruction which is in the dataset. We then execute the actions of the episode from the dataset, and verify that the states returned by the environment per each step are the same as those in the dataset (except the last state, the dataset doesn't store the confirmation state once an episode is completed). The rewards were collected on the fly and we stored them under the key "rewards" together with "states", "actions", etc. This way, we were able collect 452 trajectories with their corresponding per-step rewards (out of 1571 in the human dataset) from the WebShop environment. Following the original paper, we represent observations using only the text modality. In the human dataset, we have 452 episodes, wherein the train split has 398 episodes, 3.7k transitions and an average reward of 7.54, and the evaluation split has has 54 episodes, 406 transitions and an average reward of 8. We also use the final IL checkpoint provided by the authors of WebShop to collect datasets of different sizes, i.c. 100, 1000, 1500, 5000, 10000 episodes, where in all these datasets the average reward is 5.8-5.9. In this case, larger dataset also has a greater diversity of environment instances specified by different natural language instructions (or item descriptions).
Data Storage Following Yao et al. 2022) we store all episodes in single jjson file.
21

---

Published as a conference paper at ICLR 2024
G EXPERIMENTAL SETUP
We collect data from 200 different Procgen levels for offline training, validate the hyperparameters online to perform model selection on the another 50 levels, and evaluate the agents' online performance on the remaining levels, i.e. level_seed [250, oo). For more details on the process for collecting these datasets and their reward distributions, see Appendix [F] We also explain the architecture used for the underlying policy in Appendix I
G.1 BASELINES
1. Behavioral Cloning (BC) is trained to predict the actions corresponding to all states in the dataset, via cross-entropy loss. This baseline is parameterized by either ResNet (He et al.) 2016) (in the case of Procgen) or a BERT (Devlin et al. 2019) (in the case of WebShop), takes input the current state and outputs a probability distribution over all possible actions.
2. Batch Constrained Q-Learning (BCQ) Fujimoto et al. (2019) restricts the agent's action space to actions that appear in the dataset for a given state, in an effort to reduce distributional drift which is one of the main challenges in offline RL.
3. Conservative Q-Learning (CQL) Kumar al. (2020a) regularizes the Q-values by adding an auxiliary loss to the standard Bellman error objective, in an effort to alleviate the common problem of value overestimation in off-policy RL.
4. Implicit Q-Learning (IQL) Kostrikov et al. (2021): : uses expectile regression estimate the value of the best action in a given state, in order to prevent evaluating outto--distribution actions.
5. Behavioral Cloning Transformer (BCT) Chen al. (2021 a transformer-based version of BC, where the agent's policy is parameterized by causal transformer with a context containing all previous (state, action) pairs in the episode. The agent has predict the next action given the current state and episode history.
6. Decision Transformer (DT) Chen et al. (2021) is similar to BCT but in addition to the state and action tokens, the context also contains the return-to-go each step. The agent has to predict the action and the current return-to-ge given state and episode history. At test time, DT conditioned on the maximum possible return-to-go o for that particular game.
All our implementations have almost I million parameters with ResNet He et al. 2016) encoder and trained and evaluated in similar manner detailed in Appendix H.2.1
G.2 EVALUATION METRICS
For Procgen, we report the mean and standard deviation across either 3 or 5 model seeds for each game, as well as the inter-quartile mean (IQM) (Agarwal et al. 202 and mean normalized return averaged across all 16 games. We follow Agarwal et al. 2020) which showed that IQM is more robust metric than the mean or median when reporting aggregate performance across multiple tasks, especially for small number of runs per task. For WebShop, follow the recommended procedure in Yao et al. (2022) and report the average scores and success rates on a set of train and test instructions.
H HYPERPARAMETERS
H.1 BEHAVIOUR POLICY
Procgen To collect datasets for each of the 16 games within Procgen, we employed the Proximal
Policy Optimization (PPO) algorithm (Schulman et al. (2017) using the setup outlined in Raaleeanu & Fergus (2021). Specifically, our PPO training involved training the policy for 25 million environment steps, utilizing a set of 200 training levels. The architecture consists of a ResNet (He et al. 2016) which encodes the 64x64 RGB images into a linear embedding, which is then processed by two parallel fully-comnected layers, one for the actor and one for the critic with hidden dimension of 256. The policy is trained for 25M environment steps and checkpoints are saved regularly throughout
22

---

Published as a conference paper at ICLR 2024
this process. All of the hyperparameters are same the one used in (Raileanu & Fergus 2021
and (Cobbe et al. 2020) and are detailed in Table which were shared across all 16 games.
Table 2: Table summarizing the hyperparameters used for PPO in Procgen
Hyperparameter
Value
Y AGAE
0.999
0.95
PPO rollout length PPO epochs
256
3
PPO minibatches per epoch
8
PPO clip range PPO number of workers
0.2
Number of envs per worker
64
Adam learning rate
5e-4
Adam
le-5
PPO max gradient norm
0.5
PPO value clipping
no
return normalization
yes
value loss coefficient
0.5
entropy bonus
0.01
To provide overview of the policy's performance, Figure&8ded depicts the training returns across each game for the entire 25 million environment steps for all 3 model seeds. Our expert dataset, which obtained online, is derived from the final checkpoint curated suboptimal dataset by selecting a checkpoint having performance at 50% of the expert's level. The achieved returns for these checkpoints in each game are listed in Table
this training procedure. Furthermore, we
WebShop We employed a pre- pre-trained Imitation Learning (IL) checkpoint provided by Yao et al., 2022) to collect training trajectories by rolling out the policy in the online environment. This particular checkpoint achieves a score 59.9 (which corresponds
mean reward of 5.99) and success rate 29.1% 1% in the test set. This policy consists of I 10M parameter BERT model for encoding the current states and a list of available actions, and outputs log probabilities over the available actions.
H.2 MODEL. TRAINING
Here we list down the set of hyperparameters used in each offline learning algorithm separately. Moreover, all offline RL baselines (i.e. BCQ, CQL and IQL), plus BC, had the same encoder size and type, which was a ResNet in the case of Procgen, and a BERT encoder in the case of WebShop. All of our experiments were run on single NVIDIA V100 32GB GPU on the internal cluster, with varying training times and memory requirements.
H.2.1 PROCGEN
Hyperparameters For BC, we performed sweep over the batch size (64, 128, 256, 512) transitions and learning rate E (5e- 3, le 4, 5e - 4, 6e 5). For BCQ, CQL, and IQL, which use DON-style training setup (i.e., they have base model and a frozen target model) (Mnih et al. 2013), in addition to the hyperparameters mentioned for BC, we swept over whether to use polyak moving average or directly copy weights, in the latter case, the target model update frequency {1. 100, 1000} and in the former case, the polyak moving average constant 7 E (0.005, 0.5, 0.99). For BCQ, we also swept over the threshold value for action selection {0.3, 0.5, 0.7]. For CQL. swept over the CQL loss coefficient, which we refer to eql-alpha in our codebase, (0.5, 4.0, 8.03. Finally, for IQL, we sweep over the temperature E {3.0, 7.0, 10.0} and the expectile weight (0.7, 0.8, 0.9).
For sequence modelling algorithms, DT and BCT, we sweep over the learning rate and batch size mentioned above, well as the context length size (5, 10, 30. 50). For DT, we also sweep the return-tee to-go (rtg) multiplier E 5]. We follow similar approach in (Chen et al. 2021) to set the maximum return-to-go at inference time by finding the maximum return in the training dataset for
23

---

Published as a conference paper at ICLR 2024
gigtthh
Caunelyer
chanee
Sport Sauu Spoc
Spee
Von Ronie
Soud
Sood
Won Erhage Tan Rours
NH
660
2.3
2.5
1.4
3.5
2.00
3-3
2.5
000
0.5
colrrun
Specel
Plun Spiedd Rouuto
Space
Yian Stode toan toas Recuur
00
2.5
2.0
2.3
2.5
0.0
0.5
1.0
2.0
0.5
3.0
2.0
0.5
1.0
Sieel
1450
1.
10
Sinel
Spoed
Edeade Tai Aeriim mn
&
Mon Eroek Tot Patinouus
Sanddo Sand
Sees
00
2.5
2.0
3.5
0.0
0.5
1.0
3.5
2.0
0.3
0.5
3.0
355
2.0
0.0
0.5
2.0
stargeios
Sioed
Seed
Since
Gioed
Seee 22
Seed
Mast Tplote Than Ratunt
Mon Spose Tuan Kann
Mon Splods
Mon
E
Sevel
00
3.5
1.0
2.2
2.5
0.0
3.0
3.5
2.0
2.5
2.2
0.5
3.0
0.0
2.5
2.0
2.0
Figure 8: Training Returns for the Procgen data collecting behavioral policy PPO.
particular game and then multiplying by either or depending on the rtg multiplier value. We also the default value of 0.1 for dropout in DT and BCT used originally in (Chen al. 2021) for optimal performance.
We 3 random trials per each configuration and select the best hyperparameter by looking the min-max normalized mean train and validation results, averaged across all 16 games. Train results are calculated by rolling the final checkpoint of the policy 100 times on training level and likewise for validation results, the policy is rolled out 100 times by randomly sampling one level at a time of 50 validation levels. During our initial experiments overfitted quickly (within 10-20 epochs) on the training dataset. Therefore, to save training time and prevent overfitting, we employ early stopping by calculating the validation return after every epoch and stopping the training process if the validation return does not improve in the last 10 epochs. However, in Section 4.5l where the dataset size was either 5M 10M transitions, we used fixed epochs only.
noticed that many of these algorithms
Table 3 list the final hyperparameters for BC, BCQ, BCT, DT, CQL and IQL for 1M expert and suboptimal dataset, 10M expert dataset well for the single level experiments (except for IQL, which uses polyak averaging in single-level experiments). Since the 25M suboptimal dataset, which used in Section 4.2 has very different distribution than synthetically created 1M mixed subopt-uub dataset (which had 75% returns of experts'), we ran a hyperparameter sweep on this dataset by uniformly sampling 1M transitions and following similar procedure above. We
24

---

Published as a conference paper at ICLR 2024
Algo
Hyperparameter
1M Expert
1M
Mixed
25M Subopti-
10M Expert
Suboptimal
mal
Learning Rate
0.0005
0.0005
0.0005
0.0005
BC
Batch Size
256
256
256
256
Learning Rate
0.0005
0.0005
0.0005
0.0005
Batch Size
512
512
512
512
BCT
Context Length
30
5
5
5
Eval Return Multiplier
0
0
Learning Rate
0.0005
0.0005
0.0005
0.0005
Batch Size
512
512
512
512
DT
Context Length
10
5
5
5
Eval Return Multiplier
5
5
5
Learning Rate
0.0005
0.0005
0.0005
0.0005
Batch Size
256
256
512
256
Target model Weight Update
Direct copy
Polyak
Direct copy
Direct copy
BCQ
0.5
Target update frequency Threshold
1000 0.5
1000
1000
1000
0.5
0.5
0.5
Learning Rate
0.0005
0.0005
0,0005
0.0005
Batch Size Target model Weight Update
256
256
256
256
Direct copy
Polyak
Direct copy
Direct copy
CQL
0.99
Target update frequency Alpha
1000
1000
1000
1000
4.0
4.0
4.0
4.0
Learning Rate Target model Weight Update Batch Size
0.0005
0.0005
0.0005
0.0005
Direct copy
Polyak
Direct copy
Direct copy
512
256
512
512
IQL
0.005
Target update frequency Temperature
100
100
1000
100
3.0
3.0
3.0
3.0
Expectile
0.8
0.8
0.8
0.8
Table 3: List of hyperparameters used in Procgen experiments
25

---

Published as a conference paper at ICLR 2024
performed similar sweep for the 10M expert dataset experiment as well. The best hyperparameters for this dataset are also listed in Table
H.2.2 WEBSHIPP
Hyperparameters For BC, we performed sweep over the batch size E (1,4, 1,4,8) 8) and learning
rate E {2e 5, 2e 4, 2c 3). For BCQ and CQL, we swept over the target model update frequency E {100, 1000). For BCQ, we also swept over the threshold value for action selection {0.1, 0.5, 0.9} and for CQL
swept over (4.0, 7.0).
For scaling experiment in Section the sweep for BCQ threshold was even wider
{0.01, 0.1, 0.2, 0.3. 0.4, 0.5, 0.6, 0.7, 0.8, 0.9) with rest the hyperparameters being the same from the best hyperpa- rameter selected above.
We run trial for each combination of hyperparameters and select the best performing one by rolling out the agent all validation goal levels (goal_idx; (500, 1500) and the first 500 train goal levels (goal_idx* (500, 1500) ). Final hyperparameters are listed in Tables for BC, CQL and BCQ.
Algorithm
Hyperparameter
Human Demonstrations
IL Trajectories
Learning Rate Batch Size
0.00005
0.00005
BC
Learning Rate
0.00005
0.00005
Batch Size
1 0.005
0.005
BCQ
Target update frequency Threshold
100
1000
0.5
0.9
Learning Rate
0.00005
0.00005
Batch Size
1
1 0.005
CQL
T
0.005
Target update frequency
1000
100
Alpha
7.0
4.0
Table 4: List of hyperparameters used in WehShop experiments
I TRAINING AND TESTING ON SINGLE LEVEL IN PROCGEN
Here we show the results when training agents on expert and suboptimal transitions from level 1 and testing online only on that level. Figures [3c (overall IQM) and [1 (per-game returns) show the performance of all baselines after training and testing on expert dataset from level 1, demonstrating that not just BC, but other offline RI baselines can learn well when trained on high- return demonstrations from singleton environments. Note that offline RL methods still seem to struggle more some of the games, even when trained on demonstrations from single level. Also note that here even the final checkpoint of PPO struggles in some games (achieving reward), and that is why report performance Figure [2] show the performance of these baselines when trained on suboptimal dataset level 1. Here, 9 out of 16 games, offline RL performs comparable even better (in Bigfish, Fruitbot, Heist, Jumper) compared to BC and sequence modelling.
level 40 well in Figures[1b) and [ Figure 13d and
26

---

Published as a conference paper at ICLR 2024
Level Seed 40: Expert Dataset
Dataset Average
PPO performance on Level Seed 40
bigfish
bossfight
caveflyer
12.5
12.5
12.5
10.0
10.0
7.5
5.0
5.0
5.0
2.5
2.5
RET BCQ OGL ROL
RET BCG IGL
RCT BC0 1GL
BCT BC. Tol
dodgsball
10
40
12.5
30
20
Average Peturn
5.0
2.5
0.0
BC
BCT
DT BCQ OVL 126
BC
SCT DT BCO ONL INC
SC
SCT DT BCO ONL INL
SCT DT BCC COL IOL
helst
jumper
leaper
maze
10
10-
10.0
10
5.0
4-
2.5
0.0
8C BCT
DT BCQ COL IQL
BC
BCT
DT BCQ C2L 1QL
8C
BCT
OT
BCQ COL IQL
8C BCT OT BCQ COL IOL
miner
ninja
plunder
starpilot
15.
10.0
20
7.5
10
5.0
10
2.5
BCT
BCT
BCT
Figure 9: Performance of each baseline level 40 in each Procgen game when trained and tested on the same level using expert dataset.
27

---

Published as a conference paper at ICLR 2024
Level Seed 40: Mixed Expert-Suboptimal Dataset
Average
performance
bigfish
bossfight
caveflyer
chaser
10
12.5
12.5
10.0
7.5
7.5
5.0
5.0
5.0
2.5
0.0
BCT
BCT
SCT
climber
colnrun
dodgeball
fruitbet
40
30
10
Average Return
20
10
2-
BCT BCQ
BCT BCQ IQL
BCT IQL
BCT IQL
heist
jumper
leaper
10.0
2.5
0.0
BC BCT DT BCQ OOL IQL
BC BCT DT BCQ OOL IQL
BC BCT DT BCO OQL IQL
BC
BCT DT BCO COL IQL
miner
ninja
plunder
starpilot
12.5
25
10-
10.0
20:
7.5
15
5.0
10
2
0.0
BC
BCT
DT
OCL 1OL
BC BCT
DT
BCO OOL 10L
BC BCT DF 8C0 00L 1OL
8C
9CT
DF BCO COL
IOL
Figure 10: Performance of each baseline on level 40 in each Procgen game when trained and tested on the same level using mixed expert-suboptimal dataset.
J EFFECT OF DATA DIVERSITY ON GENERALIZATION: SUBOPTIMAL DATA
Here use the entire training log of the behavioral policy (PPO) (Appendix H. and use subset of interactions made by that agent as our offline dataset (see seed in Figure g which has 25M transitions in total. This approach of using the training trajectory of the behavioural policy and sampling mixed data out of it is also consistent with the previous literature ( (IAgarwal et al. 2020). Gulcehre et al.) 2020), (Kumar et al. 2020ah. (Fu et al. 2020)). We then uniformly sample episodes from the dataset such that we get a mixed, suboptimal-exii -eppert training dataset in Procgen and the dataset size has almost million, 5 million as well as 10 million transitions. Throughout this scaling process, keep all other hyperparameters same as well as the number of training levels constant at 200.
From Figures 14 and 18 it is evident that all the algorithms exhibit poor train and test performance, even with 10M transitions. well and even outperform BC when trained suboptimal data, experiments show that they still struggle when trained on data from multiple environments and tested on new ones, irrespective the quality of the demonstrations (i.e., whether they expert of suboptimal).
contrast with prior work showing that offline RL approaches learn
28

---

Published as a conference paper at ICLR 2024
Level Seed 1: Expert Dataset
Dataset Average
PPO performance on Level Seed
bigfish
bossfight
caveflyer
10
20
0.25
15
0.20
10
0.15
I
0.10
5-
0.00
RCT BCQ ROC
HET BCG IGE
RCT BC0 TOL
BCT BCG ToL
dodgsball
2.5
10
30
2.0
20
Average Return
1.5:
0.0
BC
BCT
DT BCQ OVL 10%
BC
BCT DT BCO ONL INC
SC SET Dr 8:0 Ol INC
8C
SCT DT BCC COL INL
helst
10.0
jumper
0.0
1e 17
leaper
10
maze
10
-0.5
7.5
-1.0
5.0
4-
-2.0
2-
0.0
-2.5
BC
BCT DT BCQ COL IQL
8C
BCT
DT BCQ COL IQL
8c
BCT DT BCQ CoL IdL
82
BCT OT
BCQ COL [0L
miner
ninja
plunder
starpilot
10
10
BCT
BCT
Figure 11: Performance of each baseline on level in each Procgen game when trained and tested on the same level using expert dataset.
29

---

Published as a conference paper at ICLR 2024
Level Seed 1: Mixed Expert-Suboptimal Dataset
Dataset Average
PPO performance on Level Seed
bigfish
bossfight
caveflyer
chaser
30
0.25
0.20
20
0.15
0.10
0.05
0.00
RCT
BCG
RET BCQ IGL
SCT BCG TGL
OCT BCG loL
dodgsball
1.0
10-
0.8
20
Average Return
10
0.2-
0.0
BC
BCT DT BCQ OOL 1X%
BC BET DT BCO ONL INC
SC SET or 800 Ol INC
80
SCT DT BCO COL INL
helst
jumper
0.0
1e 17
leaper
10
maze
10
10
-0.5
-1.0
-1.5
4-
-2.0
2-
-2.5
BC
BCT
DT BCQ COL IQL
BC
BCT
DT BCQ COL IQL
8c BCT DT BCQ CoL IdL
aic
BCT DT BCQ CoL IdL starpilot
miner
ninja
plunder
12.5
10.0
10.0
7.5
5.0
2.5
2.5
0.0
BCT
BCT
BCT
BCT
Figure 12: Performance of each baseline on level in each Procgen game when trained and tested on the same level using mixed expert-suboptimal dataset.
30

---

Published as a conference paper at ICLR 2024
Pragen Exper Celead.
Proggen Duanet
PP Perforance Dpett Datand Azerage
PPC Performares
Subast Coteuer Aeurage
D7
COL
Level Seed 40 (Expert)
(b) Level Seed 40 (Suboptimal)
0.6
Smarrnat Dinge
Inatormanse
Performance
0.0
COL
02
RC0
COL
(c) Level Seed (Expert)
(d) Level Seed (Suboptimal)
Figure 13: Aggregated performance of each baseline across all Procgen games when trained and tested on the same level using suboptimal or expert dataset. Red line represents the performance of our expert PPO checkpoint on this level.
-0.06
10 10--Raa000 Norrel and (24 Raturs
0.008
-0.11
MttsMtc Norma laed 12P
0..10
-0.13
0.00
-0.14
Tran
-0.15
-0.02
Tran
Number training
Number training levels
Number training
BC
(b) BCQ
DT
Figure 14: The Effeet of Data Size on Performance (Suboptimal). Train and test min-max normalized IQM scores for BC, BCQ, and DT the size of the suboptimal dataset increased from I to 10 million transitions. algorithms have poor train and test performance (even when using 10M transitions). While the train performance slightly increases with the dataset size, the test performance does not vary much. For remaining algorithms, see Figure 18
K SCALING NUMBER OF VALIDATION LEVELS IN PROCGEN
In this section, we compare the use of limited set of 50 validation levels in Section We contrast this approach with similar proportion of levels used for training. Specifically. if n) levels are allocated for training, then [r,, (0.1 n)) levels are reserved for validation. As illustrated in Figure 17 a notable trend emerges among all offline learning algorithms, indicating robust correlation between the two aforementioned approaches. To maintain methodological consistency with broader array of experiments presented in this paper, we advocate for the adoption of a consistent practice for this benchmark. Specifically, suggest that researchers consider employing level range of [n, n + 50) exclusively for validation purposes.
L EFFECT OF DATA SIZE ON GENERALIZATION; MORE RESULTS
In this section, perform the experiment of scaling offline learning algorithms on the 10M expert dataset and 25M suboptimal dataset from Section 4.5 on the remaining offline learning algorithms: BCT, CQL and IQL. For both datasets, observe sumilar findings, i.e. expert dataset
31

---

Published as a conference paper at ICLR 2024
Km. fomatted Iecu
Km Man-Maa Nomakkeen ige metuum
0.4
0.4
0.3
0.3
0.2
0.2
0.2
0.1
0.1
0.1
0.0
200
400
800
10k
100k
0.0
200
400
800
10k
100k
0.0
200
400
800
1k
10k
100k
Number of training levels
Number of traininng levels
Number of training level:
(a) BCT
(b) CQL
(c) IQL
Figure 15: The Effect Data Diversity on Performance BCT, CQL IQL.. Train and test performance for varying number of training levels in the IM expert datasets, aggregated across all Procgen games. The plot shows the IQM and error bars represent the 75-th and 25th percentiles computed over 3 model seeds. While the training performance does change much with the number of training levels, the test performance increases (and generalization gap decreases) with the diversity of the dataset.
Scaling Procgen 10M Expert Dataset
IQL Train
I
CQL Train
I BCT Train
IQL Test
-I-- CQL Test
-T--
BCT Test
Min.Mat Max Normalizen lizd IQM Retur
0.3
0.2
0.1
1M
5M Number transitions
10M
Figure 16: The Effect of Data Size on Performance (Expert). Train and test min-max normalized IQM scores for BCT, CQL and IQL the size of the dataset is increased from 1 to 10 million transitions aggregated across random trials. Scaling the dataset size while keeping the number of training levels fixed (to 200) leads to slight increase in both train and test returns, but the generalization gap remains about the same.
(Figure 16). all three methods have almost constant generalization gap the data size scales up. On the suboptimal dataset (Figure 18), all methods have poor train and test aggregate performance and contrary to prior works, in our setup when we sample and train offline learning algorithms on a subset of episodes from the training log of the behavioral policy, the resulting offline learning policy does not generalize all and does not perform well the 200 training levels well.
L.1 GENERALIZATION TO NEW INSTRUCTIONS USING SUBOPTIMAL DEMONSTRATIONS IN WEBSHOP
Here we train BC, CQL and BCQ on dataset collected using pre-trained behavioral cloning policy (see Appendix F22 to test the effect of scaling the number episodes (and hence the variety of goal instructions) on the performance of these baselines. Figure 19 shows that in all three baselines, the train and test performance significantly increases once the dataset has at least 500 episodes. After that point, while there is not much increase in the scores in train and test levels, there is slight increasing trend in the success rates for BCQ and CQL. Overall, similar to Procgen, BC outperforms both BCQ and COL on all datasets, thus highlighting the need for research in this area
32

---

Published as a conference paper at ICLR 2024
0.6
NI59A Www.acNNemaa Normation 0R Rattroo
Wad Nermaoed IOM maBuu
1244
0.4
0.4
0.4
0.2
0.2
0.2
0.1% Vol
0.0 200
400
800
1k
10k
100k
0.0
200
400
800
10k
100k
0.0
200
400
800
1k
10K
100k
Number of training lovels
Number of training levels
Number of training level
(a) BC
(b) BCT
(c) DT
0.6
0.6
mn Norme term munum
m- Nerm merma tard E3M muaum
0.2
ValLeNes
ValLoves
3.1% Levis
0.1H Vol Levels
3.1% Levess
0.0
200
400
800
1K
10K
100k
0.0 200
400
800
10k
100k
0.0
200
400
800
1k
10k
100k
training levals
Number traininn
Number training levals
(d) BCQ
(c) CQL
(f) IQL
Figure 17: Comparing the of 50 validation levels vs. validation levels proportional to the number of training levels when trained on 1M expert transitions with varying number of training levels in Procgen. IQM was computed using model seeds for each experiment.
0.04
Mendtee Manmaa toud KIM Ruturr
-0.10
0.02
-0.17
Moraatod
0.00
-0.14
-0.14
Trant
10M
1M
10M
10M
Number of training lovels
Number training levels
Number of fraining levels
(a) BCT
(b) COL
(c) IQL
Figure 18: The Effeet of Data Size on Performance in Procgen (Suboptimal). This plot shows the train and test min-max normalized IQM scores for BCT, CQL and IQL as the quantity of suboptimal offline dataset is increased from million transitions to 10 million transitions. As can be seen, all algorithms have very poor train and test performance (even when using 10M transitions) and at very granular level, the train performance generally increases, but test performance does not change much.
M MORE SCALING ANALYSIS
Here we try to see whether having fixed number of episodes per level (i.e. 10 episodes per level) as opposed having fixed million transitions total irrespective of the number of levels leads to better generalization gains. For this, we collect many different datasets where we specify the total number of levels [200, 400, 8001000) we want to collect the data from and on each level we collect 10 episodes (each episode has atleast transitions). Hence, each dataset can have more than million transitions in total.
We ran this experiment for maximum of 1000 seeds (beyond which the dataset size became too huge to fit in single GPU). In figure 20 we observe that in BC, BCQ, DT and BCT, the generalization gap decreases slightly. but the trend is similar to what had observed in the of fixed million transitions Section Hence the amount data per level in the underlying dataset does not affect the generalization performance of the offline RL method.
33

---

Published as a conference paper at ICLR 2024
Average Scaling number training trajectories WebShop
BC
Auerage Sucoess Rata
301
2
CQL
8
2. 3
BCQ
50
25
221
17203 1027
0220 20:0
1730
10.00
01203
1250
0725
1251
1257
1X30 0751
10203
12:7
0111
1250
Figure 19: Score and success rate of each baseline on the WebShop environment when trained on various sizes of datasets. Once the dataset consists of 500 episodes, the performance increases significantly, however, beyond that the change in performance is not much.
0.6
0.5
0.5
0.4
Mivdea Nemaatzdd FOR
0.4
0.1
Tran
0.1
0.1
0.0
200
400
800
2
0.0
200
400
800
0.0
200
400
800
3k
baining
training levels
tainiig
(a) BC
(b) BCT
(c) DT
0.5
0.5
0.5
We.eet Kome see 10N hotm
Wi-ouu Nomma aed EON Retum
0.5
0.5
0.5
0.4
0.4
10.4
Tran 10t
Tran tot
Tra 102
0.0
0.0
0.0
training
training
(d) BCQ
(e) CQL
(1) IQL
Figure 20: Scaling the training levels in the dataset while keeping the number of episodes per level fixed (10 episodes).
N PER-GAME SCORES IN PROCGEN
Figures 21 and show the performance of these baselines on each individual game when trained using 1M expert dataset. Figure 21, we compare the training returns which were calculated by averaging over 100 episodes by randomly sampling levels from the training set. The grey line shows the average return of the trajectories the training dataset. For most games, at least some these methods match the average performance of the training data. Among the different approaches, BC the most robust, while the offline RL and sequence modeling approaches fail to do well on some of the games. In Figure 22 compare the test returns which were calculated by averaging 100 episodes by randomly sampling levels from the test set. The red line shows the average test performance of the final PPO checkpoint across 100 randomly sampled test levels.
34

---

Published as a conference paper at ICLR 2024
1M Expert: per-game train returns
PPO Seed
Train Performance
Expert Dataset Average
bigfish
bossfight
caveflyer
15
10
1
BC BCT DT BCQ OOL IQL
BC BCT DT BCQ OQL IQL
BC SCT DT BCO IQL
BC SCT DT BCQ OOL IQL
climber
coinrun
dodgeball
fruitbot
30
20
Average Return
BC BCT DT BCQ COL IQL
BC BCT DT BCQ COL IQL
BC BCT DT BCQ COL IQL
8C BCT DT BCQ COL IOL
heist
jumper
leaper
maze
2
BCT
DT
OCT
OCT
miner
ninja
plunder
starpilot
12.5
30
10.0
20
7.5
10
2.5
Figure 21: Procgen Per-Game Train Results on the 1M Expert Dataset. Average episode return for each method on train levels for each Procgen game. The mean and standard deviation are computed across 5 model seeds. The grey line shows the average return the trajectories in the training dataset. The brown line shows PPO's average train return which computed over 100 randomly sampled train levels. Most of the methods match the average performance of the training data. In most cases, BC is competitive or better than the offline RL and sequence modeling approaches. For numerical comparison, refer to Table in Appendix.
35

---

Published as a conference paper at ICLR 2024
1M Expert: per-game test returns
PPO Seed 1 Test Performance
bigfish
bossfight
caveflyer
BC BCT DT BCQ OOL IQL
BC BCT DT BCQ OQL IQL
BC BCT DT BCO OOL IQL
BC
SCT DT BCQ COL IQL
climber
coinrun
dodgeball
fruitbot
25
20
1.0
Average Return
10
0.5
BC BCT DT BCQ COL KQL
BC BCT DT BCQ COL IQL
0.0
BC
BCT
DT
BCQ COL IQL
BC BCT DT BCQ COL IOL
heist
jumper
leaper
maze
2.5
2.0
1.0
0.5
0.0
BCT DT
ICT
miner
ninja
plunder
starpilot
10
20
15
10
Figure 22: Procgen Per-Game Test Results on the 1M Expert Dataset. Average episode return for each method on test levels for each Procgen game. The mean and standard deviation are computed across 5 model seeds. The red line shows PPO's average test return which was computed over 100 randomly sampled test levels. As seen in previous figure, most of the methods match the average performance the training data. However, many them fail to reach PPO's performance at test time. In most cases, BC is competitive better than the offline RL and sequence modeling approaches. For numerical comparison, refer to Table in Appendix.
36

---

average than Published as a conference paper at ICLR 2024
Environment
BC
BCT
DT
BCQ
CQL
IQL
higfish
10.94 0.75
9.99 + 0.64
10.35 + 0.69
6,23 +: 1.02
9.02 = 1.37
8.19 =: 0.54
bossfight
7.16 - 0.35
1.11 0.15
1.01 0.188
+.36
8.57
0.32
8.45 + 0.36
caveflyer
0.15
6.56
2.57 2.39
2.65 0.23
4,55
chaser
5.6 0.12
2.67 0.23
2.98 0.199
4.37 0.56
1.420.52
3.68 0.13
climber
8.27 0:22
8.35 1:55-:00
7,98 0.27
0.17
5.22 0.26
coinrun
9.48 9.48 0.14
9.32 0.06
9.42 1.1200.0
0.18
8.58 0.20
8.32
dodgehall
4.01 - 0.22
3.43++0.00
0.24
1.55
0.15
1.14
2.74 0.22
fruitbot
0.98
23.36
0.38
14.68
1.29
19.42 1.21
27.51 0.51
heist
7.78=
0.22
7.32 0.12
3.96
3.74 0.31
5.02 0.42
jumper
0.06
0.1
8.51
7.14
7.68
0.26
leaper
2.78=0.15
2.31
2.96 0.2
2.54 0.08
0.22
3.08 0.15
maze
9.0
8.08 0.2
+
+.34
7.48 0.13
1.14
1.23
0.2
0.33
6.98
8.48
ninja
0.33
7,62+:0.2
7.72
0.12
4,76:0330
5.92
5.7 0.35
plunder
5.26 = 0.22
0.112
5.57 0.07
4.47 0.56
4.78
0.36
4.56 0.17
sturpilot
20.91
14.39
14.30 0.559
26.36 1.02
0.64
Table Mean and Standard Deviation for train performances each offline learning algorithm averaged over 5 random seeds when trained on 1M Expert Dataset in Proegen.
Environment
BC
BCT
DT
BCQ
CQL
IQL
bigfish
4.38 = 0.38
2.18 + 0.13
2.37 + 0.16
3.57 + 0.34
4.1 0.37
4.85 = 0.52
bossfight
5.87 = 0.26
0.35 + 0.11
0.52 : 0.08
6.53
0.33
8.13 = 0.13
7.02 = 0.33
caveflyer
4.92 - 0.28
3.85
0.17
3.48 0.35
2.15 0.26
2.12 0.28
3.43:0..2
chaser
4.05
4.39
elimber
4.91 = 0.22
0.19
0.28
0.17
0.98
2.33 0.33
coinrun
8.26 + 0.19
7.66
8.08
7.02
7.22=0.17
7.74
dodgehall
0.98 + 0.07
0.89
0.84+ 0.04
0.76 0.09
0.85 0.0T
0.93 + 0.12
fruilbot
1: 0.62
13.93
13.56 0.73
16.99 2.12
25.22
beist
2.42 0.14
1.110.16
1.78 0.00
0.48
0.12
0.58 - 0.26
jumper
5.68 + 0.18
4.14
5.6 0.23
4.14 0.22
4.26 = 0.22
4.06 + 0.21
leuper
2.84 0.07
2.36
2.36 0.188
2.48 0.09
282 0.28
244
maze
4.46 - 0.16
4.26
3.88
2.48 0.11
3.0
0.31
miner
7.85 0.32
0.06
1.07
1.46
2.21
1.60 0.17
ninja
5.88 + 0.3
0.07
0.24
4.52 0.44
4.36 = 0.25
4.38 - 0.12
plunder
4.94 0.13
0.12
-.24
3.66 1 0.29
3.84 0.23
4.03 0.14
starpilot
17.69 0.59
1.49
10.72 = 0.32
22.21
0.7
22. 42 0.39
22.88 + 0.59
Table Mean and Standard Deviation for the test performances of each offline learning algorithm averaged over 5 random seeds when trained on 1M Expert Dataset in Procgen.
Generalization New Environments (Expert)
In most cases, BC performs similarly better than the offline RL and sequence modeling approaches on test levels. However, all the offline learning methods (state-of-the-ar offline RL, behavioral cloning, and sequence modeling approaches) perform the online RL method (PPO) on unseen environments. For more than half of the games, offline RL methods cannot reach PPO's test performance. All sequence modelling and offline RL methods fail to generalize on the game of Miner, which is one of the most stochastic games Procgen.
Tables 5 6 and show the train and test performances of each offline learning algorithm in 16 Procgen games for both types of offline data regimes, 1M expert and 1M mixed expert-suboptimal respectively. Moreover, we also plot the generalization gap for each algorithm in every game for better understanding of how well they learn from training levels and perform zero-shot on testing levels in Figures 23 and
37

---

Published as a conference paper at ICLR 2024
Environment
BC
BCQ
IQL
CQL
bigfish
7,81 = 0.91
7.83 1.19
7.47
7.53 = 1.02
8.03 = 1.04
0.777
bossfight
5.53 0.22
0.71
7.24
1.73=
7.96
caveflyer
5.06 + 0.44
5.14
4.34
3.79 0.67
2.68 0.44
+
chaser
4.05 = 0.21
2.37 0.19
2.35
1.74
1.42
0.76
elimber
6.94 0.35
0.34
0.12
0.21
1.15
2.98 0.99
coinrun
0.33
7.73
dodgeball
3.03 1: 0.05
3.03
0.44
0.13
2.06
2.15 0.15
fruitbot
1.12
0
15.38 1.09
33,362022
26.89
26,06 0.6
heist
6.43 + 0.34
4.83 0.5
4.43 0:23
2.47 0.12
1.67 0.42
1.6
jumper
7.53
0.37
0.34
6.83 0.48
6
6.87 0.27
leaper
0.15
2.22
0.27
2.47
2.68
2.77
maze
0.26
0.2
0.65
0.17
0.48
3.77 -
miner
0.39
6.67
0.34
3.56
4.05
2.36
ninja
0.1 0.1
6.710.17
6,4 0.15
0.12
3.77
4.97 0.32
plunder
5.81 0.21
0.11
49
4.18
0.44
3.93
starpilot
21.92 0.35
13.64
12.34
22166
22.8
21.57 2.17
Table 7: Mean and Standard Deviation for the train performances of each offline learning algorithm averaged over random seeds when trained on 1M Mixed Expert-Suboptine Dataset in procgen. See Appendix 4.2 for more details.
Environment
BC
BCT
DT
BCQ
IQL
CQL
bigfish
2.89 0.15
2.09 + 0.08
2.23 0.15
4.13 0.52
4.14 0.54
3.64 =: 0.36
bossfight
5.13=0.11 0.14
0.18
0.58 0.16
6.69
0.57
7.12
7.91 0.35
caveflyer
4.05 0.24
3.14
3.5
2.42
1.66
1.97 0.41
chaser
3.43
0.05
0.07
0.2
1.6
2.58
climber
4.64 0.29
3.18
0.43
0.25
0.35
0.94
coinrun
7.07
7.5
6.6320.11
0.3
7.17
dodgeball
1.19-0.L
0.05
0.94
0.55
0.11
0.01
fruitbot
0.7
11.39
1.33
144
2..0
24.46
0.52
heist
2.37 0.3
0.17
1.97
0.53 + 0.15
0.27 0.03
0.5 0.06
jumper
4.63 4.63 0.47
4.27 0.43
4.53
0.15
3.5
2.17
leaper
0.26
1.77=0.33
4.77 0.19
5.5 0.06
1.93 0.13
2.15
1.97 0.18
miner
6.56=0.00
1.28
1.28
0.51 0.15
0.8
0.43 0.08 0.08
ninja
5.23 0.12
5.37 + 0.38
5.1 0.36
4.67 + 0.43
3.23+0.0
3.77 0.26
plunder
4.59
0.17
3.39
3.86 0.25
0.26
starpilot
17.93 0.32
11.64 0.71
0.22
21.86 2.07
19.64 1.79
20.11 + 0.43
Table 8: Mean and Standard Deviation for the test performances of each offline learning algorithm averaged over random seeds when trained on 1M Mixed Expert-Suboptimal Dataset in Procgen. See Appendix 4.2 for more details.
Procgen Expert
Gap
Procgen 1M Mixed Expert-Suboptimal Detaset Generalization Gap
Min-MMx Normalized Mean Cap
Mar--Max Normalized Mcan Gap
0.4
0.2
0.2
0.0
CQL
IQL
BCQ
CQL
(a) IM Expert Dataset
1M Suboptimal Dataset
Figure 23: Min-max normalized mean Generalization Gap plots for offline learning algorithms
trained using IM expert and suboptimal dataset Procgen respectively.
38

---

Published as a conference paper at ICLR 2024
Number of opisodes per level Procgen 1M Export Data
20.
17.5
15.
179
Limbbe
dodgeball
Gra
Jumper
Iesper
15379
20
100
15.5
200
Figure 24: Number of episodes per level from the training set for each environment in the 1M expert dataset from Proegen. Red line represents the median. Lower and upper green lines represent the 25th and 75th percentile of the values respectively.
39

---

Published as a conference paper at ICLR 2024
Total number of transitions per level Procgen 1M Expert Data
bigitsh
bosutght
cavelyer
C1200
17500
25900
5000
10000
15000
0:50
2000
frunbott
23000
30000
1110000
5000
A25000
5006
4000
1
1..0
15.0
200
30
353
155
230
3.00
100
135
206
Jamppet
kugas
11400
53000
40000
3000
20000
4000
Mlander
10000
33000
73008
500.3
1300
53008
40000
30008
15009
4000
4000
7308
10000
20
100
125
200
30
300 150
200
300
250
53
100
15c
200
Figure 25: Total number of transitions per level from the training for each environment in the 1M expert dataset from Procgen. Red line represents the median. Lower and upper green lines represent the 25th and 75th percentile of the values respectively.
40

---

Published as a conference paper at ICLR 2024
Number of episades per level Procgen 1M Suboptimal Data
NgRhh
boodigtt
Cooer
1140
135
155
720
110
150
155
755
Jumser
Insper
700
110
150
plunder
starples
Figure 26: Number of episodes per level from the training set for each environment in the 1M suboptimal dataset from Procgen. Red line represents the median. Lower and upper green lines represent the 25th and 75th percentile of the values respectively.
41

---

Published as a conference paper at ICLR 2024
Total number of transitions level Procgen IM Suboptimal Data
bossfgtt
2000C
12063
12C83
20:0C
1O005
$300
18.0
1..00
15.0
1.00
Cleedoor
Jkdlgsbull
15000
12C00
C000
7904
3000C
CE00
1.0
50
3.5g
15.0
2.00
3.50
215
160
130
200
Fxdss
jarmper
Supper
40000
12003
LO00S
18000
30000
3.00
ringa
plando:
30000
28900
0003
4003
200
Figure 27: Total number of transitions per level from the training set for each environment in the 1M suboptimal dataset from Procgen. Red line represents the median. Lower and upper green lines represent the 25th and 75th percentile of the values respectively.
42

---

Published as a conference paper at ICLR 2024
Per-game Generalization Gap (Procgen 1M Mixed Sxbortt-Suopptimal Dataset)
bigfish
bossfight
caveflyer
chaser
1.0
2.0
0.50
0.5
0.25
0.0
1.0
0.00
2
-0.5
00.55
BC BCT DT BCQ COL
-1.0
BC BCT DT BCQ COL IOL
0.0 BC BCT
BCQ COL IQL
-0.50
BC
BCT DT BCQ COL IOL
climber
dodgeball
fruitbot
1.5
2.0
1.5
Average Return
0.5
1.0
2
0.0
helst
Jumper
leaper
maze
1.0
3-
3
0.5
2
2
0.0
0
BC: BCT
D1 BL. DOL (Q)
0
5C.
BCT
OTT BOQ COL 10
BC BCT BCG COL 101
100
BC. 101 BOQ COL RX.
miner
ninja
plunder
starpilot
5
1.5
4
1.0
4
3
0.5
2
0.0
0.0
BCQ
BCT
COL
Figure 28: Mean and standard deviation of Per-game Generalization Gap for every offline learning algorithm in Procgen when trained using 1M suboptimal data. Every algorithm is run for 3 random trials.
43

---

Published as a conference paper at ICLR 2024
Per-game Generalization Gap (Procgen 1M Expert Dataset)
bigfish
bossfight
caveflyer
chaser
1.5
1.0
0.5
0.0
-0.5
-1.01
COL
CQL
BCQ
BCQ
climber
coinrun
dodgeball
fruitbot
3
10.0
1.5
7.5
3
2
Average Return
1.0
5.0
2
0.5
1
2.5
0.0
8CO COL
0.0
BCQ
heist
jumper
1.0
leaper
maze
3
0.0
2
1
-1.0
8C BCT
9C0 COL K2L
0
BC OCT DT BCD COL IQL
BC BCT DT BCQ COL IQL
0
BC BCT DT
BCG BCO 1OL
miner
ninja
plunder
starpllot
6
1.5
2.0
1.5
1.0
1.0
0.5
2
2
0.5
0.0
0.0
BC BCT
BC0 COL
K2L
BC
OCT
DF
BCO COL
IQL
BC BCT DF BOO COL TOL
BC BCT DT BCQ COL
Figure 29: Mean and standard deviation of Per-game Generalization Gap for every offline learning algorithm in Procgen when trained using 1M expert data. Every algorithm is run for 5 random trials.
44


Reviews:
[{"review_id": "ZCsOn4gxhk", "reviewer": "Reviewer_f7uZ", "summary": "This paper introduces a benchmark for evaluating generalization in offline learning. Based on the benchmark, state-of-the-art offline policy learning algorithms, including BC, sequence modeling approaches, and offline RL algorithms, are tested. The results show that all the offline learning methods perform worse than online RL in both train and test environments. The results also reveal that BC is stronger to generalize to new environments than other offline learning methods.", "questions": "Can the authors share some thoughts on why offline RL does not generalize well?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper presents new results on the generalization of offline learning. As we know, it is important to understand the generalization ability of offline learning methods in order to apply offline methods to real-world problems. Although not very surprising, this paper first confirms that offline RL and sequence modeling approaches can be struggling to generalize to new environments.\n2. The results may have a broad impact on the community. Indeed, we can no longer ignore the generalization problem of existing offline learning methods. So, more investigation is needed. The results may also have an impact on our choice of offline learning methods in application scenarios.\n3. The experimental results are sufficient and convincing. In the experiments, multiple sequence modeling methods and multiple offline RL algorithms are included, two kinds of games are tested, and multiple settings are tested. \n4. The new benchmark is new and an important contribution.", "weaknesses": "1. The paper does not discuss in depth the root causes of the generalization problem. I think it would be a great credit to the paper if the authors could share some thoughts on why.\n2. There are minor problems:\n- The color of the lines in Figure 2(a) is wrong.\n- The results on Leaper are missing in Figure 11 and Figure 12.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LlaOE1CrLv", "reviewer": "Reviewer_HieC", "summary": "This paper studies the generalization abilities of offline RL algorithms across different environments. In particular, it introduced a collection of offline RL datasets of different sizes and skill-levels from the Procgen and WebShop environments. Experiments show that existing offline RL methods perform significantly worse than online RL on both train and test environments. Additional experiments show that an increase in data diversity improves generalization while an increase of the size of training data does not.", "questions": "I appreciate the authors' efforts to investigate an important question in offline RL. However, upon reviewing the current draft, I find the perspective presented to be somewhat one-sided. It is essential to acknowledge that the overall conclusions drawn in the paper might be confined to the specific benchmark dataset being utilized. There have been notable instances demonstrating the impressive generalization capabilities of offline RL agents [1] [2]. Consequently, it would be prudent to avoid overly definitive statements in this paper, given these successful examples.\n\nMoreover, I would like to highlight a gap in the current work—there is a lack of more in-depth analysis to elucidate the key question: \"why certain offline RL agents [1][2] exhibit superior generalization skills while others do not?\"\n\nA more profound theoretical analysis might be instrumental in explaining these discrepancies.\n\n[1] (Agarwal et al., 2020) An Optimistic Perspective on Offline Reinforcement Learning\n\n[2] (Kumar et al., 2023) Offline Q-Learning on Diverse Multi-task Data Both Scales and Generalizes", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This paper investigates an important problem in offline RL.\n\n- It introduced a collection of offline RL datasets of different sizes and skill-levels from the Procgen and WebShop environments.\n\n- The experiments are thorough.\n\n- The writing is clear and easy to follow.", "weaknesses": "- The novelty of the study is somewhat restricted. Given that many existing offline RL algorithms do not inherently prioritize generalization ability in their design, so the current experimental results are mostly within expected outcomes.\n\n- There are many duplicate references: \"Leveraging procedural generation to benchmark reinforcement learning\", \"Offline q- learning on diverse multi-task data both scales and generalizes\", \"Deep residual learning for image recognition.\", \"The nethack learning environment.\"\n\n- This paper primarily serves as a summary of empirical observations, and no specific solutions are proposed to address the generalization issue. IMHO, there is a lack of deeper understanding of the generalization issue of offline RL agents and the take aways for readers from this work is limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DL9PeUyjFB", "reviewer": "Reviewer_4tcZ", "summary": "This paper study and benchmark the generalization abilities of existing offline RL algorithms, revealing several interesting and helpful conclusions regarding to current offline RL learning researches.", "questions": "1. Will the author open-source their benchmark and evaluation codes?\n2. Can the author explain more about the setup of Webshop? What is the state/action/reward/objective/transition of this problem? An example would be better (can lie in the appendix).\n3. Can you explain how you fine-tune these algorithms and how the results are selected? It is thorny and important to select the best model in practice.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem studied in this paper is important and interesting. Existing offline RL benchmarks indeed require more practical metrics for evaluation.\n2. The experiments conducted in this paper seem solid and abundant. The conclusions made sound convincing.\n3. Hyperparameters are provided and the results seem reproducible.", "weaknesses": "1. The algorithms included are a bit limited. Methods like model-based learning [1-2], curriculum imitation [3], and other methods are not involved. More useful conclusions can be made when the benchmarking algorithms are expanded.\n\n2. Benchmark included is a bit limited. Most of the results are concluded from a simulated benchmark ProcGen, only a small part of experiments are conducted on the real-world dataset \"WebShop\". The author may consider expanding their tested benchmark to more real-world problems as introduced in [4]. \n\n[1] Yu T, Thomas G, Yu L, et al. Mopo: Model-based offline policy optimization[J]. Advances in Neural Information Processing Systems, 2020, 33: 14129-14142.\n[2] Yu T, Kumar A, Rafailov R, et al. Combo: Conservative offline model-based policy optimization[J]. Advances in neural information processing systems, 2021, 34: 28954-28967.\n[3] Liu M, Zhao H, Yang Z, et al. Curriculum offline imitating learning[J]. Advances in Neural Information Processing Systems, 2021, 34: 6266-6277.\n[4] Qin R J, Zhang X, Gao S, et al. NeoRL: A near real-world benchmark for offline reinforcement learning[J]. Advances in Neural Information Processing Systems, 2022, 35: 24753-24765.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uCsMQK7Mqt", "reviewer": "Reviewer_qttr", "summary": "This paper studies the problem of generalization for popular offline RL algorithms. By carrying out extensive experiments of state-of-the-art offline RL algorithms (IQL, CQL, DT, etc) on procgen and webshop environments, this paper presents the findings that although offline RL methods outperform BC in training environments with suboptimal data, they fail to generalize as well to testing environments similar to training environments.", "questions": "Why IQL cannot be used in the setting of webshop? The expectile regression part should not depend on the number of actions for each state and as long as the expectile regression can be implemented it should be fine?\n\nIn section 4.5, is it possible to extend the plot a bit to show around how much data the learning curve stops to grow? It seems that we also need more than 3 data points to draw a valid conclusion regards to the trend.\n\nIn Figure 3, why is the blue line sometimes higher than the red line? Isn't the data collected by expert PPO?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper carries out extensive experiments on procgen and another more realistic environment webshop, covering most state-of-the-art offline rl methods such as CQL, IQL and BCQ. \n\nThe problem of generalization of offline reinforcement learning is important yet relatively scarcely touched.\n\nThe paper is well-written and easy-to-follow. The format of using red text boxes makes it easy to capture important takeaways.", "weaknesses": "I have concerns that some of the offline RL baselines might not be tuned appropriately. For example, in the procgen environments, there seems to be a large gap between BC and CQL both in the train and test environment. However, CQL contains a weighted combination of TD-learning loss and behavior cloning loss so a proper tuning of the weights should make CQL at least comparable to BC.\n\nAlthough procgen is a relatively popular benchmark, it is not the primary benchmark for those offline RL methods. They are most extensively tested on continuous control tasks such as D4RL (https://arxiv.org/abs/2004.07219), so authors should explain why they did not conduct experiments in the setting of continuous control. For example, the different test environments can be attained by modifying the environment parameters such as gravity.\n\nThe conclusion of the paper is derived mostly through empirical observations. It would be important to also have theoretical understandings of why offline RL does not work as well as BC in terms of generalization, similar to the analysis done in https://arxiv.org/pdf/2204.05618.pdf.\n\nI am willing to raise my score if those concerns can be solved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)