Multi-Environment Pretraining Enables Transfer to Action Limited Datasets
Authors: David Venuto
Sherry Yang
Pieter Abbeel
Doina Precup
Igor Mordatch
Ofir Nachum
Venue: Submitted to ICLR 2023 (2023)

Abstract: Using massive datasets to train large-scale models has emerged as a dominant approach  for  broad  generalization  in  natural  language  and  vision  applications. In reinforcement learning, however, a key challenge is that available data of sequential decision making is often not annotated with actions - for example, videos of game-play are much more available than sequences of frames paired with the logged game controls. We propose to circumvent this challenge by combining large but sparsely-annotated datasets from a \emph{target} environment of interest with fully-annotated datasets from various other \emph{source} environments. Our method, Action Limited PreTraining (ALPT), leverages the generalization capabilities of inverse dynamics modelling (IDM) to label missing action data in the target environment. We show that utilizing even one additional environment dataset of labelled data during IDM pretraining gives rise to substantial improvements in generating action labels for unannotated sequences. We evaluate our method on benchmark game-playing environments and show that we can significantly improve game performance and generalization capability compared to other approaches, even when using annotated datasets equivalent to only $12$ minutes of gameplay. 

Under review as a conference paper at ICLR 2023
MULTI-ENVIRONMENT PRETRAINING TRANSFER TO ACTION LIMITED DATASETS
ENABLES
Anonymous authors Paper under double-blind review
ABSTRACT
Using massive datasets train large-scale models has emerged dominant ap- proach for broad generalization in natural language and vision applications. In re- inforcement learning, however, key challenge that available data of sequential decision making is often not annotated with actions for example, videos of game- play are much more available than sequences of frames paired with the logged game controls. We propose to circumvent this challenge by combining large but sparsely-annotated annotated datasets from targer environment of interest with fully- annotated datasets from various other source environments. Our method, Action Limited Pre'rrining (ALPT), leverages the generalization capabilities of inverse dynamics modelling (IDM) to label missing action data in the target environment. We show that utilizing even one additional environment dataset of labelled data during IDM pretraining gives rise to substantial improvements in generating tion labels for unannotated sequences. We evaluate our method on benchmark game-playing environments and show that can significantly improve game per- formance and generalization capability compared to other approaches, even when using annotated datasets equivalent only 12 minutes of gameplay.
INTRODUCTION
The training of large-scale models on large and diverse data has become a standard approach in natural language and computer vision applications (Devlin al., 2019 Brown al.) 22200, Mahajan et al. 2018) Zhai et al. 2021). Recently, number of works have shown that a similar approach can applied to tasks more often tackled by reinforcement learning (RL), such as robotics and game-playing. For example, Reed et al. (2022) suggest combining large datasets of expert behavior from a variety of RL domains in order to train single generalist agent, while Lee et al. (2022) demonstrate similar result but using non-expert (offline RL) data from suite of Atari game- playing environments and using a decision transformer (DT) sequence modeling objective (Chen al. 2021b
Applying large-scale training necessarily relies on the ability to gather sufficiently large and diverse datasets. For RL domains, this be challenge, the most easily available data for example, videos of a human playing video game or a human completing predefined task often does not contain labelled actions, i.c., game controls robot joint controls. We call such datasets action limited, because little or none of the dataset is annotated with action information. Transferring the success of approaches like DT to such tasks is therefore bottlenecked by the ability to acquire action labels, which can be expensive and time- time-consuming (Zolna et al., 2020).
Some recent works have explored approaches mitigate the issue of action limited datasets. For example, Video Pre Pre'Training (VPT) (Baker et al. (022) proposes gathering a small amount (2k hours of video) of labeled data manually which used train an inverse dynamics model (IDM) Seuyen--uuog et al.) 2008); the IDM is then used to provide action labels much larger video- only dataset (70k hours). This method is shown to achieve human level performance in Minecraft.
While VPT shows promising results, it still requires over 2k hours of manually- data; thus, similar amount of expensive labelling is potentially necessary to extend VPT to other environments. In this paper, we propose orthogonal but related approach to VPT: leveraging large set of labeled data from various source domains to learn an agent policy on limited action dataset of a target evaluation environment. To tackle this setting, we propose Action Limited Pretraining (ALPT),

---

Under review as a conference paper at ICLR 2023
Label Videos
Target Environment
with IDM
All Actinn Labes
10k Lubooa
Train Sequence Model
Train Non- Caun-Cl IDM
Source Environments
Causal Transtarmer
100M Action
Fgy 4g
Sy,
Se, 4
St, R
Figure I: The dynamics model pretraining procedure of ALPT using the source set of environments along with the limited action target environment dataset.
which relies the hypothesis that shared structures between environments be exploited by non-causal (i.e., bidirectional) transformer IDMs. This allows us to look at both past and future frames infer actions. In many experimental settings, the dynamics faceted human behavior in the same setting. It has been suggested that IDMs are therefore more data efficient and this has been empirically shown (Baker et al., 2022). ALPT thus uses the multi- environment source datasets as pretraining for an IDM, which is then linetuned on the action-limited data of the target environment in order to provide labels for the unlabelled target data, which is then used for training DT agent.
far simpler than multi-
Through various experiments and ablations, we demonstrate that leveraging the generalization ca- pabilities of IDMs is critical to the success of ALPT. as opposed to. for example, pretraining the DT model alone on the culli-environment datasets or training the IDM only on the target environment. On a benchmark game-playing environment, we show that ALPT yields as much as 5x improvement in performance, with as little as 10k labelled samples required (i.c., 0.01% % of the original labels), derived from only 12 minutes of labelled game play (Ye et al., 2021). We show that these benefits even hold when the source and target environments use distinct action spaces; i.e., the environments share similar states but common actions, further demonstrating the power of IDM pretraining.
While ALPT is, algorithmically, a straightforward application existing offline RL approaches, our results provide new perspective on large-scale training for RI. Namely, results suggest that the most efficient path to large-scale RL. methods may be via generalist inverse dynamics modelling paired with specialized agent finetuning, instead of generalist agent training alone.
RELATED WORK
In this section, we brietly review relevant works in multi-task RL, meta-learning for RL, semi- supervised learning, and transfer learning.
Multi-Task RL. It is commonly assumed that similar tasks share similar structure and properties (Caruana 1997| Ruder 2017 Zhang et al. 2014; Radford et al. 2019a). Many multi-task RL works leverage this assumption by learning shared low-dimensional representation across all tasks (Ca- landriello et al. 2014 Borsa et al.. 2016; D'Eramo et al.) 2020). These methods have also been extended to tasks where the action space does not align completely (Bräm et al.) (020)). Other meth- ods assume a universal dynamics model when the reward structure is shared but dynamics are not (Zhang al. 202 a). Multi-task RL has generally relied on task identifier (ID) to provide contex- tual information, but recent methods have explored using additional side information available in the task meta-data to establish richer context (Sodhani al., 202T). ALPT can be seen as multi-tass RL, given that we train both the sequence model and IDM using multiple different environments, but we not explicitly model context information or have access to task IDs.
Meta RL. Meta-learning is set of approaches for learning to learn which leverages set of meta- training tasks (Schmichuber) 1987 Bengio et al. 199T), from which an agent can learn either parts of the learning algorithms (eg how to tune the learning rate) or the entire algorithm (Lacombe al. 2021: Kalousis, 2002). In this setting, meta-learning can be used to learn policies (Duan et al. 2017 Finn et al. 2017) or dynamics models (Clavera et al., 2019). A distribution of tasks is assumed to be available for sampling, in order to provide additional contextual information to the policy. One such method models contextual information probabilistic context variables which condition
2

---

Under review as a conference paper at ICLR 2023
the policy (Rakelly et al. 2019). This method has been shown learn from only handful of trajectories. Meta-training can be used to leam policies offline, while using online interaction to correct for distribution shift, without requiring any rewards in the online data (Pong et al.) (022). These methods are commonly used to train on a source set of tasks, like ALPT, but usually require task labels. Meta-training tasks need to be hand-selected, and the results are highly dependent on the quality of that process,
Semi-supervised learning. Semi-supervised learning uses both labelled and unlabelled data to im- prove supervised learning performance (Zhu et al. 2009). It is especially useful when a limited amount of labelles data is given and additional labels are difficult to acquire, unlabelled data is plentiful. Early methods of this type infer unknown labels using a classifier trained on the labeled data Zhuu Ghahramani] 22022. Other methods rely on additional structural side information to regularize supervised objectives (Szummer & Jaakkola, 2001), such as the time scale of Markov random walk over a representation of the data. Many methods, especially those using deep learning, combine supervised and unsupervised learning objectives (Rasmus al. 2015). More recent meth- ods use generative models and approximate Bayesian inference to fill in missing labels (Kingma) et al., 2014). The problem of superviee learning is especially relevant in RL, where large datasets of experience containing action descriptions or rewards may hard to acquire, eg. through manual annotation of videos running robotic experiments. By using an inverse dynamics model, ALPT applies semi-supervised learning to label actions in a large dataset of experience frames, given only limited labeled action data.
Transfer Learning and Zero-shot RL. Policies learned by RL in one domain can have limited capability generalize to new settings (Oh et al. 2016). The most difficult problem is zero-shot RL, where the agent must generalize at evaluation time to new environment that was not seen in training. without acquiring any new data. Transfer learning (Taylor & Stone 2009) tackles a subset of generalization problems where the agent can access interactions from a related environment or task during training. This prior experience other environments is leveraged to improve learning in novel environments. Transfer learning has been applied across both environments (Mordatch et al.) 2016; Tzeng et al.) and tasks (Rusu et al. 2016 Parisotto et al. 2016). It has also been examined in hard exploration games, using imitation learning from human -enerrttd video data (Aytar et al.) 2018). ALPT can be seen as tackling the transfer learning problem, with limited action data from the target environment and providing pseudo-labels for actions. Notably, we consider the under- explored scenario in which the action space is not completely shared between the training and test environments.
Offiin RI. Offline RI. describes a setting that during learning, the agent has access to only fixed dataset of experience. When function approximation becomes necessary in environments with large, complex state spaces, online RL algorithms are extremely sensitive to the dataset distribution. It has been shown that even when the logged data is collected with single behavior policy, standard online algorithms fail (Fujimoto et al., 2019). This has been hypothesized to be due erroneous generalization of the state action function. One such class of methods applies random ensembles of Q-value function targets (Agarwal et al. 2020a) to alleviate this source of error. Other works suggest regularizing the agent policy to the behavior policy (Zhang et al. 202 1)). In offline RL., errors arise when needed to bootstrap the value function and algorithms must apply strong regularizations on both learned policy and value function to achieve stable performance (Wu et al., [2020] Kumar et al.) 2019; Zhang et al., (2021b] Nachum al., 2019). Decision transformer has been suggested as an offline RL strategy that has been successful in learning generalizable agents from a wide variety of logged data in multiple unrelated environments (Reed et al. 2022
3 BACKGROUND
In this section, we review the standard offline RL setting and the use of decision transformers (DT) as a sequence modelling objective for offline RL. We then define the setting of multi-environment offline RL. with action-limited data, which is our focus.
3

---

Under review as a conference paper at ICLR 2023
3.1 OFFLINE REINFORCEMENT LEARNING
We consider an agent acting within Markov decision process (MDP) defined by (S, P, PR), where is the set of states, A is the set of actions, P : S x A - Dist(S) is the transition probability kernel and R: S
[0, 1] is the scalar reward functtin.
In offline RL, the agent is given a dataset of episodes, i.e. sequences of states, actions, and rewards collected by unknown policies interacting with the environment:
St, Ot Pty
The objective is typically to use this dataset in order to learn a conditional action distribution, Pa [s acer, T<r), that maximizes the expectation of the total return, G, Exso Te+k when used to interact with the environment from which the training episodes were generated.
3.2 OFFLINE RL AS SEQUENCE MODELING
Decision transformer (DT) Chen et al. 202 is an approach to offline RL which formulates this problem sequence modeling, and then transformer-basee architectures solve it. For this purpose, the episodes in the offline dataset are augmented with the returns associated with each step:
T
Qt, Tt: G.....).
This sequence is tokenized and passed to causal transformer Po, which predicts both returns and actions using a cross-entropy loss. Thus, the learning objective for is:
( = E, Σ log acn, - log (eaalssenaa
(3)
During inference, at each timestep f, after observing St. DT uses the predicted return dis- tribution Po (s5,, Te)) to choose an optimistic estimate G, of return, before using Pola|sgr,ast G.) to select an action å, (see Lee al. (2022) for details).
3.3 MULTI-ENVIRONMENT AND ACTION LIMITED DATASETS
Our goal is to use pretraining on set of environments where labelled data is plentiful, in order to do well on target environment where only limited action-labelled data is available. Therefore, the offline RL setting we consider includes multiple environments and action-limited datasets, as detail below.
We consider set of n source environments, defined by set of MDPs: E. (M.....,, M, }
and a single target environment M.. For each source environment Ma. we have an offline
dataset of episodes generated from Ma. denoted by D, {t := (.... St, labelled with actions. The agent is trained using the offline dataset of episodes: (Ul Da).
3. fully
For the target environment, the agent has access to small labelled dataset from M.. de- noted as Dt = {T:=(( (..., St, ae, r)), and large dataset without action labels, D. = (T (t,,,,,,
ACTION LIMITED PRETRAINING (ALPT)
We now describe our proposed approach to offline RL in multi-environment and action limited settings. ALPT relies upon an inverse dynamics model (IDM) which uses the combined labelled data in order to learn a representation that generalizes well to the limited action data from the target environment. The predicted labels of the IDM the unlabelled portion of the target environment dataset are then used for training a sequence model parameterized as a decision transformer (DT). We elaborate this procedure below and summarize the full algorithm in Table
4

---

Under review as a conference paper at ICLR 2023
Table I: A summary of ALPT.
Step
Procedure
Pretraining
Train IDM on all labelled data: (Un-1 DD) Train DT on all data: (Ua=1 Da) UD: UD_ with IDM providing action labels on D,
Finetuning
Train IDM on labelled data in target environment dataset: D Train DT all data in target environment dataset: D UD. with IDM providing action labels on D
Evaluation
Use trained DT agent to interact with target environment M..
4.1 INVERSE DYNAMICS MODELING
Our inverse dynamics model (IDM) a bidirectional transformer trained to predict actions from an action-unlabelled sub-trajectory of episode. The training objective for learning IDM P3 is
[ E, ΣΣ log Pa(ac+i St , Sitk)
(4)
where length of training sub-trajectories. our experiments, we parameterize P3 using the GPT-2 transformer architecture (Radford al., 2019b) and use k 5. We obtained through experimental tuning.
4.2
ENVRN-MNVI PRETRAINING AND FINETUNING
ALPT is composed of two-stage pretraining and finetuning process.
During pretraining, we use the combined labelled datasets for all source enviromments combined with the labelled portion of the target environment dataset: (U"=1 Dạ) U UDD. to train the IDM P3- Concurrently, we also train the DT Pa on the combined labelled and unlabelled datasets for all source environments combined with the target environment datasets, by using the IDM provide action labels on the unlabelled portion D The DT training dataset is therefore: (Ui-1 Da) U DJ U U..
During finetuning, we simultaneously train both the IDM and DT exclusively on the target environ- ment dataset. We train the IDM on the labelled portion D We train DT on the full action limited dataset DJ U D by using the IDM to provide action labels on the unlabelled portion D.-
Finally, during evaluation we use the trained DT agent to select actions in the target environment M., following the same protocol described in Section 3.2
EXPERIMENTS
We evaluate ALPT on multi-game Atari setup similar to Lee al. (022). Our findings are three- fold: (1) ALPT, when pretrained on multiple source games, demonstrates significant benefits on the target game with limited action labels; (2) ALPT maintains its significant benefits even when pretrained on just single source game with disjoint action space, (3) we demonstrate similar benefits on maze navigation tasks.
5.1 EXPERIMENTAL PROCEDURE
Architecture and Training. Our architecture and training protocol follow the multi-game Atari setting outlined in Lee et al. 2002)). Specifically, we use the 40M-sized model from Lee et al. (2.22) but reduce the model's hidden size from 768 512,
only pretrain at most subset of Atari games. We used transformer layers, which is the same as 40M-sized model of 8 heads each, which is the same as the 10M-sized model. The rest of the architecture and training hyperparameters
5

---

Under review as a conference paper at ICLR 2023
Table 2: A summary of the baseline methods.
Method
Training Games
IDM
DT-1
DT-5
5
x
DT-1--DM
DT-5-RET
1
ALPT
5
remain unchanged for experiments on Atari. For the Maze navigation experiments, we modify the original hyperparameters to batch size of 256 and weight decay of 5 x 10-5. During pre- training, we train the DT and IDM for 1M frames.
Datasets As Lee et al. (2022), we use the standard offline RL Atari datasets from RL Un- plugged (Gulcehre et al. 2020). Each game's dataset consists of 100M environment steps of train- ing a DQN agent (Agarwal et al., 2020b). For the source games, we use this dataset in entirety. For the target game, we derive an action-limited dataset by keeping the action labels for randomly sampled sequences consisting of a total 10k transitions (0.01% % the original dataset, equivalent to 12 minutes of gameplay) and removing action labels in the remainder of the dataset. For the maze navigation experiments, we generate the data ourselves. The offline datasets for each maze configu- ration contain 500 trajectories with length of 500 steps until the goal state is reached. They generated using an optimal policy for each maze, with an e-greedy exploration rate of 0.5 to increase data diversity.
5.2 BASELINE METHIOSS
We detail the methods that compare ALPT to below.
Single-game variants. To evaluate the benefit multi- versus single-environment training, we assess the performance of training either DT alone DT and IDM simultaneously the target game. When training DT alone (DT1), we train it only on the 10k subset of data that is labelled, while when training DT and IDM simultancously (DT1-IDM) we use the IDM provide action labels on the unlabelled portion of the data.
Multi-game DT variants. To assess the need for IDM versus training on DT alone, we evaluate multi-game baseline (DT5) composed of DT alone. For this baseline, we pretrain DT on all labelled datasets combined from both the source and target environments before finetuning the DT model on the 10k labelled portion of the target game.
Return prediction DT variants. As an alternative way for DT to leverage the unlabelled portion of the target environment dataset, we evaluate a baseline (DT5-RET) that uses the unlabelled portion for training return prediction. The model still undergoes a pretraining and finetuning stage, first pretraining on all available data and then finetuning only data from the target game.
We give summary of the baseline methods as well as ALPT Table
5.3 How DOES ALPT PERFORM COMPARED TO THE BASELINES?
We focus our first set of multi multi-game game pretraining experiments on 5 Atari games: {Asterix, Breakout, Spacelnvaders, Freeway, Seaquest). This subset of games is selected due to having a similar shared game structure and access to high-quality and diverse pretraining data. We evaluate each choice of target game in this setting, i.c., for each game evaluate using it as the target game while the remaining 4 games comprise the source environments.
We compare pretraining regime (ALPT) with the single-game variant and standard DT baselines in Figure?? 2 We see that pretraining ALPT on the source games results in substantial downstream performance improvements. We show that there relatively minimal performance improvements when pretraining on datasets that do not include any non-target environments (DT1-IDM). Utilizing ALPT results in improvements up 500% higher than the single-game training regime. The

---

Under review as a conference paper at ICLR 2023
Asterix
Breakout
Freeway
35
3500
ALPT
160
ALPT
ALPT
Evalliaton Pefomaacce
3000
Evelluttoon Performance
140
E 1oldioon Performance
D13 DIS-RET
0T3-DM
DTS RET
2500
OTS-RET
200
DTI IDM
20
2000
1 mm VAmgy
80
1500
1000
500
20
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Prames
1e6
Prames
1e6
Frames
1e6
Seaquest
Spaceinvaders
3500
1200
Laliuationn Peffrmence
3000
Evalustion Per Performence
2500
800
0T5
600
DTS-RET
DTS-RET
DTT-DM
400
DTT-DMM
mmm
2000
300
200
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Frames
166
Frames
1e6
Figure 2: Game performance across the ALE environments for the baseline and Multi Game video pretraining method. This figure shows the evaluation game performance (Episodic Return) of our DT policies during finetuning on the limited action target dataset. We show the performance across the different pretraining regimes. Higher score is better. The shaded area represents the standard deviation over random seeds. The r-axis shows the number labelled game frames used during finetuning. We evaluate ALPT on 16 episodes of length 2500 each following (Lee et al. 2022)
performance difference is especially stark in Breakout and Seaquest. Additionally, we show that performance is not recovered under pretraining only the sequence model (DT5) on the action rich environments, indicating that most generalization benefits are occurring due to the IDM pretraining. We also show that performance using DTI and DT5-RET is poor, highlighting the need for explicit re-labelling to achieve good performance during sequence model finetuning.
5.4 IS DYNAMICS MODELLING IMPROVED UNDER MORE OR LESS SOURCE DATASETS?
As experimental ablation, we explore if having more or less source data train our dynamics model results better performance. In this set of experiments, we expand or reduce the set of source environments during pretraining of both our DT and IDM. Results presented in Figure We find that only limited number of source environments is needed for downstream performance improvements. We do not find that performance is significantly improved by increasing the environ- ment diversity of the training dataset, highlighting the versatility of pretraining the IDM even given single related dataset. These results indicate both the practicality of ALPT and fast adaptation ability during finetuning.
5.5 CAN ALPT HELP WHEN SOURCE AND TARGET HAVE DISJOINT ACTION SPACES?
The previous experiments have included at least one source game during pretraining whose the action space overlaps with that the target environment. The next set of experiments aims to explore pretraining source environments where the action space (A.,) is disjoint with the target environment action space (A.), that is, Arn.A = D. To do so, we use Freeway as our single source enviroument dataset. In Freeway, the action space consists of {Up, Down). In contrast, game such as Breakout has an action space consisting of [Left, Right). Surprisingly. using Freeway as a source environment and Breakout as a target environment still yields significant benefits for ALPT.

---

Under review as a conference paper at ICLR 2023
Breatsut
20000
27900
100
23000
2000
20000
2003
2022
0.0
Phoonie
NammeTriaGame
Enduro
4020
3000
2330
6000
2036
3030
3020
0.0
0.2
0.4
2.5
0.5
3.0
0.0
0.3
0.8
3.5
0.8
3.5
0..0 0.45
12.8 012
012
020
Figure 4: Game performance across the ALE environments for the baseline and Multi Game video pretraining method. This figure shows the evaluation game performance (the reward obtained during the episode) during finetuning on the limited action target dataset. We show the performance across the different pretraining regimes. Higher score is better. The z-axis shows the number of labelled game frames used during finetuning. The shaded area represents the standard deviation over random seeds.
We present this result in Figure 4 as well as a variety of other choices for the target environment, none of which share any actions with the source environment Freeway.
We hypothesize that performance improvements are not unexpected due to the already known broad generalization capabilities of DT architectures. It may also be the case that, despite the action spaces being disjoint, they still exhibit similar structure. For example, a top/down action space left/right action space are structurally opposite each other in terms of movement, differing only orientation. This similar structure is potentially learned and leveraged during finetuning.
Breakout Game Diversity
5.6 Do ALPT'S BENEFITS PERSIST IN OTHER DOMAINS ?
175
ALPT-2 Games
Evaluation Perfomance
ALPT-10
We now demonstrate ALPT's benefits on an action-limin limited navigation task. This corre- sponds a scenario where we have densely an- notated navigation maps for set of source re- gions but only a sparsely annotated navigation map for a target region. We would like to evalu- whether ALPT, pretrained source regions with abundant action labels, can generalize to navigating in a target region (of a different lay- out) with limited action labels.
DT1-IDM
125
100
75
50
25
0.0
0.2
0.6
0.8
1.0
Figure 3: Increasing or decreasing the number of games in the source dataset does not appear to have a measurable effect on target game per- formance. ALPT-10 utilizes ( ('Asterix', 'Break- out', 'Freeway', 'Seaquest", 'Spacelnvaders', 'At- lantis', 'DemonAttack: 'Frostbite', 'Gopher", "TimePilot") and ALPT-2 utilizes ('Asterix', 'Breakout") ")
Maze Navigation Environments. To answer the above question, we consider a gridworld navigation task where an agent seeks to navi- gate to goal location in 20 x 20 2D maze from a random starting location to random goal location using 4 discrete actions: (Up, Down, Left, Right]. The agent receives a re- ward of I at the goal state and 0 otherwise.
To collect the offline training datasets, we follow Yang et al. (2022): Zhang et al. (2018) to algorith- mically generate maze layouts with random internal walls that form blocked tunneled obstacles as shown in Figure 55] We start with blocked obstacles, and generate one source maze from which collect 500 trajectories with full action labels. We then different random seed generate
8

---

Under review as a conference paper at ICLR 2023
20X20 Maze
1.0
Evaluation Reward
ALPT Blocked
0.6
DTI-IDM
0.4
I
0.0
0
1000
2000
3000
4000
5000
Steps
(b)
Figure 5: (a) An example diagram of the Blocked (above) and Tunneled (below) mazes. The green cell is the goal state. (b) Evaluation performance while training on the 20x20 Maze dataset. Higher score is better. The shaded area represents the standard deviation over 3 random seeds. The action limited target dataset contains 250 labelled actions these experiments.
the target maze, from which collect 500 trajectories with only 250 action labels (0.5% of the full action labels). We then train the IDM ALPT-Blocked on both the source and target datasets, labeling the missing actions from the target game, and train DT all at the same time (no separate finetuning stage). ALPT-Blocked and other baselines are evaluated in the target maze only.
Results. Performance in the target maze environment with limited action labels is presented in (b).. ALPTTBlocked trained on both source and target mazes allows us to solve the target task twice as fast compared only training on the target maze without access another source maze. To further illustrate the benefit of multi-enviromment training on more diverse data, we introduce the tunnelled maze, and train LLPT-Blocked+Tunnellle source blocked maze and source tunneled maze, respectively, target blocked maze. Training on both tunnelled and blocked mazes enables greater dataset diversity, which further improves generalization, leading to even faster convergence on the target task. These preliminary results on navigation suggest that multi-environment pretraining can benefit a broad set of tasks.
on 500 trajectories with full actions from
well as 250 action samples of the
CONCLUSION
We explored the problem of learning agent policies from action limited datasets. Inspired by the paradigm of large-scale, multi-enviromment training, we proposed ALPT, which pretrains an inverse dynamics model (IDM) multiple environments to provide accurate action labels for decision transformer (DT) agent on an action limited target environment dataset. Our experiments and abla- tions highlight the importance of pretraining the IDM, opposed pretraining the DT agent alone. Our results support the importance of generalist inverse dynamics models as an efficient way to implement large-scale RL. As more labelled data becomes available for training offline RL agents, ALPT provides an efficient way of bootstrapping performance on new tasks with action limited data.
6.1 LIMITATIONS
The largest limitation of ALPT the assumption that we would have plentiful labelled data from related environments. One interesting fact uncover is that this data does not have to be based on the same action space as the desired target environment, indicating the versatility ALPT and its ability to ingest diverse source environment training data. We also caution that have only evalu- ated ALPT so far on limited, self-contained video game tasks and simple navigation environments. We hope that more labelled data becomes available in RL domains, ALPT will have wider appli- cability, allowing RL agents to scale and bootstrap to new environments. It would also be useful to investigate further how much labelled data from a limited set of source environments is required to be able to handle a much larger set of unlabelled datasets.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on of- fline reinforcement learning. In Proceedings of the 37th International Conference Machine Learning, ICML'20, 20, 2020a.
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference Machine Learning, pp. 104-114, PMLR, 2020b.
Yusuf Aytar, Tobias Pfaff, David Budden, Tom Le Paine, Ziyu Wang, and Nando de Freitas. Playing hard exploration games by watching youtube. NIPS' 18, 2018.
Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos, 2022.
Y. Bengio, S. Bengio. and J. Cloutier. Learning synaptic learning rule. In IJCNN-9I-Sen -eeattee International Joint Conference on Neural Networks, 1991.
Diana Borsa, Thore Graepel, and John Shawe-Taylor. Learning shared representations in multi-task reinforcement learning, 2016.
Timo Bräm, Gino Brunner, Oliver Richter, and Roger Wattenhofer. Attentive multi-task deep rein- forcement learning. In Machine Learning and Knowledge Discovery in Databases, 2020.
Tom Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020.
Daniele Calandriello, Alessandro Lazaric, and Marcello Restelli. Sparse multi-task reinforcement learning. In Advances in Neural Information Processing Systems, 2014.
Rich Caruana. Multitask learning. 1997.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling, 2021a.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Picter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Advances in Neural Information Processing Systems, 2021b.
Ignasi Clavera, Anusha Nagabandi, Simin Liu, Ronald S. Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning adapt in dynamic, real-world environments through meta- reinforcement learning. In International Conference on Learning Representations, 2019.
Carlo D' D'Erao, Davide Tateo, Andrea Bonarini, Marcello Restelli, and Jan Peters. Sharing knowl- edge in multi-task deep reinforcement learning. In International Conference on Learning Repre- sentations, 2020.
Jacob Devlin, Ming- Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep
bidirectional transformers for language understanding. Association for Computational Linguis- tics, 2019.
Yan Duan, John Schulman, Xi Chen, Peter I.. Bartlett, Ilya Sutskever, and Pieter Abbeel. RL^2: Fast reinforcement learning via slow reinforcement learning, 2017.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. Proceedings of the 34th International Conference on Machine Learning, 2017.
10

---

Under review as a conference paper at ICLR 2023
Scott Fujimoto, David Meger, and Doina Precup. Off Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp.
2019.
Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Thomas Paine, Sergio Gómez, Konrad Zoina, Rishabh Agarwal, Josh S Merel, Daniel Mankowitz, Cosmin Paduraru, et al. RI unplugged: A suite of benchmarks for offline reinforcement learning. Advances in Neural Information Process- ing Systems, 33:7248-7259 2020.
Alexandros Kalousis. Algorithm selection via meta-learning. PhD thesis, University of Geneva, 2002.
Diederik P. Kingma, Danilo J. Rezende, Shakir Mohamed, and Max Welling. Semi-supervised learning with deep generative models. In Proceedings of the 27th International Conference Neural Information Processing Systems, 2014.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Sys- tems, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Proceedings of the 34th International Conference on Neural Informa- fion Processing Systems, NIPS 20, 2020.
Thomas Lacombe, Yun Sing Koh, Gillian Dobbie, and Ocean Wu. A meta-learning approach for au- tomated hyperparameter tuning in evolving data streams. In 2021 International Joint Conference on Neural Networks (JJCNN, 2021.
Kuang-Huck Lee, Olir Nachum, Mengjiao Yang. Lisa Lee, Daniel Freeman, Winnie Xu, Sergio Guadarrama, Ian Fischer, Eric Jang, Henryk Michalewski, and Igor Mordatch. Multi-game deci- sion transformers, 2022.
Dhruv Mahajan, Ross B. Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In Computer Vision ECCV 2018 15th European Conference, Proceedings, Part II, 2018.
Igor Mordatch, Nikhil Mishra, Clemens Eppner, and Pieter Abbeel. Combining model-based policy search with online model learning for control of physical humanoids. 2016 IEEE International Conference Robotics and Automation (ICRA), 2016.
Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice: Policy gradient from arbitrary experience, 2019.
D. Nguyen-Tuore J. Peters, M. Seeger, and B. Schölkopf. Learning inverse dynamics: A com- parison. In Advances in Computational Intelligence and Learning: Proceedings of the European Symposium Artificial Neural Networks, 2008
Junhyuk Oh, Valliappa Chockalingam, Satinder Singh, and Honglak Lee. Control of memory, active perception, and action in minecraft. In Proceedings of the 33rd International Conference on International Conference on Machine Learning, 2016.
Emilio Parisotto, Lei Jimmy Ba, and Ruslan Salakhutdinov. Actor-mimic; Deep multitask and transfer reinforcement learning. In ICLR (Poster), 2016.
Vitchyr H Pong, Ashvin V Nair, Laura M Smith, Catherine Huang, and Sergey Levine. Offline meta- reinforcement learning with online self-supervision. In Proceedings of the 39th International Conference on Machine Learning, Proceedings of Machine Learning Research, 2022.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. 2019a.
Alec Radford, Jeff Wu. Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019b.

---

Under review as a conference paper at ICLR 2023
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off off-policy metu-reiinorrement learning via probabilistic context variables. In Proceedings of the 36th Inter- national Conference Machine Learning, Proceedings of Machine Learning Research, 2019.
Antti Rasmus, Harri Valpola, Mikko Honkala, Mathias Berglund, and Tapani Raiko. Semi- supervised learning with ladder networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems, 2015.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg. Tom Recles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent, 2022.
Sebastian Ruder. An overview of multi-task learning in deep neural networks, 2017.
Andrei A. Rusu, Sergio Gomez Colmenarejo, Çaglar Gülçehre, Guillaume Desjardins. James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distil- lation. In ICLR (Poster), 2016.
Jürgen Schmidhuber. Evolutionary principles in self-referential learning. (on learning how to learn: The meta-meta.... hook. Technical report, PhD Thesis, 1987.
Shagun Sodhani, Amy Zhang, and Joelle Pineau. Multi-task reinforcement learning with context- based representations. In Proceedings of the 38th International Conference Machine Learning, Proceedings of Machine Learning Research, 2021.
Martin Szummer and Tommi Jaakkola. Partially labeled classification with markov random walks. In Advances Neural Information Prucessing Systems, 2001.
Matthew E. Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 2009.
Eric Tzeng, Coline Devin, Judy Hoffmman, Chelsea Finn, Pieter Abbeel, Sergey Levine, Kate Saenko, and Trevor Darrell.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning, 2020.
Mengjiao Yang, Dale Schuurmans, Pieter Abbeel, and Ofr Nachum. Chain of thought imitation with procedure cloning. arXiv preprint arXiv:2205. 108/6, 2022.
Weirui Ye, Shaohuai Liu, Thanard Kurutach, Picter Abbeel, and Yang Gao. Mastering atari games with limited data. In Advances in Neural Information Processing Systems, 2021.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers, 2021.
Amy Zhang, Shagun Sodhani, Khimya Khetarpal, and Joelle Pineau. Learning robust state abstrac- tions for sadden-parametee block {mdp}s. In International Conference on Learning Representa- fions, 2021a.
Chi Zhang, Sanmukh Rao Kuppannagari, and Viktor Prasanna. (BRAC)+: +: Going deeper with
behavior regularized offline reinforcement learning, 2021b. URL htttp::///penrevieee
net/ wtw.orrrmmiddddM.eeeee...
Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. A study on overfitting in deep reinforcement learning. arXiv preprint arXiv:/804 /80400693, 2018.
Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang. Facial landmark detection by deep multi-task learning. In Computer Vision ECCV 2014, 2014.
Xiaojin Zhu and Zoubin Ghahramani. Learning from labeled and unlabeled data with label propa- gation. Technical report, 2002.
12

---

Under review as a conference paper at ICLR 2023
Xiaojin Zhu, Andrew B. Goldberg, Ronald Brachman, and Thomas Dietterich. Introduction to Semi- Supervised Learning. 2009.
Konrad Zolna, Alexander Novikov, Ksenia Konyushkova, Caglar Gulcehre, Ziyu Wang, Yusuf Ay- tar, Misha Denil, Nando de Freitas, and Scott Reed. Offline learning from demonstrations and unlabeled experience, 2020.
13


Reviews:
[{"review_id": "xcwxeHytBdR", "reviewer": "Reviewer_NLz9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper tackles the problem of limited availability of labelled datasets for reinforcement learning tasks. The paper proposes to circumvent this challenge by combining large but sparsely-annotated datasets from a target environment of interest (without densely annotated large scale datasets) with fully annotated datasets from various other source environments. The proposed method employs the generalization capabilities of inverse dynamics modelling (IDM) to label missing action data. The method is evaluated on five games of the atArI benchmark suit.", "summary_of_the_review": " The claims in the paper are generally well founded with adequate baselines and ablations. The paper demonstrates impressive performance improvements. However, the paper should include more detailed dissuasions on some claims (see above).", "strength_and_weaknesses": "Strengths:\n* The proposed method is interesting and novel. \n*  The proposed method seem to improve performance significantly.\n* The paper includes adequate baselines and ablations.\n* The paper is well written and easy to understand.\n\n\nWeaknesses:\n* Disjoint action spaces: the performance difference between ALPT and DT1-IDM is limited. The paper should clarify whether the performance difference comes mainly from using an IDM? The paper should also include a discussion on how the proposed method is able to leverage datasets with disjoint action spaces for improved performance.\n* The paper should provide a more detailed explanation and a discussion on why including more source games does not improve performance on the target environment (in Section 5.4).\n* The paper uses only five games as they have a similar shared game structure.  This should be quantified in more detail. How similar does the game structure have to be? In general using only 5 games from the Atari suit is quite limited and the paper should ideally include more games.\n* It would be interesting to include results on the performance difference when training purely on the fully label target environments as a percentage of recovered performance. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "wTxeAQqQu4", "reviewer": "Reviewer_qasR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is motivated by the limited availability of actions in datasets when training RL agents. To better leverage the large amount of data not paired with actions, this paper proposes to learn a model to annotate actions for the dataset. More concretely, this paper proposes Action Limited PreTraining (ALPT), which learns an inverse dynamics model (IDM) to label missing actions for datasets in the target environment. ALPT is evaluated in Atari environments and maze navigation environments. ALPT has two stages: the pre-training stage, which trains a DT model and IDM simultaneously on datasets collected in source environments, and the finetuning stage, which finetunes both models on datasets in the target environment. ", "summary_of_the_review": "I would recommend a major revision of the paper and would recommend rejecting the paper. Although the paper is well-motivated, there are key descriptions missing in the methodology part, which makes the paper hard to evaluate. Moreover, the experiments hardly show that finetuning indeed helps.", "strength_and_weaknesses": "Strength:\n - The paper is well-motivated and focuses on realistic problems. The action-limited problem focused in this paper stems from the widely studied learning-from-observation problem. \n - The paper is well-organized and easy to follow.\n - The motivation is clearly stated. \n\nWeakness:\n\n - Unclear methodology description.\n     - In section 4.2, the paper mentions that IDM and DT are trained simultaneously. However, it is still unclear what “simultaneously” means. Are IDM and DT trained iteratively? If so, what are the training steps for IDM (or DT) before switching to another model? What are the total training steps?\n     - In finetuning, is DT initialized with the model pre-trained on all source environments? If so, it would be better to mention it clearly in section 4.2. Moreover, one ablation is initializing DT randomly and training it from scratch on the target datasets. \n \n- Concerns regarding the Atari experiments\n     - In Figure 2, what is the meaning of the x-axis? Does it mean the number of labeled frames used in fine-tuning? \n     - In Figure 2, the evaluation curves of ALPT are quite bumpy across environments. What is the number of episodes used to get each evaluation point on the curve? Will a larger number of evaluation episodes help smooth the evaluation curve?\n     -  **Moreover, in Figure 2, ALPT does not seem to improve along finetuning, which indicates that the finetuning is not helpful. In Asterix, the performance even goes down after finetuning What may cause such experiment observations? One ablation to do is to freeze the IDM model and only finetune DT.\n     - In Figure 2, the DT-IDM baseline seems to have no shaded area. In Figure 4, the breakout experiments have no shaded area. Are they run with 3 random seeds?\n     - Although ALPT improves upon baselines, the absolute rewards in Atari environments are still much lower than the reported values in the multi-game decision transformer’s appendix (https://arxiv.org/pdf/2205.15241.pdf page 24). For instance, DT’s performance in Asterix would have a reward of around 14,706 and ALPT has a reward of 2,500. It is reasonable that DT would perform better since it has the full dataset with actions. However, the large performance gap still raises concerns about the effectiveness of the proposed method.\n\n- Lacking experimental setup descriptions.\n     - For figure 3, what are the selected source environments in “ALPT-2 Games” and “ALPT-10 Games”?  \n     - The paper mentions that all the hyperparameters of DT are mostly the same as the settings in multi-game DT. It would be better to mention the key hyperparameters in the appendix for references, such as the number of heads and the number of layers. \n     - What is the architecture of the IDM model?  \n- Missing key related works in the literature review. \n     - It would be better to posit this paper as an offline RL paper and review literature related to learning from demonstrations, learning from observations, offline RL, and multi-task behavior cloning. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "oDbtQ04p-G", "reviewer": "Reviewer_hAS5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studied the problem of learning agent polices with datasets that have limited labeled actions. Action Limited PreTraining (ALPT) is presented, which pretrain an inverse dynamics model (IDM) on multiple environments to provide accurate action labels for decision transformer (DT) agent on an action limited target environment dataset. Experiments are conducted on game-playing benchmark among a couple of different environments. ", "summary_of_the_review": "This paper studied the problem of learning agent polices with datasets that have limited labeled actions. Action Limited PreTraining (ALPT) is presented, which pretrain an inverse dynamics model (IDM) on multiple environments to provide accurate action labels for decision transformer (DT) agent on an action limited target environment dataset. Experiments are conducted on game-playing benchmark among a couple of different environments. \nThis work is well motivated. The research problem that's explored is valuable. Pretraining towards generic agents / representations on large-scale datasets has emerged as a dominant approach broadly in learning community. While, one big challenge exists in control and/or RL tasks is the lacking of explicitly labeled actions. Especially, when leveraging real-world video datasets, e.g. Ego4D for control tasks, the gap between semantic level 'activity' vs. physical level 'actions' brings ambiguous and large distribution discrepancies. So, to seek effective pretraining regimes that can leverage such massive datasets without actions labels is important. \nHowever, my main concern is that the proposed approach is hard to scale up. However, on contrary the main motivation of this work is for large-scale pretraining. To achieve the goal, ALPT have to first pretrain two transformers, i.e. one bidirectional T for IDM and a DT, and these two Transformer need to be trained one by one (as training of DT will need the action labels generated by the bidirectional T). Then, ALPT needs an extra finetuning stage that have to the same thing, i.e. finetune two transformer one by one. I am wondering, all these process should be time and computation consuming, especially when dealing with massive scale datasets. In experiments, only a couple of environments with small scale datasets are used for evaluation, which is hard to convince me that the proposed approach can indeed address lacking-of-label issue in large scale pretraining. ", "strength_and_weaknesses": "Strength:\nThis work is well motivated. The research problem that's explored is valuable. Pretraining towards generic agents / representations on large-scale datasets has emerged as a dominant approach broadly in learning community. While, one big challenge exists in control and/or RL tasks is the lacking of explicitly labeled actions. Especially, when leveraging real-world video datasets, e.g. Ego4D for control tasks, the gap between semantic level 'activity' vs. physical level 'actions' brings ambiguous and large distribution discrepancies. So, to seek effective pretraining regimes that can leverage such massive datasets without actions labels is important. \n\nWeakness:\nMy main concern is that the proposed approach is hard to scale up. However, on contrary the main motivation of this work is for large-scale pretraining. To achieve the goal, ALPT have to first pretrain two transformers, i.e. one bidirectional T for IDM and a DT, and these two Transformer need to be trained one by one (as training of DT will need the action labels generated by the bidirectional T). Then, ALPT needs an extra finetuning stage that have to the same thing, i.e. finetune two transformer one by one. I am wondering, all these process should be time and computation consuming, especially when dealing with massive scale datasets. In experiments, only a couple of environments with small scale datasets are used for evaluation, which is hard to convince me that the proposed approach can indeed address lacking-of-label issue in large scale pretraining. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "mGcLSuaCYTN", "reviewer": "Reviewer_PQGj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an approach to train offline reinforcement learning methods with a mix of data belonging to the target environment or task of interest (e.g., a game) and data belonging to different, but related environments/tasks (e.g., different games). In particular, the paper proposes to use a small set of labeled data from the target environment, plus a large set of unlabeled data from the target environment and a large set of labeled data from the source environments. By using an approach similar to Vide PreTraining, an Inverse Dymanics Model is trained on labeled data and used to annotate unlabeled data. All the data is used to train the final policy. This approach results in the proposed ALPT method. Experiments compare ALPT with a series of baselines to show the appropriateness of the different steps of the proposed approach. Other experiments aim to give insights into the role of dynamics modeling, the effect of using source and target environments with disjoint action spaces, and the benefit of ALPT in other domains such as maze navigation.", "summary_of_the_review": "In summary, while I found the investigation and analysis very interesting, in my opinion, the quality of the paper is limited by the aspects outlined above. I believe a careful revision of those aspects would make the submission much more solid.", "strength_and_weaknesses": "STRENGHTS:\n - While the proposed technique is conceptually simple and the technical novelty with respect to previous works such as VPT (Baker et al., 2022), the proposed investigation seems to be useful to me. Indeed, it is reasonable to assume that labeled examples from similar tasks will become are available or will become available with time. In this case, it can be useful to be able to leverage such annotations to improve performance on a related task. The experimental analysis is also carried out trying to answer interesting questions concerning the role of each step of the proposed training pipeline, the effect of the number of source datasets used for dynamics modeling, the performance of the ALPT in the presence of disjoint action spaces, and the benefits of the proposed technique in other domains.\n\nWEAKNESSES:\n - While I found the investigation overall interesting, one may object that the choice of tasks make the analysis and conclusions less general than what observed in other works. Indeed, Baker et al (2022) considered the Minecraft game as its wide variety of things to do, build, and collect, can make results more applicable to varied real-world applications, while Lee et al. (2022) considered a set of 46 Atari games to show the abilities of their model to generalize across diverse scenarios. In contrast, the proposed investigation considers a subset of 5 Atari games (asterix, breakout, freeway, seaquest, and spaceinvaders) which, while maintaining a certain degree of variety, are arguably easier to tackle than Minecraft or the full set of 46 Atari games. In my opinion, this choice makes the paper and results somewhat weaker as compared to the two aforementioned works.\n - The paper reports reasonable ablations and comparisons. However, the main assumption is that a large quantity of labeled data is available for the source environments, while only a small set of labeled data in the target environment is available. Not surprisingly, the single-game variants, which are trained on the 10k subset of the data perform worse than methods trained on the set of 100M source + 10K target action labels. In order to assess the role of having more data as compared to being able to leverage data from diverse scenarios, one could consider the case in which single-game instances are trained on the target environments considering large sets of 100M labeled examples. This scenario is reasonable if we assume that we can label 100M examples on the target environment with the same effort required to label 100M examples on different source environments. More importantly, this setting would allow to assess the mere role of just using more data in the training procedure.\n - Another point which makes the experimental analysis a bit weak is the absence of clear comparisons with state-of-the-art approaches. For instance, one would expect comparisons with the VPT approach. If I understood correctly, this may be very similar to the \"single-game\" variants, but, if this is the case, this should be explicitly pointed out in the paper. Also, it could be interesting to see what happens if decision transformers are replaced by the methods considered for comparisons in Lee et al. (2022), such as C51, CQL, etc. These comparisons would help the reader to better understand the benefit of the proposed approach over current techniques.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Jjows3IvNjq", "reviewer": "Reviewer_aonF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper explores the problem of learning agent policies from action limited datasets and proposes ALPT, which pretrains an inverse dynamics model (IDM) on multiple environments to provide accurate action labels for a decision transformer (DT) agent on an action limited target environment dataset. The experiments and ablations highlight the importance of pretraining the IDM, as opposed to pretraining the DT agent alone. The results support the importance of generalist inverse dynamics models as an efficient way to implement large-scale RL. In addition, as more labelled data becomes available for training offline RL agents, ALPT provides an efficient way of bootstrapping performance on new tasks with action limited data.", "summary_of_the_review": "Considering the above pros and cons, my recommendation of the paper is marginally below the acceptance threshold.", "strength_and_weaknesses": "##########################################################################\n\nPros:\n\n- The paper proposes ALPT to use the multi-environment source datasets as pretraining for an IDM, which is then fine-tuned on the action-limited data of the target environment in order to provide labels for the unlabelled target data, which is then used for training a DT agent, which is an interesting idea for learning agent policies from action limited datasets.\n- Through various experiments and ablations, this paper demonstrates that leveraging the generalization capabilities of IDMs is critical to the success of ALPT, as opposed to, for example, pretraining the DT model alone on the multi-environment datasets or training the IDM only on the target environment. In addition, on a benchmark game-playing environment, we show that ALPT yields as much as 5x improvement in performance, with as little as 10k labelled samples required (i.e., 0.01% of the original labels), derived from only 12 minutes of labelled game play (Ye et al., 2021).\n- The paper is well-written and the comparison of benchmark methods is also interesting to read.\n##########################################################################\n\nCons:\n\n- Although this paper provides an interesting perspective on large-scale training for RL, it's just a straightforward application of existing approaches, seems the novelty is not enough.\n- This paper only compared ALPT with the single-game variant and standard DT baselines, lacking enough experiments to support the proposed hypothesis.\n- Missing code link to reproduce experiments.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject