Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper tackles the problem of limited availability of labelled datasets for reinforcement learning tasks. The paper proposes to circumvent this challenge by combining large but sparsely-annotated datasets from a target environment of interest (without densely annotated large scale datasets) with fully annotated datasets from various other source environments. The proposed method employs the generalization capabilities of inverse dynamics modelling (IDM) to label missing action data. The method is evaluated on five games of the atArI benchmark suit.
strength_and_weaknesses: Strengths:
* The proposed method is interesting and novel. 
*  The proposed method seem to improve performance significantly.
* The paper includes adequate baselines and ablations.
* The paper is well written and easy to understand.


Weaknesses:
* Disjoint action spaces: the performance difference between ALPT and DT1-IDM is limited. The paper should clarify whether the performance difference comes mainly from using an IDM? The paper should also include a discussion on how the proposed method is able to leverage datasets with disjoint action spaces for improved performance.
* The paper should provide a more detailed explanation and a discussion on why including more source games does not improve performance on the target environment (in Section 5.4).
* The paper uses only five games as they have a similar shared game structure.  This should be quantified in more detail. How similar does the game structure have to be? In general using only 5 games from the Atari suit is quite limited and the paper should ideally include more games.
* It would be interesting to include results on the performance difference when training purely on the fully label target environments as a percentage of recovered performance. 
clarity,_quality,_novelty_and_reproducibility: Quality: The paper is interesting and novel.

Clarity: the paper is clear and well written.

Originality: To the best of my knowledge, the proposed idea is novel.
summary_of_the_review:  The claims in the paper are generally well founded with adequate baselines and ablations. The paper demonstrates impressive performance improvements. However, the paper should include more detailed dissuasions on some claims (see above).
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studied the problem of learning agent polices with datasets that have limited labeled actions. Action Limited PreTraining (ALPT) is presented, which pretrain an inverse dynamics model (IDM) on multiple environments to provide accurate action labels for decision transformer (DT) agent on an action limited target environment dataset. Experiments are conducted on game-playing benchmark among a couple of different environments. 
strength_and_weaknesses: Strength:
This work is well motivated. The research problem that's explored is valuable. Pretraining towards generic agents / representations on large-scale datasets has emerged as a dominant approach broadly in learning community. While, one big challenge exists in control and/or RL tasks is the lacking of explicitly labeled actions. Especially, when leveraging real-world video datasets, e.g. Ego4D for control tasks, the gap between semantic level 'activity' vs. physical level 'actions' brings ambiguous and large distribution discrepancies. So, to seek effective pretraining regimes that can leverage such massive datasets without actions labels is important. 

Weakness:
My main concern is that the proposed approach is hard to scale up. However, on contrary the main motivation of this work is for large-scale pretraining. To achieve the goal, ALPT have to first pretrain two transformers, i.e. one bidirectional T for IDM and a DT, and these two Transformer need to be trained one by one (as training of DT will need the action labels generated by the bidirectional T). Then, ALPT needs an extra finetuning stage that have to the same thing, i.e. finetune two transformer one by one. I am wondering, all these process should be time and computation consuming, especially when dealing with massive scale datasets. In experiments, only a couple of environments with small scale datasets are used for evaluation, which is hard to convince me that the proposed approach can indeed address lacking-of-label issue in large scale pretraining. 
clarity,_quality,_novelty_and_reproducibility: 1. The technical contribution and novelty is limited. As what I mentioned before, the motivation of this work is to address issues for large-scale pretraining. However, both the techniques and experiments do not well validate the claim. The key novelty of the proposed approach is to leverage source and part of labeled target environments to train a model that can generate action labels. It closely related with widely used ideas of generating pseudo labels in transfer learning and semi-supervised learning.
2. Lack of implementation details and the evaluations are limited. No source codes are provided.
Some questions:
1) When training IDM, it is saying that an bidirectional transformer is used, and is said that the model architecture as GPT. I am wondering GPT should be a causal transformer not bidirectional transformer, would you please clarify?
2) it it interesting to see the different effects of training with IDM vs. forward dynamics model and even a combination of both forward and inverse dynamics model. I am wondering how and why that induce the choice of only IDM.
3) In sec. 5.5, only a pair of tasks are evaluated. When performing a single task with very simple action space, it very likely that the finetuning stage can already well fit the distribution. I am wondering a multitask setting or zero-shot/few-shot setting. At least without the IDM finetuning. 
summary_of_the_review: This paper studied the problem of learning agent polices with datasets that have limited labeled actions. Action Limited PreTraining (ALPT) is presented, which pretrain an inverse dynamics model (IDM) on multiple environments to provide accurate action labels for decision transformer (DT) agent on an action limited target environment dataset. Experiments are conducted on game-playing benchmark among a couple of different environments. 
This work is well motivated. The research problem that's explored is valuable. Pretraining towards generic agents / representations on large-scale datasets has emerged as a dominant approach broadly in learning community. While, one big challenge exists in control and/or RL tasks is the lacking of explicitly labeled actions. Especially, when leveraging real-world video datasets, e.g. Ego4D for control tasks, the gap between semantic level 'activity' vs. physical level 'actions' brings ambiguous and large distribution discrepancies. So, to seek effective pretraining regimes that can leverage such massive datasets without actions labels is important. 
However, my main concern is that the proposed approach is hard to scale up. However, on contrary the main motivation of this work is for large-scale pretraining. To achieve the goal, ALPT have to first pretrain two transformers, i.e. one bidirectional T for IDM and a DT, and these two Transformer need to be trained one by one (as training of DT will need the action labels generated by the bidirectional T). Then, ALPT needs an extra finetuning stage that have to the same thing, i.e. finetune two transformer one by one. I am wondering, all these process should be time and computation consuming, especially when dealing with massive scale datasets. In experiments, only a couple of environments with small scale datasets are used for evaluation, which is hard to convince me that the proposed approach can indeed address lacking-of-label issue in large scale pretraining.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper explores the problem of learning agent policies from action limited datasets and proposes ALPT, which pretrains an inverse dynamics model (IDM) on multiple environments to provide accurate action labels for a decision transformer (DT) agent on an action limited target environment dataset. The experiments and ablations highlight the importance of pretraining the IDM, as opposed to pretraining the DT agent alone. The results support the importance of generalist inverse dynamics models as an efficient way to implement large-scale RL. In addition, as more labelled data becomes available for training offline RL agents, ALPT provides an efficient way of bootstrapping performance on new tasks with action limited data.
strength_and_weaknesses: ##########################################################################

Pros:

- The paper proposes ALPT to use the multi-environment source datasets as pretraining for an IDM, which is then fine-tuned on the action-limited data of the target environment in order to provide labels for the unlabelled target data, which is then used for training a DT agent, which is an interesting idea for learning agent policies from action limited datasets.
- Through various experiments and ablations, this paper demonstrates that leveraging the generalization capabilities of IDMs is critical to the success of ALPT, as opposed to, for example, pretraining the DT model alone on the multi-environment datasets or training the IDM only on the target environment. In addition, on a benchmark game-playing environment, we show that ALPT yields as much as 5x improvement in performance, with as little as 10k labelled samples required (i.e., 0.01% of the original labels), derived from only 12 minutes of labelled game play (Ye et al., 2021).
- The paper is well-written and the comparison of benchmark methods is also interesting to read.
##########################################################################

Cons:

- Although this paper provides an interesting perspective on large-scale training for RL, it's just a straightforward application of existing approaches, seems the novelty is not enough.
- This paper only compared ALPT with the single-game variant and standard DT baselines, lacking enough experiments to support the proposed hypothesis.
- Missing code link to reproduce experiments.
clarity,_quality,_novelty_and_reproducibility: The quality and clarity are good but the originality is only marginally significant or novel, and missed code links to reproduce experiments.
summary_of_the_review: Considering the above pros and cons, my recommendation of the paper is marginally below the acceptance threshold.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper is motivated by the limited availability of actions in datasets when training RL agents. To better leverage the large amount of data not paired with actions, this paper proposes to learn a model to annotate actions for the dataset. More concretely, this paper proposes Action Limited PreTraining (ALPT), which learns an inverse dynamics model (IDM) to label missing actions for datasets in the target environment. ALPT is evaluated in Atari environments and maze navigation environments. ALPT has two stages: the pre-training stage, which trains a DT model and IDM simultaneously on datasets collected in source environments, and the finetuning stage, which finetunes both models on datasets in the target environment. 
strength_and_weaknesses: Strength:
 - The paper is well-motivated and focuses on realistic problems. The action-limited problem focused in this paper stems from the widely studied learning-from-observation problem. 
 - The paper is well-organized and easy to follow.
 - The motivation is clearly stated. 

Weakness:

 - Unclear methodology description.
     - In section 4.2, the paper mentions that IDM and DT are trained simultaneously. However, it is still unclear what “simultaneously” means. Are IDM and DT trained iteratively? If so, what are the training steps for IDM (or DT) before switching to another model? What are the total training steps?
     - In finetuning, is DT initialized with the model pre-trained on all source environments? If so, it would be better to mention it clearly in section 4.2. Moreover, one ablation is initializing DT randomly and training it from scratch on the target datasets. 
 
- Concerns regarding the Atari experiments
     - In Figure 2, what is the meaning of the x-axis? Does it mean the number of labeled frames used in fine-tuning? 
     - In Figure 2, the evaluation curves of ALPT are quite bumpy across environments. What is the number of episodes used to get each evaluation point on the curve? Will a larger number of evaluation episodes help smooth the evaluation curve?
     -  **Moreover, in Figure 2, ALPT does not seem to improve along finetuning, which indicates that the finetuning is not helpful. In Asterix, the performance even goes down after finetuning What may cause such experiment observations? One ablation to do is to freeze the IDM model and only finetune DT.
     - In Figure 2, the DT-IDM baseline seems to have no shaded area. In Figure 4, the breakout experiments have no shaded area. Are they run with 3 random seeds?
     - Although ALPT improves upon baselines, the absolute rewards in Atari environments are still much lower than the reported values in the multi-game decision transformer’s appendix (https://arxiv.org/pdf/2205.15241.pdf page 24). For instance, DT’s performance in Asterix would have a reward of around 14,706 and ALPT has a reward of 2,500. It is reasonable that DT would perform better since it has the full dataset with actions. However, the large performance gap still raises concerns about the effectiveness of the proposed method.

- Lacking experimental setup descriptions.
     - For figure 3, what are the selected source environments in “ALPT-2 Games” and “ALPT-10 Games”?  
     - The paper mentions that all the hyperparameters of DT are mostly the same as the settings in multi-game DT. It would be better to mention the key hyperparameters in the appendix for references, such as the number of heads and the number of layers. 
     - What is the architecture of the IDM model?  
- Missing key related works in the literature review. 
     - It would be better to posit this paper as an offline RL paper and review literature related to learning from demonstrations, learning from observations, offline RL, and multi-task behavior cloning. 
clarity,_quality,_novelty_and_reproducibility: The motivation of the paper is clear. The quality of the literature review, methodology, and experiments can still be largely improved. 
summary_of_the_review: I would recommend a major revision of the paper and would recommend rejecting the paper. Although the paper is well-motivated, there are key descriptions missing in the methodology part, which makes the paper hard to evaluate. Moreover, the experiments hardly show that finetuning indeed helps.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an approach to train offline reinforcement learning methods with a mix of data belonging to the target environment or task of interest (e.g., a game) and data belonging to different, but related environments/tasks (e.g., different games). In particular, the paper proposes to use a small set of labeled data from the target environment, plus a large set of unlabeled data from the target environment and a large set of labeled data from the source environments. By using an approach similar to Vide PreTraining, an Inverse Dymanics Model is trained on labeled data and used to annotate unlabeled data. All the data is used to train the final policy. This approach results in the proposed ALPT method. Experiments compare ALPT with a series of baselines to show the appropriateness of the different steps of the proposed approach. Other experiments aim to give insights into the role of dynamics modeling, the effect of using source and target environments with disjoint action spaces, and the benefit of ALPT in other domains such as maze navigation.
strength_and_weaknesses: STRENGHTS:
 - While the proposed technique is conceptually simple and the technical novelty with respect to previous works such as VPT (Baker et al., 2022), the proposed investigation seems to be useful to me. Indeed, it is reasonable to assume that labeled examples from similar tasks will become are available or will become available with time. In this case, it can be useful to be able to leverage such annotations to improve performance on a related task. The experimental analysis is also carried out trying to answer interesting questions concerning the role of each step of the proposed training pipeline, the effect of the number of source datasets used for dynamics modeling, the performance of the ALPT in the presence of disjoint action spaces, and the benefits of the proposed technique in other domains.

WEAKNESSES:
 - While I found the investigation overall interesting, one may object that the choice of tasks make the analysis and conclusions less general than what observed in other works. Indeed, Baker et al (2022) considered the Minecraft game as its wide variety of things to do, build, and collect, can make results more applicable to varied real-world applications, while Lee et al. (2022) considered a set of 46 Atari games to show the abilities of their model to generalize across diverse scenarios. In contrast, the proposed investigation considers a subset of 5 Atari games (asterix, breakout, freeway, seaquest, and spaceinvaders) which, while maintaining a certain degree of variety, are arguably easier to tackle than Minecraft or the full set of 46 Atari games. In my opinion, this choice makes the paper and results somewhat weaker as compared to the two aforementioned works.
 - The paper reports reasonable ablations and comparisons. However, the main assumption is that a large quantity of labeled data is available for the source environments, while only a small set of labeled data in the target environment is available. Not surprisingly, the single-game variants, which are trained on the 10k subset of the data perform worse than methods trained on the set of 100M source + 10K target action labels. In order to assess the role of having more data as compared to being able to leverage data from diverse scenarios, one could consider the case in which single-game instances are trained on the target environments considering large sets of 100M labeled examples. This scenario is reasonable if we assume that we can label 100M examples on the target environment with the same effort required to label 100M examples on different source environments. More importantly, this setting would allow to assess the mere role of just using more data in the training procedure.
 - Another point which makes the experimental analysis a bit weak is the absence of clear comparisons with state-of-the-art approaches. For instance, one would expect comparisons with the VPT approach. If I understood correctly, this may be very similar to the "single-game" variants, but, if this is the case, this should be explicitly pointed out in the paper. Also, it could be interesting to see what happens if decision transformers are replaced by the methods considered for comparisons in Lee et al. (2022), such as C51, CQL, etc. These comparisons would help the reader to better understand the benefit of the proposed approach over current techniques.
clarity,_quality,_novelty_and_reproducibility: I found the paper not always very clear and easy to follow. For instance: 
 - Figure 1 reports that the trained IDM is "non-causal". The choice of a non-causal transformer IDM is not well discussed in the paper.
 - What's "Dist(S)" reported in section 3.1?
 - The second paragraph of Section 4.2 (During pretraining, ...") is a bit hard to follow because of different confusing statements. E.g., "we use the combined labelled dataset, i.e., D_d for all source environments M_d...", however, D_d is not the combined labelled dataset, but one of the datasets, right?. In contrast, the notation reported in Table 1 with the union of different D_d sets is more clear. I would suggest revising the whole section to improve clarity.
 -  The descriptions of the different baselines described in section 5.2 are hard to follow. I would suggest adding a summary table explaining the different properties of the baselines.
 - Figure 2 reports "evaluation game performance". However, this evaluation measure is not mentioned or introduced in the main text.
 - The "Breakout" graph in Figure 4 does not have shaded lines (standard deviation).

The technical novelty of the paper is not very high. Indeed, as the authors state, ALPT is a straightforward application of existing approaches. Nevertheless, I found the conceptual novelty of the paper good.
summary_of_the_review: In summary, while I found the investigation and analysis very interesting, in my opinion, the quality of the paper is limited by the aspects outlined above. I believe a careful revision of those aspects would make the submission much more solid.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper was reviewed by five experts in the field and received a mixed score. The main concerns are the limited novelty, unconvincing experiments, and lack of clarity. The authors did a good job of rebuttal and addressed many of the concerns. However, the reviewers and ACs still feel that more work is needed to get it to the best version. AC also agrees that this work can be much stronger with additional experiments and implementation details. While this paper clearly has merit, the decision is not to recommend acceptance. The authors are encouraged to consider the reviewers' comments when revising the paper for submission elsewhere.


justification_for_why_not_higher_score: 1. Limited novelty
2. The experiments are weak
3. The framework is not very general for real-world applications. 
justification_for_why_not_lower_score: NA
Decision: Reject