Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces Video Pre-Training (VPT), which is a new algorithm to learn foundation models for complex, long-horizon embodied agents in the popular Minecraft game. First, an inverse dynamics model is trained with human contractor data in Minecraft to predict keyboard and mouse actions from videos. Second, the inverse dynamics model is used to process large amounts of YouTube videos to obtain noisy action labels. Third, an agent imitates from the human players in YouTube by behavior cloning on the predicted actions. Finally, the policy can be further finetuned with RL on challenging tasks. 
strengths_and_weaknesses: # Strengths

* Novelty: this paper proposes to use an inverse dynamics model learned from human contractor data to automatically label much larger amounts of YouTube data. The idea is simple, powerful, and novel. 
* Performance: the VPT foundation models exhibit impressive zero-shot behaviors. The VPT model fine-tuned through RL is able to craft diamond tools, which is an extremely long-horizon task that takes up to 24,000 actions. Minecraft is a very challenging domain, so the performance is impressive and solid. 
* It is surprising that keyboard and mouse action space can be learned effectively and works quite well. This is a promising direction for solving other challenging video games. 
* This paper is well-written and easy to follow. 

# Weaknesses

While the final performance of VPT is impressive, I hope to see more ablations studies: 

1. On the effect of pretraining. From figure 9 we know that IDM can be quite data efficient. So what if the IDM is only trained on the house building contractor data, and then train BC on the labeled early game data as well as the house building data? This may tell us whether we need the "pretraining stage" in the paper.  

2. On the effect of data filtering. The paper states that "With enough data, a large enough model, and enough training compute, a BC model trained on both unclean and clean videos would likely still perform well" - is there any concrete evidence on how much data filtering is worth in terms of model capacity, data size, and training budget? 

3. On the sequence length of IDM. There is a 3D Conv to aggregate nearby frames (similar to frame-stacking). Why do we still need a long sequence (128) to train the IDM? Will the performance drop significantly with smaller sequence lengths? 

4. On the model scalability of IDM. Figure 19 provide some scalability analysis of IDM on the "full" dataset. However, according to Figure 9, we do not need such large amount of data. So is such a large IDM still necessary? Some scalability analysis on smaller contractor dataset may be illustrative.
questions: The questions are listed in the "Weakness" section above to ask for more ablations. 
limitations: The authors have addressed the limitations adequately.
review_title: null

final_score: rating: 9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces Video Pre-training (VPT), a foundation model pre-trained on large-scale unlabelled videos in Minecraft along with a small amount of human demonstrations. The small amount of data is used to train an inverse dynamics model that can identify the action (M&K inputs directly which is quite neat) taken between frames, and can be used to relabel the larger unlabelled subset. VPT shows some promising zero-shot behaviour, but its performance is significantly improved by fine-tuning with Reinforcement Learning. 

The experiments in this paper are of an unprecedented scale in RL, performed on a popular/relevant domain of Minecraft where the range of possible capabilities are numerous, and has large implications for the RL field in general. More generally, this paper points towards a unifying scale and fine-tune paradigm for RL. A large scale foundation model learns a "common sense" layer for the agents, and the agents can be later on made goal-directed by fine-tuning on a relevant reward function. The bitter lesson strikes again as we see that to get to agents with such general capabilities, all we needed was a clean large-scale dataset and simple/existing methods like IDM and PPO sufficed.
strengths_and_weaknesses: Strengths: 
* The experiments in this paper are of an unprecedented scale in RL, performed on a popular/relevant domain of Minecraft where the range of possible capabilities are numerous, and has large implications for the way RL field in general. More generally, this paper points towards a  unifying scale and fine-tune paradigm for RL. 
* It uses the native mouse and keyboard interface for its agents, thus the results in this paper should translate well to other domains as well. 
* The trained agents are able to demonstrate very sophisticated capabilities such as crafting diamonds which seemed quite out of reach for AI agents until now. 

Weaknesses: 
* Because of the scale of experiments, the design choices are not carefully ablated. Is IDM the way to learn from contractor videos, we discard a ton of info. Similarly, is predicting only actions in VPT optimal (how about tokenized frame predictions)? PPO seems to be a legacy choice as well, and we don't get to see how it performs other off-policy or model-based methods. 
* Reward functions are very carefully human designed to get the right behaviour. This seems antithetical to the scaling paradigm, at internet scale, it should be possible to infer reward functions directly.  
questions: * The decreasing zero-shot performance with increasing model scale seems puzzling. Why do you think that happens to be the case? Are the model under-trained? 
* How were the optimal model/data/compute scales determined? Were these ad-hoc choices or did you refer to Kaplan or Chinchilla scaling laws?


limitations: The limitations should point out that a VPT like recipe would only work on RL domains where large-scale unlabelled data is accessible. It should also point to the reliance of carefully designed reward functions.
review_title: null

final_score: rating: 9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors use ~2,000 hours of hand-labeled data to train an Inverse Dynamics Model (IDM) to predict the mouse and keyboard actions from raw video of Minecraft games. This model benefits from using both the future and past video frames when computing per-frame predictions (i.e. “acausality”).

The authors then use this "IDM" to predict the mouse and keyboard action labels for 70,000 hours of raw unlabelled video of Minecraft games sampled from the internet. The authors call the labels predicted by the IDM on this data “pseudo-labels”.

Given the 70,000 hours of pseudo-labelled gameplay, the authors train a model to predict future actions from past actions using Behavioural Cloning (BC). This model is referred to throughout the work as a Video PreTraining, or VPT model.

The authors show that their VPT model already achieves “non-trivial zero shot performance”, and has thus successfully learnt a prior over the behaviour distribution. To achieve their strongest results, the authors then incorporated this VPT into two further downstream experiments: fine tuning using more specialised datasets covering a specific desired behaviour, and applying reinforcement learning.

The authors found that factoring the problem into two parts: an inverse dynamics model to predict pseudo labels, and a model for behavioural cloning, were key to achieving good performance. The VPT model trained by the authors using this technique is able to provide a very strong baseline behavioural prior, and further fine tuning allows the authors to obtain very compelling results in Minecraft gameplay that appear far beyond the current capabilities of RL to learn from scratch. Perhaps the most impressive demonstration of results is that the agent was able to create "diamond tools", which require tens of thousands of complex actions be taken by the agent.

strengths_and_weaknesses: Strengths:
* The work is very well-executed and demonstrates very impressive results on a challenging learning task, namely Minecraft with the full action space. The excellent performance through training a large behavioural prior is the paper's main novelty, and this largely results from the author's careful execution, and careful application of learning methods. This also clearly required resolving numerous technical challenges, which are mostly enumerated in depth in the supplemental material.
* The outlined method could be generalised to other learning tasks, unlocking the potential of large amounts of unlabelled video data that is increasingly available. This would not be without challenges (inverse dynamics is hard), but the approach seems promising and worthy of further study.
* The paper is clearly written, making it easy to follow and understand the work. The level of detail is appropriate across the sections, and the supplemental material appears to contain sufficient details required to reproduce the work.

Weaknesses:
* The described method may currently be limited in its generality - prediction of inverse dynamics is a difficult problem in its own right, and is somewhat simplified by the world studied by the authors - though this is a compelling direction for further research.
* It would be nice to see a little more depth in the results and conclusions section, and more discussion of future work.
* A little more detail about the architectures and models in the main body of the paper could be welcome.

questions: * Deep in the appendix it is mentioned that the BC model and the IDM are identical. This information would be useful in the main body of the paper. i.e. - “The behavioural cloning model architecture is the same as the IDM architecture described in Appendix 971 D.1 except that we modify the architecture so that it is causal”. Minor nit: in the appendix, it is written “residual transformer layers”, where I believe these are  referred to as attention layers in the main body. (attention layers seemed clearer.

* Minor nit: I didn’t really understand figure 3 (right) - what is being compared exactly here?

* Minor nit: IDM Architecture [appendix] - there’s a long wordy description of the model, but maybe a picture or table could be nice?

* Minor nit: Figure 13 (and others) in the appendix appear almost “washed out” :) 
limitations: The authors could certainly elaborate further on the generalisability of the methods presented - how important is it that e.g. screen resolution and settings match up so closely? To what extent could we hope to learn an inverse kinematics model on a more general or noisy distribution of data? 

For example, it seems unlikely to me that this could currently work on real-world video (which would be an exciting prospect for e.g. autonomous-driving). Generally, the "results and conclusions" paragraph could benefit from more thought and further deeper discussion.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the authors use a small amount of labeled data and a large amount of unlabelled data for solving Minecraft tasks. The authors aim to build a general-purpose framework for sequential decision-making tasks. The model has zero-shot generalization ability.
strengths_and_weaknesses: (+) The results are impressive. Minecraft tasks are challenging. This paper shows a promising way to solve such tasks.

(-) The idea is not novel. The action is predicted based on a set of historical observations in Eqn 1. Even though the method part has three stages as shown in Fig2, the technical contribution is limited in the method part.

(-) This paper did not cite works that are quite strongly related to this paper, such as [1], [2], and [3].
[1] Pre-Trained Language Models for Interactive Decision-Making
[2] A Generalist Agent
[3] Can Wikipedia Help Offline Reinforcement Learning?


questions: Using large models and massive data always seems helpful for better results and zero-shot generalization. This has become an obvious "fact" based on recent papers, such as DALLE [4], CLIP[3], LID[1], and GATO[2]. So what are the main contribution or novelty left after excluding the large models and massive data part? 

This is a good paper, trying to solve a challenging problem. Minecraft tasks are challenging, and there are no methods that can solve such challenging tasks. However, if the good performance mostly comes from the usage of large data and large models, this paper is not convincing enough to be accepted. If the authors could provide more evidence of what makes this paper different from other decision-making works in terms of the method part (except for the large data part), I would change my mind.

[3] Learning Transferable Visual Models From Natural Language Supervision
[4] Hierarchical Text-Conditional Image Generation with CLIP Latents

limitations: (-) The results highly depend on the policy trained on the small amount of labeled data. If this policy is not good enough, it cannot provide reliable predictions for the unlabelled data. 

(-) The unlabelled video data cannot be used directly. The authors still need carefully design the filtering procedure to ensure the data are clean enough for training a better policy.

(-) This is a good paper with impressive results, but it seems the novelty is limited and not a good match for NeurIPS.
review_title: null

Meta-review: metareview: The authors have introduced Video Pre-Training (VPT), a semi-supervised learning approach that allows relatively small volumes of labeled data to train an inverse-dynamics model that is subsequently applied to predict the action labels associated with a far larger, unlabeled dataset. They then train an agent in a supervised regime with respect to these labels to achieve strong performance in Minecraft, which requires reasoning over very long time horizons.

Overall there is clear consensus among the reviewers that this paper is novel, technically sound and of broad interest to the NeurIPS community. The authors have also proactively engaged with reviewer feedback to improve manuscript clarity. I am confident in recommending this paper for acceptance.
confidence: Certain
Decision: Accept