VideoWorld: Exploring Knowledge Learning from Unlabeled Videos
Authors: name: Zhongwei Ren; name: Yunchao Wei; name: Xun Guo; name: Yao Zhao; name: Bingyi Kang; name: Jiashi Feng; name: Xiaojie Jin
Published: 2025-01-16 18:59:10

Abstract: This work explores whether a deep generative model can learn complex
knowledge solely from visual input, in contrast to the prevalent focus on
text-based models like large language models (LLMs). We develop VideoWorld, an
auto-regressive video generation model trained on unlabeled video data, and
test its knowledge acquisition abilities in video-based Go and robotic control
tasks. Our experiments reveal two key findings: (1) video-only training
provides sufficient information for learning knowledge, including rules,
reasoning and planning capabilities, and (2) the representation of visual
change is crucial for knowledge acquisition. To improve both the efficiency and
efficacy of this process, we introduce the Latent Dynamics Model (LDM) as a key
component of VideoWorld. Remarkably, VideoWorld reaches a 5-dan professional
level in the Video-GoBench with just a 300-million-parameter model, without
relying on search algorithms or reward mechanisms typical in reinforcement
learning. In robotic tasks, VideoWorld effectively learns diverse control
operations and generalizes across environments, approaching the performance of
oracle models in CALVIN and RLBench. This study opens new avenues for knowledge
acquisition from visual data, with all code, data, and models open-sourced for
further research.