Merlin:Empowering Multimodal LLMs with Foresight Minds
Authors: name: En Yu; name: Liang Zhao; name: Yana Wei; name: Jinrong Yang; name: Dongming Wu; name: Lingyu Kong; name: Haoran Wei; name: Tiancai Wang; name: Zheng Ge; name: Xiangyu Zhang; name: Wenbing Tao
Published: 2023-11-30 17:57:34

Abstract: Humans possess the remarkable ability to foresee the future to a certain
extent based on present observations, a skill we term as foresight minds.
However, this capability remains largely under explored within existing
Multimodal Large Language Models (MLLMs), hindering their capacity to learn the
fundamental principles of how things operate and the intentions behind the
observed subjects. To address this issue, we introduce the integration of
future modeling into the existing learning frameworks of MLLMs. By utilizing
the subject trajectory, a highly structured representation of a consecutive
frame sequence, as a learning objective, we aim to bridge the gap between the
past and the future. We propose two innovative methods to empower MLLMs with
foresight minds, Foresight Pre-Training (FPT) and Foresight Instruction-Tuning
(FIT), which are inspired by the modern learning paradigm of LLMs.
Specifically, FPT jointly training various tasks centered on trajectories,
enabling MLLMs to learn how to attend and predict entire trajectories from a
given initial observation. Then, FIT requires MLLMs to first predict
trajectories of related objects and then reason about potential future events
based on them. Aided by FPT and FIT, we build a novel and unified MLLM named
Merlin that supports multi-images input and analysis about potential actions of
multiple objects for the future reasoning. Experimental results show Merlin
powerful foresight minds with impressive performance on both future reasoning
and visual comprehension tasks.