Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary
  Resolution
Authors: name: Zuyan Liu; name: Yuhao Dong; name: Ziwei Liu; name: Winston Hu; name: Jiwen Lu; name: Yongming Rao
Published: 2024-09-19 17:59:51

Abstract: Visual data comes in various forms, ranging from small icons of just a few
pixels to long videos spanning hours. Existing multi-modal LLMs usually
standardize these diverse visual inputs to a fixed resolution for visual
encoders and yield similar numbers of tokens for LLMs. This approach is
non-optimal for multimodal understanding and inefficient for processing inputs
with long and short visual contents. To solve the problem, we propose Oryx, a
unified multimodal architecture for the spatial-temporal understanding of
images, videos, and multi-view 3D scenes. Oryx offers an on-demand solution to
seamlessly and efficiently process visual inputs with arbitrary spatial sizes
and temporal lengths through two core innovations: 1) a pre-trained OryxViT
model that can encode images at any resolution into LLM-friendly visual
representations; 2) a dynamic compressor module that supports 1x to 16x
compression on visual tokens by request. These design features enable Oryx to
accommodate extremely long visual contexts, such as videos, with lower
resolution and high compression while maintaining high recognition precision
for tasks like document understanding with native resolution and no
compression. Beyond the architectural improvements, enhanced data curation and
specialized training on long-context retrieval and spatial-aware data help Oryx
achieve strong capabilities in image, video, and 3D multimodal understanding
simultaneously. Our work is open-sourced at https://github.com/Oryx-mllm/Oryx.