Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolution

Authors: Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Jiwen Lu, Yongming Rao
Primary area: foundation or frontier models, including LLMs
Keywords: Multi-modal Large Language Model, Multi-Modal Understanding, Arbitary Resolution

Abstract:
Visual data comes in various forms, ranging from small icons of just a few pixels to long videos spanning hours. Existing multi-modal LLMs usually standardize these diverse visual inputs to fixed-resolution images or patches for visual encoders and yield similar numbers of tokens for LLMs. This approach is non-optimal for multimodal understanding and inefficient for processing inputs with long and short visual contents. To solve the problem, we propose Oryx, a unified multimodal architecture for the spatial-temporal understanding of images, videos, and multi-view 3D scenes. Oryx offers an on-demand solution to seamlessly and efficiently process visual inputs with arbitrary spatial sizes and temporal lengths through two core innovations: 1) a pre-trained OryxViT model that can encode images at any resolution into LLM-friendly visual representations; 2) a dynamic compressor module that supports 1x to 16x compression on visual tokens by request. These designs enable Oryx to accommodate extremely long visual contexts, such as videos, with lower resolution and high compression while maintaining high recognition precision for tasks like document understanding with native resolution and no compression. Beyond the architectural improvements, enhanced data curation and specialized training on long-context retrieval and spatial-aware data help Oryx achieve strong capabilities in image, video, and 3D multimodal understanding simultaneously.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729164606173

thread_type:
official_review

<thread_object>
creator: reviewer_qcfr

<content>
summary:
The primary goal of this work is to address the limitations of current multi-modal LLMs in processing visual inputs of varying lengths. To tackle this issue, the paper proposes two key improvements based on existing architectures: First, it enhances the current visual encoder to accept arbitrary inputs by enlarging and rescaling the positional encoding. Second, it implements input downsampling based on the type of visual input, supplemented by additional attention mechanisms to compensate for information loss. To ensure broader applicability, this work also carefully curates diverse datasets to enhance the model's capabilities. The results demonstrate that the proposed approach shows significant advantages over similarly sized models across various benchmarks.

strengths:
1. The writing is clear, engaging, and easy to understand.
2. The two proposed improvements to the existing architecture are concise, straightforward, and easily implementable.
3. The approach further enhances the performance ceiling of similarly sized models across various benchmarks, contributing positively to the advancement of the community.

weaknesses:
1. The title and introduction seems somewhat overclaiming. Initially, I interpreted "ON-DEMAND" in the title as implying a dynamic adjustment of spatiotemporal compression based on input; however, the method only implements fixed downsampling based on input type, which was disappointing after reviewing the methodology.
2. The proposed improvements appear somewhat rudimentary. Both the adjustments to positional encoding and the downsampling with attention seem like baseline approaches, lacking deep exploration or inspiration for future work. While I appreciate the complexity of the experiments and the contributions to the community, there are notable deficiencies in novelty.
3. There is a lack of comparison regarding processing times for different methods when handling long videos or mixed data. Although the paper emphasizes the method's efficacy for compressing long videos, I did not find any specific comparisons regarding processing times.

questions:
A discussion and comparison of existing methods for compressing visual signals would significantly enhance the paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730604167842

thread_type:
official_review

<thread_object>
creator: reviewer_o36j

<content>
summary:
This paper proposes a series of improvement on the vision encoder design for multi-modal large language models, enabling better understanding of images, videos, and 3D multi-view inputs. The core contribution is a ViT-based vision encoder that supports native resolution inputs, accommodating varying resolutions and aspect ratios. Additionally, the authors introduce a dynamic compression technique with a cross-attention design to reduce the information loss caused by the pooling operations. The results demonstrate the model's effectiveness across video, 3D, and image understanding tasks.

strengths:
1. This paper implements and open-sources a powerful ViT-based encoder that supports native input resolution. This can benefits the community since it offers a more flexible and effective alternative to dynamic partitioning or fixed resolution approaches.
2. The reported results are very promising, especially on tasks requires high-resolution details or long context understanding.

weaknesses:
1. The main concern is the lack of detailed implementation regarding the pretraining of the Oyrx ViT. It would be valuable to understand whether the Oyrx ViT remains effective without pretraining—for instance, by modifying the ViT architecture as Oyrx ViT and directly fine-tuning it on multi-modal conversation data. Additionally, more specifics about the pretraining process, such as the dataset size, batch size, and training objectives, should be provided.
2. The cross-attention downsampling module is similar to prior work [1]. The authors should discuss how their approach differs from the cross-attention architecture proposed by mini-Gemini[1].  It seems that the only apparent difference is that mini-Gemini uses a CLIP encoder for low-resolution features.
[1] Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models https://arxiv.org/abs/2403.18814

questions:
1. What is the performance of Oyrx ViT without pretraining? If pretraining is essential, could the authors provide details on the amount of data used and the specific task mixture involved in the pretraining process?
2. In section 3.1.3, how to utilize the correspondence information provided by TrackingAnything?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730708782084

thread_type:
official_review

<thread_object>
creator: reviewer_hrjh

<content>
summary:
This paper introduces Oryx, a unified multimodal large language model (MLLM) designed for comprehensive spatial-temporal understanding across diverse visual inputs, including images, videos, and multi-view 3D scenes. 

Key innovations include the OryxViT encoder, which processes inputs with variable aspect ratios and resolutions, and a dynamic compressor that supports on-demand compression ratios from 1x to 16x based on task requirements. It allows the model to handle extended temporal lengths while maintaining performance.

The paper also highlights enhanced data curation and specialized training strategies that bolster Oryx’s capabilities in image, video, and 3D multimodal understanding. Comprehensive evaluations across multiple benchmarks demonstrate Oryx’s competitive performance, often surpassing larger models in specific tasks.

strengths:
1. **Innovative Architecture**:   
The paper introduce OryxViT and the Dynamic Compressor to handle various visual inputs with arbitrary resolutions and temporal lengths, which address a critical limitation in current MLLMs.

2. **Support for Native Resolution**:   
The OryxViT encoder preserves the integrity of visual content by processing inputs at their original resolution, which is essential for detailed tasks.

3. **Efficiency and Scalability**:   
By supporting on-demand compression, Oryx can effectively manages computational resources. It’s suitable for processing both high-resolution images and long-duration videos without compromising performance.

4. **Comprehensive Evaluation**:   
The model is extensively tested across a wide range of benchmarks, including NextQA, Perception Test, MMBench-Video, and ScanQA. Demonstrate its robust performance and state-of-the-art results among open-source models.

5. **Advanced Training Strategies**:   
The paper also introduce an effective data curation and training pipelines, including long-form temporal training and spatial-aware knowledge acquisition through coarse correspondences to boost MLLM training.

weaknesses:
1. **Writing Issues**:
    - **Lack of Focus:** The first part of the paper mainly focus on addressing the varying resolutions of visual inputs, while the second part shifts to enhanced data curation for supporting different modalities (e.g. 3D scene understanding) in MLLMs, which create a sense of discontinuity.
    - **Excessive Technical Detail**: While the technical innovations are commendable, the method part contains too much technical details that can disrupt the flow and make it difficult to read.
2. **Ambiguity in Performance Gain:**
    - It also remains unknown whether the observed performance improvements on Long-form temporal/3D understanding mainly come from the model design or from the introduction of the new long-form videos dataset and the Coarse Correspondences dataset. A more detailed analysis is required.
3. **More experiment to justify design choice:**
    - There are too many manual design choices within this paper that haven’t been justified by exp.
        
        1. A more detailed analysis of the computational complexity of the Variable-Length Self-Attention approach compared to traditional methods would be helpful. How does the memory footprint scale with varying sequence lengths, and what optimization strategies are in place to handle potential memory bottlenecks? 
        
        2. How does the significant reduction in token length for long videos (downsampling by a factor of 16) affect the model's ability to capture fine-grained visual details essential for understanding complex scenes or actions? Why downsample (1x, 4x, 16x) is a reasonable choice? How would other downsample strategy affect performance?
        
        3. Does using a shared MLP for projecting features from different downsampling ratios adequately capture the unique characteristics of images, short videos, and long videos, or could specialized expert projection layers yield better performance? 
4. **Failure cases analysis**
    - Including analysis of failure cases would be appreciated. This analysis could provide valuable insights into the limitations of the Oryx model and offer transparency on how it performs under various challenging scenarios.

questions:
Given the current weaknesses of this paper, I am unable to provide a positive score. However, addressing the issues above could prompt me to reconsider the rating.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731139933376

thread_type:
official_review

<thread_object>
creator: reviewer_cene

<content>
summary:
Oryx MLLM paper proposes unified solution to process images, spatio-temporal videos and multi-view 3D input. It introduces a dynamic compressor module that performs token compression and adaptive positional embedding enables native resolution input / video length.

strengths:
- The one-fit all solution that unifies solution to video of any length and image of varying resolution, 3D input makes is a significant contribution in it's usability especially in real world deployment. 
-The code/ training details have been elaborated and released which enables reproducibility and usage.
- Exhaustive benchmarks provided for the long video understanding and the 3D correspondance understanding.

weaknesses:
- The effect of ORyxViT and the compressor module on the inference throughput needs to be provided.
- Need clarity on what is short and long video to decide the downsample layers: Line 223
- It would be great to see apple-apple comparison of SOTA VLM/ Video-LLM outputs for the qualitative examples that were added.

questions:
- Covered above in weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732474027483

thread_type:
official_comment

<thread_object>
creator: area_chair_fcus

<content>
comment:
Dear Reviewers,

This is a friendly reminder that the discussion period will end on Nov 26th (Anywhere on Earth). If you have not already, please take a careful look at the other reviews and author responses, and comment on whether your original rating stands. Thank you.

Best, AC
</content>
</thread_object>

</thread>
<thread>
creation date:
1732553618244

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response by Authors

comment:
We would like to express our gratitude to the reviewers for their positive feedback and insightful suggestions. We have uploaded a revised version of our paper and highlighted the modifications in blue color. The modifications are concluded below:

1. We optimize the flow in the method part and simplify the technical details to make the method more concentrated. We adjust the architecture and the data curation part to make the logic more clear. (Reviewer hrJh)

2. We add an apple-to-apple comparison with state-of-the-art MLLM in Appendix A. (Reviewer ceNe)



3. We add the failure cases and the relevant analysis in Appendix B. (Reviewer hrJh)



4. We test the inference speed, memory efficiency, and the memory-resolution curve to strongly demonstrate the efficiency of OryxViT in Appendix C.2 (Reviewer ceNe, hrJh)



5. We add an ablation about the proposed training data about long-form videos and 3D data in Appendix C.3 (Reviewer hrJh)



6. We evaluate the design of the MLP adapter in Appendix C.4 (Reviewer hrJh)



7. We add a detailed analysis on the downsampling strategy and function in Appendix C.5 (Reviewer hrJh, qcfr)



8. We provide details about training OryxViT in Appendix D.2 (Reviewer o36J) 



We hope the revised version can better address your concern and we sincerely thank your effort in reviewing the paper.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents the Oryx series, a unified multi-modal framework for spatio-temporal understanding that handles diverse visual inputs using OryxViT for native resolution processing, a Dynamic Compressor for efficient data handling, and a joint training strategy.  The initial scores were 5,5,6,6.  Strengths included innovative approach that supports native input resolution, extensive experiments, and promising results.  Weaknesses included incremental improvements, some overclaiming statements, and some missing comparisons and experimental analysis.  The rebuttal and discussion largely addressed these concerns, and two reviewers increased their scores, leading to a final score of 6,6,6,6.  The AC agrees with the unanimous decision of the reviewers to accept the paper.