CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modular Fusion
Authors: Shoubin Yu
Jaehong Yoon
Mohit Bansal
Venue: ICLR 2025 Poster (2025)
Keywords: Video-Language Reasoning
Video Question Answering
Multimodal Fusion
Parameter-Efficient Fine-tuning
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Despite impressive advancements in recent multimodal reasoning approaches, they are still limited in flexibility and efficiency, as these models typically process only a few fixed modality inputs and require updates to numerous parameters. This paper tackles these critical challenges and proposes CREMA, a generalizable, highly efficient, and modular modality-fusion framework that can incorporate many new modalities to enhance video reasoning. We first augment multiple informative modalities (such as optical flow, 3D point cloud, audio, thermal heatmap, and touch map) from given videos without extra human annotation by leveraging sensors or existing pre-trained models. 
Next, we introduce a query transformer with multiple parameter-efficient modules associated with each accessible modality. 
It projects diverse modality features to the LLM token embedding space, allowing the model to integrate different data types for response generation. Furthermore, we propose a novel progressive multimodal fusion design supported by a lightweight fusion module and modality-sequential training strategy. It helps compress information across various assisting modalities, maintaining computational efficiency in the LLM while improving performance. We validate our method on seven video-language reasoning tasks assisted by diverse modalities, including conventional VideoQA and Video-Audio/3D/Touch/Thermal QA, and achieve better/equivalent performance against strong multimodal LLMs, including OneLLM, BLIP-2, and SeViLA while reducing over 90% trainable parameters. We provide extensive analyses of CREMA, including the impact of each modality on reasoning domains, the design of the fusion module, and example visualizations.

Published as a conference paper at ICLR 2025
CREMA: GENERALIZABLE AND EFFICIENT VIDEO- LANGUAGE REASONING VIA MULTIMODAL MODULAR FUSION
Shoubin Yu" Jachong Yoon'
Mohit Bansal
UNC Chapel Hill
(shoubin, jhyoon, mbansal] } (cs. unc edu
ABSTRACT
Despite impressive udvancements in recent multimodal reasoning approaches, they are still limited in flexibility and efficiency, as these models typically process only a few fixed modality inputs and require updates to numerous parameters. This paper tackles these critical challenges and proposes CREMA, A, generalizable, highly efficient, and modular modality-fusee fuuioon framework that can incorporate many new modalities to enhance video reasoning. We first augment multiple informative modalities (such as optical flow, 3D point cloud, audio, thermal heatmap, and touch map) from given videos without extra human annotation by leveraging sensors or existing pre- traitrened models. Next, introduce query transformer with multiple parameter-efficient modules associated with each accessible modality. It projects diverse modality features the LLM token embedding space, allowing the model to integrate different data types for response generation. Furthermore, we propose novel progressive multimodal fusion design supported by lightweight fusion module and
training strategy. It helps compress information across various assisting modalities, maintaining computational efficiency the LLM while improving performance. We validate our method on video-language reasoning tasks assisted by diverse modalities, including conventional VideoQA and
Thermal QA, and achieve better/equivalent perfor- mance against strong multimodal LLMs, including OneLLM,, BLIP-2, and Se ViLA while reducing over 90% trainable parameters. We provide extensive analyses of CREMA, including the impact of each modality on reasoning domains, the design of the fusion module, and example visualizations.
INTRODUCTION
We humans understand the world through various senses, such sight, sound, touch, and heat, allowing us to understand our environment and act accordingly. This concept has inspired the field of multimodal learning that connects various perceptions, including vision-language (Alayrac et al., 2022; Li et al., 2023b; Zang et al., 2023; Radford et al., 2021), audio-video (Han et al., 2020; Tang al., 2022), and 2D-3D joint vision (Li al., 2020: Hou al.. 2021; 2023; Lei et al., 2024). In particular, recent Multimodal Large Language Models (MLLMs) (Yu et al., 2023b; Li et al., 2023b; Liu et al., 2023a; Tang et al., 2023a) have shown promising versatility in handling multiple forms of input data, such as vision, audio, and text. These models are crucial in real-world applications that require a comprehensive understanding of multiple modalities to make decisions in various contexts. For example, autonomous vehicles rely visual road signs, sirens, and LIDAR for navigation and safe driving. Embodied AI takes visual, heat, and touch information to complete household tasks. Similarly, educational AI enhances the learning experience by integrating various information, such as videos, speech, and textbooks.
Despite their recent advancements, deploying generic MLLM that handles multiple diverse modali- ties is still very challenging in terms cost and flexibility. For different types of inputs, MLLMs
"Equal contribution.
Project Page: https:/CCRMAA- Videoll J.M. Vidthol..Miittuuiiii....

---

Published as a conference paper at ICLR 2025
Large Language Model
Epoch 0
Epoch
Epoch
Language Querias
Self-gated Multimodal Quary Fusion
FC
Multimodal Q-Former
AV
D
FF
AN
AH
A
1:30
Model-Specfic
Multimodal Encoders
Visual
Audio
3D
Mullmodia Inputs
Gradient
Stop
Video
Dapth
Optical
Surface Normals
Touch
Thermal
Audio
3D
Magnitudes
Gradient
Map
Flow
Map
Hastmap
Cloud
Figure 1: Overview of the CREMA architecture & training, Left: Multimodal encoders, Q- former, and LLM are kept frozen in the process. For each modality input, we extract tokens using corresponding moddlitt-sspciite and compress the obtained multimodal tokens. In the end, the LLM uses modality-fusiol tokens to generate responses. Right: We present a modality-sequentin training and modality-adaption early exist strategy, further boosting the training efficiency while allowing faster modality adaptation.
adaptation module. Then, we employ the fusion module to blend
have required extremely large computational budgets to update the LLM with individual encoders for modalities. Alternatively, recent efficient MLLMs using separate projection modules (Zhang et al., 2023; Sun et al., 2023a; Li et al., 2023b; Han et al., 2023) provide a more efficient and flexible way for multimodal reasoning. However, as each modality module contains hundreds of millions of pa- rameters for training. this approach is still computationally intensive, and balancing well as fusing various types of inputs becomes even more complex and costly when more modalities are introduced. Such challenges also exist in very recent pioneering works (Liu al., 2023c; Panagopoulou et al., 2023; Lu et al., 2023); these models aim to integrate more diverse sensory data for compositional understanding via partial updates to the models, yet still require notable training resources to adapt to different modalities (7B for Unified-IO 2 (Lu et al., 2023)). Moreover, they focus primarily on fixed modality pairs (like 3D-text and visual-text), limiting their adaptability new data forms and broader applications. This raises the crucial research question: Can enable MLLMs to efficiently leverage multiple modalities for Video-Languase Reasoning at lower costs, similar to how humans do?.
In this paper, we present CREMA: a generalizable and highly efficient MLLM framework designed for video- lideo-language reasoning with many modalities. The proposed framework enables us to incor- porate any new set of modalities, including video, depth map, optical flow, surface normals, audio, thermal heatmap, touch map, 3D point cloud, notably with very few trainable parameters (4~10M for each new modality) as compared to BLIP-2 (Li al., 2023b) (~ (~108M) and SeViLA (Yu et al., 2023a) (~216M) but higher performance due to assistance of diverse modalities. As illustrated in Figure 1 Left, given frozen pre-trained vision-language modules on top of the Q-Former (Iii et al., 2023b) architecture, including linear projectors, low-rank adapters (Hu al., 2022), and learnable queries. Our parameter- efffeccreeeffi modular design ensures that the pre-trained backbone remains unchanged and enables updates with new modalities and more advanced LLMs in the future without complex architecture changes. To enrich the input modalities, we utilize pre-trained models to extract features from raw videos, e.g., depth maps and optical flow.
backhone, our approach introduces modality-adaptive
Furthermore, despite the effectiveness of our versatile video reasoning framework for multimodal data, handling multiple modalities is not always beneficial; some modalities may be redundant or irrelevant to the reasoning tasks, and optimizing with all modalities simultaneously can lead to certain deficiency, such as suboptimal convergence of the modality inputs. Besides, the L.M receives longer input contexts, which include token embeddings from all modality queries, resulting in increased computations to produce responses. To address these remaining concerns, propose new modality fusion technique that effectively integrates various modality tokens through a novel self-gated attention module and performs alternative modality training within minibatch, preventing interference in the learning of modality representation. Furthermore, we present a modality- adaptive early exit strategy to bypass the training of a specific modality if it is considered as converged.
2

---

Published as a conference paper at ICLR 2025
This further increases the efficiency of CREMA while allowing faster adaptation and maintaining comparable performance. (Please see Figure Right). In the end, we allow the model to maintain GFLOPs while achieving enhanced reasoning ability, even when the LLM processes many modalities.
We validate CREMA on various video-language reasoning benchmarks assisted by diverse modali- ties, including VideoQA (NExT-QA (Xiao al., 2021), Perception (Päträucean et al., 2023)), 3D-VideoQA (SQA3D (Ma et al., 2023)), Audio- VideoQA (MUSIC-AVQA (Li et al., 2022), VG- GSound (Chen et al., 2020)), Touch-VideoQA GTouch&Go (Yang et al., 2022)), Thermal-Videode (Thermal-IM (Tang et al., 2023b)). CREMA surpasses other strong multimodal video reasoning baselines, improving fine-tuning performance by +2.4% on MUSIC-AVQA, +2.6% on SQA3D, and +0.6% on NEXT-QA while reducing by 90% % of the trainable parameters. CREMA also outperforms general-purpose baselines in the zero-shot setting. We further provide comprehensive analyses of applying CREMA with different LLM,, varying sets of modalities, different modality fusion strategies, benefits of adding more modality, and qualitative analysis with input/response visu- alizations to highlight the efficiency/dfleciiveess of CREMA in various video-language reasoning across diverse modalities. Our contributions are summarized as follows:
We propose highly efficient and generalizable modality-extensible learning framework, coined CREMA, which learns multiple modality-adaptivy modules to understand given data through augmented senses.
We present novel modality fusion and training design that efficiently weighs modalities, integrating useful modality features into response generation.
CREMA's 's design allows easy embracing of any new modalities by adding additional modality- adaptive modules without any need to modify the existing framework.
We show the efficacy of CREMA on seven video reasoning datasets by achieving better/equivalent performance while reducing over 90% of trainable parameters than strong baseline models.
With CREMA framework, we conduct extensive studies to provide a comprehensive analysis of how multimodal information helps video-centric language reasoning tasks.
2 RELATED WORKS
Learning with Multiple Modalities. Beyond conventional unimodal learning. leveraging additional modalities, such as visual and audio, in learning models is increasingly popular and has demonstrated remarkable success in solving diverse tasks (Zhu et al., 2024; Liu et al., 2023b; Lu et al., 2023; Moon et al., 2023; Wang et al., 2024a). Vision Language Models (Huang et al., 2023; Li et al., 2023a; Gong et al., 2023; Chen et al., 2023b) are the most prevalent branch of multimodal learning that combine vision and language by training on massive data to understand and generate outputs involving visual and text-based information. Audio-Language Models (Chuang et al., 2020; Castellon et al., 2021; Wang et al., 2023) have been proposed for various audio-associated language tasks, e.g., spoken question answering and speech synthesis. Also, 2D-3D Joint Vision (and Language) Models (Li et al., 2020; Hou et al., 2021: 2023; Lei et al., 2024) aim to combine features of both two-dimensional (2D) and three-dimensional (3D) data to interpret and analyze both modalities, allowing for a more comprehensive understanding of visual information. However, these approaches are not scalable to other tasks involving different modality inputs since they focus on handling fine-grained problems with pre-defined modalities.
Multimodal Large Language Model. Very recently, several works propose integrated pipelines using more than two different data sources for general-purpor reasoning (Zellers et al., 2022; Han et al., 2023; Li al., 2023b; Girdhar et al., 2023; Liu al., 2023c). MERLOT-REVERS! (Zellers et al., 2022) introduces a new training objective that learns from audio, subtitles, and video frames. Given modality-specine
features extracted from the encoder/tokenizer for these inputs, the joint transformer learns to predict the masked text and audio. Therefore, incorporating new types of inputs can be challenging since it requires new pre-training steps for all other modalities. X-InstructBLIP (X-BLIP) (Panagopoulou et al., 2023) integrates various modalities into framework using frozen LLMs, employing modelity-specific Q- Q-Formers as adapters to connect different encoders. However, this method needs train the individual Q-Former for each modality to enable modality-aligma instruction tuning, which is still sesource-intensive MultiPLY (Hong et al., 2024) is a multisensory embodied LLM for interaction within 3D environments with fixed modality set. X-VILA (Ye et al.,
3

---

Published as a conference paper at ICLR 2025
2024) is an omni-modality model aimed at cross-modality alignment, understanding, and generation. concentrates on large-scale cross- modalltyy alignment. OneLLM (Han et al., 2023) presents a universal encoder and projection module to align various modalities with language, its flexibility is limited in adapting new modalities, as the pre-trained projection may be impaired with unseen input format. Our CR EMA method adopts efficient and modular approach, using parameter-efficient adapters for each modality, and enhances flexibility in combining any new modalities.
3 CREMA: GENERALIZZBLLE AND EFFICIENT VIDEO-LANGUAGE REASONING VIA MULTIMODAL MODULAR FUSION
We first provide a preliminary of the Q-Former framework for connecting multimodal inputs with the LLM in Section 3.1. Next, we define the problem for compositional VideoQA and introduce our CREMA method for efficient multimodal compositional video reasoning in Section 3.2. Finally, we describe the training and inference process in Section 3.3.
3.1 PRELIMINARIES: Q-FORMER
To connect various types of sensory inputs with the LLM, we adopt the Q-Former architecture originally proposed in BLIP-2 (Li et al., 2023b), transformer (Vaswani al., 2017) module that bridges the modality encoder and the LLM, similar to Perceiver (Jaegle et al., 2021). It receives modality features Z from the encoder along with learnable queries v and produces fixed-length tokens output. This design enables the Q-Former to compress image tokens into fixed-length set of query tokens, facilitating efficient processing of video inputs while preserving critical holistic information. It further projects obtained tokens q into the LLM's embedding space via fully connected layer to make them compatible. In the end, serves as soft visual prompts (Jia et al., 2022) for the LLM. CREMA method adopts several lightweight form-adaptive modules top of the Q-Former to integrate knowledge from different data types (e.g., video frames, audio, 3D point cloud, touch map etc.) efficiently.
3.2 MULTIMODAL COMPOSITIONAL VIDEO REASONING VIA EFFICIENT MODULAR ADAPTATION AND FUSION
Multimodal Encoders. Our proposed method, CREMA, illustrated in Figure 1, aims to generate responses using both language (i.e., questions) and various multimodal inputs (e.g., images, audio, depth data), denoted as MI [M1, M2,... Mn). where n represents the number of accessible modalities. Throughout this paper, audio, 3D point cloud, optical flow, surface normals, and depth map. in total (up to four different modalities once), but we note that our approach is able to process a larger number different data types if needed. We first encode input data for each modality using the corresponding encoder. Here, we adopt several classes of publicly available pre-trained encoders for modalities, which are kept frozen, to improve training efficiency and learning generalization. Universal encoder (Girdhar et al., 2023; Han et al., 2023) can be employed as well, but lacks the flexibility of other new modalities that are not pre-trained before. Next, we add a fully connected layer (dashed box in Figure 1) for each data type when dimension misalignment happens. It maps different modality representations into a unified feature space while avoiding incompatibilities between varying encoder architectures. We then obtain a set of multimodal features and in the next step, the Q-Former will extract informative features in Z into the query embeddings. More details are discussed in Section 4.1 and Section A.2.
CREMA method handles six modalities: video RGB frames,
Multimodal Q-Former. Previous studies (Panagopoulou et al., 2023; Yu et al., 2023b; Zhang et al., 2023; Hong et al., 2023) have shown the capability of the Q-Former architecture to integrate various modalities with LLMs. However, they necessitate the individual Q-former for each modality, leading to significant parameter demands. While Q-Formers are moderately scaled at 188 million parameters, which is less than LLMs, this size becomes substantial with increasing modalities. For example, processing six different modalities would require about 1B parameters, highlighting the cost and complexity of scaling the modality with additional Q-Formers.
Hence, to deploy a lightweight, universal module capable of integrating various sensory repre- sentations, introduce Multimodal Q-Former. This architecture integrates Modality-specific
4

---

Published as a conference paper at ICLR 2025
Multi- Suati--uery Adapter (MMQA) for each modality. As illustrated in the middle of Figure 1, MMQA consists of Low-Rank Adaptation (LoRA) (Hu et al., 2022) modules", learnable queries, and linear projections. The intuitive design of our approach enables efficient and flexible adaptation any specific modalities. Let 2m be the features extracted from the data Mn of the m modality. The multimodal Q-Former propagates Z. with the corresponding MMQA module capture the relevant information, producing query embeddings qm- Given the learnable input queries Um vn we compute a linear projection layer containing the modality-specific LoRA as follows:
V ++1
We + AWW. m.
(I)
AW m = Am m B E E
(2)
where 3 represents the original linear projection parameters of the Q-Former. AW indicates a low-rank adapter for W with rank r < d. Here, d is the feature size of the Q-Former. With the hidden dimension r = 64, updating only a small number of parameters of AW for each modality while freezing the Q-Former backbone is sufficient for the model to capture rich modality representation. In addition, the proposed approach can effortlessly integrate new modalities. Upon the arrival of new modality, our method can simply append appropriate MMQA modules without modifying the existing architecture, ensuring sustained support for previously integrated modalities.
Self-gated Multimodal Query Fusion. Our approach, which concatenates modality-adaptive queries from lightweight MMQA modules, efficiently manages multimodal reasoning tasks. However, the LLM faces increased training/inference time and computational costs due to extra input tokens proportional to the number of modalities. To prevent the query token size from growing linearly with each new modality, we introduce a novel self-gated multimodal query fusion module (Assistance Modality Fusion in Figure 1). We define the token embedding of video queries qv q1 major and others g,v = {q:) to be supportive ones. Next, we merge supportive query embeddings through linear projection layer (:: 0) to match the dimension with qv. Motivated by Ramachandran et al. 201 we then perform attention on the merged query embeddings viv v via self-gated operation and fuse them with qv:
VV ((22 ... qn]] 0)
(3)
q = concat (gv: (sigmoid (q.v) qvv)),
where [;-...: denotes channel-wise concatenation. This design mirrors human perception in video reasoning tasks, where visual cues are primary but are assisted by other modalities for a richer understanding. In the end, CREMA performs cross-modal reasoning; the model aggregates multimodal query embeddings with the language query via simple concatenation and feeds the concatenated embeddings into the Large Language Model (LLM) obtain the final response a, such that a LLM (concat q, I)).* The proposed design of modality fusion in CREMA successfully reduces computational costs when incorporating new modalities, achieving comparable or even enhanced performance compared to CREMA without the modality fusion.
3.3 MODALIJT-SSQQURNTH AND MODULAR TRAINING OF CREMA
Leveraging the proposed highly efficient modality-speciiff multi-query adapter (MMQA) and the self-gated multimodal query fusion approach, CREMA effectively processes wide range of sensory input types without requiring tailored architecture designs. However, different input modalities possess distinct characteristics and varying quantities of information. Directly optimizing the model on diverse modality inputs may lead to suboptimal convergences, suffering from over/under- fitting on dominant/insigiifcant data types. To address the susceptibility of optimization when training on various modality inputs simultaneously, as shown in Figure Right, we propose a simple yet effective remedy: modality-seuuental modular training and adaptive early exit. Inspired by the alternating update of multimodal representation learning (Zhang et al., 2024), we propose a sequential, modality- adaptive optimization process for each iteration. Instead of performing a joint back- proopppppaggtion step for all modalities simultaneously, we decompose it into sequential modality-specin updates.
We implement LoRA modules at the query and value linear projections for each self-attention layer. 3 For the rest of the paper, unless otherwise stated, we omit the layer index for readahility. "To let the I.I. be aware of the difference of each modality, we insert modal-specific prefix tokens before modality queries. We omit the notation from the equation for simplicity.
5

---

Published as a conference paper at ICLR 2025
Table 1: Fine-tuning Results on Audio- Question Answering (MUSIC-AVQA), We report simple notations for each modality and question type: V: Video RGB frames, A: Audio, F: optical Flow, D: Depth, and N: surface Normalization. Cnt.: Counting, Com. Comparative, Loc.: Location, Ext.: Existential, and Tem.: Temporal. We bold the best and underline the second-best numbers.
Audio
Visual
Audio-Visual
Trainable Params.
Method
Modality
Avg.
GFLOPs
Cut. Com, Avg. Cut. Avg. Ext. Loc. Cut. Com. Aem.Avg
AVQA (Li et 2022) A.A LAVISH (Lin et al., 2023) V. A
80.3 60.0
77.3
74.5 77.8
76.1 81.4
68.7 69.9
64.6
67.1
70.9
73.5
18M
85.6 65.9
8 80.2
81.1
80.6
84.6 69.2
78.8
65.6
69.1
73.8
76.9
2IM
86.7 58.5
16.3
87.2
90.5
81.5
72.0
81.3
64.3
70.1
74.2
78.9
10RM
L30K
BLIP-2 cLi al. 2023b)
A.A
86.3 58.4
76.0
87.6
93.0
90.3
80.4
68.3
826
63.8
69.8
73.5
78.4
216M
1.78K
V.
86.0
59.9
76.4
84.9
88.8
81.8
79.7
65.1
73.6
78.1
324M
3.21K
V
88.3 60.6
82.3
84.4
85.2
84.8
8488
71.7 80.8
63.8
70.6
74.6
78.7
4M
L,3K
CREMA (Ours)
V.A
89.0 61.4
83.0
84.7
85.0
84.8
84.4
73.2
84.8
63.2
71.3
75.6
79.4
9M
1.78K
V.A.F
87.1
61.0
81.5
84.2
90.3
87.2
83.4
74.2
82.6
68.7
21.7
76.4
80.5
21M
1.81K
D,N.D.N
$8.3
64.2
83.2
86.2
91.3
88.7
84.9
74.9
85.2
68.6
73.5
77.7
81.7
38M
1.84K
This approach selectively updates the trainable weights corresponding to the target modality input, ensuring more focused and efficient learning for each modality. Specifically, given the minibatch data M = (M, ..... M,) containing modalities, we propagate M through the model but update only the trainable modules corresponding to the target modality M,, (i.e., the MMQA module for m and the fusion module). Note that this sequential training within the minibatch helps the model remain robust to the order of modalities during training. Furthermore, we extend this to a early exit strategy, allowing the model to bypass the training of specific modality if it is considered as converged. We use the average gradient magnitude of the MMQA module weights as metric for the early exit of each modality input. Specifically, at each training epoch j + 1, we determine whether the model exits the training of the target modality if it satisfies the following equation: ز + 1] > T average(g]: ji)², where the temperature parameter T and g = [#1,... Ij+1] denotes the list of the averaged gradient magnitudes obtained at the end of each training epoch. In the end, CREMA effectively learns rich information from various modalities for video- vidoo-langagg reasoning, while mitigating unnecessary over-convergence imbalanced training across different modalities.
EXPERIMENTS
In this section, we first outline the overall experimental setup in Section 4.1 and show the results of the proposed CREMA on various cross-modal Video QA & reasoning tasks in Section 4.2. We further provide more insights about the MLLM backbone, training strategy, design of the fusion module, and the impact of new modalities across tasks in Section 4.3. More experiments on extra datasets (VGGSound (Chen et al., 2020) and ferceptionTest (Päträucean et al., 2023)), training memory comparison, the impact of sequential training, LoRA rank, MMQA initialization, and qualitative examples are included in Appendix (Section B).
4.1 EXPERIMENTAL SETUP
Datasets & Benchmarks. We evaluate CREMA on the following video reasoning and QA tasks: SQA3D (Ma et al., 2023), MUSIC-AVQA (Li et al.. 2022), and NExT-QA (Xiao et al., 2021). We further evaluate CREMA on TouchQA and ThermalQA collected by ourselves based on public video- touch (Touch&Go (Yang et al., 2022)) and lideo-therma data (Thermal-IM (Tang et al., 2023b)). See Appendix (Sections A. 1 and A.3) for more details.
Implementation Details. Pre-trained Visual Experts: We employ frozen pre-trained visual experts to extract modalities features from raw videos. Specifically, we use ZoeDepth (Bhat et al., 2023), Unimatch (Xu et al., 2023), and NLL-AngMF (Bae et al., 2021) to estimate depth, flow, and normals, respectively. Modality Encoder: We use frozen modality-specific encoders to encode each modality to embedding space. We adopt ViT-G (Sun et al., 2023b) for visual (frames, depth, norm. flow, touch, and thermal), BEATs (Chen et al., 2023a) for audio, and follow data extraction in 3D-LLM (Hong et al., 2023) and ConceptFusion (Jatavallabhula et al., 2023) for 3D point cloud. See Appendix (Section A.2) for details. Baselines & Model Implementation: We extend 3D-LLM and
'We omit the target modality index for brevity.

---

Published as a conference paper at ICLR 2025
Table 2: Fine-tuning Results on 3D Situated Question Answering (SQA3D). V: Video RGB frames,
: Bird-Eye View image, P: 3D Point cloud, D: Depth, and N: surface Normalization.
Trainable Params.
Method
Modality
What
How
Can
Which Others
Avg.
GFLOPs
MCAN (Yu et al., 2019)
V
28.8
59.6
44.0
68.3
40.7
40.4
43.4
56M
ClipBERT (Lei el al,, 2021)
V
30.2
60.1
38.7
63.3
42.4
42.7
43.3
135M
ScanQA (Azuma et al., 2022)
P
31.6
63.8
46.2
69.5
43.8
45.3
46.5
38M
45.1
62.7
48.6
63.3
45.8
49.8
51.4
108M
1.30K
3D-LLM (Hong et al., 2023)
V.
47.7
61.0
49.0
63.6
49.0
49.6
52.3
216M
1.67K
V.P.D.N
45.2
62.4
44.8
64.2
45.8
49.4
52.0
434M
5.47K
V
44.9
62.1
48.1
67.7
48.4
49.8
51.8
4M
1.30K
CREMA (Ours)
V.P
46.2
63.6
46.4
63.0
48.7
50.1
52.1
8M
1.69K
V.P.D.N
47.6
51.0
65.1
48.4
56.4
54.6
38M
1.83K
Table 3: Fine-tuning Results Video Question Answering (NExT-QA). Question types abbreviated as: P.&N.: Prev & Next, Pre.: Present, Cnt.: Count, Loc.: Location, and Otr: Other.
Methods
Modality
Causal
Temporal
Descriptive
Avg.
Truinable Params.
GFLOPs
How Why Avg.
P&&N. Pre. Avg.
Cut. Loc. Otr. Avg.
VQA (78) (Ko et al,, 2023)
72.7
69.2
75.8
72.0
SM
MirasoL3B (Piergiovanni al., 2023) V
73.2
3B
SeViLA (Yu 2023a)
21.3
25.3
74.2
216M
69.9
13.9
72,9
65.4
71.9
68.1
64.9
91.8
80.3
81.2
72.6
10SM
1.30K
BLIP-2 (1. al., 2023b)
V.E.D
70.8
74.2
73.3
65.0
71.4
67.6
61.5
93.2
81.6
81.4
72.7
216M
324M
5.03K
V.F.D.N
71.7
74.2
73.5
65.7
72.6
68.5
65.5
92.5
82.1
73.3
432M
6.12K
67.3
73.9
72.1
63.0
70.2
65.9
61.9
93.2
80.3
81.6
71.6
4M
L.30K
CREMA (Ours)
V.F
69.3
74.1
72.8
64.4
70.5
66.9
67.2
92.
80.9
82.2
72.4
SM
2.22K
V.F.D
70.4
74.4
73.3
72.4
V.E.D.N
21.3
75.5
74.4
67.3
22.5
69.4
66.1
92.9
79.7
81.6
73.9
28M
2.47K
BLIP-2 with individual Q-Formers for each new modality our baseline. We fully fine-tune these Q-Formers. Our Multimodal Q-Former is initialized from BI.I--2 pre-trained one. We set 64 LoRA rank and 32 query tokens for all MMQA modules. More details in Appendix (Sections A.4 and A.5).
4.2 MAIN EXPERIMENTAL RESULTS
MUSIC-AVQA: CREMA achieves superior audio-video reasoning ability. Table 1, recent parameter-efficien approaches, AVQA and LAVISH, perform reasonably well on audio and video QA tasks (MUSIC-AVQA), but are less impactful due to their restricted language capability, BLIP- 2 achieves higher accuracy by training modality-specifff Q-Formers with powerful language model, FLAN-T5XL. However, it fails to incorporate multiple modalities, and degrades audio-video reasoning ability when combining V modality with other modalities: A, F, D, and N. Our method constantly improves the average accuracy with more modality, outperforming LAVISH (+4.8 %pp) and BLIP-2 (+2.4~3.66%) %)p), by using only 14.-1 trainable parameters compared BLIP-2.
SQA3D: Our method is significantly efficient yet outperforms publicly available, strong MILM baselines on 3D-associated video reasoning. As shown in Table we evaluate the fine-tuning performance on SQA3D, and CREMA with video frame inputs (V) obtains improved accuracy compared to baselines on single modality inputs, MCAN, ClipBERT, and ScanQA. We also measure the performance of 3D-LLM, a strong multimodal learning method that shares almost all of its structures with BLIP-2, except for the 3D encoder. Although 3D-LLM enhances its performance by integrating multiple modalities, this brings a considerable increase in parameters to update multiple Q-Formers for each modality. Meanwhile, our CREMA method with V, P. D, modalities surpasses all baselines, achieving the best average accuracy by updating only proposed MMQA modules, which uses 991.2% fewer parameters (38M) for training than 3D-LLM (434M).
NEXT-QA: CREMA consistently achieves superior performance against strong vision-language reasoning methods on the NEXT-QA dataset. As shown in Table 3, SeViLA, BLIP-2, and CREMA adopt - (3B) but achieve reasoning capabilities comparable to the LLAMA-7B model. CREMA with obtains a slightly lower fine-tuning performance compared to BLIP-2 since they perform fine-tuning of the entire Q-Former framework. However, the ability of our proposed
7

---

Published as a conference paper at ICLR 2025
Table 4: Fine-tuning Results on TouchQA and ThermalQA. Table 3: Zero-shot Evaluation on Multimodal Compositional QA T: Touch map, H: thermal tasks (SQA3D and MUSIC-AVQA). Heatmap.
Method
Modality
Acc.
Params,
Tr:
Method
Modality
Ace.
Total Params.
Method
Modality
Acc.
Params.
CMC
V.T
44,3
12M
SQA3D
MUSIC-AVQA
TouchOA
BIJP-2
V.T
77.4
78.2
216M
Unified QA P
GPT-3
41.0
41.0
175.08
11.08
X-InstrucBLIIP
43.5
22.7
14.1B
13.2B
CREMA
(Ouns)
V.T.N
V.T
V
79.1
78.0
8M
4M
3D-LLM
34.5
36.9
3.18
7.8B
A
V.A
44.5
34.8
14.48
7.8B
OneLLM
39.4
7.8B
OncLLM
V
48.4
7,88
CMC
V.H
12M
V.P
37.9
7.8B
V,A
7.88
ThermalOA
BLIP-2
V,H
54.4
55.2
108M
216M
CREMA
V
39.6
37.3
4.18
3.1B
CREMA (Ouns)
A
51.0
31.0
4.1B
3.2B
CREMA
V
54,9
(M
(Ours)
V. P
40.0
4.1B
V.A
52.6
4.28
V.H.D
56.7
20M
framework to incorporate a variety of new modalities enhances its compositional understanding: CREMA with V, D, surpasses BLIP-2 and SeViLA with 87~94% less parameters for training.
TouchQA & ThermalQA:
To further demonstrate the generalizability of our framework over unique/rare sensory inputs, we evaluate CREMA on video-touch and video-thermal QA tasks. In Table 4, BLIP-2 shows competitive performance but struggles to integrate data from different modalities for reasoning, resulting degraded performance when using two modalities. In contrast, CREMA achieves superior performance using only 20M trainable parameters. This advantage is particularly significant when compared to the parameter-efficient multimodal reasoning method, CMC, showing considerable margin of performance improvement despite similar trainable parameters.
Zero-shot Evaluation: In addition to the fine-tuning evaluation, CREMA method also achieves superior zero-shot performance compositional video reasoning. We perform zero-shot evaluations on SQA3D and MUSIC-AVQA in Table 5. Note that Unified QA (Khashabi et al., 2020) and GPT-36 with caption generated from 3D point cloud input perform well, attributed their considerable model size and pre-trained data. We also observe that OneLLM (Han et al., 2023), universal multimodal reasoning framework equipped with Llama2-7B (Touvron et al., 2023) for the LLM backbone, degenerates the performance when integrating different modalities: V, P. On the other hand, CREMA demonstrates a distinct advantage in zero-shot compositional reasoning across modalities, improving performance when combining video frames and 3D point clouds. We also compare with X-InstructBLIPP (Panagopoulou et al., 2023), a strong reasoning framework integrating various modalities with -odality-specific Q-Formers adapters to connect different encoders, and OneLLM on audio-video reasoning tasks. As shown, our method outperforms both X-BLIP (13B) and OneLLM (7.8B), by +8.1 % and +10.3 % (V, A).
CREMA with Different MILM Backbone: version of CREMA
Table 6: Comparison between CREMA with Mistra-7B and other popular Video-LLMs with comparable size LLMs. * our reproduced.
We further built with VideoChat2 (Li et al., 2024) using stronger Mistral-7B as an LLM backbone and pro- cessed more frames (16 frames following the VideoChat2 setting). As shown in Table 6, observed that our CREMA achieved impressive fine-tuning performance (79.4%) outperforming other strong video-LLMs by incorporating op- tical flow (F) and depth map (D) on NEXT-QA while using a remarkably smaller number of trainable parameters, demonstrating the gener- alization ability across MLLM and efficiency of our proposed framework. We leave the CREMA with even stronger MLLMs (c.g., BLIP-3 (Xue et al., 2024), Qwen-V VI. (Wang et al., 2024b)) with more diverse modalities for further studies.
Tr.
Model (Modality)
LLM
Acc.
Params.
Ladeo-LLaMA (V)
Vicuna-7B
60.6
216M
Video-LLaVA (V)
Vicuna-7B
66.3
VideoChat2 (V)
Vicuna-7B
68.6
200M
ILaMA-VQA (V)
LLaMA2-7B
72.0
5M
MotionFpic - VoT (V)
Vicuna-7B
76.0
> 100M
LLaVA-NeXT (V)
Qwenl.5-7B
78.2
7B
VideoChat2 (V)
Mistral-7B
78.4
200M
CREMA (V, F)
Mistral-7B
78.9
21M
CREMA (V. D)
Mistral-7B
79.4
45M
We borrow the results of GPT-3 from the official technical report (Ma et al., 2023).
8

---

Published as a conference paper at ICLR 2025
4.3 QUANTITATIVE ANALYSIS
We have extensively validated the versatility of the proposed CREMA approach over broad range of video-related reasoning tasks, assisted by various supportive modalities. Now, we provide in-depth analyses addressing the following two research questions:
RQ 1: Why did the other baseline struggle to enhance video reasoning with more modality?
To enable an LLM to comprehend information from non-linguistic modality inputs, we need to project them into a unified language embedding space. However, this becomes increasingly challenging as the number of different modalities grows, each representing distinct attributes and sensory information. This issue is observed in models like 3D-LLMM and BLIP-22 (See their degraded or on-par performance when injecting new modalities in Tables 1 and 2). which introduce individual attention modules to produce modality-speciff query embeddings used for LLM reasoning. Due to the lack of shared representation space and an intelligent fusion design, these embeddings often suboptimally converge, making them less compatible with other modalities in terms of representation distribution and resulting in insufficient compositional video reasoning capabilities. Furthermore, as discussed in Section 3.3, employing a universal optimization policy for learning different modality inputs can cause the model to over-converge on a few dominant modalities, which is also another critical issue in solving the many-modee video reasoning problem.
RQ 2: How does CREMA address challenges and help video reasoning with more modalities? We note that CREMA exhibits improved compositional reasoning abilities using more diverse modalities, evident in higher average performance. The model benefits from the following crucial innovations. 1) we compel a regularization effect on the model through parameter-efficient updates, often leading to stable and better generalization than fine-tuning large models (Zhao et al., 2021; Ding et al., 2022; Fu et al., 2023). addition, the modular adaptation from the unified backbone using MMQA and the proposed novel modality fusion approach allow the model to produce compatible yet informative modality token embeddings sampled within a stable multimodal representation space. 2) Our proposed modality-sequenlie training and modality-adaptive early exit approach help CREMA to optimize multiple modality inputs effectively. In particular, we demonstrate the efficacy of our proposed addativy- early exit strategy. Let the early stop indicator be les - = + 1|/T average(gl: .]). implying that CREMA A exits to learn the specific modality data once the corresponding les reaches 1. We observe that our adaptive early exit reduces training tine/computations by 25 ~ 60%, as demonstrated in Table 7 and Figure 2. without necessitating additional complex computations or calibration data. Furthermore, CREMA with a poddality-ddaptinn early exit facilitates faster model convergence, which is evident in performance increase of 1.0 1.1 % in both datasets, compared to CREMA trained for a similar number of epochs. Please see Section B.4 for the ablation of the
training.
RQ3: Ablations for the modality fusion module of CREMA. As the architectural design of the fusion module affects the fusion quality of token embeddings, we investigate our CREMA with different fusion strategies: given a concatenated multimodal token embedding Q obtained by a multimodal Q-Former (i.e., concat), Linear reduces the token size of Q through a linear projection, Mixture-of- -fpextsts (MoE) adopts a MoE layer inside Q-Former to extract a few token embeddings, and Cross-Attention adopts extra prompts shown in Table 8, our proposed self-gated fusion module achieves competitive performance with Concat, while other variants decrease the average accuracy despite incorporating additional modalities besides video. Also, it performs efficiently compared to Cross-attention and Linear, requiring more computationally expensive operations combine modality. We expect that the modality fusion via MoE or Cross-atie aateenaonnoonn alters the original model architecture of the Q-former through additional parametric layers. Thus, it struggles unlock the capabilities of those powerful designs under the parameter-efficient fine-tuning setup and the limited scale of downstream data.
input and computes the cross-attention with Q. As
RQ4: The impact of new modalities on easy/hard questions. We further delve into how adding extra modalities beyond video RGB frames (V) can enhance video reasoning problems. Following the previous work (Buch et al., 2022), which splits the dataset into the easy/hard groups based on the performance of the reference model to find subsets requiring less/more modality information, we classify question inputs in compositional video reasoning tasks (SQA3D and MUSIC-AVQA) based the zero-shot performance of CREMA method with only V; i.e., if the model predicts correctly + easy, otherwise > hard, indicating that input examples in hard may need additional knowledge to find appropriate answers. After that, we fine-tune our CREMA and test obtained

---

Published as a conference paper at ICLR 2025
Table 7: Ablation of Modality-adaptive Early Table Average accuracy & GFLOPs (on Exit on MUSIC-AVQA (vs. BLIP-2 w/ V, A, and SQA3D (vs. 3D-LLM w/ V, N).
NEXT-QA) of our method with different modal- ity fusion strategies. We use F, D, N on NExT- QA and V, P. D on SQA3D, Concat indicates that
we concatenate multimodal query tokens.
Avg. Epochs
Avg. Epochs
Method
Acc.
Params.
Acc.
Params,
BLIP-2 (ar)
77.8
324M
8
51.2
432M
Fusion
NEXT-QA
SQA3D
GFLOPS
3D-LLM
78.1
324M
20
52.0
432M
20
Video-Ouly (V)
1.30K
CREMA
79.3
38M
8
52.8
38M
10
Concal
73.5 (+1.0)
(4.0
6.14K
(Ours)
80.5
20
Cross-Atention
(-111)
(-20)
2.33K
Lincar
71.1 (31
(-
188K
+Adaptive
Early Exit
80.5
38M
11
54.3
38M
15
Self-Gated (Ours)
(4.9.)
(2.0(+9.0)
54.6 (-2.8)
(-077
2.47K
1,02K
Table 9: Acuracy of CREMA method easy and hard questions across datasets and modali- ties.
Ireiccttr
0.8
Modality
Easy Acc.
Hard Acc.
0.7
SQA3D
RGB 3D Depth
V V,P
75.0
37.2
0.6
75.8 (40.8)
37.3 (+0.1)
V.D.D.N
72.6 (-2:4)
42.1 (+4.9)
Norm
MUSIC-AVQA
V VA
85.7
71.2
87.2
71.4 (+0:2)
V.A.D.D..
87.9 (+2.2)
75.2 (+4.0)
Training epochs
NExT-QA
Figure 2:
Mudality-adaptive Early Exit of CREMA on SQA3D. CREMA stops to update MMQA modules for specific modality once the corresponding indicator value reaches 1.0.
V V.D
90.6
41.5
90.2 (-0.4)
43.1 (4-1-0)
V.E.D V.E.D.N
90.5 0-0.1)
44.7 (+3.2)
89.1
50.0 (45.5)
subsets. As shown in Table 9, adding new modalities brings improvement over both easy and hard subsets. However, performance gain on the easy subset is less effective it is already dominant to the video frame inputs. whereas information from additional modalities benefits the prediction of the hard (+4.9%p SQA3D and +4.0%p on MUSIC-AVQA). In NEXT-QA, adding new modalities marginally decreases the easy subset, but significantly boosts the hard (+8.5%p) It indicates that leveraging extra modalities can be an effective modality augmentation strategy, mitigating overfitting. Furthermore, CREMA method can be an efficient tool to determine modality importance for video reasoning benchmark designs.
CONCLUSION
This paper introduces CREMA, an efficient and powerful framework for multimodal compositional video reasoning. We introduce parameter-efficient modality-adaptive modules atop multimodal Q-former to seamlessly incorporate any new modalities like video, optical flow, audio, 3D point cloud, etc. Since our CREMA method does not require modifying the backbone architectures, can easily upgrade our framework with new and stronger language models in the future without damaging its ability existing modalities. We demonstrate the efficacy of our method multimodal QA benchmarks, surpassing baselines' performance with a notable reduction in trainable parameters. We present multimodal fusion module and training strategy to keep low computational costs in LLM while achieving better performance, when we integrate more modalities.
various
ETHICS STATEMENT
The intended use of CREMA is to conduct ladoulan reasoning with the help of diverse assistance modalities. This does not have any particular potential for misuse beyond the general potential for AI technology to be used in harmful ways. Because it based on MLLM to answer video questions, CREMA has the potential to hallucinate. Note that potential is shared widely in
10

---

Published as a conference paper at ICLR 2025
the MLLMs/Vide--LLMs (Li et al., 2023b; 2024). The fact that CREMA answers questions with diverse modality augmentation, enhances the system performance while delimiting model overfitting thus mitigating model hallucination. This is crucial to build reliable, trustworthy video-language reasoning systems that assist educational AI.
ACKNOWLEDGEMENT
We thank the reviewers, and Jaemin Cho, Abhay Zala, Han Lin, Yi-Lin Sung, and Ziyang Wang for their valuable feedback and input for the paper. This work was supported by the National Institutes of Health (NIH) under other transactions 10T20D038045-01, ARO Award W91 10220, DARPA KAIROS Grant 1A8750-19-221005 ONR Grant N00014-23-1-2356 DARPA ECOLE Program DARPA MCS Grant
No. ARR0012390000 DRL21 11263. The views, opinions, and/or findings contained in this article are those of the authors and not of the funding agency.
and NSF-AI Engage Institute
REFERENCES
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In Proceedings of the IEEE International Conference Computer Vision and Pattern Recognition (CVPR), 2022.
Gwangbin Bac, Ignas Budvytis, and Roberto Cipolla. Estimating and exploiting the alcatoric uncer- tainty in surface normal estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13137-13146,
Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. Zoedepth: Zero-shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302.12288 2023.
Shyamal Buch, Cristóbal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, and Juan Carlos Niebles. Revisiting the" video" in video-language understanding. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Rodrigo Castellon, Chris Donahue, and Percy Liang. Codified audio language modeling learns useful representations for music information retrieval. In ISMIR, 2021.
Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio- visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 721-725. IEEE, 2020.
Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, Wanxiang Che, Xiangzhan Yu, and Furu Wei. BEATs: Audio pre-training with acoustic tokenizers. In Proceedings of the International Conference on Machine Learning (ICML), 2023a.
Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, and Jing Liu. Vast: A ailoon-audio-uuiiill---ee omni-modality foundation model and dataset. In Advances Neural Information Processing Systems (NeurIPS). 2023b.
Yung-Sung Chuang, Chi-Liang Liu, Hung- Yi Lee, and Lin-shan Lee. Speechbert: An audio-and-text jointly learned language model for end-to-end spoken question answering. In Interspeech, 2020.
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Mathhis Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models. arXiv preprint 2:Xx:2200300600 2022.
11

---

Published as a conference paper at ICLR 2025
Zihao Fu, Haoran Yang, Anthony Man-Cho So, Wai Lam, Lidong Bing, and Nigel Collier. On the effectiveness of parameter-effiiiet finetuning, In Proceedings of the AAAI National Conference on Artificial Intelligence (AAAI), 2023.
Rohit Girdhar. Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space bind them all. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. Multimodallgpt: A vision and language model for dialogue with humans. 2305.04790, 2023.
Yuan Gong, Andrew Rouditchenko, Alexander H Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James Glass. Contrastive audio-visual masked autoencoder. arXiv preprint arXiv:2210.07839, 2022.
Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. Onellm: One framework to align all modalities with language. arXiv preprint arXiv:2312.03700 2023.
Tengda Han, Weidi Xie, and Andrew Zisserman. Self-supervised co-training for video representation learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Yining Hong, Zishuo Zheng, Peihao Chen, Yian Wang, Junyan Li, and Chuang Gan. Multiply: A multisensory object-centric embodied large language model in 3d world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26406-26416, 2024.
Hou, Saining Xie, Benjamin Graham, Angela Dai, and Matthias Nießner. Pri3d: Can 3d priors help 2d representation learning? In Proceedings of the International Conference Computer Vision (ICCV), 2021.
Ji Hou, Xiaoliang Dai, Zijian He, Angela Dai, and Matthias Nießner. Mask3d: Pre-training 2d vision transformers by learning masked 3d priors. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In Proceedings of the International Conference Learning Representations (ICLR), 2022.
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). 2023.
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Per- ceiver: General perception with iterative attention. In Proceedings of the International Conference on Machine Learning (ICML), 2021.
Krishna Murthy Jatavallabhula, Soroush Saryazdi, Ganesh Iyer, and Liam Paull, gradslam: Automag- ically differentiable slam. arXiv preprint arXiv:1910.1060 2019.
Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Shuang Li, Ganesh Iyer, Soroush Saryazdi, Nikhil Keetha, Ayush Tewari, et al. Conceptfusion: Open-set multimodal 3d mapping. In Robotics: Science and Systems (RSS), 2023.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie. Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
12

---

Published as a conference paper at ICLR 2025
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannanch Hajishirzi. Uniftedqa: Crossing format boundaries with single qa system. arXiv preprint arXiv:2005.00700, 2020.
Chris Dongjoo Kim, Byeongchang Kim, Hyummin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Pupers), 2019.
Dohwan Ko, Ji Soo Lee, Wooyoung Kang, Byungseok Roh, and Hyunwoo J Kim. Large language models are temporal and causal reasoners for video question answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017.
Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for video-and-language learning via sparse sampling. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Weixian Lei, Yixiao Ge, Jianfeng Zhang, Dylan Sun, Kun Yi, Ying Shan, and Mike Zheng Shou. Vit- lens: Towards omni-modal representations. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03726, 2023a.
Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu. Learning to answer questions in dynamic audio-visual scenarios. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 19108-19118, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In Proceedings of the International Conference on Machine Learning (ICML), 2023b.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang. Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 22195-22200 2024.
Yong-Lu Li, Xinpeng Liu, Han Lu, Shiyi Wang, Junqi Liu, Jiefeng Li, and Cewu Lu. Detailed 2d-3d 3d joint representation for -bjjett interaction. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Tsung-Yl Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings. Pari V 13, pp. 740-755. Springer, 2014.
Yan-Bo Lin, Yi-Lin Sung, Jie Lei, Mohit Bansal, and Gedas Bertasius. Vision transformers are parameter-effiient audio-visual learners. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, arXiv preprint arXiv:2310.03744 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS), 2023b.
Shikun Liu, Linxi Fan, Edward Johns, Zhiding Yu, Chaowei Xiao, and Anima Anandkumar. Prismer: A vision-language model with an ensemble of experts. arXiv preprint arXiv:2303.02500, 2023c.
13

---

Published as a conference paper at ICLR 2025
Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action. arXiv preprint arXiv:2312.17172 2023.
Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song- Chhu Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes. In Proceedings of the International Conference on Learning Representations (ICIR), 2023.
Seungwhan Moon, Andrea Madotto, Zhaojiang Lin, Tushar Nagarajan, Matt Smith, Shashank Jain, Chun-Fu Yeh, Prakash Murugesan, Peyman Heidari, Yue Liu, et al. Anymal: An efficient and scalable any-modality augmented language model. arXiv preprini arXiv:2309. 16058, 2023.
Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using million captioned photographs. Advances in neural information processing systems, 24, 2011.
Artemis Panagopoulou, Le Xue, Ning Yu, Junnan Li, Dongxu Li, Shafiq Joty, Ran Xu, Silvio Savarese, Caiming Xiong, and Juan Carlos Niebles. X-instructblip: A framework for aligning x-modal instruction- aware representations to Ilms and emergent cross-modar reasoning. arXiv preprint arXiv:2311.18799, 2023.
Viorica Păträucean, Lucas Smaira, Ankush Gupta, Adrià Recasens Continente, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang, et al. Perception test: diagnostic benchmark for multimodal video models. arXiv preprint arXiv:2305. 13786, 2023.
AJ Piergiovanni, Isaac Nobel, Dahun Kim, Michael S Ryoo, Victor Gomes, and Anelia Angelova. Mirasol3b: A multimodal autoregressive model for time-aligned and contextual modalities. arXiv preprint arXiv:2311.05699 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning (ICML), 2021.
Prajit Ramachandran, Barret Zoph, and Quoc Le. Swish: self-gated activation function. arXiv 2017.
preprint
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of elip-filtered 400 million image-text pairs. In NeurIPS Workshop, 2021.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 1556-2565, 2018
Guangzhi Sun, Wenyi Yu, Changli Tang, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Fine-grained audio-visual joint representations for multimodal large language models. arXiv preprint arXh::2310.0586; 2023a.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv: vrxiv:2303.15389 /5389, 2023b.
Zineng Tang, Jaemin Cho, Yixin Nie, and Mohit Bansal. Tvlt: Textless vision- languaee transformer. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Mohit Bansal. Any-to-any generation via composable diffusion. In Advances Neural Information Processing Systems (NeurIPS). 2023a.
Zitian Tang, Wenjie Ye, Wei-Chiu Ma, and Hang Zhao. What happened 3 seconds ago? inferring the past with thermal imaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17111-17120, 2023b.
14

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Andong Wang, Bo Wu, Sunli Chen, Zhenfang Chen, Haotian Guan, Wei-Ning Lee, Erran Li Li, Joshua B Tenenbaum, and Chuang Gan. Sok-bench: A situated video reasoning benchmark with aligned open-world knowledge. In The JEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2024a.
Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang. Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111 2023.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing lision-langaag model's perception of the world at any resolution. arXiv preprini arXiv:2409. 12/91, 2024b.
Junbin Xiao, Xindi Shang. Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. Proceedings of the IEEE International Conference Computer Vision and Puttern Recognition (CVPR), 2021.
Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, Fisher Yu, Dacheng Tao, and Andreas Geiger. Unifying flow, stereo and depth estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael S Ryoo, et al. xgen-mm (blip-3): A family of open large multimodal models. arXiv preprint arXiv:2408. 08872, 2024.
Fengyu Yang, Chenyang Ma, Jiacheng Zhang, Jing Zhu, Wenzhen Yuan, and Andrew Owens. Touch and go: Learning from human-collected vision and touch. arXiv preprint arX1v:2211.12498, 2022.
Hanrong Ye, De-An Huang, Yao Lu, Zhiding Yu, Wei Ping, Andrew Tao, Jan Kautz, Song Han, Dan Xu, Pavlo Molchanov, et al. X-vila: Cross-modality alignment for large language model. arXiv preprint arXiv:2405. 19335, 2024.
Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. Self-chained image-language model for video localization and question answering. In Advances in Neural Information Processing Systems (NeurIPS), 2023a.
Wenyi Yu, Changli Tang, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma. and Chao Zhang. Connecting speech encoder and large language model for asr. arXiv preprint arXiv:2309. 13963, 2023b.
Zhou Yu, Jun Yu. Yuhao Cui, Dacheng Tao, and Qi Tian. Deep modular co-attention networks for visual question answering. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Yuhang Zang, Wei Li, Jun Han, Kaiyang Zhou, and Chen Change Loy. Contextual object detection with multimodal large language models. arXiv preprint arXhv:2305.18279 2023.
Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati. Jack Hessel, Ali Farhadi, and Yejin Choi. Merlot reserve: Neural script knowledge through vision and language and sound. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023.
15

---

Published as a conference paper at ICLR 2025
Xiaohui Zhang, Jachong Yoon, Mohit Bansal, and Huaxiu Yao. Multimodal representation learning by alternating unimodal adaptation. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In Proceedings of the International Conference on Machine Learning (ICML), 2021.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny, Minigpt-4: Enhancing vision-language understanding with advanced large language models. In Proceedings of the International Conference on Learning Representations (ICLR), 2024.
16

---

Published as a conference paper at ICLR 2025
APPENDIX
In this Appendix, we present the following:
Additional information about our experimental setup (Section A), including benchmarks details (Section A.I), details of multimodal encoder and visual expert models (Section A.2), MMQA module pre-training data pre-pree preprrcesssig (Section A.3), baseline implementation details (Sec. A.4), and CREMA implementation details (Sec. A.5).
Additional experiments (Section B), including fine-tuning results on VGGSound& Periedtionneeeeeeneeeee (Section B.1). comparison of training memory (Section B.2), zero-shot per-task performance on MUSIC-AVQA (Section B.3), the impact of MMQA pre-training (Section B.5), the impact of LoRA rank (Section B.6), more qualitative visualization (Section B.7), and extra discussion model design (Section B.8).
Limitation and broader impact of this work (Section C), as well as license information (Section D) for the datasets, codes, and models that used in this paper.
License information (Section D) for the datasets, codes, and models that we used this paper.
A EXPERIMENTAL SETUP
In this section, we present additional information the used datasets/benchmarkk (Sec. A.1), multimodal encoder and visual expert models (Section A.2), baseline implementation (Sec. A.4), and CREMA implementation details (Sec. A.5).
A.1 BENCHMARK AND DATASET
We evaluate the CREMA framework on three video reasoning and QA tasks, focusing on both conven- tional VideoQA (requires video and language) and compositional VideoQA (requires video, language and other modalities). These include: (1) SQA3D (Ma al., 2023): Another compositional Video QA task, requiring the understanding of video, 3D scenes, and text. Designed for 3D situated QA, it includes 33K questions and 650 3D scenes corresponding to ego-centric videos. We apply extra depth maps it and report results the test part following (Hong al., 2023). (2) MUSIC-AVON (Li et al., 2022): compositional Video QA benchmark that involves reasoning across video, audio, and text. This dataset contains 9288 videos and 45K questions. We follow X-InsturetBI.I IIP (Alayrac et al., 2022) to evaluate our CREMA A and on other baselines on the high-quality real video part. We enhance with optical flow
input and report findings the test (3) NExT-QA (Xiao et al., 2021): A conventional Video QA benchmark for causal and temporal reasoning with video and text inputs. consists of 5440 videos and 52K questions. We include optical flow, depth map, and surface normals extracted from raw videos as additional modalities. Our results are based on the validation partition following previous work (Yu al., 2023a). (4) ThouchQA: we build a question and answering dataset based on Touch&Go (Yang et al., 2022). As the original data only contains material class annotations, we reformulate the classification task as a QA task, by asking the model: What material is the person touching? Our TouchQA dataset contains 714 training data and 3212 test
Table 10: Baseline models fine-tuning hyperparameters.
Batch Size Learning Warmup Epoch
Gradient
Model (Dataset)
Modality Frames
per GPU
Rate
Accumulation Step
V
4
16
3e-05
1000
20
2
3D-LLM (SQA3D)
V,P
P
4
16
3e-05
1000
20
2
V.P.D.N D.N
4
3e-05
1000
20
2
A
4
8
3e-05
1000
20
2
4
8
3e-05
1000
20
2
BLIP-2 (MUSICCAAAAA
V.A
3e-05
1000
20
2
A.A.F
3e-05
1000
V
4
16
3e-05
1000
10
V.
3c-05
1000
2
BLIP-2 (NEXT-QA)
V.E.D
3e-05
1000
V.E.D.N
3e-05
1000
17

---

Published as a conference paper at ICLR 2025
data. (5) ThouchQA: Similar the TouchQA dataset, we build the ThermalQA dataset public video-thermal heatmap dataset, Thermal-IM (Tang et al., 2023b). Thermal-IM contains action labels in each video and was originally designed to predict human pose 3 seconds ago according to both video and thermal heatmap. We reformulate it as QA task as well by asking the model: What action might have occurred before this video? and the answer is action label. Our Thermal dataset contains 1131 training data and 391 test data. (6) Perception'Test (Paträucean et al., 2023): a multimodal benchmark designed to comprehensively evaluate the perception and reasoning skills of multimodal video models. We use the multi-choice QA part of this benchmark, which contains 1955 train data and 5260 validation data. (7) VGGSound (Chen al., 2020): a large-scat scale audio-video dataset. contains over 200K videos with audio sounds. It was originally for audio-vides classification. We formulated the classification tasks as the the open-ended QA task over 300 audio classes.
A.2 MULTIMODAL ENCODERS AND VISUAL EXPERT MODELS
We apply multiple encoders encode multimodal raw input discussed in Section 3.2. For visual inputs (video RGB frames, depth map, optical flow, and surface normals), we follow BLIP-2 (Li et 2023b) and X-InstructBIIIP (Panagopoulou al., 2023) to utilize VIT-G (Sun al., 2023b) to encoder visual information. As depth map, optical flow, and surface normals raw data are with different channel numbers to the ViT model required. We transform those extra visual information to the RGB domain first to adapt the ViT model. For audio, we use the same BEATS (Chen et al., 2023a) encoder as X-InstructBLIIP (Panagopoulou al., 2023). For the 3D point cloud, we follow data preprocessing in 3D-LLM (Hong et al., 2023) and ConceptFusion (Jatavallabhula et al., 2023) that first extract pixel-aligned dense features for rendered images features and then fuse 2D features into 3D maps using gradslam (Jatavallabhula et al., 2019).
We employ plug-and-play frozen experts to extract diverse modalities features, including depth map, optical flow, and surface normals, from raw vidcos. For optical flow estimation, utilize the SotA For depth map estimation, leverage the SotA ZoeDepth-NK (Bhat et al., 2023) model. For surface normals estimation, we follow Prismer (Liu et al., 2023c) to use NLL-AngMF (Bae et al., 2021) that pre- pre-trained on ScanNet (Dai al., 2017). We decode video into frames to extract per- frame depth map/optical flow/surface normals. We set 3 fps, I fps, and 3 fps to decode SQA3D, MUSIC-AVQA, and NEXT-QA videos respectively.
Unimatch (Xu et al., 2023) model CCMFFlow-ccale--rrrreeee
A.3 MMQA PRE-TRAINING DETAILS
As discussed in Section 3.3, we conduct extra lightweight pre-training for MMQA module to obtain a good initialization. We follow audio-perainiing settings X-InstructBLIP (Panagopoulou et al., 2023) with AudioCaps (Kim et al., 2019), but excluded caption data as our work is more focusing video reasoning. In this case, we obtained a QA-related subset from AudioCaps for MMQA-QA pre-training. Similarly, we utilized the 3D data released from 3D-LLM (Hong et al., 2023) and also took the QA format part for MMQA-3D pretraining. We pre-trained with le learning rate and epoch for efficient initialization.
A.4 BASELINE MODEL IMPLEMENTATION
We conduct experiments with 4 x 48GB A6000 GPUs, we report baseline model training hyperpa- rameters in Table 10. We follow hyperparameter settings that have been searched to yield the best performance in SeViLA (Yu et al., 2023a) with the same backbone model. To prompt L.M,, we design different prompts for open-ended QA tasks (SQA3D, MUSIC-AVQA) and multi-choice QA (NEXT-QA) following previous works (Yu et al., 2023a; Han et al., 2023). For open-ended QA, we let LLM generate responses without extra constraints and then compare the generated answers with ground-truie answers for accuracy calculation. We list prompts design for each dataset in Table 12. We utilize the same multimodal encoders, and multimodal information from the same estimator for fair comparison.
18

---

Published as a conference paper at ICLR 2025
Table 11: CREMA fine-tuning hyperparameters.
Batch Size Learning
Gradient
Dataset
Modality
# Frames
Warmup Epoch
GPU
Rate
Accumulation Step
P
2c-4
1000
20
2c-4
1000
20
SQA3D
V, V, D
2e-4
1000
20
2e-4
1000
20
V, D.D.D.
2e-4
1000
20
A
24
2e-4
1000
20
1
V
24
2c-4
1000
20
MUSIC-AVQA V.A
24
2c-4
1000
20
A.A.F
24
2c-4
1000
20
A,
2c-4
1000
20
V
16
le-4
1000
10
V.F
16
le-4
1000
10
V.D
4
16
le-4
1000
10
NEXT-QA
V.N
4
16
le-4
1000
10
V.E.D
4
16
le-4
1000
10
1
V.D.D.N
4
le-4
1000
10
2
Table 12: Prompt designs for each dataset.
Dataset
LLM Prompt
SQA3D
Model information, answer question using a single Based on the frames ond andlo information, answer the question using single word or phrase.
on the fromes
phrase.
MUSIC-AVQA
VGGSound
Based on the frames audio information, answer the question using single word or phrase.
NEXT QA
Considering the information presented the frame, select the correct answer from the options
Perception Test
Considering the information presented in the frame, select the correct answer from the options map information, unswer question using single or phruse. Based on the fromes ond thermal heatmap information, answer the question using single word or phraze.
TouchQQ
frames
ThermalQA
A.5 CREMA IMPLEMENTATION DETAILS
CREMA framework adopts BLIP-2 (Li et al., 2023b), an image-language model with 4.1B parameters and pre-trained on 129M images in total, including COCO (Lin et al., 2014), Visual Genome (Krishna et al., 2017), CC12M (Sharma et al., 2018), SBU (Ordonez et al., 2011), and 1 11MM images from LAION400M (Schuhmann et al., 2021). 1). See Appendix for details. we also report our CREMA framwork training hyperparameters in Table 11. The experiments are conducted on the same 4 48GB A6000 GPUs machine.
In the zero-shot setting, conducted evaluations on SQA3D (video + point cloud) and MUSIC- AVQA (video + audio). Since these tests include only two modalities at time, we bypassed the Self-Gated Multimodal Query Fusion module and directly concatenated the video tokens with the corresponding modality tokens. This ensures no parameter mismatch or interference and every modality is independent during inference.
Table 13: Fine-tuning Results on VGGSound. The numbers in the bracket represent results after excluding out-of-vocabulary results caused by formulating the classification the open-ended QA.
Method
Modality
Ace
Trainable Params.
47.0
324M
CAV-MAE (Gong al., 2022)
V. A
65.5
324M
Mirasol3B TTM (Piergiovanni al., 2023)
V. A
66.4
3B
MirasoI3B (Piergiovanni al., 2023)
V.
69.8
3B
51.5 (53.1)
4M
CREMA (ours)
V, A
62.4 (67.0)
9M
19

---

Published as a conference paper at ICLR 2025
Table 14: Fine-tuning Results on PerceptionTest.
Method
Modality
Ace
Trainable Params.
V
67.1
108M
BLIP-2 (Li et al., 2023b)
V,
68.2
216M
V, D.
67.9
324M
V
66.6
4M
CREMA (ours)
V, F.
68.7
20M
Table 15: Comparison of Training Efficiency on Video Question Answering (NExT-QA). Tested on the single A6000 GPU.
Model
Modality
BatchSize
Training Memory
BLIP-2 (Li et al., 2023b)
V, D. V.F.D,N
46.3GiB
CREMA
4
18.6GiB
B EXTRA EXPERIMENTS
In this section, we provide additional experiments and analysis, including zero-shot per-task perfor- mance MUSIC-AVQA (Section B.3), the impact of MMQA pre-training (Section B.5), the impact of LoRA rank (Section B.6), and more qualitative visualization (Section B.7).
Table 16: Zero-shot Per-task Results CREMA method on Audio-Video Question Answering (MUSIC-AVQA), We report simple notations for each modality and question type: V: Video RGB frames, A: Audio, Cnt.: Counting, Com.: Comparative, Loc.: Location, Ext.: Existential, and Tem.: Temporal.
Audio Question
Visual Question
Audio- Visual Question
Modality
Avg.
Cnt.
Com.
Avg.
Cnt.
Loc.
Avg.
Ext.
Loc.
Cnt.
Com.
Tem.
Avg.
50.4
53.2
51.0
18.5
23.9
39.9
13.0
27.1
49.6
4.3
29.1
31.0
73.4
51.2
68.6
51.4
44.4
48.0
76.4
43.5
38.7
47.1
26.3
47.7
51.0
A.V
75.5
51.6
70.4
55,5
42.6
49.2
76.2
44.2
45.1
48.2
26.3
49.5
52.6
EXTRA RESULTS ON VGGSOUND AND PERCEPTIONTEST
As listed in Table 13 and Table 14, we report extra fine-tuning performance on VGGSound (Chen al., 2020) and Perception Test Test (Päträucean et al., 2023). On VGGSound, we formulate the original audio-video classification task an open-ended QA task. We calculate accuracy by directly matching generated answer and the ground truth. Note that our original outputs have 2.9% (V) and 7.9% (V, A) predicted answers on test data that are out-of vii-of-voccaulary. We report both numbers with and without those vocabulyy data in Section B.I. It shows that combined with audio (A) brings notable and consistent improvement. Our CREMA shows comparable performance with fewer trainable parameters compared with other methods. On PerceptionTTest our CR EMA shows consistent improvement when adding new modalities while the baseline method struggles with this.
B.2 COMPARISON OF TRAINING MEMORY
We further compare the training memory of our CREMA and baseline method Table 15. Our method requires much less training memory when leveraging the same modality input and batch size. Thus, our method is more friendly to incorporate more modalities during the training in the view of computation resources.
20

---

Published as a conference paper at ICLR 2025
Table 17: Ablation of -odality-sequential Training of CREMA.
Method
Dataset
Modality
Accuracy
Trainable Params.
MUSIC-AVQA
V.A.F.
80.6
38M
CREMA (Joint)
SQA3D
D,
52.7
38M
NEXT-QA
D,
73.0
28M
MUSIC-AVQA
F,
81.7 (+1.)
38M
CREMA (Sequential)
SQA3D
V,
54.6 (+1.9)
38M
NEXT-QA
F. D.N
73.9 (40.9)
28M
Table 18: The impact of Modality-Specifii LoRA Pre-training. We report zero-shot performance.
Dataset
Modality
w/o PT
w PT
MUSIC-AVQA (Avg.)
A
28.1
31.0 36.1
SQA3D (Avg.)
P
37.3
B.3 EXTRA ZERO-SHIOT RESULTS MUSIC-AVQA
As listed Table 16, we report extra zero-shot performance on MUSIC-AVQA by more fine- mne-grained task/question types. It shows that video (V) combined with audio (A) brings notable and consistent improvement across most question types, highlighting the compositional video reasoning ability of proposed CREMA.
B.4 THE EFFECT OF HODALITY-SEQUENTTE TRAINING
We validate the effectiveness of the modality-sequentee training of our approach in Table 17. We report the fine-tuning performance three datasets: MUSIC-AVQA, SQA3D, and NEXT-QA. As shown, our sequential training mechanism effectively improves multimodal video reasoning capabilities without additional architectures the computationally and memory- nemory-intensive gradient modification strategy. This advantage is distinguishable from MLA (Zhang et al., 2024), multimodal representation learning method that sequentially trains shared classification head different modalities. Since MIAA uses a unified trainable module (i.e., the shared classifier head) to train multimodal data, it employs the orthogonal gradient projection technique different sensory inputs to mitigate the forgetting of previously learned modality information. In contrast, our model leverages substantial capacity to capture crucial information from various modalities through lightweight, yet effective modality-speciff multi-quer adapter (MMQA) modules, avoiding modality forgetting. Furthermore, we propose modality-adaptive dynamic early exit strategy, enabling our model to converge quickly and mitigate over-convergence issues in different modality data by utilizing the gradients lightweight
modules.
B.5 THE IMPACT OF MMQA PRE-TRAINING.
As listed in Table 18, we demonstrate the impact of MMQA module pre-training on SQA3D and MUSIC-AVQA datasets. shows that such an efficient MMQA pre-training brings a significant boost (+1.2% on SQA3D with 3D point could (P), +2.9% on MUSIC-AVQA with audio (A)) to the zero-shot performance for each single modality. It demonstrates the effectiveness of our MMQA and the pre-training process.
Table 19: The impact of the rank r of modality-adaptive LoRA module in CREMA.
Music-AVQA (A)
SQA3D (P)
Trainable Params.
#Rank
(A) (P)
32
69.4
47.0
3.8 M / 2.7 M
64
68.0
47.3
5.0 M / 3.9
128
67.7
46.3
7.4 6.3
21

---

Published as a conference paper at ICLR 2025
Scene:
Video:
Video:
Audio: -fute first, then and viokn fast- Optical
Depth:
Question: What facing the wall my right? V: black X
Question:
sound comning middle
V-P:black X
P:: brown X
P+ green
V: yes X
V+A:yes X
A: you X
V+A+F:
Audio: sounds fist,
Depth:
Optical Flow:
Question: right, soo mysair? Vi X
Question: Where the first sounding instrument? V: X V-A. micdle X A: lof X
V+P: no X no X
V+P+D:
V+A+F: right y
Figure 3: Qualitative examples for multimodal compositional video reasoning from SQA3D (Left) and MUSIC-AVQA (Right). The correct predictions are marked by green check marks.
B.6 RANK OF LORA MODULE.
We investigate the impact the rank r of our modality-adaptiinn LoRA modules in CREMA. As shown in Table 19, adjusting the rank size within reasonable range brings an insignificant change in the number of trainable parameters (c.g., +1 ~ 2 M). We also find that larger rank size does not guarantee improved performance. For MUSIC-AVQA, (A), and it always outperforms the best-performing baseline (AVQA, 64.2%) by a significant margin. CREMA trained 3d point cloud data performs similarly to the rank size of 32 and 64 SQA3D. We set r = 64 as the default for all experiments, showing the robustness of our MMQA module in selecting r various modality inputs and evaluation tasks. But believe that our CREMA method with proper can further improve its reasoning ability.
fine-tuned the model audio inputs
B.7 QUALITATIVE ANALYSIS
Beyond the numerical comparison of the effect integrating different sets of modalities for our CREMA method, we investigate our model's generated responses according to diffcrent types of input examples. In Figure 3 Left, CREMA with 3D point cloud inputs (P) fails to find the chair and respond to the color of the wall, brown, as its 2D scene image features are incorporated in 3D point cloud features. CREMA with Video (V) and V, P also predict incorrect chair color, black. However, with the assistance of depth information, the method can capture objects accurately and also find the designated chair. Similarly, in Figure Right, optical flow inputs help to find musicians with their poses playing instruments, so our CREMA method can tell the middle instrument is not being played at the beginning, but from the left. Figure 4, we present additional visual examples from SQA3D and MUSIC-AVQA, demonstrating how the integration of multiple input modalities enhances model predictions. For instance, depth maps in the top-left example reveal the distance of objects, enabling the model to discern that the clock is closer than the pillow. Similarly, on the middle left, depth maps indicate an open door through depth field analysis, aiding in question answering. In the MUSIC-AVQA examples on the right, the optical flow captures motion, which is essential for deducing which instrument is being played. Specifically, the bottom right illustration shows that the initial static behavior of the left people implies that the right instrument is not played initially. This evidence highlights the benefit of incorporating diverse modalities for improved model reasoning ability.
22

---

Published as a conference paper at ICLR 2025
3D Scenn::
Video:
Video:
Audio: "playing gultar
Depthi
Optical Flow:
Which the acoustic puitar?
Question: the clock cioser to the baby mobilo or the pillow? V: pillow X
pillo Pipillow X
VaP+D: baby mobile
X
viatin
X
viatin
V+A+F: ertu
Video:
Videe:
Audio: -vtolin sounds fist, thon the Bute-
Optical
Question: the open V: closed X
Question: Where is the first soundiin instrument?
ceP:clased X
P: closed
X
V+P+0: open
V: left X V+A middle X
A: midll X
V-A-F:right
Figure 4: Qualitative examples for multimodal compositional video reasoning from SQA3D (Left) and MUSIC-AVQA (Right). The correct predictions are marked by green checks.
Question: Is the first sound coming from the middle instrument?
Video
Predicted Answer: Yes
Correct Answer: No
Attention Map
Question: Is the first sound coming from the middle instrument?
Video
Predicted Answer: No
Correct Answer: No
Optical
Flow
Attention
Map
Figure 5: Visualization on attention map under different modality combinations. Top: with audio and video. Bottom: with audio, optical flow, and video. We omit audio for simplicity. We highlight attention regions that may affect model prediction with red boxes.
B.8 EXTRA DISCUSSIONS ON MODEL. DESIGN.
Prioritizing different modalities. We conducted additional experiments comparing different prior- itization strategies, including prioritizing other modalities and treating all modalities equally (i.e., no fusion, directly concatenating tokens) in Table 20. It shows that prioritizing video achieves the best performance, and prioritizing other modalities treating all modalities equally results in lower performance, validating the effectiveness our design.
Single FC layer Design. conducted additional experiments comparing the FC layer to a lighter- weight alternative, interpolation to project dimension dimension-mismatttiee mismatched features between multimodal encoder and multimodal Q-Former in Table 21. The results show that the FC layer improves
23

---

Published as a conference paper at ICLR 2025
Setting
NEXT-QA
Major: V + Supportive: D.F.N
73.9
Major:
D Supportive: V.F.N
62.1
No Prioritizing: V, F
73.5
Table 20: Ablation on prioritizing different modalities.
Connector
Parameters
MUSIC-AVQA
One-layer FC
0.3M
79.4
Interpolation
OM
78.9
Table 21: Ablation on the projection layer between multimodal encoder and multimodal Q-Former.
performance while adding a negligible number of parameters. We attrihute this improvement to the FC layer's ability provide more dynamic and learnable projection between multimodal encoders and the Multimodal QFormer, which better aligns features compared to interpolation.
Query Token Length. We conducted experiments analyzing the impact of query token length on performance, trainable parameters, and computational cost in Table 22. We find increasing the number of query tokens indeed improves accuracy as more fine-grained features are captured. However, also leads to increasing computational costs (GFLOPs), We find that 32 query tokens frame strike a good balance between performance and efficiency. This design aligns with BL.IP-2, ensuring strong performance without excessive computational overhead.
C LIMITATION & BROADER IMPACTS
Limitations. Our CREMA is mainly based BLIP-2 structure which was originally pre-trained only on image-language data. This may lead to limited temporal modeling/knowlldgee trained model. It might not handle well fine-grained temporal events (e-g., open the door close the door). Moreover, a potential limitation could be incorporating multimodal inputs such as depth and flow, which requires computing these modalities and introduces additional overhead.
the pre-
Broader Impacts. The CREMA framework leverages a pre-trained vision-language model backbone with the proposed adapter modules integrate multiple modality inputs through universal frame- work. Similar to most works leveraging pre-trained vision-language models, this might occasionally yield unexpected inappropriate responses, potentially reflecting societal biases related to gender, race, or sexuality. More studies of vision vasion-language models are needed to evaluate and mitigate these negative biases, and toxic output.
D LICENSE
We will make our code and models publicly accessible. We use standard licenses from the community and provide the following links to the licenses for the datasets, codes, and models that used in this paper. For further information, please refer to the specific link.
SQA3D: Apache
Modalities
# Quey Token
NEXT-QA
Trainable Param.
GFlops
V
16
70.8
4M
1.0K
V
32
71.6
44M
1.3 K
V
64
72.0
44
2.1 K
V.F
16
71.8
8M
1.4K
VIF
32
72.4
8M
2.2K
V.F
64
72.9
8M
6.2K
Table 22: Ablation the number of query tokens.
24

---

Published as a conference paper at ICLR 2025
MUSIC-AVQA: MIT
NEXT-QA: MIT
AudioCaps: MIT
3D-LLM: MIT
LAVIS: BSD 3-Clause
Touch-and-Go: CC BY
Thermal-IM: BSD 3-Clause
Perception Apache
VGGSound: CC BY
PyTorch: BSD-style
Huggingface Transformers: Apache
Torchvision: BSD 3-Clause
25


Reviews:
[{"review_id": "9Hf74JrplD", "reviewer": "Reviewer_d9Rm", "summary": "The paper introduces CREMA, an efficient and generalizable framework for video-language reasoning that enhances understanding through multiple modalities, including video, depth, audio, and 3D point cloud data, among others. CREMA employs a modular fusion approach, with lightweight, modality-adaptive modules that allow for easy integration of new modalities with minimal added parameters. The framework also incorporates a novel self-gated attention fusion technique to reduce computational demands. Additionally, it proposes a modality-sequential modular training and adaptive early exit strategy to boost training efficiency and enable faster adaptation to new modalities. CREMA demonstrates superior performance across multiple benchmarks, such as SQA3D, MusicQA, NExT-QA, TouchQA, and ThermalQA, highlighting the benefits of integrating diverse input modalities for improved video reasoning.", "questions": "Please refer to the weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- This paper proposes a framework capable of handling multiple modalities and addresses the issue of token quantity increasing with the number of modalities.\n\n- A single Q-former is used to process multiple modalities, avoiding the large increase in parameters typically associated with multi-modal input. Each modality requires only a small amount of modality-specific parameters, and since the parameters for each modality within the Q-former are independent, processing different modalities does not cause interference.\n\n- The modality-sequential and modular training approach accommodates the differences across various modalities, preventing overfitting or underfitting to any specific modality.\n\n- The paper demonstrates through multiple benchmarks that the proposed framework effectively integrates information from diverse modalities, thereby enhancing video reasoning capabilities.", "weaknesses": "- I believe the main focus of this paper is on ensuring that the number of tokens input into the LLM does not increase linearly with the number of modalities, while maximizing parameter sharing across modalities to avoid excessive parameter growth. However, I feel that the teaser image does not effectively highlight these key points.\n\n- My biggest concern lies with the Self-gated Multimodal Query Fusion. This module concatenates tokens from different modalities along the channel dimension, meaning that the input modalities during inference must match those used in training exactly—neither more nor less—otherwise, there will be a parameter mismatch within the Self-gated Multimodal Query Fusion. Many videos, for example, may not contain point cloud information; however, if point cloud data was included as input during training, it must also be part of the input during inference. This limitation significantly restricts the flexibility of input modality types.\n\n- Additionally, the description of the zero-shot setup is not clear enough. Before performing zero-shot evaluation on SQA3D and MUSIC-AVQA, which datasets were used to train and optimize the model's new parameters? Furthermore, as mentioned above, I believe that the Self-gated Multimodal Query Fusion limits the model's zero-shot reasoning capabilities, as different combinations of input modalities would require different models. This implies that different models were likely used for zero-shot evaluation on SQA3D and MUSIC-AVQA. Therefore, the authors should clarify which specific model was used for evaluation in each experiment.\n\n- Some related works on integrating multiple modalities are missing, such as MultiPLY[1] and X-VILA[2], both of which are multimodal LLMs capable of handling various input modalities. The authors should discuss the relationship with these works.\n\n[1]. MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World\nYining Hong, Zishuo Zheng, Peihao Chen, Yian Wang, Junyan Li, Chuang Gan\n\n[2]. X-VILA: Cross-Modality Alignment for Large Language Model\nHanrong Ye, De-An Huang, Yao Lu, Zhiding Yu, Wei Ping, Andrew Tao, Jan Kautz, Song Han, Dan Xu, Pavlo Molchanov, Hongxu Yin\n\n\n--- \n\nPost-rebuttal:\n\nMost of concerns are addressed, I raise my rating to 6.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z0BuShLS6D", "reviewer": "Reviewer_dN8u", "summary": "This paper proposes CREMA, a generalizable and modular modality-fusion framework that augments multiple modalities without extra human annotation and incorporates them into a query transformer, enhancing video reasoning. It introduces a progressive multimodal fusion design, maintaining computational efficiency and improving performance. Validated on 7 video-language reasoning tasks, CREMA outperforms or matches strong multimodal LLMs while significantly reducing trainable parameters, demonstrating its effectiveness and innovation.", "questions": "Please refer to the weakness", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is clear writing and easy to follow.\n2. Few current works focus on integrating multiple modalities, so the authors' motivation is commendable.\n3. I appreciate the paper's innovation. Although it may not introduce many new structures, the modality-adaptive early exit strategy appears to have broad application potential. It's the first time I've seen the use of gradients to determine whether to exit early, and it is also the first method to apply early stopping by modality. Therefore, I acknowledge the paper's innovative approach.", "weaknesses": "1. Overall, I believe this paper is worthy of acceptance and presents no significant issues. My only curiosity, as mentioned by the authors in the limitations section, is whether the method can be applied to more advanced baselines such as LLava, rather than just BLIP. If feasible, I would appreciate the authors addressing this point, which could lead me to adjust my score upwards.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qhbTdAVcUw", "reviewer": "Reviewer_5ndk", "summary": "This paper proposes a method, \"CREMA,\" that addresses the problem of video understanding with diverse modalities, including optical flow, point clouds, audio, etc. CREMA first uses modality-specific encoders to encode each modality. Then CREMA introduces a Q-former to extract features from each modality. Before feeding the features into LLMs, CREMA further leverages a self-gating modality fusion guided by the video features. Such an approach has the advantage of significantly less trainable parameters and competitive performance across multiple datasets, including MUSIC-AVQA, SQA3D, etc.", "questions": "* Q1: Why CREMA is called a video-language model? For example, SQA3D mainly uses RGBD as input, and the authors call CREMA a video-language model because the visual information is formatted as a video sequence.\n\n* Q2: Although the authors have compared the trainable parameter number, it is arguable what is the number of total parameters, as LORA is used. The questions is: what is the total number of parameters, and what is the speed of inference?\n\n* Q3: It is interesting to see that modalities of depth or surface normal are used, or even helpful, for MUSIC-AVQA and NExT-QA. I suggest the authors provide analysis or visualizations of how such modalities benefit the models.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* S1: The presentation of this paper is straightforward and clear.\n\n* S2: The proposed fusion approach with Q-former (architecture) and modality-sequential training (training recipe) are both reasonable and looks simple for other researchers to follow.\n\n* S3: The evaluation covers various domains, including audio, point clouds, optical flows, etc. The approach CREMA has demonstrated competitive performance across these scenarios, especially when the number of modalities is large.", "weaknesses": "* W1: This paper lacks sufficient quantitative or qualitative analysis on why multi-modality assists the model. For example, the MUSIC-AVQA performance in Table 1 can benefit from depth and surface normal information, which is not very intuitive. Therefore, some visualizations or other formats of analysis the authors see fit will greatly enhance the motivation here. I saw Figure 3 and the analysis provided by the authors. However, it is unclear whether the learned Q-former indeed considers these modalities, as indicated by Sec. B.7. Since the author uses self-gate to fuse the modalities, is it possible to analyze the model's reliance on certain modalities with attention scores?\n\n* W2: Following the previous point, the increased number of trainable parameters with more modalities makes it complicated to confirm whether the additional modalities are indeed helpful. For example, adding depth and normal information increases the trainable parameters from 9M to 38M.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b7uzKUJWXx", "reviewer": "Reviewer_wvMi", "summary": "The paper proposes CREMA, a flexible and efficient framework for video-language reasoning that incorporates multiple modalities, including optical flow, audio, thermal maps, and 3D point clouds. CREMA addresses the limitations of current multimodal models that require extensive parameters and fixed modality inputs by introducing a modular, parameter-efficient design. This framework allows seamless integration of new modalities while reducing computational costs, validated by superior performance on seven diverse reasoning tasks compared to baseline models.", "questions": "●Q1: In Line 177, the paper states that the Q-Former \"extracts the most informative features from the input modality and removes any irrelevant information\" by generating fixed-length tokens. However, the fixed-length constraint may risk omitting crucial details, particularly for modalities rich in information. To substantiate the claim of extracting only the most informative features, it would be beneficial to include empirical evidence or an ablation study comparing different token lengths and their impact on performance across modalities. \n\n●Q2: In Lines 194-195, the authors mention adding a fully connected layer (shown as a dashed box in Figure 1) for each data type when dimension misalignment occurs. Could you clarify why a fully connected layer was chosen over a potentially lighter-weight approach like interpolation? A fully connected layer seems more computationally intensive, so I am curious about the specific advantages it offers in this context. To clarify the advantages of this design choice, it would be helpful for the authors to provide a brief comparison, perhaps in terms of computational cost and performance, with lighter-weight options such as interpolation. \n\n●Q3: In Line 233, the authors select video queries as the \"major\" modality, with other modalities as \"supportive,\" explaining this choice as mirroring human perception in video reasoning tasks. Could you clarify the rationale behind prioritizing video in this way? Additionally, was a sensitivity analysis conducted to verify the impact of this design choice? I am curious whether this prioritization consistently benefits performance across tasks or if certain scenarios might require a different modality emphasis. To support this prioritization, the authors could consider presenting results from ablation studies or sensitivity analyses across various tasks and modality combinations, demonstrating whether prioritizing video consistently enhances performance or if other scenarios might benefit from different modality emphasis. \n\n● Q4: In Line 259, the authors describe decomposing the back-propagation process for different modalities in each iteration. Could this approach limit the model’s ability to capture interactions between modalities, which is critical for vision-language tasks? It seems more like a trade-off for efficiency rather than a true remedy, as mentioned. This decomposition may prevent the model from fully learning cross-modal interactions and effectively fusing information across modalities. Could you clarify this design choice and its potential impact on performance?\n\n● Q5: In Line 456, the paper mentions achieving a 'regularization effect on the model through parameter-efficient updates.' Could you elaborate on the specific mechanisms or components within CREMA that contribute to this regularization effect? Additionally, how does this approach enhance model generalization across various modalities?\n\n● Q6: Could CREMA also accommodate remote sensing imagery as an input modality? Remote sensing images, captured from satellites or drones, provide detailed information on Earth’s surface across multiple spectral bands. If CREMA can process this type of data, would specific adaptations be needed to handle its unique spatial and spectral characteristics?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper introduces CREMA, a novel, parameter-efficient framework that enables the seamless addition of new modalities without altering the core architecture—a significant advantage over existing models like BLIP-2 and SeViLA, which rely on fixed modality inputs and require extensive parameters. CREMA effectively integrates diverse modalities, such as 3D, thermal, and audio data, by projecting them into a unified representation space interpretable by the model for reasoning.\n\nKey architectural innovations, including self-gated multimodal query fusion and sequential modality training, bring practical improvements to multimodal reasoning tasks, particularly in video-language applications. CREMA demonstrates broad applicability and efficiency across seven video-language reasoning tasks, achieving notable accuracy gains in VideoQA and 3D reasoning. Through reductions of over 90% in parameter requirements and optimizations like modality-sequential training and adaptive early exit, CREMA marks a significant advancement in multimodal reasoning, validated through extensive fine-tuning and zero-shot evaluations.", "weaknesses": "●  Prioritizing certain modalities as primary lacks quantitative backing, which could benefit from sensitivity analysis to validate this design choice across diverse tasks.\n\n●  The Q-Former generates fixed-length tokens for each modality to extract the most informative features and remove irrelevant information. However, this fixed-length constraint could risk omitting valuable details, particularly in modalities with high information density. \n\n●  The decomposition of back-propagation by modality, while efficient, may limit the model’s ability to fully capture interactions between modalities, impacting the quality of multimodal reasoning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nfBx2riDUZ", "reviewer": "Reviewer_bc5T", "summary": "This work presents a multi-modal LLM pipeline CREMA to joint learning from different modalities: visual, depth map, optical flow, audio etc that are synchronized with a video input. Built on top of existing multimodal encoders and LLM, it proposes modal-specific queries and a modality fusion module to incorporate inputs from different modalities while keeping a low trainable parameter scale. The model is evaluated on tasks require multimodal inputs: audio-video QA, 3D situated QA, touchQA/thermal QA etc. It outperforms existing methods that are using the multimodal inputs such as OneLLM and 3D-LLM.", "questions": "-  Motivation behind adaptive early exit is not clear. The equation used on line 284 says if the gradient for a modality is larger than a threshold, then it will exit training. Shouldn’t it be smaller than a threshold since the gradient scale will be small after convergence?\n- Why using a sigmoid function in the fusion module in equation (3)? Seems it only does a scaling to the original q^{\\bar}_\\V which may not be necessary", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "-\tThe proposed model is a general framework for language QA based on multimodal video inputs. It achieves impressive performance on a wide range of tasks: audio-video QA, 3D situated QA, touchQA/thermal QA etc. \n-\tSome ablation studies are conducted for the choice and early exit strategy and modality fusion method (Table 7&8).", "weaknesses": "-\tThe performance of the proposed model seems to be dependent on the used multimodal encoders (ZoeDepth, Unimatch, NLL-AngMF to estimate depth, flow, normal, BEATs to encode audio, and ViT-G to encode visual). The comparison to existing methods might be unfair due to different encoders are used. More explanations are needed to verify this.\n-\tThe overall novelty is limited. The proposed model-specific queries and modality fusion module are subtle technical changes that does not bear a strong novelty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)