Modularized Self-Reflected Video Reasoner for Multimodal LLM with Application to Video Question Answering
Authors: Zihan Song
Xin Wang
Zi Qian
Hong Chen
Longtao Huang
Hui Xue
Wenwu Zhu
Venue: ICML 2025 poster (2025)
Keywords: Video Question Answering
Multimodal LLM
Modular Network
Self-reflected Training
Primary area: deep_learning->large_language_models

Abstract: Multimodal Large Language Models (Multimodal LLMs) have shown their strength in Video Question Answering (VideoQA). However, due to the black-box nature of end-to-end training strategies, existing approaches based on Multimodal LLMs suffer from the lack of interpretability for VideoQA: they can neither present reasoning paths nor indicate where the answers are derived from the video. To address this issue, we propose **MSR-ViR** (**M**odularized **S**elf-**R**eflected **Vi**deo **R**easoner), which for the first time integrates modular networks to Multimodal LLMs, capable of providing VideoQA with explicit reasoning paths for more interpretability. Specifically, a **MoST-Grounding** (Modularized Spatial-Temporal Grounding) network is proposed to decompose complex questions via tree-structured policies, localizing relevant temporal and spatial segments within videos through step-by-step reasoning. The proposed MoST-Grounding network provides explicit visually grounded information for Multimodal LLMs with clear reasoning paths, thus enhancing interpretability for the predicted answers. To further improve the reasoning quality, we design an **Alternate Self-reflection Training Strategy** to jointly optimize policy generation and Multimodal LLMs. Experiments on real-world datasets demonstrate the superiority of our proposed MSR-ViR framework in video understanding, reasoning transparency, and providing explicit localization evidence for answers.

Modularized Self-Reflected Video Reasoner for Multimodal LLM with Application to Video Question Answering
Zihan Song'
Xin Wang's Zi Qian 2
Hong Chen
Longtao Huang²
Hui Xue Wenwu Zhu
Abstract
1. Introduction
Video Question Answering (VideoQA) is a representative task in video understanding, aiming answer questions based on the content of given video. Leveraging their rich external knowledge and strong generalization capabili- ties, multimodal large language models (Multimodal LLMs) have emerged as powerful tools for tackling video under- standing tasks such as VideoQA, video captioning and so on. However, existing approaches based on Multimodal LLMs suffer from the following issue in VideoQA tasks: the classic end-to-end training approaches operate as black- box systems, which inherently suffer from the lack of inter- pretability. Falling short in terms of transparency, they are unable to unveil the reasoning paths or pinpoint the specific segments of the video from which the answers are derived.
Multimodal Large Language Models (Multimodal L.Mss) have shown their strength in Video Ques- tion Answering (VideoQA). However, due to the black-box nature of end-to-end training strate- gies, existing approaches based on Multimodal LLLMs suffer from the lack of interpretability for VideoQA: they can neither present reason- ing paths nor indicate where the answers are de- rived from the video. To address this issue, we propose MSR- ViR (Modularized Refleeeflet Video Reasoner), which for the first time inte- grates modular networks to Multimodal LLMs, capable of providing VideoQA with explicit rea- soning paths for more interpretability. Specifi- cally, MoST-Grounding (Modularized Spatial- Temporal Grounding) network is proposed to de- compose complex questions via rre-structured policies, localizing relevant temporal and spatial segments within videos through step-by-step rea- soning. The proposed MoST- GronnGGgon network provides explicit visually grounded information for Multimodal L.MMs with clear reasoning paths, thus enhancing interpretability for the predicted answers. To further improve the reasoning qual- ity. we design an Alternate Self-reflection Train- ing Strategy to jointly optimize policy genera- tion and Multimodal LLMs. Experiments on real- world datasets demonstrate the superiority of our proposed MSR-ViR ViR framework in video under- standing, reasoning transparency, and providing explicit localization evidence for answers.
To solve this issue, we marry modular network with Multi- modal LLMs for interpretable VideoQA. In particular, propose the Modularized Self-Reflected Video Reasoner (MSR-ViR) framework, which is able to obtain clear rea- soning paths when answering the questions. The pro- posed MSR- ViR framework contains a Modularized Spatial- Temporal Grounding (MoST-Groundingg) module, to- gether with a reinforcement learning- based Alternate Self- reflection Training strategy train Multimodal L.M for VidcoQA, as is shown in Figure I (c). Concretely, by following a tree-structured execution policy generated by a Question Parser, MoST-Grounding first decomposes a complex question into several small parts. Then through step-by-step spatial-temporal grounding and reasoning, progressively derives the most relevant visual information in the video, which will be fed Multimodal LLM to answer the question. The execution policy provides clear reasoning path from the question to the answer, providing interpretability for making predictions, while the spatial- temporal grounding results can provide visual evidence for the predicted answers. As for model optimization, on the one hand, the proposed Alternate Self-reflection Training strategy optimizes policy generation via letting the Multi- modal LLM undergo Supervised Finetuning (SFT) based on the execution results of the policies generated by the Ques- tion Parser in order to better understand the video content. On the other hand, the predicted result of the Multimodal
'Department of Computer Science and Technology, Beijing National Research Center for Information Sci- ence and Technology. Tsinghua University, Beijing, China "Alibaba Group, Hangzhou, China. Correspondence to: Wenwu Zhu
Xin Wang <xin_ <xin_wang@@ waang@@ssngguua.du..u <wwzhu @ wwwzhu@isinghua..de. wsi.ggu..cdd.c..>
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Modularized Self-Reflected Video Reasoner for Multimodal LLM with Application to Video Question Answering
Video
Question Parser
Quevtion
Video
Video
Question
Modular Network
Question
(MeST-Grounding
Belf- Reflection
Question
Localizer
Spetieliferpocei decunding Semslts
Modular Network
Ancre/BLatribttion
Image Captioner
Internediate
Oxpeicea of
Multimodal LLM
LLM
Multimodal LLM
Answez
Answer
Answer
(a) Grounding-based Method
(b) Modular Method
(c) MSR-ViR (ours)
Figure 1. Comparison between existing works and our method. (a) shows the common framework of existing grounding based methods, and (b) shows the common framework of existing modular methods. (c) is our MSR-ViL ViR framework.
LLM also serves as a feedback to train the Question Parser through reinforcement learning. The training of the Ques- tion Parser and the Multimodal LLM alternates in a cyclical manner, and both are optimized during this alternate self- reflected training process.
egy to generate reasoning paths.
We provide a theoretical proof of the upper bound computational complexity of our MSR-ViR frame- work, demonstrating that the computational overhead is strictly bounded.
We train the Multimodal LLM with both classic methods and our proposed model over the training sets of commonly used VideoQA datasets, NEXT-QA (Xiao et al., 2021) and STAR (Wu et al., 2021), and evaluate them on their corre- sponding test sets. We also conduct zero-shot experiments on long-form VideoQA datasets EgoSchema (Mangalam al., 2023) and VideoMME (Fu al., 2024a) for further evaluation. The results show that our proposed MSR-ViR framework significantly outperforms classic training meth- ods as well as other grounding based VideoQA methods. Furthermore, we conduct evaluations on NEXT-GQA (Xiao et al., 2024), a widely-used grounding based VideoQA dataset, whose results demonstrate that the proposed MSR- ViR framework is able to not only improve the performance of VideoQA but also more accurately localize the temporal segments relevant to the questions compared to baseline methods. We also prove a theoretical upper bound over the computational complexity of MSR-ViR demonstrating that the additional computational overhead introduced can be strictly bounded and reasonable.
We conduct extensive experiments to demonstrate that i) MSR-ViR framework can significantly outper- form baseline methods, and ii) MSR-ViR can accu- rately localize the temporal segments, providing visu- ally grounded evidence for the predicted answers.
2. Related Works
Video Understanding with Multimodal LLMs. Multi- modal LLMs have been widely used for video understand- ing tasks (Zhang et al., 2023; Lin et al., 2024; Maaz et al., 2024; Li et al., 2023b; 2024a; Zhang et al., 2024c; Song et al., 2024; Yao et al., 2024; Li et al., 2025). Most Mul- timodal L.LMs for videos are built on open-source L.MMs such as LLaMA (Touvron et al., 2023) and Vicuna (Chiang et al., 2023), and adapters are utilized to align encoded vi- sual information with the textual space. However, classic end-to-end training methods of Multimodal LLMs remain black boxes, resulting in a lack of interpretability they are unable to provide inference process as well as grounded evidence of the answer in the video.
Our contributions can be summarized as follows:
Grounded VideoQA with LLMs. Grounded VideoQA aims to indicate where in the video the answer originates while answering questions. Most existing grounding- VideoQA methods (Wang et al., 2024f; Xiao et al., 2024; Qian et al., 2024b; Yu et al., 2024; Wang et al., 2024a) attempt to localize time segments relevant to the question within the video in the first place and then sample frames from the identified segments to serve as inputs to Multimodal LLMs, as is shown in Figure (a).
We propose modularized VideoQA framework with self-reflection training. To the best of our knowledge, this the first VidcoQA work incorporating modular networks into Multimodal LLMs for interpretability.
(retrieval-
We propose modular network MoST-Grounding to decompose complex questions into small parts and derive the most relevant visual information. We also propose an Alternate Self-reflection Training Strat-
2

---

Modularized Self-Reflected Video Reasoner for Multimodal LLM with Application to Video Question Answering
Inputloutput
Ground-outh
Supervision
Feadback
Order
Training
Frozen Model
Quastion
Video
Question
Parser
Policy
Why loos bo woxtigcia badlwarlo love Ioueu alke uggraatiovaan bans?
Question
Video
MoST BoTT-Groun Module
Polley
Parser
Modutartzatio
Global Representation
Iespord att trumes
Spakal
Errenabel
Muttimodal LLM
Modularized Spatial-Tempora Grounding Module
(MoST- MooST-Gronnding
Multimodal Supervised
Tampori Localter
Spaital Localiber
Freaze Muttinodal LLM
Finetuning
Activete Maltieede ULM Freaze Quection Panser
LocataCky
Activots Quaction
Terporalittann
Spetn
Question
Vidco
Quastiio
Parser
Video
Tonperetturate
Polky!
refoertie
Multimodal Language Model
Muttinodal
Muttimodal
Groundlnddr
Reinforcement Learning
(a) Alternate Self-reflection Training Strategy
MSR-ViR Forward Propagation
Figure 2. Framework of MSR-ViR. The left part (a) shows Alternate Self-reflection Training Strategy, including (i) Multimodal LLM Supervised Finetuning and (ii) Question Parser Reinforcement Learning. The right part (b) demonstrates forward propagation details of MSR-ViR during Multimodal LLM supervised finc-tuning.
Other work like GroundVQA (Di & Xie, 2024) integrates query grounding and answer generation into unified model to achieve grounded VideoQA. Grounded-basse methods address the issue of providing visual evidence in videos to some extent, but still they lack interpretability as they typically rely on black-box models to perform temporal localization without clear reasoning path, especially for questions with complicated structures.
reflection training strategy. Figure 2 demonstrates our over- all framework. We will first introduce the Question Parser in Section 3. and then introduce our MoST-Grounding module Section 3.2. In Section 3.3, we will present how to process various information from both MoST-Grounding module and naive inputs based Multimodal LLM. Then, our proposed Alternate Self-reflection Training Strategy is introduced in Section 3.4. Finally, the computational com- plexity of the framework will be discussed in Section 3.5.
Modular VideoQA with LLMs. Modular methods utilize various smaller models according to execution policies gen- crated by certain LLM to handle sub-tasks derived from the original complex question, and another LLM integrates the outputs of these smaller models to produce the final answer (Min al., 2024; Zhang et al., 2024a; Surís al., 2023; Wang al., 2024c;g), shown in Figure (b). While this approach enhances interpretability, the unimodal LLMs used can only receive video information through video cap- tions, potentially missing temporal context and detailed information with video. Additionally, policies generated without training might be unreasonable. affecting accuracy of question answering. Appendix A presents some other works that are related to our work.
3.1. Question Parser
Many video language questions actually involve a "multi- step" reasoning process rather than the end-to-end "one- step" processing. As the example shown in Figure 2, our MSR- ViR framework mirrors the "multi-step" reasoning process how humans tackle VideoQA tasks: when facing video together with complex question, we first utilize Question Parser decompose the question into several sub-questions, allowing to identify the relevant video seg- ments and regions, together with tree-structured reasoning process, to explicitly help answer the question.
Given a question q, our Question Parser Q aims to generate the policy p = Q(q), which serves as the execution plan for the subsequent MoST-Grounding module (illustrated in Section 3.2) to invoke various small modules for temporal and spatial localization. Considering the diversity of ques-
3. The Proposed MSR- ViR Framework
In this section, we describe our proposed framework MSR- ViR, modularized VideoQA framework with alternate self-
3

---

Modularized Self- Seff-Reflected Video Reasoner for Multimodal LLM with Application to Vidco Question Answering
To address complex semantic scenarios, both the temporal localizer and spatial localizer consist of several types of small modules that can be dynamically assembled accord- ing to the policy. To be specific, there are 7 small modules in our MoST-GGrounding module. As the core module of temporal localizer, DetectAct module temporally local- izes a simple action described by short query (like "car approaching") in the video. In this module we utilize uni- fied video temporal grounding model UniVTG (Lin et al., 2023). Spatial localizer contains Locate0b module localizing an object described by short query (like "man wearing cap") in a video frame. In this module we take advantages of an open- -occbuaayy real-time object detec- tor YOLOO-Wort (Cheng et al., 2024). Details about other modules are presented in Appendix C.
Question which object did the person sit on after they put blanket?
Temporal Localizer
O'eeraon put down Blarket")
Question Parser
Prompt
Tesporaltacal ('atter"))
Polccy
Temporaltruncace
"tempora1__ooe Tocatizer"; I
"wodule": "cepporaltrrancade
[1
"radlle: "semporal Jecalize".
"params": ('Aftar", ("modulle";
"detect_arr". "parass": ["the person put the slanket"
Spatial Lzealizer
Locateooj "persor"
"patial 1o4t1izer: "modblle" trancatl,
Soutialiocal iee ("stccing on!)
rodule":
"ssatial, loralitze,,
"soraus": ("streing on", ('modute";
"ocate_ : "porans": ("the person'
Spstiallruncate
In our execution policies, both temporal localizer and spatial localizer will dynamically assemble corresponding modules (m²} and {m²). respectively. With the policy p generated by Question Parser, F, and F, in Equation (1) would be instantiated as follows:
Figure 3. Prompting the Question Parser to generate modulariza- tion policies.
tion content and structure, as well as the limited training data for question policies, we utilize a large language model as the Question Parser and take advantage of its in-context learning capabilities (Brown et al., 2020). As shown in Figure 3, we carefully design the prompt as instruction and stimulate the Question Parser to generate policies in the uniform JSON format, organizing and chaining the small modules in a specific structure. The JSON structure allows MoST-Grounding module to recursively call each module, ultimately generating the spatial-temporal grounding results. The complete prompt for the Question Parser presented in Appendix B.
Fepp() = ((mm))))) = (2)
=
After instantiating with the policy, the modules within the temporal localizer are called first locate and extract the temporal-grounded frames from the video. Subsequently, the modules within the spatial localizer are invoked to gen- erate the corresponding spatial-grounded frame for each temporal- cemporal-grounded frame. In this modular manner, MoST- Grounding extracts several temporally and spatially local- ized video frames from the video, which will serve as visual input our Multimodal LLM.
3.2. Modularized Spatial-Temporea Grounding Module
3.3. Multimodal LLM Answerer
MoST bosT-Grounding module is the core component of our framework, recursively invoking various small modules ac- cording to the modular policy generated by the Question Parser to achieve temporal and spatial localization for com- plex questions. Our MronnGrounding module consists of two parts: temporal localizer F, and spatial localizer F., each containing several small modules for temporal and spatial localization, respectively.
After MoST-Grounding localizes the temporal segments and spatial regions relevant to the question, a Multimodal LLM is needed to understand the textual and visual information in order answer the question. Formally, the answer of question g given the video can be written follows:
ū(g, 1) = F(u Usr by
where and 6,, represent the video segments and bounding box generated from MoST-Grounde module, and F(...) denotes the forward propagation of Multimodal LLM. To better enhance the video understanding ability of the Multi- modal LLM, we extend the input of the Multimodal LLM in Equation (3) with the following two strategies. Firstly, provide additional global representation of the video to the Multimodal LLM by compressing several uniformly sampled frames through average pooling. This is necessary because the MoST-Groundhe rately localize the segments relevant to the question, and the Multimodal LLM might overlook essential information if it
Given a concept a and video e {vi, 112,...,?7 PT) con- taining T frames, temporal localizer aims to generate the most relevant segments
{u....., } from the video v where 1 < i j j < T. Later the output of the temporal localizer, namely video segments Ug, along with concepts e, are processed to the spatial localizer, generating most relevant visual bounding box 6,, within video segments Ug- Formally, MoST-Grounding module M is written follows:
module may not always accu-
4

---

Modularized Self-Reflected Video Reasoner for Multimodal LLM with Application to Vidco Question Answering
relies solely on the grounding results. Secondly, in addition to the original question and the aforementioned visual in- formation, we provide the Multimodal LLM with a guiding prompt that explains the specifie meanings of various visual input components. With these two designs, Equation (3) would be modified with:
and reference model Teet- Pw denotes the positive policy, p denotes the negative policy and q is the input question. my is our Question Parser to be trained, while wef is a refer- ence model initialized with the Question Parser itself but remain frozen. Through DPO training, the probability of generating positive policies increases, while the probability of generating negative policies decreases. In other words, the Question Parser learns to generate more reasonable poli- cies. We prompt the Question Parser to view the same question from multiple perspectives, generating different modular policies. The MoST- MoST-Grounding module executes each policy, producing their respective grounding results. The Multimodal LLM then computes the corresponding losses. We classify the policy with the smaller loss as posi- tive and the one with the larger loss as negative, training the Question Parser according to Equation (6).
D(q.v) F(Pblg, Va, g.)))
where Pt(...) represents the guiding prompt and Je de- notes the global video representation. Specific input format of the Multimodal L.M is presented in Appendix F.
With the grothd-truth of question q being y, the supervised finetuning loss of the Multimodal LLM is defined as a cross entropy loss LeB:
(CE(ő(g.V).)) - Σ y ylog(ū(g, v)),
(5)
Our training strategy alternates between SFT of the Multi- modal LLM and reinforcement learning for the Question Parser, optimizing with the loss functions in Equation (5) and Equation (6), respectively. While training one large model, the other model's parameters remain frozen. Dur- ing this process, the Multimodal LLM periodically pauses to adapt based on the modular policies from the Question Parser. After set of training period, the Question Parser utilizes these refined policies to further train the Multimodal LLM, allowing both models to optimize continuously. See Appendix D for detailed training process.
(0.0.))E
where is the target answer, D is the dataset. By opti- mizing the loss function in Equation (5), the Multimodal LLM undergoes supervised finetuning on VideoQA datasets, learning to answer questions based on all the provided in- formation.
3.4. Alternate Self-reflection Training Strategy
As discussed in Section 3.1, we can teach the Question Parser to generate modularization policies from complex questions by providing examples in the prompt. However, relying solely on in-context learning does not ensure the quality of the policies. To address this issue, we propose the Alternate Self-reflection Training Strategy, which enables the Question Parser to improve the quality of its policies through reinforcement learning.
3.5. Computational Complexity
Considering that our framework introduces an LLM as the Question Parser and several smaller grounding models as sub-modules in MoST-Grounding of the upper bound computational complexity of our frame- work. We present the following propositions:
we provide an analysis
We assume that for a given question, provided that all our modules remain unchanged, reasonable modular policy is more likely to accurately localize the correct temporal seg- ments and spatial regions. Consequently, the loss computed by the Multimodal L.M is likely to be smaller. There- fore, we provide feedback to the Question Parser using the loss noticed during the training process of the Multimodal LLM,, thereby guiding it through reinforcement learning training. Specifically, we utilize Direct Preference Opti- mization (DPO) (Rafailov et al., 2024) to train the Question Parser. Different from previous RLHF methods, DPO di- rectly optimizes a language model without explicit reward models, making the training process straightforward and sta- ble. Specifically in DPO, the policy objective is formulated as:
Proposition 3.1. Given parameters of Multimodal LLM Pt, Pg. Ps. Ps. video with N input frames and resolution H x W, text input with length 1, the complexity of Multimodal Answerer is O H22 + P2N2 + Pyl + P.Nt).
Proposition 3.2. Given small modules in MoST-Grounding with complexity Ct C2. and large models in Ques- tion Parser and Multimodal LLM with complexity Ct. Cz. Cs. question with length L, prompt for Ques- tion Parser with length le- video with length T and resolution H x W, the complexity of MSR-ViR is O (QTTH22 + o2T² + C,H22 + +OL2 Cglg).
Detailed proof of these propositions is provided in Ap- pendix H. It is worth noting that C1 C2 are significantly smaller than C1, C2, C3. and N is small constant in our framework. Therefore, compared to its baseline Multimodal LLM, the additional complexity introduced by our frame- work mainly stems from CzLE. which is the complexity of Question Parser. This computational overhead is bounded
(DPO(TT: Tree) = -E E [log a (fo(pw. 4) - fo(pi. g))l, (6)
where fo(2, v)= 3 log
is the reward of policy x for question y implicitly defined by the language model m
(uly))
5

---

Modularized Self-Reflected Video Reasoner for Multimodal LLM with Application to Video Question Answering
Table Experiments on NEXT-QA and SuuR-sub. models are finetuned on the corresponding training set. The first part contains small vision- language models, and in the second part models or methods are based on Multimodal LLMs. Qwen- VL and LLaVA-NeXT are our direct baselines, MSR-ViE ViRg is our framework based Qwen- VL and MSR- ViR, our framework based on LLaVA-NeXT. Bold number denotes the best result. Model size and inference speed could be found in Appendix G.2.
NEXT-QA
STAR-sub
Method
Temporal
Causal
Descriptive
Average
Interaction
Sequence
Average
65.0
50.6
52.8
MIST
56.6
54.6
66.9
57.1
55.5
54.2
54.9
CoVGT
57.4
58.8
69.3
60.0
HiTeA
58.3
62.4
75.6
63.1
Intern Video
58.5
62.5
75.8
63.2
62.7
65.6
64.4
BLIP-2
64.9
69.7
79.4
69.6
65.4
69.0
67.2
TGB
66.5
72.8
81.2
72.1
InstructBLIP
70.5
71.5
79.8
72.5
SeViL.a
69.4
74.2
81.3
73.8
63.7
70.4
67.1
GCG
72.6
74.2
80.7
74.6
Qwen-Vi VL
68.4
71.3
80.6
71.9
60.4
65.5
63.0
ILaVA-NeXT
69.5
73.3
79.7
73.1
67.6
72.1
69.9
(oRs)
69.9
73.4
81.5
73.6
64.8
68.0
66.4
MSR-ViRR, 72.2
74.6
80.9
74.9
68.9
73.1
71.0
because Question Parser utilizes prompt with fixed length Ip irrelevant to the complexity of VideoQA scenario. It is also reasonable, as Question Parser generates modular poli- cies as clear reasoning paths enhancing interpretability of our framework. We also conduct inference speed experi- ments on NEXT-QA dataset, and the results are presented in Appendix G.2.
from NEXT-QA, providing ground-truth temporal clips for validation and test sets evaluate temporal grounding accu- racy. It is worth mentioning that for STAR, we create a sub- set (STAR-sub) with Interaction and Sequence questions (82.5% of STAR), excluding Prediction and Feasibility types as they lack temporal and spatial grounding videos, making them unsuitable for our framework.
Baselines. On NExT-QA and STAR, baselines include vision-language models ATP (Buch et al., 2022), MIST (Gao et al., 2023), CoVGT (Xiao et al., 2023), HiTeA (Ye et al., 2023), Intern Video (Wang et al., 2022), Multimodal LLMs BLIP2 (Li et al., 2023a). InstructBLIP (Dai et al., 2023) and grounding-based Multimodal L.MSs TGB (Wang et al., 2024f), SeViLa (Yu al., 2024), GCG (Wang et al., 2024a). On EgoSchema and NEXT-GQA, our baselines include vision-language models VGT (Xiao et al., 2022), VIO- LETv2 (Fu al., 2023), Temp[CLIP] (Radford et al., 2021), FroZenBiLM (Yang et al., 2022) (which achieve grounded VideoQA with the method in (Xiao et al., 2024)). grounding- based method TGB, SeViLa, LangRepo (Kahatapitiya et al., 2025) and modular method LLoVi (Zhang al., 2024a), MoReVQA (Min al., 2024). On VideoMME, our base- lines include Video LLMs Video-LLaVA (Lin et al., 2024). ShareGPT4Video (Chen et al., 2024), Long VA (Zhang et al., 2024b), Video-CCAM (Fei et al., 2024b) and VITA 1.5 (Fu et al., 2024b). We utilize Qwen-VL. (Bai et al., 2023) and LLa VA- (Zhang et al., 2024c) as direct baselines on NExT-QA. STAR and EgoSchema, while Qwen2-VL (Wang et al., 2024b) and L.L.VA-Vida (Zhang al., 2024e) our direct baselines VidcoMME.
4. Experiments
In this section, we first introduce the basic setups of our experiments in Section 4.1. Next, we introduce our ex- periments on VideoQA datasets NExT-QA, STAR in Sec- tion 4.2, after which experiments on long-form VideoQA datasets EgoSchema and VideoMME will be discussed in Section 4.3. Then in Section 4.4 we present experi- ments grounded VideoQA dataset NEXT-GQA. Finally, we present the ablation study in Section 4.5.
4.1. Experiments Setups
Datasets. We conduct experiments VideoQA datasets NExT-QA (Xiao et al., 2021). STAR (Wu et al., 2021) and long-form VideoQA datasets EgoSchema (Mangalam et al., 2023) and VideoMME (Fu et al., 2024a), together with a grounded VideoQA dataset NEXT-GQA (Xiao al., 2024). NEXT-QA and STAR contain various reasoning tasks in- cluding spatial-temporal reasoning, logical attribution and so while EgoSchema and VideoMME include longer videos and more complex questions, making them partic- ularly well-suited for evaluating video understanding and reasoning capabilities of models. NEXT-GQA is derived
Implementations. We implement our method based on
6

---

Modularized Self-Reflected Video Reasoner for Multimodal LLM with Application to Vidco Question Answering
Table Zero-shot experiments on EgoSchema. Sub. shows the results on EgoSchema subset and Full shows the results on EgoSchema full set. The numbers in parentheses represent the improvement of our method compared to their direct baselines. Bold number denotes the best result.
Table 3. Zero-shot experiments on VideoMME.. Short, Medium. Long represents 3 subsets with different video length. The num- bers in parentheses represent the improvement of our method compared to their direct baselines. Bold number denotes the best result.
Method
Size
Sub.
Full
Method
Short
Medium
Long
Avg.
ScViLa
3B
25.7
22.7
Vidco-LLaVA
45.3
38.0
36.2
39,9
LLoVi
7B
50.8
33.5
ShareGPT4
48.3
36,3
35.0
39,9
LangRepo
7B
60.8
38.9
LongVA
61.9
51.4
45.4
52.9
Video-CCAM
62.2
50.6
46.7
53.2
Qwen-VL VL
7B
44,6
36.8
VITA 1.5
67.0
54.2
47.1
56.1
MSR- (our)
7B
49. 49.0(+4.4)
38.7(+1.9)
Qwen2-VL VL
65.2
52.2
48.3
55.3
LLaVA-NeXT
7B
54,8
43.3
MSR (our))
66.8
55.4
51.3
57.9(+2.6)
MSR-ViR, (our)
7B
61.2(+6.4)
46.0(+2.7)
LLaVA-Videe ViR-ViiLy
69.7
56.6
49.3
58.5
72.3
60.7
52.6
61.9(+3.4)
SWIFT framework (Zhao et al., 2025) and utilize LoRA (Hu et al., 2022) during supervised finetuning. We utilize a large language model Qwen2-7B (Yang et al., 2024) for our Question Parser. Qwen- VL (Bai et al., 2023), LLaVA- NeXT (Zhang et al., 2024c), Qwen2-VL (Wang et al., 2024b) and LLa LLLVA-Videe (Zhang et al., 2024e) are utilized for our Multimodal LLMs, denoted MSR-ViRg. MSR- ViRi, MSR- ViRgz and MSR-ViBe LiRLy respectively. Following the classic training strategy, we uniformly sample 4 frames from videos for Qwen-VLL 8 frames for VA. 32 frames for Qwen2-VL. and LLaVA- Video to implement our direct baselines. For MSR- - we sample 2 temporal- and 2 opatial-grounder frames. For MSR-ViR, we sample 8 temporal- 8 spatial-grounded frames. For MSR-ViRqq and MISR-ViRLV we sample 16 temporal- and 16 spatial- grounded frames. As for our Alternate Self-reflection Train- ing Strategy, the period for alternating training between two LLMs is 200 steps, with the gradient accumulation step set to 16. We conduct 5 epochs of SFT on NExT-QA and STAR for Qwen-VI. LLaVA-NeXT and our MIRRVViR framework, selecting the best model according to the results on vali- dation set. On EgoSchema, we present zero-shot results. We conduct epoch of SFT on a subset of LLaVA-Video 178K (Zhang et al., 2024e) for Dw Vi-VViieee and our corresponding MSR-ViK framework, and present zero-shot results on VideOMME.
ral understanding ability of our method comparing to pre- vious grounding- methods. Besides, for Interaction questions where spatial information is relatively important, our method MSR-ViR, also presents the best performance. Comparing MSR-ViRq and MSR-ViR, with their own di- rect baselines, we prove that our framework help enhance VideoQA abilities of Multimodal LLMs by providing them with most relevant grounded information, ignoring redun- dant information that may impair understanding.
4.3. Experiments on Long-form VideoQA
To further evaluate the performance of our method in com- plex scenarios, we conduct zero-shot experiments on a long- form VideoQA dataset EgoSchema (Mangalam al., 2023). The long videos and complex questions in EgoSchema makes it an ideal benchmark for testing model's abil- ity to accomplish intricate VideoQA tasks. We selected several methods with comparable LLM sizes to ours as baselines, with the results shown in Table 2. Size refers to the number of parameters in the largest LLM utilized in each method. As shown, our method MSR-ViL ViR, achieves the best results both the EgoSchema subset and the full set, demonstrating its superiority. Additionally, compared to their respective direct baselines, both MSR-ViR, and MSR-ViR ViR, show significant improvements, indicating that our method enhances the ability of Multimodal L.Ms to understand long videos and complex questions.
4.2. Experiments on VideoQA
We compare our MSR-ViR framework with existing vision- language models, Multimodal LLMs and grounding- based methods on NExT-QA and STAR-sub. As shown in Table I, MSR-ViR, -VRR, achieves best results the overall NExT-QA and STAR-sub dataset together with most subsets formed by different types of questions. Particularly, MSR-ViR sur- passes TGB and SeV,La, which also utilize grounding- based Multimodal LLMs, on Temporal questions where temporal information essential, demonstrating the superior tempo-
To further demonstrate the superiority of our method, we also conduct experiments on a longer VideoQA dataset VideoMME (Fu et al., 2024a), which contains hour-long videos. Limited by total frame number, Qwen-VI. and VL- are not suitable for this dataset, so we utilize Qwen2-VL and LLaVA-
and the results are demon- strated in Table 3. As shown, both MSR- MSR-ViRg2 and MSR- ViRly significantly outperform the relative baseline. It could
7

---

Modularized Self-Reflected Video Reasoner for Multimodal LLM with Application to Video Question Answering
Table Experiments on NEXT-GQA. VGT. VIOLETv2, Frozen- BiLM, Temp[CLIP). TGB, SeViLa, MSR-ViS (ours)) and MSR- (ours) are finetuned on the training set. Method in gray lines utilize significantly larger LLMss (Palm-2 and GPT-4). Bold num- ber denotes the best result excluding gray methods.
Table 5. Ablation study on NExT-QA and NEXT-GQA, Bold num- ber denotes the best result. SR means self reffeceetc,,,n,, RP means reasoning path, prompts means instruction prompts, m² represents spatial modules and go means global representation of the video.
NEXT-QA
Method
m
IoU @ 0.3
IoU @0.5
Acc @GQA
Tem.
Cau.
Des.
Avg.
MSR- MSR-ViRQ
69.9
73,4
81.5
73.6
VGT
3.0
3.6
1.7
14.4
w/o SR
67.2
72.5
80.5
72.1
V
3.1
4.3
1.3
12.8
w/o m
67.0
72.5
81.4
72.2
Temp|CLIP]
12.1
17.5
8.9
16.0
w/o prompts
68.3
72.4
82.4
72.8
FrozenBiLM
9.6
13.5
6.1
17.5
Jr
66.9
70.1
78.0
70.4
TGB
19.9
23.3
11.2
only w/ gr
63.3
68.4
77.5
68.3
LangRepo
8.7
6.0
11.2
NEXT-GQA
SeViLa
21.7
29.2
13.8
16.6
Acc @ @QA
Ace @GQA
IoU
LLoVi(Mistral-7BB
8.7
6.0
11.2
mIoU
@0.5
LLoVi(GPT-4)
20.0
15.3
24.3
MoPeVQA(Palm-2)
19.7
15.4
39.6
MSR- MSR-ViRQ
69.9
18.5
22.8
16.4
SR
68.3
17.9
22.2
15.7
(
22.8
33.0
16.4
18.5
w/o RP
66.8
14.4
18.5
11.4
ASS-ViRR(ouus"
23.4
33.6
16.4
18.6
ods implemented with significantly larger LLMs (LLoVi with GPT-4 for example) for fair comparison. MSR-ViRo ViRq achieves best results on mloP, while MSR-ViR, achieves best results on mloU, indicating that our method grounds the temporal segment relevant to the question more precisely than existing grounding-based and modular methods. The best result of Ace@ GQA demonstrates that our method can perform VideoQA tasks more effectively, while also pro- viding more reasonable temporal evidence indicating which specific segment of the video the answer derives from.
also be seen that the improvement is more significant longer subsets, where grounding and reasoning are more essential for VideoQA, further demonstrating the superiority of our MSR-ViR framework.
4.4. Experiments on Grounded VideoQA
To further confirm MSR-ViR is capable of more accurately grounding the relevant information thus enhancing VideoQA ability of Multimodal LLMs, we conduct experiments on the NEXT-GQA (Xiao al., 2024) dataset. NEXT-GQA not only contains the answer to the question, but also presents ground-truth time span, indicating where the answer is derived from the video, in other words, the most relevant time period to the question. The dataset re- quires VideoQA models to provide "evidence" of their an- swer, evaluating the grounding accuracy with IoP (Intersec- tion over Prediction) and IoU (Intersection over Union). It also measures @ce@ GQA, which is the proportion of ques- tions that are correctly answered, and the same time IoP between predicted time span and ground-truth time span is larger than 0.5.
4.5. Ablation Study
As demonstrated in Table 5, to further validate the effective- ness of modules and designs in our MSR-V VIR framework, we conduct ablation study on NEXT-QA and NEXT-GQA dataset for MSR- MSR-ViR, concerning the following questions:
Is Alternate Self-reftection Training Strategy necessary? We remove the self-reflection training process, only fine- tuning our Multimodal LLM without training the Question Parser, and the results are shown by w/o self-reflection in Table 5. The average accuracy on NExT-QA declines by 1.5, and accuracy on each subsets decreases to varying degrees. The grounded accuracy as well as loU of temporal ground- ing also decline as shown in experiments on NEXT-GQA. This demonstrates the necessity of Alternate Self-refiection Training Strategy. To further verify the necessity of reason- ing path, we utilize MoST-Grounding to directly provide grounding results based on original questions without rea- soning paths, shown in Table 5 as "woo RP".
We compare our MIR framework with existing grounding-based methods and modular methods, together with vision-language models and Multimodal LLMs which utilizes NG+ method in (Xiao al., 2024) for training. The results are shown in Table 4. Here we only present the results of IoU and Ace@GQA in Table 4, and the com- plete results can be found in Appendix G.I. Methods in the first part are models implemented with NG+, while the second part includes grounding- based methods and the third part contains modular methods. We de-emphasize meth-
Is spatial.localizer localizer necessary in MoST-Grounding mod- ule? Most existing grounding- methods only consider
8

---

Modularized Self-Reflected Video Reasoner for Multimodal LLM with Application to Video Question Answering
Why did the boy lift his hands up above his head nearing the end while turning?
DetectAct( "turning"
TemporatLocalize/ "while
TemporatTruncate
for camera X
LocateObyl' "the lifting hands up")
SpatialTruncate
Temporal grounding result
Spatial grounding result
(a) Without Self-reflection Training
DetectAct( boy lift his hands up}
TemporalBetwoen
TemporalTruncate
Temporailoocalize "the end"")
play with toy
LocateObi("the boy")
SpatialTruncate
(b) With Self-reflection Training
Figure An example of MSR-ViR reasoning process. (a) is the reasoning process of MSR-ViR without self-reflection training. (b) is the reasoning process of MSR-ViR with self-reflection training.
temporal grounding, including all small modules in it, only providing our Mul- timodal LLM with temporal grounding results, denoted by w/o spatial modules. The average accuracy on NExT-QA drops by 1.4, proving that spatial grounding results pro- vided by spatial localizer contain useful information for Multimodal LLM answer the question correctly.
remove the spatial localizer
are presented in Appendix G.4.
Figure 4 demonstrates the reasoning process of AISR-ViB with and without self-reflection training. As shown Fig- ure 4, after self-reflection training, Question Parser gen- erates more reasonable policies accurately grounding the questions in the videos, leading correct answers. More visualization examples can be found in Appendix E.
Are our designs in training Multimodal LLM necessary? In Section 3.3, we introduce two designs for our Multi- modal LLM training: global representation and instruction prompts. We remove these two designs separately and con- duct tests on NEXT-QA. The results show that the average accuracy on NEXT-QA decreases varying degrees for both, indicating that the two designs proposed for train- ing Multimodal I.LMs are effective. We also attempt to make Multimodal LLMs answer questions utilizing only the information from global representation, results of which are shown in Table as "only w/ g...
5. Conclusion
In summary, we propose Modularized Self-Reflected Video Reasoner IMSR-iiY, self-reflected framework that inte- grates a Modularized Spatial-Temporal Grounding (MoST- Grounding) module into Multimodal LLM for inter- pretable VideoQA. Modularization policies generated by Question Parser demonstrates clear reasoning paths en- hancing interpretability of our framework, while spatial- temporal grounding results present visual evidence for an- swers. Through the proposed alternate self-refection train- ing process, policies are gradually refined, becoming more reasonable. Extensive experiments demonstrate that MSR- ViR significantly improves VideoQA capabilities of Multi- modal LLMs while grounding answers in videos more accu- rately. Future work could explore further enhancements to the design of modular network and its execution efficiency.
In addition the above questions, investigate the choice replace UniVTG
of small modules in MoST-Grounding with different grounding models and conduct experiments on the NEXT-GQA dataset, the results of which are presented in Appendix G.3. We also conduct ablation study on frame sampling strategy, and the results
in MoST-Groundie

---

Modularized Self-Reflected Video Reasoner for Multimodal LLM with Application to Video Question Answering
Acknowledgements
Chiang, W.-L.,, Li, Z., Lin, Z.., Sheng, Y., Wu, Z., Zhang, H., Zheng, I.., Zhuang, S., Zhuang. Y., Gonzalez, J. E.,, Stoica, I,, and Xing, E. P. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https: wws.m..og//bll......oo 2023- 01-000003--00000 -i00viina/....
This work supported by National Natural Science Foun- dation of China No. 60.62222209 Beijing National Research Center for Information Science and Technology under Grant
No.BNR2023TD03006.
Choudhury, R., Ninuma, K., Kitani, K. M., and Jeni, L. A. Video question answering with procedural programs. In European Conference on Computer Vision, pp. 315-332. Springer, 2024.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which feel must be specifically highlighted here.
Dai, W., Li. J., Li, D., Tiong, A. M. H., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S. Instructblip: Towards general-purpose purpose vision-language models with instruc- tion tuning, 2023. URL https::/arxxv..orrt /abs/
References
2305 06500.
Ayyubi, H., Liu, J., Asgarov, A., Hakim, Z. I. A., Sarker, N. H., Wang, Z., Tang, C.-W., Alomari, H., Atabuzza- man, M., Lin, X., et al. Enter: Event based interpretable reasoning for videoqa. arXiv preprint 2rXiv:2501.14194 2025.
Di, S. and Xie, W. Grounded anestion--nnswerin long egocentric videos. In Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition, pp. 12934-12943, 2024.
Duan, X., Wang, X., Zhang, Z., and Zhu, W. Parametric visual program induction with function modularization. International Conference on Machine Learning, pp. 5643-5658. PMLR, 2022.
Bai, J., Bai, S., Yang. S., Wang. S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: A frontier large vision- language model with versatile abilities. arXiv preprint arXiv:2308. 12966, 2023.
Fan, Y., Ma, X., Wu, R., Du, Y., Li, J., Gao, Z., and Li, Q. Videoagent: A memory-augmented multimodal agent for video understanding. In European Conference on Computer Vision, pp. 75-92. Springer, 2024.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Fei, H., Wu, S., Ji, W., Zhang, H., Zhang, M., Lee, M.-L., and Hsu. W. ofdeo-of-thoughtt Step-by-step video rea- soning from perception to cognition. In International Conference on Machine Learning, pp. 13109-13125. PMLR, 2024a.
Buch, S., Eyzaguirre, C., Gaidon, A., Wu, J., Fei-Fei, L., and Niebles, C. Revisiting the "video" in video-language understanding. In Proceedings of the IEEE/CVF con- ference on computer vision and pallern recognition, pp. 2917-2927, 2022.
Fei, J., Li, D., Deng, Z., Wang, Z., Liu, G., and Wang, H. Video-ccam: Enhancing video-language understand- ing with causal cross-attenticn masks for short and long videos. arXiv preprint arXiv:2408. 2tn::2408.14222.. 14023, 2024b.
Chen, H., Wang. X., Chen, H., Song, Z., Jia, J.,, and Zhu, W. Grounding- prompter: Prompting Ilm with multimodal in- formation for temporal sentence grounding in long videos. arXiv preprint 172112211..17117, 2023.
Feng, W., Wang, X., Chen, H., Zhang, 7.,, Chen, H., Song, Z., Zhou, Y., Yang, Y., Wu, H., and Zhu, W. Llm4vg: Large language models evaluation for video grounding. arXiv preprint arXiv:23 14206, 2023.
Chen, I.., Wei, X., Li, J., Dong, X., Zhang, P., Zang, Y., Chen, Z., Duan, H., Tang, Z., Yuan, L., et al. Sharegpt4video: Improving video understanding and gen- eration with better captions. Advances in Neural Infor- mation Processing Systems, 37:19472-19495, 2024.
Fu, C., Dai, Y., Luo, Y., Li, L., Ren, S., Zhang, R., Wang, Z., Zhou, C., Shen, Y., Zhang, M., et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal 1lms in video analysis. arXiv preprint arXiv:2405.21075, 2024a.
Cheng, T., Song, I.., Ge, Y., Liu, W., Wang, X., and Shan, Y. Yolo-world: Real-time open- open-vocabulary object detection. In Proceedings of the IEFE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 16901-16911, 2024.
Fu, C., Lin, H., Long, Z., Shen, Y., Zhao M., Zhang, Y., Dong, S., Wang, X., Yin, D., Ma, L., et al. Vita: Towards open open-source interactive omni multimodal Ilm. arXiv preprint arXiv:2408.08: 052/1, 2024b.
10

---

Modularized Self-Reflected Video Reasoner for Multimodal LLM with Application to Video Question Answering
Fu, T.-J., Li, L., Gan, Z., Lin, K., Wang, W. Y., Wang, L.., and Liu, Z.. An empirical study of end-to-end video- language transformers with masked visual modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22898-22909 2023.
Li, K., He, Y., Wang, Y., Li, Y., Wang, W., Luo, P., Wang, Y., Wang, L., and Qiao, Y. Videochat: Chat-centric video understanding. arXiv preprint wrXiv:2305.06355 2023b.
Li, K. Wang, Y., He, Y., Li, Y., Wang, Y., Liu, Y., Wang, Z., Xu, J., Chen, G., Luo, P., et al. Mvbench: A compre- hensive multi-modas video understanding benchmark. In Proceedings of the IEEE//VF Conference on Computer Vision and Pattern Recognition, pp. 22195-22206, 2024a.
Gao, D., Zhou, L., Ji, L., Zhu, L.. Yang, Y., and Shou, M. Z. Mist: Multi-modal iterative ppatial-temporal transformer for long-form video question answering. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 14773-14783, 2023.
Li, Z., Xu, Q. Zhang, D., Song, H., Cai, Y., Qi, Q., Zhou, R., Pan, J., Li, Z.,, Tu, V., et al. Groundinggpt: Language enhanced multi-modal grounding model. ings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 2024b.
Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang. L., and Chen, W. LoRA: Low-rank adap- tation of large language models. In International Confer- ence on Learning Representations, 2022. https: / /openrevieww net / wiw.ummidddnneeeeeeeeeee
Proceed-
Lin, B., Ye, Y., Zhu, B., Cui, J.. Ning. M., Jin, P., and Yuan, L. Video-llave Learning united visual representation by alignment before projection. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 5971-5984, 2024.
Huang, B., Wang, X., Chen, H., Song, Z., and Zhu, W. Vtimellm: Empower Ilm to grasp video moments. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 14271-14280, 2024.
Kahatapitiya, K., Ranasinghe, K., Park, J., and Ryoo, M. S. Language repository for long video understanding. In Workshop on Video- LadeuaLa Models @ NeurlPS 2024, 2025. URL https: wwwp:ppwweeeeeeeeeeeeeee forum? id=CTaWnveTNz.
Lin, K. Q.. Zhang, P., Chen, J., Pramanick, S., Gao, D., Wang, A. J., Yan, R., and Shou, M. Z. Univtg: To- wards unified video- Lalouala temporal grounding. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2794-2804, 2023.
Khademi, M., Yang, Z., Frujeri, F., and Zhu, C. Mm- reasoner: A multi-modal knowledge-aware framework for knowledge-basse -ased visual question answering. In Find- ings of the Association for Computational Linguistics: EMNLP 2023, pp. 6571-6581, 2023.
Liu, Y., He, J., Li, W., Kim, J., Wei, D., Pfister, H., and Chen, C. W. R2 RR-tunnin:: Efficient image-to-video transfer learning for video temporal grounding. In European Conference on Computer Vision, pp. 421-438. Springer, 2024.
Lan, X., Yuan, Y., Wang, X., Wang, Z., and Zhu, W. A sur- vey on temporal sentence grounding in videos. ACM Transactions on Multimedia Computing, Communica- tions and Applications, 19(2):1-33, 2023.
Maaz, M., Rasheed, H., Khan, S., and Khan, F. Video- chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12585-12602, 2024.
Lei, J., Berg, T. L.,, and Bansal, M. Detecting moments and highlights in videos via natural language queries. Advances in Neural Information Processing Systems, 34: 11846-11858, 2021.
Mangalam, K., Akshulakov, R., and Malik, J. Egoschema: A diagnostic benchmark for very long-fore forg-form video lan- guage understanding. Advances in Neural Information Processing Systems, 36:462122462444 2023.
Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., and Li, C. "LaVA-onevision Easy visual task transfer. Transac- tions on Machine Learning Research, 2025. ISSN 2835- 8856. URL https:// http:///pperreeeeeeeee n forum? id=zKv8qULVGn.
Min, J., Buch, S., Nagrani. A., Cho, M., and Schmid, C. Morevqa: Exploring modular reasoning models for video question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13235-13245, 2024.
Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: bootstrap- ping language-image pre-training with frozen image en- coders and large language models. In Proceedings of the 40th International Conference on Machine Learning, pp. 19730-19742, 2023a.
Nie, J., Wang, X., Hou, R., Li, G., Chen, H., and Zhu, W. Dynamic spatio-temporal graph reasoning for videoqa with self superviid event recognition. IEEE Transac- tions on Image Processing, 2024.

---

Modularized Self-Reflected Video Reasoner for Multimodal LLM with Application to Video Question Answering
Qian, L.,, Li, J., Wu, Y., Ye. Y., Fei, H., Chua, T.-S., Zhuang, Y., and Tang, S. Momentor: Advancing video large language model with fine-grained temporal reasoning. In International Conference on Machine Learning, pp. 41340-41356. PMLR, 2024a.
Wang. H., Lai, C., Sun, Y., and Ge, W. Weakly supervised gaussian contrastive grounding with large multimodal models for video question answering. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 5289-5298, 2024a
Qian, R., Dong, X., Zhang, P., Zang, Y., Ding, S., Lin, D., and Wang, J. Streaming long video understanding with large language models. Advances in Neural Information Processing Systems, 37:1 19336-119360 2024b.
Wang. P., Bai, S., Tan, S., Wang, S., Fan, Z.., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al. Qwen2-vl: vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint (rXiv:2409.12199 2024b.
Qian, Z.., Wang, X., Duan, X., Chen, H., and Zhu, W. Dy- namic spatio-temporal modular network for video ques- tion answering. In Proceedings of the 30th ACM Interna- tional Conference on Multimedia, pp. 14466-4477, 2022.
Wang, X., Zhang, Y., Zohar, O., and Yeung-Levy, S. Videoa- gent: Long-fore form video understanding with large language model as agent. In European Conference on Computer Vision, pp. 58 38-76. Springer, 2024c.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.
Wang. X., Li, H., Zhang, Z., Chen, H., and Zhu, W. Mod- ular machine learning: An indispensable path towards new-generation large language models. arXiv preprint arXiv:2504.20020, 2025.
PMLR, 2021.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C.D Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
Wang. Y., Li, K., Li, Y., He, Y., Huang, B., Zhao, Z., Zhang, H., Xu, J., Liu, Y., Wang, Z., et al. Internvideo: General video foundation models via generative and discrimina- tive learning, arXiv preprint arXiv:2212.03/91, 2022.
Ren, S., Yao, I.., Li, S., Sun, X., and Hou, L. Timechat: A time-sensitive multimodal large language model for long video understanding. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14313-143233 2024.
Wang, Y., Meng, X., Liang, J., Wang, Y., Liu, Q., and Zhao, D. Hawkeye: Training video-text llms for grounding text in videos. arXiv preprint 243. 2024d.
Wang, Y., Wang. Y., Chen, K., and Zhao, D. Stair: spatial- temporal reasoning with auditable intermediate results for video question answering. In Proceedings of the Thirty- Eighth AAAI Conference Artificial Intelligence and Thirty-Sixth Conference Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Ed- ucational Advances in Artificial Intelligence, pp. 19215- 19223, 2024c.
Rennie, S. J., Marcheret, E., Mrouch, Y., Ross, J., and Goel, V. Self-critical sequence training for image captioning. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 7008-7024, 2017.
Shi, Y., Di, S., Chen, Q., and Xie, W. Unlocking video- Ilm via agent-of-thoughss distillation. arXiv preprint arXiv:2412.0166 2024.
Wang. Y., Wang, Y., WU, P., Liang, J., Zhao, D., Liu, Y., and Zheng, Z. Efficient temporal extrapolation of multi- modal large language models with temporal grounding bridge. In Proceedings of the 2024 Conference on Em- pirical Methods in Natural Language Processing, pp. 9972-9987, 2024f.
Song, E., Chai, W., Wang, G., Zhang, Y., Zhou, H., Wu, F. Chi, H., Guo, X., Ye, T., Zhang, Y., et al. Moviechat: From dense token to sparse memory for long video under- standing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 18221- 18232, 2024.
Wang, Z., Yu, S., Stengel-Eskin, E., Yoon, J., Cheng, F., Bertasius, G., and Bansal, M. Videotree: Adaptive tree- based video representation for I1m reasoning on long videos. arXiv preprint arXiv:2405. 1at.:2405.99999 2024g.
Surís, D., Menon, S., and Vondrick, C. Vipergpt: Visual inference via python execution for reasoning. In Pro- ceedings of the IEEE/CVF International Conference Computer Vision, pp. 11888 11898, 2023.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F. et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302. 139711 2023.
Wu, B., Yu, S., Chen, Z., Tenenbaum, J. B., and Gan, C. Star: A benchmark for situated reasoning in real-world videos. In Annual Conference on Neural Information Processing Systems, 2021.
12

---

Modularized Self-Reflected Video Reasoner for Multimodal LLM with Application to Video Question Answering
Xiao, J.. Shang, X., Yao, A., and Chua, T.-S. Next-qa: Next phase of question-answering explaining temporal actions. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9777-9786, 2021.
Zhang, P., Zhang, K., Li, B., Zeng, G., Yang. J., Zhang, Y., Wang, Z., Tan, H., Li, C., and Liu, Z. Long con- text transfer from language to vision. arXiv preprint arXiv:2406. 16852, 2024b.
Zhang, Y., Li, B., Liu, h., Lee, Y. j.. Gui, L., Fu, D., Feng, J., Liu, Z., and Li, C. Llava-next: A strong zero-shot video understanding model, April 2024c. URL htttp:/://aaaaall.ithhia io/ blog/2024- -ext
Xiao, J., Zhou, P., Chua, T.-S., and Yan, Video graph transformer for video question answering. In European Conference Computer Vision, pp. 39-58. Springer, 2022.
Xiao, J., Zhou, P., Yao, A., Li, Y., Hong, R., Yan, S., and Chua, T.-S. Contrastive video question answering via video graph transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Zhang, Y., Wang. X., Chen, H., Fan, J., Wen, W., Xue, H., Mei. H., and Zhu, W. Large language model with curriculum reasoning for visual concept recognition. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 6269-6280, 2024d.
Xiao, J. Yao, A., Li, Y., and Chua. T.-S. Can i trust your answer? visually grounded video question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13204-13214, 2024.
Zhang, Y., J., W., Li, B., Ma, Z., Liu, Z., and Li, C. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713,, 2024e.
Yang. A., Miech, A.. Sivic. J., Laptev, I., and Schmid, C. Zero-shot video question answering via frozen bidirec- tional language models. Advances in Neural Information Processing Systems, 15:124-141 2022.
Zhao, Y., Huang, J., Hu, J., Wang, X., Mao, Y., Zhang, D., Jiang, Z.,, Wu, Z., Ai, B., Wang, et Swift: a scalable lightweight infrastructure for fine-tuning, In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 29733-29735, 2025.
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., et al. Qwen2 technical report. arXiv preprint a4Xiv:2407. 10671, 2024.
Zhong, Y., Hu, Z.-Y., Lyu, M., and Wang, L. Beyond embed- dings: The promise of visual table in visual reasoning. In Proceedings of the 2024 Conference on Empirical Meth- ods in Natural Language Processing, pp. 6876-6911, 2024.
Yao, Y., Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., Cai, T., Li, H., Zhao, W., He. Z., et al. Miniepm-v: A gpt-4v level milm on your phone. arXiv preprint arXiv:2408. 01800, 2024.
Ye, Q.. Xu, G., Yan, M., Xu, H., Qian, Q., Zhang, J., and Huang, F. Hitea: Hierarchical temporal-aware video- language pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15405- 15416, 2023.
Yu, S. Cho, J., Yadav, P., and Bansal, M. Self-chained image-languas model for video localization and question answering. Advances in Neural Information Processing Systems, 36, 2024.
Zhang, C., Lu, T., Islam, M. M., Wang, Z., Yu, S., Bansal, M., and Bertasius, G. A simple IIm framework for long- range video question-anse annwering......ng...... In Proceedings of the 2024 Conference on Empirical Methods Natural Lan- guage Processing, pp. 21715-21737, 2024a.
Zhang, H., Li, X., and Bing, I.. Video-Ilama: An instruction- tuned audio-visual language model for video understand- ing. In Proceedings of the 2023 Conference on Empir- ical Methods in Natural Language Processing: System Demonstrations, pp. 553-553,100 2023.
13

---

Modularized Self-Reflected Video Reasoner for Multimodal LLM with Application to Video Question Answering
A. Other Related Works
Video Grounding with Multimodal LLM. In order to explore whether Multimodal LLMs are capable of understanding the temporal information in videos, some works have attempted to study the temporal perception ability of Multimodal LLMs for tasks like video grounding (Lan et al., 2023). For example, VT (Huang et al., 2024) introduces boundary perception training process on multi-event datasets, improving the performance of Multimodal LLMs tasks such as video grounding and dense video captioning. HawkEye (Wang et al., 2024d) constructs a large-scale video-text corpus with regment-level captions and negative spans, on which a coarse-grained segment representation method and recursive grounding strategy are utilized to train Multimodal LLMs. Other works (Ren et al., 2024; Qian et al., 2024a; Li et al., 2024b; Chen et al., 2023; Feng et al., 2023) also explore the grounding ability of Multimodal LLMs.
Visual/Video Question Answering with Modular Method. The modular approach (Wang et al., 2025), which decomposes complex questions into multiple sub-questions and solves each of them through dedicated models, has been attempted in visual/video question answering tasks (Qian et al., 2022; Duan et al., 2022; Nie et al., 2024; Zhang et al., 2024d; Zhong et al., 2024; Khademi et al., 2023; Wang et al., 2024e; Fan et al., 2024; Choudhury et al., 2024; Shi et al., 2024; Ayyubi et al., 2025; Fei et al., 2024a). Specifically, MM-Reasoner (Khademi et al., 2023) leverages vision APIS and LLMs to extract and utilize query-specific knowledge for visual question answering. (Zhong et al., 2024) introduces Visual Table, a novel visual representation that provides detailed object descriptions and knowledge in structured text, significantly boosting performance in visual reasoning tasks. ViQ (Choudhury et al., 2024) and ENTER (Ayyubi al., 2025) utilize LLM to generate executable programs, leveraging external tools for modular video question answering. VideoAgent (Fan et al., 2024) and MotionEpic (Fei al., 2024a) utilize Meaii--of-Thuugee which involves multi-round conversation with LLM leveraging several expert small models, while AoTD distills knowledge from CoT into Video LLMs to improve instruction tuning. Different from existing methods, MSR-ViR is the first framework that not only integrates a modular network into a Multimodal LLM but also jointly optimizes them with self-reflection training for reasoning path refinement and QA accuracy improvement.
B. Prompt for Question Parser
We carefully design prompt for our Question Parser to generate policies from given questions from in-context learning. The complete prompt is presented in Figure 5, Figure 6, Figure 7. We first inform the Question Parser of some basic information and an introduction to the functions of each module. Then we tell it the general template of the policy and several variants under special circumstances. Finally, through a few examples, we teach the Question Parser how to generate a policy based on the question.
C. Module Implementation
Table Illustration of our modules. Type is the type of the module, where T represents modules temporal. localizer and S represents modules in localize. Vision-Language Model denotes the small vision-language model use in the module. As for the input and output, V denotes the input video, / denotes the video frame, TS denotes a time period, BBOX denotes a bounding box, Qa denotes the action query, Qo denotes the object query, p preposition representing certain temporal relationship and is word phrase representing certain spatial relationship.
Module Name
Vision-Langusae Model
Type
Input
Ouput
DetectAct
T
UniVTG
V, Qu
TS TS(ou))
TemporalLocalize
T
p. (
TemporalBetween
T
TS1, TS2 V.TS
(
TemporalTruncaa
T
I2,
LocateOhj
S
YOLO-Worke YOLO-World
I, Q,Qo
BBOX BBOX (out)
SpatialLocalize
S
I. (B0 1, B BBOX,, BBOX,,
Spatial/rrunaat
S
kout
All small modules in our MoST-Grounding module are listed in Table 6, the detailed implementation of which are as follows:
14

---

Modularized Self-Reflecte Video Reasoner for Multimodal LLM with Application to Video Question Answering
You question parser. You decompose complex questions into smaller and by doing answer the question. The question is given together with a video clip, and now hope to temporally and spatially ground the question the video clip before answering The question perser need decompose the question into several modules, and by solving each module step by step, the question can be grounded in the video. Now have and only have these modules:
easier
detect act(video (ivivve__eahhh,...__e_e act query): This module works as temporal localizer. Provided with the path video and query of action, the module will temporally locate query into the video, and return timespan: [start time, end _ttmee;
2. temporai_ locpriiieecaiieeepepp.... act time): This module locate timespan in the video. act time usually the return value of detect_ act, which looks like [start time, end time), and prep preposition ("before", "when". "after" or something like that). The module determine the final timespan according the preposition and the act time, also returning in the format [start_ time, end_time]:
3. temporal between(act time1, act time2): This module actually special form of temporal localize, where the preposition is "between". act time1 and act time2 are two timespans detected by "detect actt" module, and this module locate the timespan between the two actions, resulting in returning [start time, end. time] well.
temporal truncate(video, path, tp_rees resultt; This module truncate clip of video according to the provided timespan result: [start time, end_ time]. The module samples 2 frames between the start time of the video and the end time of the video, thus returning list of frames: [frame1, frame2]:
5. locate obj(image, obi_ query): This module works spatial localizer, which spatially locate area: [x1, y1, x2, yZ].
the image
according to the provided obj_ query, returning a bounding box
obj_query,
6. spatial_ localize(image, rel, ressett:::: This module locate specific area in an image. resultt the return value of module focate_obf which bounding box, and is verb or preposition that describes the relationship between the expected result and the given sp_result. The return of this module is the expected result that also bounding box. spatial, truncate(lmage, sp_result): This module truncate the bounding box sp_ resuretuttt the provided Image, resize the bounding box area to the same size of the original image, and returning the resized image. Among these modules, 2, 3,4 temporal modules 5, 6, are spatial modules. What you provide policy according to the given question using the above 7 modules, pollcy always has two parts:
do is to
temporal_joo localizer and spatial localizer, each utilizing one some Generally speaking, reasonable policy will be format as follows:
above modules.
"eemporl_locallzer"" localizer":
"module": "temporal_trunent truncate",
"params": [[
"module": "temporal "amppoat_localize"
"params": [$(a prepasition), (" ("module": "ddtecttact", "params": (5(description of an event)]]]
)
spatiaal "spatiallocaalizer"" l
"module": "spatial_truncate""
"params":
"module": "spatial_localize"
localize",
"params": [$(a relationship word), ("module": "locate_obi", "params": [S(description object))))
)]
within $() you consider carefully. They necessarily directly origin question: sometimes you need to think deep into the question and try to think of reasonable sentence or word to fill in $0) part. temporal that needs specifically located, just simplify the
you
{
"temporal_ Jocalizer":
cemmporalllccalizer"
"module": "temporal truncate"
"params":
"spatial_ localizer":
"module": "spatial_ truncate".
"params": [[
"module": "spatial localize",
"spattal_locallze.
"params": ($(a relationship word), ("module": "locate_obj"" "params": (S(description objecti]]]
}
Figure 5. Prompt for Question Parser (Part I).
15

---

Modularized Self-Reflecte Video Reasoner for Multimodal LLM with Application to Video Question Answering
Also, you think there spatial that needs specifically located, just simplify the format
{
"module": taapporrattuuucate."
"params": I
"module": "temporal "temporal_localiee".
localize",
"params": [$(a preposition), ("module": "detect_act", "params": (S(description of an event))))
)
"spatial_lowe localizer":
"module": sspatial_truncatee
"params":
you think there are more than one area to be spatially located, you can add params to the "spatial "spatial_truncate" module:
campooral_ocallzerre {
"module": "temporal "ariporrr_tuuucate"""
"params":
"module": "temporal_locallli localize",
"params": ($(a preposition), ("module": "detect_act", "params": [S(description of event)])]
)]
"spatial_ localizer";
"module": "spatial_ "spatialltrunnatt",
"params": I[
"module": "spatial_locar localize",
"params": [$(a relationship word), ("module": "locate_obj" "params": [S(description of an
object A)]}] }
"module": localill,,,
"params": [š(a relationship word), ("module": "locate_oke _obj",, "params": [S(description of an
object B)])] )]
}
Except what I have mentioned above, you are not allowed to change the format the policy other strange ways.
Here examples:
1.question: What is the boy holding after his mother entering the room?
policy:
"module": leeeppora_ttunccat
"params":[[
"module": "temporal_locaa localizee"
"params": ["after", ("module": "detect_act", "params": ["a woman enters the room") "]]]
}}
"spatial_iow localizer":
"module": "spatial_truncatt"
"params":
"module": "spatial_localiz""
"params": ["holding",, ("module": "locate_obj"" "params": ["a boy"]]]
}]
Figure Prompt for Question Parser (Part IT).
16

---

Modularized Self-Reflected Video Reasoner for Multimodal LLM with Application to Video Question Answering
2.question: Why is the girl crying?
policy:
"lemporallocalizee"
localizer": {
"module": "temppral_ truncate",
"params" [K
"module": Campporl__ocallinnn
"params": ("when", ("whhen"""wwhn" (wwhhen,("mmdule"""
("module": "detect_act", "params": ["a girl crying "[]]
"spatial_loce" localizer":
"module": appatial_tuunctte
"params":
"module": "sspttal_locallize
("srrosnn"" ("module". "locate _ojite__""" "params": ["a girl"])]
3.question: What is the animal on the left the farmer?
policy:
"temporal_localieer" locallizerlllir
"module": campporal_truncatee
"params":0
"spatial_ localizer"::
"module": "rannatetru
"params":
"module":
"params": ["left
(""eft of", ("module": "locate_obj", , "params": ("a farmer"]]
4.question: Which object the person put down before they took the sandwich?
policy:
"temporal localizer":
Nemporaa_localizer""
"module":
"params":
"module": "temporal_ ".aaaiiaa_occaizee..
"params": ["before", ("module": "detect_act" "params": ["the person took the sandwich") "]}]
""mofure,""mmduuee"..
):
"spatial_locallzer"" localizer":
"module": "sspatia__truncate
"params":
"module": "spatial_locale localize",
"params":
("putting down", ("module": "locate_ob) "params": ["the person
Expand your thinking, and don't be confined to the words and phrases in the original question. Remember that your goal in decomposing the question is to temporally and spatially locate it within the video. Therefore, feel free to think creatively about how breaking down the question might help find the parts of the video that are truly relevant to the question. Based on the original question, you can make some reasonable extensions and provide appropriate policies.
Note that for the "Why" type question, It is always not enough to just locate the event and the object mentioned the origin question. You can try to think about what events and objects might related the question try to locate them.
Now give you question, and you will give me corresponding policy.
Figure 7. Prompt for Question Parser (Part III).
17

---

Modularized Self-Reflected Video Reasoner for Multimodal LLM with Application to Video Question Answering
DetectAct. Define the UniVTG model Mr, text encoder as E,, video encoder Eg. V (V1, 02,--y or is input video with T frames sampled lfps, and Qa is query desribing action. We have:
TS (V((E.(),,E((0.)))
where TS L/ Le] represents time period.
Temporallocalie p is preposition representing certain temporal relationship. TS = 11 .te (6)))) is an input time span. We have:
TS(m))
E {when, while, as}
TS(out)
())) min (2t( (( T)}, mmxx(0, (0, 2 (()))) (()))
p E (after)
(8)
p E {before}
where T'(luut) [t!out) ( {lout) is an output time span. T' is the video duration.
TemporalBetween. Given two imput time spans TS1 tiel and TS2 [t2er tzel have:
Tg(out) [min(( (2,), max (20)]
(9)
where 75(out) = [y(out) (out)] is an output time span.
=14,, Temporal Trrncar,.runaate.. Given an input video V (vi, U2, u2, DT) and a time span TS = [/,,44) Le]. define 8 = 14,,. e= [1e] We get / = (4, L2, ..),, where:
1 V(s (o-all-1))
(10)
and denotes the number of sampled frames.
jocateobj. Given the YOL-O-World model Ms, input image J, query of an object Qo. and an image encoder F,
together with a text encoder ET,, we have:
BBOX = Ex
(11)
where BBOX- = (Ii, 02, W/) is output bounding box.
SpatialLocalize Given an input image 1, an input bounding box BBOX = (I1, 81, 22, U2) and a word or phrase representing certain spatial relationship r, we have:
maax(0, 2x x2), 11, 21 y/2,
p E (left)
[22, M1, min(w, 20g 11), V2l.
P E (right)
[21. U2, 22, min (h, 2y/2 y1 y1)l.
p E S
XBOX (out)
(12)
[2), max(0, 2y 1/2, 22, vi],
p E Sup
[max(0, 2r: - x22), max (0, 2y, - y22,
min (w. 2x x1 min(h, 2y² 91)l.
p E Ssurownd
where Saown (bottom, down, below, under, beneath, sit on, stand on, lying on). Sup = (top, above, up, carry, lift, on),
Ssuround = [next to, beside, near, surround). Particularly, if E (hold, touch, contact, take), we have:
BBOX = SpatiaalLocalizee[IBBBB BBOX (hand) "surround"),
(13)
where BBOX (hand) = slocateOjj(A. "hand").
SpatialTruncate. Given an input image and a list bounding boxes BBOX,, BBOX,,... where BBOX, (Z41. 442 Vi2). we have:
flout) RESTZE, 251, min Wig. max 2:2; max Yi2));
(14)
where RFSTTZF,(T) is the operation that resizes an image I' into the shape of
18

---

Modularized Self-Reflected Video Reasoner for Multimodal LLM with Application to Video Question Answering
D. Detailed Alternate Self-reflection Training Strategy
The detail our Alternate Self-reflection Training Strategy is demonstrated Algorithm 1.
Algorithm 1 Altemnaee Self-reflection Training Strategy
1:
Input: Question parser Q, Moounngg module M, temporal localizer Ft. spatial localizer F.. modules in
temporal localizer {m]] 14 1=11 modules spatial localizer {m?)- Multimodal LLM F, instruction prompt Pt,
dataset D {Vv, qi 1/ J ==1 total training steps S, gradient accumulate step s, alternate training period P
2: Initialize: Q, M, CACHE
3: Freeze: M. [m]] {m]). Q, Activate: F
4: for t = ,,..., do
5: for j = 8 do
6: i < ((t - 1)s jj 1)% + 1, Prepare data (U), ges ys), Derive global representation 9r.
7:
Generate policy p Q(q:)
8:
Set Ftlp - (0:i:())(((m())))))) ((
( -(mm),(m)))) p) (+), derive ot Cg from p
9: M execution: vs, b. = =M(u,,
Cr, C.) = =Fs((
10: F forward propagation: i(a; uz) F(P(( Us. bw..
11:
Optimize F with loss: Lce in Equation (5)
Add (vi, qús Us) to CACHE
end for
14:
if t%P = then
15:
Freeze F, activate Q, initialize Q, Tere Q
16:
for i 1...., sP do
17:
Prepare data (vi, qs, (s,3)) v:)
18:
Generate policies P1, P2 = ((1))
Forward propagation to get LCE1 fCE2 for Pt. P2 respectively
if LCEI < LcE2 then
21:
Pio < -1, Pt p2
22:
else
23:
Pwo < P2, + P1
24:
end
25:
Optimize No with loss: Lroo in Equation (6)
26: end for
Q + no. clear CACHE, freeze Q, activate F
28: end if
29:
end for
E. More Visualization Examples of MSR-ViR Reasoning
Here we present some more visualization results on inference examples of our MSR-ViR framework in Figure 8 and Figure 9.
F. Multimodal LLM Input Format
The specific input of our Multimodal LLM illustrated in Figure 10 together with our instruction prompt. Global representation tokens are encoded and aligned global video representation g.- Similarly, lemporal-grounderr video tokens and spatial-grounded video tokens are encoded and aligned video segments v, and bounding boxes b.. respectively. Special tokens glob, tp and sp designed help the Multimodal LLM understand different types of tokens.
19

---

Modularized Self-Reflected Video Reasoner for Multimodal LLM with Application to Video Question Answering
Q: Why does the child pass the toy back to the person in black after holding it and walking?
DetectAct( 'child hold the toy')
TemporalBetween
TemporalTruncate
"child pass toy to the person")
want to start playing with toy
X
LocateObji "child")
SpatialTruncate
(a) Without Self-reflection Training
DetectAct( toy and walk")
Temporail.ccalize"aafii "after"""
TemporafTruncate
LocateObi(("chile
SpatialTruncate
do hold
grounding result
Spatial grounding result
(b) With Self-reflection Training
Figure MSR-ViE inference example 1.
how does the man parachuting control the parachute?
DetectAct( 'the man perachuting )
Temporall "when
TemporalTruncate
X
using the sticks
LocateOliji "the parachute")
SpatialTruncate
Without Self-reflection Training
DetactAct( man parachuting"
TemporalLocallzon "whilee)
Temporal
Dbj] parachuting
SpatialTruncate
tuga the string
Temporal grounding result
Spetial grounding result
(b) With Self-reflection Training
Figure MSR-ViR inference example 2.
20

---

Modularized Self-Reflecte Video Reasoner for Multimodal LLM with Application to Video Question Answering
<glob>
</glob>
<p>
</tp>
</sp>
Global representation token
Temporal- video token
Spalial-groundde video token
[Instruction Prompt]
Question token
This is video. The first two images you see contain some global information about the video. The third and fourth images you see are from clip of the video that might be related to the question will give you later. The last two images you see are small areas the video frames that might contain information answer my question. need to evaluate information from these images critically and answer my question.
Figure Specific input of our Multimodal LLM instruction prompt.
G. More Experiments
G.1. Complete Results NEXT-GQA
Table 7 presents the complete results NExT-GQA dataset. Our framework MSR-ViR achieves best results on IoP, IoU and Acc@@ GQA.
Table Complete results of the experiments on NEXT-GQA. Bold number denotes the best result excluding gray methods.
Method
mloP
I.P@0.3
IoP@0.5
mIoU
IoU@0.3
loU@0.5
Acc@GQA
VGT
25.3
26.4
25.3
3.0
3.6
1.7
14.4
VIOLETv2
23.6
25.1
23.3
3.1
4.3
1.3
12.8
Temp[CLIP]
25.7
31.4
25.5
12.1
17.5
8.9
16.0
FrozenBiLM
24.2
28.5
23.7
9.6
13.5
6.1
17.5
TGB
19.9
23.3
11.2
LangRepo
20.3
20.0
8.7
6.0
11.2
SeViLa
29.5
34.7
22.9
21.7
29.2
13.8
16.6
LLoVi(Mistral-7BB
20.7
20.5
8.7
6.0
11.2
LLoVi(GPT-4)
37.3
36.9
20.0
15.3
24.3
MoReVQA(Palm-2)
37.8
37.6
19.7
15.4
39.6
(SSsViRggours)
30.0
39.8
25.0
22.8
33.0
16.4
18.5
MSR-ViR
29.6
39.0
24.1
23.4
33.6
16.4
18.6
G.2. Experiments on Computational Efficiency of MSR-ViR
For further comparison between our MSR-ViL ViR framework and baselines, including end-to-end Multimodal LLMs and other grounding- based methods, we conduct experiments on NExT-QA test their inference speed, and the results, together with parameter size and accuracy on NEXT-QA, are demonstrated in Table 8. The inference speed is tested on one NVIDIA A100 GPU. Our framework's additional parameters mainly stem from the Question Parser, and MoST-Groundee than 0. IB. For comprehensive comparison, test total parameters, inference speed, and accuracy using Question Parsers of different sizes (Qwen2-7B and Qwen2-1.5B) With Qwen2-7B Question Parser, the inference speed of MSR- ViR is about twice that the direct baseline, consistent with our complexity estimates. With Qwen2- Question Parser, although accuracy slightly drops, still outperforms the direct baseline with fewer additional parameters and less computational overhead.
contributes less
21

---

Modularized Self-Reflected Video Reasoner for Multimodal LLM with Application to Vidco Question Answering
Table Model size and inference speed on NExT-QA dataset. denotes the method is designed for high-efficiency inference.
Method
Parameter Size
Inference Speed
Acc on NExT-QA
BLIP-2
4.1B
1.21 s/ sample
69.6
TGB*
4.3B
1.62 s/ sample
72.1
InstructBLIP
7.9B
1.75 s/ sample
72.5
SeViLa
8.3B
2.79 s/ sample
73.8
Qwen-VL
9.6B
1.32 sample
71.9
MSR-Vin (5R-ViR parser)
11.2B
2.35 sample
73.1
MSR-ViRg parser)
16.7B
3.10 / sample
73.6
LLaVA-NeXT
7.1B
2.19 s/ sample
73.1
MSR-ViR, (1.5B parser)
8.7B
4.29 s/ sample
74.2
MSR-ViR, parser)
14.2B
4.96 s/ sample
74.9
G.3. Ablation Study on Grounding Model
In MoST-Grouue SooT-GGrun module, we utilize small grounding model UniVTG as our temporal grounding module. To demonstrate the effectiveness of UniVTG, we further conduct ablation study the choice of temporal grounding model. We utilize R--Tuning (Liu et al., 2024) and Moment-DETR (Lei et al., 2021) replace UniVTG and test NEXT-GQA dataset, and the results are shown in Table 9. MSR-ViRo with UniVTG achieves the best results NEXT-GQA.
Table Ablation study for temporal grounding models on NEXT-GQA. This the test result of MSR-ViRe ViRg with different temporal grounding models UniVTG, Rt2 Tuning and Moment-DETK
Grounding Model
Acc@ QA
Acc@GQA
mloP
IoP@0.5
mIoU
1.5
UniVTG
69.9
18.5
30.0
25.0
22.8
16.4
1e-Tunnng
67.3
16.6
28.7
23.2
22.7
15.9
Moment-DETR
67.4
17.2
28.6
24.1
21.4
14.7
G.4. Ablation Study on Frame Sampling Strategy
In Section 4.1, we provide detailed implementations MSR- ViR and its direct baseline, including the leveraged frame sampling strategy. Here we present further ablation study on frame sampling strategy for Qwen- VLen-VL., LLaVA-NeXT, MSR- ViRo and MSR-ViR, ViR, on NEXT-QA dataset to show that our implementations are optimized for each model. The results are shown in Figure 11. We utilize the sampling strategy with the highest accuracy on NExT-QA. It could also seen that with the same number of sampled frames, MSR-ViR significantly outperforms its direct baseline, further demonstrating the effectiveness of our proposed framework.
H. Upper Bound Computational Complexity of MSR-ViR
To enhance interpretability of Multimodal LLMs, MSR- ViR incorporates several additional models (as illustrated in Sec- tion 3.1 and Appendix C), thereby elevating the computational complexity the reasoning framework. For comprehensive evaluation, we furnish the upper bound computational complexity of our reasoning framework MSR-ViRQ. With this upper bound, we demonstrate that the computational complexity of our reasoning framework lies within reasonable and controllable range. As transformers are widely utilized in our framework, we first furnish the proof of the following lemma:
Lemma H.l. For a given transformer with Nr, layers, hidden size d, intermediate size d (in feed forward networks) and a given input with I tokens, the computational complexity of inference is O (Nr (4d2 2dl² (d))
Proof. The computational time bottlenecks a transformer lie in the multi-head self-attention and feed-forward network, so we calculate the computational complexity of these two components respectively.
22

---

Modularized Self-Reflected Video Reasoner for Multimodal LLM with Application to Video Question Answering
Qwen-Vl MER-VIRg
LLAVA-ROTT
2-272
MSR-VR,
74-
3-2/2
72-
61412
Accuracy on NEXT-QA
71-
73
Accuncy on
20-
72
69-
68-
2
2
8 10
14
16
38
20
Total Number of mample Frames
Total Number of Sampled Frames
(a) Qwen-VI. VL and MSR- MSR-ViRg-
(b) LLaVA-NeXT and MSR- ViRL.
Figure Ablation study on frame sampling strategy on NEXT-QA, For MSRRViR, the number in the figures represents (temporal- grounded frames spatial-grounded frames global representation frames).
In multi-head self-attention, the calculation of the Q, K. V matrices is required first, which involves matrix multiplication between a matrix of shape x d and matrix of shape d d. So the computational complexity of this process O (3d21).
Next, assume the number of heads in multi-head self-attention h. So the dimension of each attention head will be dg d/h. For each attention head:
QQT Vlg
Attention( K, V) = softmax
V.
So the computational complexity of one attention head is 12 + 12d, = 2/2 For h heads, the overall computational complexity will be 2/2 x h 2/2 The final linear layer involves the matrix multiplication between matrix of shape I d and matrix of shape d d, thus the computational complexity is Id? Overall, the computational complexity of multi-head self-attention is 3d + 2dl² + dil 4d + 2d1
The feed-forward network involves two linear layers, the hidden size of which is dyz, so the computational complexity of feed-forward network is 2dd, fl. With N₂ layers, the computational complexity will be multiplied by NL. Up to this point, we have proved Lemma H.l: the computational complexity of a transformer is (NL (4d1 + 2d12 + 2d)
Next, we will calculate the computational complexity each modules in MSR- ViRo. For the Question Parser Qwen2-7B: Proposition H.2. Given an input with tokens, the computational complexity of Qwen2-7B is O (P,12). where P1 is constant relevant to parameters of Qwen2-7B.
Proof. Trivially, the computational bottlenecks of Qwen2-7B lie in transformers. According to Lemma H.1, the com- putational complexity of Qwen2-7B is O (Nz (4d2 2d12 ()) Considering d, Nr, and djf as constants and eliminating low-order terms, the complexity expression of Qwen2-7B could be simplified as O (P,12).
Proposition H.3. Given a video with length T and resolution x W, and a query with ly tokens, the computational complexity of UniVTG is O (PTTH"W2 + P + Palg + P,Tl,), where Pr. Ps, PA and Ps are constants relevant to parameters of UniVTG.
Proof. Uni incorporates CLIP model, which involves visual encoder and text encoder, encode the input videos and queries. Then a multimodal encoder is utilized for cross-modal interaction. All sub-modules of UniVTG are based on transformers. We will calculate the computational complexity of the three encoders respectively. Assume the parameters of transformers in the visual encoder are dy, byddf the parameters of transformers in the text encoder are dr, Lr, dyfr and the parameters transformers in the multimodal fusion encoder are dy, Lp, dsfr- (The parameters of transformers are as illustrated in Lemma H.
23

---

Modularized Self-Reflected Video Reasoner for Multimodal LLM with Application to Vidco Question Answering
For the visual encoder, the image is first input into a convolution layer Conv2d(3, dy, kemneLsize==s, s), stride=(s, 8)). The complexity of this step would be HW x² 8 x x 3 x dy 3dy HW. Then the encoded image of shape #W x dy is fed 2²
HW into a vision transformer. Let my 3f be the number of visual tokens. According to Lemma H.1, the computational complexity of the vision transformer would be (Any + 2n + 22yddddfty) For video with length T, the total complexity of the visual encoder would be O (T (3dy HW (y + 2n 22nydydrb))) For the text encoder. given the length of query 44- the computational complexity would be O (Lr (41, + 21 + 21gdrd11z)) according to Lemma H.1.
For the multimodal encoder, given the length of input (including visual tokens and text tokens) n T4 the computational complexity would be O (LF (Andz + dp + 2ndpdjsn)) according to Lemma H.1.
Overall, the computational complexity of UniVTG would be: (3dy HW + (ynydy + 2n + 2y + Lr( Al 22 + 21,drd L + +g + 2 + +1 + (1)++1)drdsss where ny 8" all the parameters as constants and eliminating low-order terms, the complexity expression of UniVTG could be simplified as O (P,TH P,T + P + P.TTg).
Hw Considering
Proposition H.4. Given an image with shape H x W, and a query with length b. the computational complexity of YOLO-World is O HP& + P+P)). where Pg and Py are constants relevant to parameters of YOLOO
Proof. YOLO-Worke consists of multiple convolution structures. Since the calculation process of these structures is repetitive and cumbersome, the detailed calculation process of the computational complexity of YOLO-World is omitted here. Evidently, the complexity of the convolution calculation is proportional to HW, and the complexity of the text encoding calculation is proportional to l (according to Lemma H.1). So the computational complexity of YOLO-World is approximately O (PoHW + P.(t).
Given the number of input frumes N, the shape of inpul frames H x W and the length of text input I, the computational complexity of Qwen-VL-P2 is O (22 + PgN2 + Piol + PuNNi), where Ps. Ps, P1o and Pı1 are constants relevant to parameters of Oown-VVL-7h
Proposition H.5.
Proof. Qwen-VL utilizes a vision transformer to encode the input images, after which a cross attention layer with learnable query embeddings is used to project the visual tokens into the space of text tokens. Finally QwenLM gives the output according to the input visual tokens and text tokens. Assume the parameters of the vision transformer are dyt, LVT, dfivr the parameters of QwenLM are dæ, Lq, ffffa- Ny is the number of queries in the cross attention layer.
As we have mentioned in the proof of Proposition H.3, the computational complexity of the vision transformer would be O H3dvrHW + LvT (Anyrrrd + 2n and s is the kernel size and stride in the convolution layer.
HW is the number of visual tokens + 2nyrdvrdsjvn (vr)) where nyT _2
The calculation of the computational complexity of the cross attention layer is similar to the proof of Lemma H.I. which is omitted here. The computational complexity of the cross attention layer is O (nvrdyrdq + 2n,nyrdo + 2ny0B + 2nvrd2). nyrdvrdq is the complexity of the linear layer before cross- attention, projecting the visual tokens in the space of dimension dyr to the space of dimension do-
Finally for QwenLM, the total length of the input would be na Nn, + I. According to Lemma H.I, the computational complexity of QwenLM would be Lo (Angd& + 2ngdo + 2nododrsa).
Overall, the computation complexity of Qwen-VL would be O(N(3dyTHW + Lyr(AnvretT + 2ntrddvr + 2ny7dyrdfivz) + NuTdvrdQ + 2no + 2n, + 2nvTdE) + LQ(4 + +1)dg + 2(N + +f2 + 2(NNn, + (ddqdff)); where nyr- = HW Considering all the parameters of the model as constants and eliminating low-order terms, a
the complexity of Qwen- VL would be O (P' + P&N2 + Puut2 + P1)N
Propusition H.6. Given a video with lenghh T and resolution H x W. length of our prompts for the Question Parser Ip and a question with length L,, the computational complexity of MSR- ViR is O (cTTH2 + o2T22 C, H2W2 + C2 + Cz1g)
24

---

Modularized Self-Reflected Video Reasoner for Multimodal LLM with Application to Video Question Answering
where Ci 0g are constants relevant to paramelers of small models MSR-ViRg and C1 C2 and C3 are constants relevant parameters of large models in MSR-ViRQ-
Proof. All other modules only involve operations with a complexity of O(1), such resizing and cropping. Therefore, only the above four parts (Question Parser, UniVTG, YOLO-World, Qwen-VL) need to be considered when calculating the complexity of MSR-Vise ViRo. When MSR-ViRo answers question, it invokes the Question Parser and Qwen- VL for one time respectively. In MoST-Grounding, considering the most complex policy where Tempora Been module is utilized to invoke UniVTG twice, while in the spatial localizer, YOLO- Woold needs to be invoked twice for each image for spatial localization.
Let the complexity of the Question Parser be Tp, the complexity of UniVTG be Ty, the complexity of YOLO- World be Ty and the complexity of VWen-VL be Tq. We sample N frames temporal-grounded frames, N frames as spatial-grounder frames. The length of the question is L, and the length of the query for UniVTG and YOLO-World is lg. The length of the video is T, and the resolution of which is H x W. Then, the upper bound complexity of Qwen- VL would be:
T = Tr 2Tu + 2NTy TQ
+2 + 2P.2 2P,T1, 2NP + 2N + 2P + 22 + PIoL2 2,,
P. ([[++1) +0TT+++++P77771
worth noting that L is usually the length of a very short sentence, thus the terms with l could be eliminated. Also notice that the length of question L is significantly smaller than the length of prompts lp. After eliminating low-order terms:
T (( + HWWP +2P,NH 4P,N2 + PioL + NL 2 P12 + P1o.. +2,TTH + +2
= = + +
where = 2P2 = 2P are constants relevant to parameters of the small model UniVTG, and C₁ = 2N.s. C2 Pio (We set N = 2 in
and Cg = Pi are constants relevant to parameters of the large models Qwen2-7B and MSR- MSR-ViRg. so actually C₁ = 4Ps).
According to Proposition H.6, the upper bound computational complexity of MSR-ViR ViRg is O(c,TH²W2 + T²T + C MHH C2L² + Cglg). As ei and 02 are constants from small model, they significantly smaller than Ci and C2. which is approximately (c, T222 + c21²).
So the execution time of MoST-Grounding although it proportional to TH2W² and T According to Proposition H.5, if we fix N as a constant, complexity for Qwen-VI. would be O(P12 framework mostly comes from Czly, as Ip is significantly larger than L.
would be strictly bounded,
P.).. After comparison, it is obvious that the computational overhead of
In another word, the Question Parser is the main source of computational overhead. This is acceptable for our framework, as the modular policies generated by the Question Parser provide interpretable reasoning path. Besides, this part of computational complexity is irrelevant to the length and resolution of the video, and the length of question itself, making strictly bounded no matter how complex the scenario is. According to the complexity calculation results, the inference time of our framework should be approximately correctness of this. Overall, the additional computational overhead introduced by our framework is reasonable and strictly bounded.
3 times that of the baseline. The experimental results in Table verify the
25


Reviews:
[{"review_id": "aA77GKK1Cv", "reviewer": "Reviewer_mAE3", "summary": "The paper introduces a framework MSR-ViR designed to improve interpretability of multimodal LLMs in VideoQA. Unlike traditional end-to-end multimodal LLMs that function as black boxes, MSR-ViR integrates modular networks to provide explicit reasoning paths. MSR-ViR serially combines (1) a question decomposer that divides complex a question into sub-steps (via few-shot prompting of a LLM), (2) a temporal localizer that localizes the relevant temporal segments (via UniVTG model), (3) a spatial localizer that identifies the spatial segments (via YOLO-World model), (4) and a multimodal LLM that is trained to answer questions given the segments. To further improve this pipeline, a Self-Reflection Training was proposed. The LLM is fine-tuned to output better policy / sub-steps, guided by the loss from the multimodal LLM. Finally, the LLM and the multimodal LLM are trained in a cyclical manner. Through experiments on benchmarks (NExT-QA, STAR, EgoSchema, and NExT-GQA), MSR-ViR demonstrates good performance in video understanding and localization accuracy compared to baseline methods.\n\n### Update After Rebuttal\nThanks the authors for their clarification which address most of my concerns. I raise my score to accept and expect that the authors will revise the paper according to the suggestions from reviewers.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for taking time to review our paper and providing insightful feedback and suggestions. We address the weaknesses and questions as follows:\n\n### **Claims And Evidence**\n\nProviding interpretable reasoning paths for black-box Multimodal LLMs in the scenario of VideoQA is one of the major claims in our paper. Although the reasoning path is generated by a text-only LLM, the reasoning process of MSR-ViR is multimodal, as presented in Figure 8, 9, 10. MoST-Grounding localizes the temporal segments and spatial regions related to the question from the video step by step according to the reasoning path. This reasoning process requires not only the textual information in the question but also the visual information in the video to perform step-by-step grounding, and thus it is multimodal. \n\nDue to the lack of label annotations, we are unable to directly evaluate the reasoning path. However, this does not mean the reasoning path is not critical in our framework. Keeping small modules in MoST-Grounding fixed, the reasoning path determines the final spatial-temporal grounding result input to the Multimodal LLM. Therefore, it is crucial for whether the Multimodal LLM can correctly answer the questions and whether the framework can provide accurate grounding results. For this reason, the evaluation of metrics such as QA accuracy, Acc@GQA, IoU is an indirect evaluation of the reasoning path. For example in Table 4, through these metrics, we verify that self-reflection training can improve the quality of the reasoning paths generated by the Question Parser.\n\nTo further verify the importance of the reasoning path, we provide another ablation study by using MoST-Grounding to directly provide grounding results with UniVTG and YOLO-World based on the original question without the reasoning path. The results on NExT-GQA are shown in the following table. It can be seen that without the guidance of the reasoning path, both the accuracy in answering questions and the accuracy of the grounding results have significantly decreased. This further illustrates the necessity of the reasoning path. \n\n|| Acc@QA | Acc@GQA | mIoU | IoU@0.5 |\n| ---------------------------------------- | ------ | ------- | ---- | ------- |\n| MSR-ViR$_{Q}$|69.9|18.5|22.8|16.4|\n| MSR-ViR$_{Q}$ without self-reflection training |68.3|17.9|22.2|15.7|\n| MSR-ViR$_{Q}$ without reasoning path|66.8|14.4|18.5|11.4|\n\n### **Experimental Designs Or Analyses**\n\nFrom the experiment results, it can be seen that the MSR-ViR is capable of providing reasonable reasoning paths, and thereby improves the accuracy of Multimodal LLM in VideoQA. As can be seen in Table 4, self-reflection training plays a key role in improving QA accuracy, which is in line with expectations and is also one of the main contributions of this paper. From the additional ablation study in the above table, it can also be seen that reasoning paths and self-reflection training have significantly improved the performance of grounded-QA, alleviating error accumulation of temporal localizer and spatial localizer. Directly using a video LLM with grounding capability provides a solution to grounding-based VideoQA, but the black-box end-to-end video LLM contradicts the core issue to be addressed in our paper - the issue of interpretability, and thus is not within the scope of discussion in our paper.  For the issue of insignificant performance improvement, please refer to Q1 in our rebuttal to reviewer u3Mw. We claim that the impact of grounding for Multimodal LLM on VideoQA datasets is limited, consistent with previous grounding-based method. Besides, we further conduct experiments on long-form VideoQA dataset VideoMME, where grounding and reasoning are more important, to demonstrate the superiority of our framework.\n\n### **Relation To Broader Scientific Literature**\n\nWe thank the reviewer for the suggestions of missing references, and discuss these works as follows:\n\n[A] introduces Visual Table, a novel visual representation that provides detailed object descriptions and knowledge in structured text, significantly boosting performance in visual reasoning tasks, while [B] presents MM-Reasoner, which leverages vision APIs and LLMs to extract and utilize query-specific knowledge. Other works explore the use of outputs for reinforcement learning of the model, such as [C] that improves image captioning quality by normalizing rewards with test-time inference outputs, leading to significant performance gains. These works share some similarities with certain ideas in our work. However, they are all focused on image tasks, rather than VideoQA task which involves more complex scenarios and requires more grounding and reasoning. We will add detailed discussion to the final version of our paper.\n\n### **Other Comments Or Suggestions**\n\nWe will make revisions to the layout of Figure 2 for clarity and comprehensibility.", "claims_and_evidence": "Over-claim on the explicit reasoning paths: One major claim of this work is that the model can provide reasoning paths. However, the path is provided by text-only LLM and based on question only (e.g., Figure 3), instead of multi-modal reasoning path. Besides, the experiments are mostly concerned about QA accuracy, leaving textual reasoning paths less critical.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Complex pipeline yet less significant performance improvement: The LLM is trained to better fit with video QA and thus output a more reasonable step plan. The performance improvement is not significant (Table 1-3) and mostly from LLM self-reflection training (Table 4). One possibility is that the combined modules (temporal localizer and spatial localizer) accumulate errors. What if we combine it with an end-to-end video LLM with grounding capability?", "relation_to_broader_scientific_literature": "Missing references: The overall framework can be considered as using external modules / tools to obtain structured and additional information for supporting multimodal LLM reasoning. It would be great if the authors could discuss the works in this direction, such as [A, B]. Besides, one key improvement of this work comes from the reinforcement learning supervised by the loss of generating answers. Similar key idea was proposed by [C] which tries to improve caption quality via sentence metric. \n\n[A] Beyond Embeddings: The Promise of Visual Table in Visual Reasoning, EMNLP 2024\n\n[B] MM-Reasoner: A Multi-Modal Knowledge-Aware Framework for Knowledge-Based Visual Question Answering, EMNLP 2023\n\n[C] Self-critical Sequence Training for Image Captioning, CVPR 2017", "essential_references_not_discussed": "See above.", "other_strengths_and_weaknesses": "Strengths:\n1.\tThis work tries to provide explicit reasoning steps for video QA using multimodal LLMs. This direction is promising and worth exploration.\n2.\tThis work proposes a reinforcement learning strategy that helps improve the reasoning plan and thus final QA accuracy. This design is simple and effective.\n3.\tThe paper writing and figures are generally good.\n\nWeaknesses: See above sections.", "other_comments_or_suggestions": "The layout of Figure 2 can be further improved for better reading experience.", "questions_for_authors": "I would consider raising the scores depending on the answers to the concerns / weaknesses above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "muxFI5iz0t", "reviewer": "Reviewer_NjDQ", "summary": "This paper introduces MSR-ViR, a novel framework designed for interpretability of video question answering. Multiple modular networks are integrated wth a multimodal large language model in the proposed method. \n\nTo refine its reasoning, the framework utilizes an Alternate Self-reflection Training Strategy, optimizing both the policy generation and the multimodal LLM.\n\nExperiments on different datasets show that the proposed method can improve video understanding and the accuracy of localizing evidence for answers.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for taking time to review our paper and providing insightful feedback and suggestions. We address the weaknesses and questions as follows:\n\n### **Claims And Evidence**\n\n1.The Question Parser's prompt is meticulously crafted. It details each module's function and includes carefully-chosen basic examples, covering common and corner cases, to guide policy generation. As seen in Figures 4,5,6, the generated policy is a concise JSON string, not some heavy structure. Although the prompt is long due to multiple policy examples, this is essential for the Question Parser to produce reasonable policies.\n\n2.Please refer to \"Experimental Designs Or Analyses\" part.\n\n### **Theoretical Claims**\n\nWe are sorry about the typo in Preposition 3.1. Preposition 3.1 should be:\n\nGiven parameters of Multimodal LLM $P_1$, $P_2$, $P_3$ and $P_4$, video with $N$ input frames and resolution $H \\times W$, text input with length $l$, the complexity of Multimodal Answerer is $O\\left(P_1NH^2W^2 + P_2N^2 + P_3l^2 + P_4Nl \\right)$.\n\nAs we have only selected Qwen-VL as the base model to analyze the computational complexity, $P_1$, $P_2$, $P_3$ and $P_4$ represent constants relevant to parameters of Qwen-VL. Specifically, according to the proof of Preposition G.5 in Appendix G, given the parameters of the vision transformer $d_{VT}, L_{VT}, d_{ff_{VT}}$, the parameters of QwenLM are $d_Q, L_Q, d_{ff_Q}$, the number of queries in the cross attention layer $n_q$ and the kernel size and stride in the convolution layer $s$, we have: $P_1 = \\frac{2L_{VT}d_{VT}}{s^4}$, $P_2 = 2L_Qd_Qn_q^2$, $P_3 = 2L_Qd_Q$, $P_4 = 4L_Qd_Qn_q$.\n\n### **Experimental Designs Or Analyses**\n\n|| Parameter Size | Inference Speed (s / sample) | Acc on NExT-QA |\n| ------------------------ | :------------: | :--------------------------: | :------------: |\n|BLIP-2|4.1B|1.21|69.6|\n|LSTP*|4.3B|1.62|72.1|\n|InstructBLIP|7.9B|1.75|72.5|\n| SeViLa|8.3B|2.79|      73.8      |\n| Qwen-VL|      9.6B      |             1.32             |      71.9      |\n| MSR-ViR$_Q$(1.5B parser) |     11.2B      |             2.35             |      73.1      |\n| MSR-ViR$_Q$(7B parser)   |     16.7B      |             3.10             |      73.6      |\n| LLaVA-Next|      7.1B      |             2.19             |      73.1      |\n| MSR-ViR$_L$(1.5B parser) |       8.7B       |             4.29             |      74.2      |\n| MSR-ViR$_L$(7B parser)   |     14.2B      |             4.96             |      74.9      |\n\nWe test the inference speed of our method and Multimodal LLM-based baselines on an NVIDIA A100 GPU. Results are presented in the above table. We omit GCG from the table as its repository lacks inference code. LSTP, designed for high-efficiency inference with optical flow, is marked. For BLIP-2, LSTP, InstructBLIP, SeViLa and Qwen-VL, we sample 4 frames from the video. For LLaVA-Next, we sample 8 frames from the video. The settings of MSR-ViR keep consistent with those in our paper.\n\nOur framework's additional parameters mainly stem from the Question Parser; MoST-Grounding contributes less than 0.1B. For comprehensive comparison, we test total parameters, inference speed, and accuracy using Question Parsers of different sizes (Qwen2-7B and Qwen2-1.5B). With Qwen2-7B, the inference speed of MSR-ViR is about twice that of the direct baseline, consistent with our complexity estimates. This computational overhead is reasonable, as reasoning-based methods typically require more time to answer questions due to the step-by-step reasoning process, like GPT-o1 over GPT-4o, and DeepSeek-R1 over DeepSeek-V3. With Qwen2-1.5B, although accuracy slightly drops, it still outperforms the direct baseline with fewer additional parameters and less computational overhead.\n\nAlthough MSR-ViR introduces additional parameters and computational overhead, we address its advantages as follows:\n\n- MSR-ViR provides interpretable reasoning path which is refined by self-reflection training, while previous methods fail to do so. As MoST-Grounding utilizes faster small modules, the computational overhead mainly comes from Question Parser that provides interpretability of our framework, and the computational overhead is reasonable.\n- As for the issue of insignificant improvement in Table 1, please refer to Q1 in our rebuttal to reviewer u3Mw. We claim that the impact of grounding for Multimodal LLM on VideoQA datasets is limited, consistent with previous grounding-based method. Besides, we further conduct experiments on long-form VideoQA dataset VideoMME, where grounding and reasoning are more important, to demonstrate the superiority of our framework.\n- Table 3 demonstrates that MSR-ViR achieves significantly higher grounding accuracy and grounded-QA accuracy than previous methods. The IoU is even higher than that of LLoVi and MoReVQA which utilize significantly larger LLMs GPT-4 and Palm-2. This demonstrates that MSR-ViR is capable of more accurately grounding and answering questions.", "claims_and_evidence": "1. MSR-ViR provides interpretable VideoQA with explicit reasoning paths: The paper proposes the grounding modules to decompose complex questions and localizes relevant video segments. However, the tree-structured policy generated by the Question parser seems a little heavy. The full prompt is provided in the supplementary, comprising three pages.\n\n2. MSR-ViR enhances the VideoQA abilities of Multimodal LLMs: This claim is not well supported. In Table 1, the author should also provide the detailed model size and inference speed. So the readers will better understand how much to pay to obtain such performance.\n\n3. The MoST-Grounding module with both temporal and spatial localizers is effective: This is supported by the ablation study. The study removes the spatial localizer, showing a drop in average accuracy on NExT-QA, indicating the usefulness of spatial grounding. Further,  the evaluation on NExT-GQA demonstrates accurate temporal grounding.", "methods_and_evaluation_criteria": "The proposed method uses multiple 'expert' modules to help localize frames both temporally and spatially. Further,  self-referential training is achieved by reinforcement learning on the Question Parser.", "theoretical_claims": "There are no significant theoretical claims in this paper. The author attempts to provide the computational complexity analysis in Section 3.5. \n\nHowever, it is not clear what P1 - P3 mean in Proposition 3.1, and where does P4 come from?", "experimental_designs_or_analyses": "1. The author should explicitly show the full parameter sizes in Table 1. It is not clear how the model sizes are different between different models. The reason why the model size is important is because the proposed method includes multiple off-the-shelf expert models to attend to specific frames both temporally and spatially. Including the model size will help the readers to understand the cost.\n2. Further, the experimental results lack the comparison of training and inference efficiency with other methods. With many add-on modules, what is the throughput or the inference speed when comparing with other models?\n3. Another concern is about the improvement in Table 1. From my point of view, the improvement is not significant when comparing with the direct baselines. For example, when comparing to LLaVA-Next, the performance increases 1.8% on the NExt-QA benchmark. Consider the increased model size and lower inference speed, how would the author address their advantages?", "relation_to_broader_scientific_literature": "This paper is majorly related to the multimodal LLM and video question answering.", "essential_references_not_discussed": "I do not have recommendations on essential references.", "other_strengths_and_weaknesses": "Please refer to previous comments.", "other_comments_or_suggestions": "I do not have other comments or suggestions.", "questions_for_authors": "I do not have other questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kDaGwtCZys", "reviewer": "Reviewer_uxoU", "summary": "The paper addresses the interpretability problem in VideoQA by introducing the Modularized Self-Reflected Video Reasoner (MSR-ViR) framework, which decomposes complex questions into smaller parts through its Modularized SpatialTemporal Grounding (MoST-Grounding) module and employs a reinforcement learning-based Alternate Self-reflection Training strategy to train a Multimodal LLM. By following a tree-structured execution policy from a Question Parser, MoST-Grounding progressively isolates relevant visual information from the video, improving both spatial-temporal grounding and reasoning. This yields transparent reasoning paths, as well as visual evidence for predicted answers. A theoretical analysis demonstrates bounded computational overhead, and experiments show that MSR-ViR not only outperforms baselines but also accurately localizes temporal segments, enhancing interpretability in VideoQA datasets such as EgoSchema, NExT-QA, STAR, and NExT-GQA.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for taking time to review our paper and providing insightful feedback and suggestions. We address the questions as follows:\n\n### **Supplementary Material**\n\nWe will open-source our code after the review and provide detailed documentation to facilitate the reproduction and application of our framework.\n\n### **Essential References Not Discussed**\n\nThe CVPR paper addresses the challenge of open-ended VideoQA in long egocentric videos. The authors propose a novel approach that integrates query grounding and answer generation into a unified model to achieve grounded VideoQA. However, the proposed unified multimodal model is still black-box, lacking interpretability. Different from this paper, our work provides interpretability in VideoQA tasks for Multimodal LLMs. We will add detailed discussion to the final version of our paper.", "claims_and_evidence": "Yes, the claims are supported by strong performances as well as ablations on verifying whether each design choice is necessary.", "methods_and_evaluation_criteria": "There are multiple components in the method: a question parser to decompose questions into sub-questions, two modularized temporal localizer and spatial localizers to extract temporal-grounded and spatial-grounded frames. Then the frames are fed into a multimodal LLM for supervised fine-tuning, and lastly, an Alternate Self-reflection training strategy to improve the question parser by using a DPO over multiple parsing and preferring the one with the lowest multimodal LLM loss.  \n\nThe benchmarks and metrics are standard and complete, ranging from accuracies of different categories in NExT-QA, STAR-sub, and EgoSchema (sub and full), as well as mIoU, IoU, Io, and accuracy of NExT-GQA.", "theoretical_claims": "The reviewer briefly checked the proofs of propositions 3.1 and 3.2 and they seem correct.", "experimental_designs_or_analyses": "The experimental design is comprehensive.", "relation_to_broader_scientific_literature": "This paper leverages grounding to improve video QA accuracies while providing interpretability because of modularization, which is an important contribution compared to previous models, which mostly rely on captions or do not provide any interpretability.", "essential_references_not_discussed": "The authors should discuss a relevant paper from CVPR 2024: [1]\n\n[1] Di, Shangzhe, and Weidi Xie. \"Grounded question-answering in long egocentric videos.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m1U9lSFoX5", "reviewer": "Reviewer_u3Mw", "summary": "This paper enhances the interpretability and reasoning capabilities of Multimodal Large Language Models (MLLMs) in video question answering.\nThe authors propose a modular system that constructs explicit reasoning paths and extracts precise spatial-temporal information.\nThe framework is further optimized using a Reinforcement Learning-based self-reflection mechanism.\nEvaluations on STAR, NExT-QA, and other VideoQA benchmarks show that the MoST-Grounding network achieves superior performance and interpretability.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for taking time to review our paper and providing insightful feedback and suggestions. We address the weaknesses and questions as follows:\n\n### **Methods And Evaluation Criteria**\n\n#### **Q1 Lack of Novelty**\n\nWe highlight the contributions of our paper as follows: \n\nMSR-ViR enhances interpretability in Multimodal LLMs for VideoQA by integrating a modular network for interpretable reasoning. An alternate self-reflection training strategy is proposed wherein Multimodal LLM provides feedback to perform DPO training on Question Parser that generates reasoning paths. To the best of our knowledge, this is the first work that not only integrates a modular network into a Multimodal LLM but also jointly optimize them with self-reflection training for reasoning path refinement and QA accuracy improvement. Prior modular-based works(VideoAgent, AoTD) directly use the results of the modular system without optimizing or refining them, and some of them are built on Uni-modal LLM(MoReVQA, ProViQ). As for the limited performance improvement issue, we address as follows:\n\n- Grounding-based methods are not capable of significantly improving the accuracy of Multimodal LLMs in answering questions, as they basically only change the input without changing the model's internal structure. More relevant visual information helps improve QA accuracy, but the improvement is limited. This aligns with existing grounding-based approaches, where improvements remain modest (e.g., SeViLa +1.2%, GCG +2.1%). The improvement of our method is comparable to that of the existing works.\n- It can be observed that the performance improvement on EgoSchema is more significant compared to that on NExT-QA. We claim that MSR-ViR shows greater performance gains on longer video datasets, where grounding and reasoning are more essential. For further validation, we train Qwen2-VL and LLaVA-Video with baseline method and MSR-ViR on a subset of LLaVA-Video-178K and test them on NExT-QA and a longer VideoQA dataset VideoMME. The results demonstrate the superiority of MSR-ViR, especially on longer videos.\n\n|NExT-QA|Tem.| Cau.| Des.| Avg.|\n|--------------|----------|----------|----------|----------|\n|Qwen2-VL|74.6|78.2|83.1|77.8|\n|MSR-ViR$_{Q2}$|75.9(+1.3)|80.2(+2.0)|84.5(+1.4)|79.6(+1.8)|\n|LLaVA-Video|79.0|82.6|86.3|82.1|\n|MSR-ViR$_{LV}$|81.8(+2.8)|84.8(+2.2)|87.3(+1.0)|84.3(+2.2)|\n\n|VideoMME|Short|Medium|Long |Avg.|\n|--------------|----------|----------|----------|----------|\n|Qwen2-VL|65.2|52.2|48.3|55.3|\n|MSR-ViR$_{Q2}$|66.8(+1.6)|55.4(+3.2)|51.3(+3.0)|57.9(+2.6)|\n|Llava-Video|69.7|56.6|49.3|58.5|\n|MSR-ViR$_{LV}$|72.3(+2.6)|60.7(+4.1)|52.6(+3.3)|61.9(+3.4)|\n\n#### **Q2 Self-Reflection Learning for Multiple-choice QAs**\n\nThe loss here refers to SFT loss. For multi-choice QAs, this loss simplifies to the correctness of the selected option. For policy pairs that yield the same option, we do not use them for DPO training. About 50% of policy pairs could be used in the first epoch, adequate for DPO training of Question Parser.\n\n#### **Q3 Handling Non-Spatial-Temporal Questions**\n\nThe interpretability in our work is considered from the perspective of spatial-temporal grounding. Non-spatial-temporal questions constitute a special scenario for our framework. In these cases, MSR-ViR asserts that answering the questions necessitates the entirety of video information. In other words, the grounding result would be the full video. Future work could expand MSR-ViR with non-spatial-temporal modules for further interpretability.\n\n### **Theoretical Claims**\n\nWe are sorry about the typo. Please refer to \"Theoretical Claims\" part in our rebuttal to reviewer NjDQ for more details.\n\n### **Experimental Designs Or Analyses**\n\n#### **Q4 Frame Sampling**\n\nThe current frame sampling rates are optimized. Due to character limitation, we present several ablation on frame sampling for NExT-QA in this link: https://anonymous.4open.science/api/repo/frame-ablation-C4EB/file/frame-ablation.png?v=54b8a357. For MSR-ViR, the number represents (temporal + spatial + global).\n\n#### **Q5 Base Models**\n\nWe conduct experiments for Qwen2-VL and LLaVA-Video on NExT-QA and VideoMME. Please refer to Q1.\n\n### **Essential References Not Discussed**\n\nSTAIR generates policies through small models trained with supervision on a single dataset, thus the types of questions it could solve are severely limited. ProViQ and ENTER utilize LLM to generate programs. However, they are based on Uni-modal LLM, and may fail to generate executable programs due to lack of training. VideoAgent and MotionEpic utilize Chain-of-Thought which involves multi-round conversation with LLM, while AoTD distills knowledge from CoT into Video LLM to improve instruction tuning. The idea of jointly optimizing Multimodal LLM and modular system via self-reflection training in our work represents a novel approach not explored in these works. We will add detailed discussion to the final version of our paper.", "claims_and_evidence": "Yes. The claims are clearly proved.", "methods_and_evaluation_criteria": "Below are some concerns about the proposed method.\n\nQ1. **Lack of Novelty**: While the authors claim that MSR-ViR differs from grounding-based methods and modular approaches like MoReVQA, its core methodology—decomposing questions, performing spatial-temporal grounding, and using a model (LLM or MLLM) to generate answers—has been extensively explored in prior works such as STAIR [1] and VideoAgent [2]. The substitution of the final answering model adds limited novelty. Although the self-reflection learning method introduces a new perspective, the overall improvements appear limited.\n\nQ2. **Self-Reflection Learning for Multiple-choice QAs**: The authors use DPO in the self-reflection process, treating policies with smaller losses as positive. Is the loss referred to here the SFT loss? If so, for multiple-choice data, such as NExT-QA and STAR-sub, various polices may yield the same choice and the same loss. How do the authors distinguish between positive and negative samples in such cases? Please clarify if my understanding is incorrect.\n\nQ3. **Handling Non-Spatial-Temporal Questions**: How does MSR-ViR address questions that do not require spatial-temporal information but still demand interpretable answers, such as, \"What makes the video humorous?\"\n\n---\n[1] STAIR: Spatial-Temporal Reasoning with Auditable Intermediate Results for Video Question Answering. Yueqian Wang, et al. AAAI 2024.\n\n[2] VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding. Yue Fan, et al. ECCV 2024.", "theoretical_claims": "What does $P_4$ mean in Proposition 3.1?", "experimental_designs_or_analyses": "Q4. **Frame Sampling:** LLaVA-NeXT uniformly samples 8 frames, Qwen-VL samples 4 frames, while MSR-ViR Q samples 4 frames for spatial-temporal information and 2 for global information, and MSR-ViR L samples 16 frames for spatial-temporal information and 2 for global information. Does this unequal sampling strategy introduce bias in performance comparisons? Clarifying whether the frame sampling rates are optimized for each model or standardized for fair evaluation would strengthen the validity of the results.\n\nQ5. **Base Models:** The base models, LLaVA-NeXT and Qwen-VL, appear outdated. Replacing them with current mainstream models, such as LLaVA-Video or Qwen2-VL, would provide a more convincing and up-to-date comparison, reflecting the latest advancements in the field.", "relation_to_broader_scientific_literature": "This paper distinguishes itself from conventional MLLMs, which typically generate answers directly without intermediate reasoning processes. By incorporating the MoST-Grounding module, the proposed approach provides the model with more precise spatial-temporal information, thereby enhancing the interpretability of the QA process and improving the overall performance of the model.", "essential_references_not_discussed": "Q6. The modular or agentic system, which decomposes complex questions into simpler sub-questions and extracts spatial-temporal information for better VideoQA performance and interpretability, has also been explored in several prior works, including STAIR [1], VideoAgent [2], ProViQ [3], AoTD [4], ENTER[5] and MotionEpic [6].\n\n---\n[1] STAIR: Spatial-Temporal Reasoning with Auditable Intermediate Results for Video Question Answering. Yueqian Wang, et al. AAAI 2024.\n\n[2] VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding. Yue Fan, et al. ECCV 2024.\n\n[3] Video Question Answering with Procedural Programs. Rohan Choudhury, et al. ECCV 2024.\n\n[4] Enhancing Video-LLM Reasoning via Agent-of-Thoughts Distillation. Yudi Shi, et al. CVPR 2025.\n\n[5] ENTER: Event Based Interpretable Reasoning for VideoQA. Hammad Ayyubi, et al. Arxiv, 2024.\n\n[6] Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognition. Hao Fei, et al. ICML 2024.", "other_strengths_and_weaknesses": "**Strength:**\n\n1. The paper is well-written, with a clear motivation and logical flow, making it easy to follow.\n\n2. The use of Reinforcement Learning for the question parser effectively circumvents the challenge of supervising intermediate reasoning steps, aligning with contemporary approaches like Deepseek-R1.\n\n**Weaknesses:**\n\nSee above.", "other_comments_or_suggestions": "None.", "questions_for_authors": "See above. I would consider raising my score if the authors address my concerns.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents MSR-ViR, a new method for interpretable video question answering, combining multiple modular networks with a multimodal LLM. It employs an Alternate Self-reflection Training Strategy to enhance reasoning by optimizing policy generation and the multimodal LLM. Experiments across various datasets demonstrate improved video understanding and answer evidence localization accuracy. All reviewers agree to accept the paper.
Decision: Accept (poster)