TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models

Authors: Ziyao Shangguan, Chuhan Li, Yuxuan Ding, Yanan Zheng, Yilun Zhao, Tesca Fitzgerald, Arman Cohan
Primary area: datasets and benchmarks
Keywords: visual temporal reasoning, video understanding, benchmark, vision-language benchmark, video-language models, evaluation

Abstract:
Existing benchmarks often highlight the remarkable performance achieved by state-of-the-art Multimodal Foundation Models (MFMs) in leveraging temporal context for video understanding.
However, *how well do the models truly perform visual temporal reasoning?*
Our study of existing benchmarks shows that this capability of MFMs is likely overestimated as many questions can be solved by using a single, few, or out-of-order frames. 
To systematically examine current visual temporal reasoning tasks, we propose three principles with corresponding metrics:
(1) *Multi-Frame Gain*,
(2) *Frame Order Sensitivity*,
and (3) *Frame Information Disparity*.
Following these principles, we introduce **TOMATO**, **T**emp**O**ral Reasoning **M**ultimod**A**l Evalua**T**i**O**n, a novel benchmark crafted to rigorously assess MFMs' temporal reasoning capabilities in video understanding.
TOMATO comprises 1,484 carefully curated, *human-annotated* questions spanning *six* tasks (i.e. *action count, direction, rotation, shape & trend, velocity & frequency, and visual cues*), applied to 1,417 videos, including 805 self-recorded and -generated videos, that encompass human-centric, real-world, and simulated scenarios. 
Our comprehensive evaluation reveals a human-model performance gap of 57.3% with the best-performing model.
Moreover, our in-depth analysis uncovers more fundamental limitations beyond this gap in current MFMs. While they can accurately recognize events in isolated frames, they fail to interpret these frames as a continuous sequence.
We believe TOMATO will serve as a crucial testbed for evaluating the next-generation MFMs and as a call to the community to develop AI systems capable of comprehending the human world dynamics through the video modality.

Decision: accept (poster)
Reviewer scores: 5, 8, 8, 6

Reviews and discussion:
<thread>
creation date:
1730518312099

thread_type:
official_review

<thread_object>
creator: reviewer_xcjw

<content>
summary:
The paper introduces a novel video benchmark specifically designed to assess the temporal reasoning capabilities of multimodal foundation models. The benchmark is thoughtfully formulated, encompassing a wide range of dynamic and continuous reasoning scenarios over time. By evaluating state-of-the-art video foundation models, the paper highlights fundamental challenges these models face in Video Question Answering (VideoQA). This work provides valuable insights into the limitations of current multimodal models in handling temporal visual tasks, promising to advance research in both the video and large language model (LLM) communities. However, despite these contributions, I still have some concerns and will adjust my rating depending on whether they can be further addressed.

strengths:
- Well-Established Benchmark: The benchmark is rigorously constructed with three quantitative principles that emphasize the complexity of temporal reasoning tasks, ensuring a challenging and relevant evaluation.

- Extensive and Comprehensive Coverage: The benchmark includes a wide range of tasks and scenarios, providing thorough coverage of diverse temporal reasoning challenges, which enhances the evaluation scope and relevance.

- In-Depth SOTA Model Comparison: The paper presents an extensive and detailed comparison of state-of-the-art video foundation models, providing valuable insights into their capabilities and limitations for video question answering.

- Clear and Cohesive Writing: The paper is well-written, presenting its ideas and findings in a clear and organized manner, making it accessible and informative for a broad audience.

weaknesses:
- Sensitivity in Mathematical Formulation: The mathematical formulation of the three principles appears sensitive to certain parameters, particularly the choice of human-picked frames and the number of input frames. For instance, using 16 frames as the multiple-frame input seems arbitrary; different frame counts could significantly impact evaluation scores and model comparison across benchmarks. This choice requires further justification, along with ablation studies on multi-frame input numbers within the three principles, to confirm robustness and fairness.

- Data Imbalance Across Tasks: The sample sizes across the six tasks are imbalanced, with some categories, like Visual Cues, containing only 70 questions—significantly fewer than other categories. This imbalance could impact the benchmark's representativeness and reliability. Addressing this imbalance would improve the benchmark's overall consistency and rigor.

- Fairness in Benchmark Comparisons: Comparisons with other benchmarks may not be entirely fair due to the random sampling of 200 QAs for evaluations. This approach may inadvertently filter out complex temporal reasoning questions. To ensure fairness and maintain the benchmark's integrity, evaluating all QAs from other benchmarks would be preferable.

- Insufficient Text-Only Baseline in Table 5: The current text-only baseline presented in Table 5 may not fully capture the model's reasoning capabilities. Converting video content into textual descriptions for input into a model like GPT-4 would offer a more meaningful comparison with visual inputs. Including this enhanced baseline would provide deeper insights and enrich the discussion in Table 5.

questions:
- Will the benchmark be publicly available?

- How do you generate the queries for each task in the benchmark? Please include this information in the main paper, which may be inspiring to the readers.

- Why not reporting the results of open-source models after fine-tuning them on the temporal reasoning questions. e.g, a training set of the proposed benchmark? It is possible that the model learns to reason from dynamics in the video after fine-tuning. Please justify this.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730547376007

thread_type:
official_review

<thread_object>
creator: reviewer_nxcv

<content>
summary:
This paper present a new benchmark for visual temporal reasoning. They start by defining three principles for video temporal understanding, namely SINGLE FRAME INSUFFICIENCY, FRAME ORDER SENSITIVITY and  FRAME CONTRIBUTION PARITY. Three principles are then quantified by a metric to reflect the headroom provided by indivisual benchmarks. Finally, a new new benchmark named TVBench with 1484 carefully curated examples are presented and a diverse set of general purpose MFMs are benchmarked against TVBench.

The paper is easy to follow and the key concepts are delivered sound and clear. I appreciate the details the authors provided in the appendix about the analysis and the dataset.

strengths:
The paper is easy to follow and the key concepts are delivered clearly. The authors start by setting three principles and studied the existing video benchmarks around these three principles quantitatively, revealing the opportunities and curating datasets around them, I think the approach to the problem and processes are rigorous and effective. Their final results on evaluating MFMs on their datasets also show big opportunties for current system to improve in terms of temporal reasoning.

weaknesses:
I have some concerns and questions for discussion.

* Clarification on Table 6. It is not very clear to me what "True" & "False" stand for in the table. More explanation and details on the experiment setups are needed. This leads to the question of how the conclusion "More Capable Models Are More Likely to Exploit Shortcuts Through Common Sense." is drawn. Does "True" and "False" mean different split of datasets? If so, those videos and their QAs are different, how could we compare their relative performance drop and reach the conclusion?

* L479 "Models Excel in First-Person Over Third-Person Perspective Temporal Reasoning Video Understanding." The conclusion of model excels FPV over TPV videos on comparing experiment results of 88 FPV QA v.s. 668 TPV QA. Since these are two set of different videos and QAs, it is hard to compare the final score side by side and draw the above conclusion in my opinion. More rigorous study or careful experiment design is needed, for instance, creating a matched set of FPV and TPV videos covering the same scenarios would make more sense.

* References to the visual temporal reasoning principles are needed. Some principles discussed in the manuscript for temporal reasoning in video understandings have been studied and revealed in previous literatures, e.g. "Single Frame Insufficiency" has been discussed in [1]; "Frame Order Sensitivity" has been explored/discussed in many literature, e.g. [2]; [3][4] studied "FRAME CONTRIBUTION PARITY" as well. Having a brief literature review section for each principle, and discuss the difference and contributions of current work could strengthen the manuscript. 

[1] Lei, J., Berg, T.L. and Bansal, M., 2022. Revealing single frame bias for video-and-language learning. arXiv preprint arXiv:2206.03428.

[2] Qian, R., Li, Y., Yuan, L., Gong, B., Liu, T., Brown, M., Belongie, S., Yang, M.H., Adam, H. and Cui, Y., 2021. Exploring temporal granularity in self-supervised video representation learning. arXiv preprint arXiv:2112.04480.

[3] Huang, De-An, Vignesh Ramanathan, Dhruv Mahajan, Lorenzo Torresani, Manohar Paluri, Li Fei-Fei, and Juan Carlos Niebles. "What makes a video a video: Analyzing temporal information in video understanding models and datasets." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7366-7375. 2018.

[4] Liu, X., Pintea, S.L., Nejadasl, F.K., Booij, O. and Van Gemert, J.C., 2021. No frame left behind: Full video action recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 14892-14901).

questions:
* The author seem not mention it in the paper - will the benchmark be made publicly available at some point?
* The author selectively evaluate some proprietary MFMs in Table5. It would be interesting to report the results of different model sizes in the same model family  (e.g. Gemini 1.5 pro v.s. Gemini 1.5 flash; Claude 3.5 Sonnet v.s. Opus) to understand the scaling effect of current SOTA MFMs on the proposed benchmarks.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730670158992

thread_type:
official_review

<thread_object>
creator: reviewer_tp5z

<content>
summary:
This paper investigates whether current multimodal foundation models can perform visual temporal reasoning, which is a complex task requiring models to incorporate visual and temporal information simultaneously. The authors show that current benchmarks on video understanding may be insufficient to truly test the model's capabilities in visual temporal reasoning and introduce TVBench which consists of challenging video and question pairs that expose a significant gap between multimodal foundation models and humans, hence exhibiting limitations in current multimodal models.

strengths:
- The paper tackles an important aspect of video understanding, which with benchmarks that are not carefully designed, might be overestimated. 
- The dataset seems to consist of well designed and curated data, as their design philosophy matches the results on the three metrics they propose. This is likely due to the rigorous quality check, and the diversity of videos (including simulations) that the authors include in the data.
- Intensive experiments and the analyses provide valuable insights into the performance of multimodal foundation models on video data.

weaknesses:
- If the authors could include an experiment to test the effect of language prior - i.e. the model's accuracy when only given the text part of the dataset questions, and report the difference, that might also help reinforce the soundness of their benchmark design.
- If the authors could provide some information on the resolution and scale of the videos, that might be helpful especially because the authors discuss models not benefiting from more than 8 frames.

questions:
1. What would happen if corresponding accuracy is zero in the denominator for the three metrics? Might need an epsilon term, or it might just be sufficient to subtract the two accuracies.
2. Why might there be a performance plateau beyond 8 frames? Is it in the design of common foundation models? Is it likely due to there being too much information to process (for example, the way the models process more than 8 images) ? Do various models show the plateau behavior exactly at 8 frames? What about at smaller frames? 
3. How does the performance of zoomed-in views differ when only tested on simulation videos with less noise?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730691818310

thread_type:
official_review

<thread_object>
creator: reviewer_rs5b

<content>
summary:
This paper tackles the problem of benchmarking temporal reasoning using multimodal foundation models. First, the paper establishes some principles that temporal reasoning benchmarks must follow in order to test temporal reasoning capabilities, and shows that existing benchmarks fail to cater to these principles. Second, a new manually annotated benchmark, comprising six question types with some counterfactual modifications, is proposed. Third, several existing models are benchmarked demonstrating a big difference between human performance and best model's performance. And finally, some ablation studies are presented that show how one can improve models capabilities on this benchmark.

strengths:
The paper is written well and easy to follow. It addresses an important area in video understanding research, where several benchmarks are known to have shortcomings in actually requiring all the frames of a video [1].
- The benchmark is manually curated in a three-stage process, potentially yielding very high-quality QA annotations
- Several models including proprietary ones lag significantly behind humans. This verifies that the benchmark is indeed difficult for existing models.
- A commendable effort is made in conceptualizing the principles defining how to check whether a benchmark requires temporal reasoning, and its shown that TVBench is way ahead of other benchmarks in adhering to these principles as shown in Tables 1-4
- Some of the ablations / analysis about existing models' capabilities are interesting, i.e. the plateauing at 8 frames compared with humans, exploiting common sense shortcuts etc.

[1]: Revealing Single Frame Bias for Video-and-Language Learning

weaknesses:
The main weakness in my opinion is that the defined principles and ways of measuring benchmarks' adherence to these principles is not very rigorous.

1: Single frame insufficiency: 
- Why is the definition limited to a single frame? What about QA pairs that can be answered in just a few frames, say 2 frames? In the abstract, it's argued that such a scenario is bad for temporal reasoning benchmark, however, this principle will consider such a question as good.
- In the second implementation, a single hand-picked frame is compared with uniformly sampled 16 frames. This seems a bit unfair to the 16 frame case. Shouldn't we compare hand-picked 16 frames with hand-picked 1 frame to get a true sense of single frame insufficiency? 

2: Frame order sensitivity:
The definition assumes that frame order sensitivity is essential for a good temporal reasoning benchmark. However, a powerful model human can easily figure out the temporal order in a lot of cases. So, I don't know if this needs to be a hard requirement.

 
3: Frame contribution parity:
Implementation-wise, again why only a single frame is considered? The proposed implementation will not give any signal for a QA pair requiring 2 frames.

For example, a benchmark that simply contains state transition questions requiring exactly 2 frames will get perfect scores on these three principles. However, such a benchmark is not very valuable. In effect, these principles can be considered necessary conditions (second one is a bit questionable), but not sufficient conditions for a benchmark to be considered as requiring temporal reasoning.

Lastly, the way of evaluating these principles using pre-existing models creates a chicken and egg problem, as the models themselves may not be capable of answering the said questions. For example, a very hard benchmark that GPT-4o cannot answer will fail all the principles if we use GPT-4o to evaluate the principles.

A couple of unrelated issues are:
- Many of the proposed question types already are part of existing works, so it’s unclear how this benchmark would be any different in satisfying the principles. Is the difficulty coming from the annotation process and counterfactuals?
- The benchmark may contain a lot of unrealistic questions because of (i) counterfactuals and (ii) simulated data that are used to artificially make the benchmark harder. These may affect the usefulness in real-world applications.

questions:
Audio inputs (speech) are not discussed in this benchmark. What happens if audio (speech) is included as part of video inputs to the models? Will the results on principles and and on TVBench change?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733291966617

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response - Part 2/2

comment:
We greatly appreciate the reviewers' comments and feedback, which have been invaluable in improving our work. We believe we have addressed all the raised concerns, questions, and suggestions raised by the reviewers, including:

- Reviewer rS5B: \
(1) Principles are not very rigorous: We have clarified our principles are intended to serve as a **diagnostic** tool and explained the motivation for each principle at [openreview response link](https://openreview.net/forum?id=fCi4o83Mfs&noteId=cpfjngP43a).\
(2) Question overlap with the existing benchmark: We clarified that although there exists overlap in the *Action Count*, in inclusion of six reasoning types, TVBench aims to _comprehensively_ evaluate all aspects of visual temporal reasoning of current models. Details about QA re-annotation can be found in Section 4.3 and task examples in Table 6.\
(3) Contain unrealistic questions: We have clarified the “unrealistic” questions (i) constitute a small portion of the benchmark, and (ii) are intended to evaluate model capabilities in handling out-of-distribution tasks. We included additional analysis to illustrate this point in the revised Section 6.3, revealing that models (i) **fail to truthfully leverage the visual input while being over-reliant on common sense**, and (ii) **are highly susceptible to noisy information in the input**.

- Reviewer tp5Z:\
(1) Potential ethical concern for video collection: We clarified that all videos sourced from YouTube are licensed under CC 4.0 and more license information can be found in Appendix F. \
(2) Experiments to test language prior: We clarified that our text-only baseline (Random (GPT-4o)) in Table 5 is intended to address this.\
(3) Suggestions including resolution and mathematical stability: We have included video resolution information in Table 9 and added epsilon term in Section 3 in our updated manuscript.\
(4) Analysis with smaller number of frames: We have added model performance across 0, 1, 2, 4, 6, 8 frames in Figure 5 in our updated manuscript. 

- Reviewer NXCv:\
(1) Table clarifications: We have added additional clarifications on Table 6 in our updated manuscript.\
(2) Comparison of FPV and TPV requires a more rigorous study: We clarified that theses study intend to suggest general trends in model performance for future research and have carefully rephrased our findings (Appendix B Scenario Analysis). \
(3) References for the principles: We have included references for each principle in our updated manuscript (Section 3).

- Reviewer xcjw: \
(1) 16 frames as the multiple-frame input seems arbitrary: We have studied 1, 8, 16, 24, 32 frames as the multi-frame input, and experiments have shown that as performance stabilizes around 8 frames, 16 frames provide sufficient window for analysis (Figure 4 in Appendix E.3). \
(2)  Insufficient Text-Only Baseline in Table 5: We have clarified our result in pilot study ([openreview response link](https://openreview.net/forum?id=fCi4o83Mfs&noteId=q51rylzyHo)) and have added additional analysis in Section 6.3, revealing that models **lack the basic ability to interpret frames as a continuous sequence**. \
(3) Fairness in benchmark comparison: We have clarified that the selection of 200 QAs for each benchmark is done through proportional random sampling across all reasoning types (if applicable).
</content>
</thread_object>

</thread>
<thread>
creation date:
1733292685377

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response - Part 1/2

comment:
We express our sincere gratitude to the reviewers for their insightful comments and constructive feedback.

We appreciate the reviewer’s recognition of key aspects of our work:
- **The proposed benchmark addresses important shortcomings in existing benchmarks**: \
“It addresses an important area in video understanding research” (reviewer rS5B). \
“The paper tackles an important aspect of video understanding, which with benchmarks that are not carefully designed, might be overestimated.” (reviewer tp5Z). 
- **The proposed benchmark is rigorously constructed with three quantitative principles**: \
“The benchmark is manually curated in a three-stage process, potentially yielding very high-quality QA annotations” (reviewer rS5B). \
“The dataset seems to consist of well designed and curated data … This is likely due to the rigorous quality check … ” (reviewer tp5Z). \
“I think the approach to the problem and processes are rigorous and effective” (reviewer NXCv). \
“The benchmark is rigorously constructed with three quantitative principles that emphasize the complexity of temporal reasoning tasks” (reviewer xcjw).
- **The experiments and analysis are comprehensive and insightful**:\
“Some of the ablations / analysis about existing models' capabilities are interesting” (reviewer rS5B). \
“Intensive experiments and the analyses provide valuable insights into the performance of multimodal foundation models on video data.” (reviewer tp5z).\
“The benchmark includes a wide range of tasks and scenarios, providing thorough coverage of diverse temporal reasoning challenges” (reviewer xcjw).
- **The paper is well-written and easy to follow**:\
	“The paper is written well and easy to follow.” (reviewer rS5B). \
	“The paper is easy to follow and the key concepts are delivered clearly.” (reviewer NXCv). \
	“Clear and Cohesive Writing: The paper is well-written, presenting its ideas and findings in a clear and organized manner, making it accessible and informative for a broad audience.” (reviewer xcjw).
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposed a new benchmark to evaluate the visual temporal reasoning of existing models in video understanding task. It targets to resolve limitations in current video understanding benchmarks, which most questions can be answered without visual temporal reasoning. It proposed three quantitative principles (e.g., Multi-Frame Gain, Frame Order Sensitivity, and Frame Information Disparity.) and rigorously construct corresponding metrics based on them. A new benchmark with carefully designed questions are proposed. Several existing MFMs are benchmarked and results show big gap between these MFMs and human performance, exhibiting limitations in these MFMs.

Strength:
1. It addresses an important problem.
2. The metrics are rigorously designed.
3. Experiments are extensive and comprehensive.
4. Paper is well written.

Weakness:
1. Based on the discussion between the reviewer rS5B, the proposed principles serve as diagnostic tools which reflect some aspects of visual temporal reasoning but may not be comprehensive. The authors may want to make this point clear in their paper.