Adaptive Memory Mechanism in Vision Transformer for Long-form Video Understanding

Authors: Zhenshun Liu, Zijian Lei, Kejing Yin, William K. Cheung
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Key-Value Cache, Vision Transformer, Video Understanding

Abstract:
In long-form video understanding, selecting an optimal Temporal Receptive Field (TRF) is crucial for Vision Transformer (ViT) models due to the dynamic nature of diverse video motion contents, which varies in duration and velocity. A short TRF can result in loss of critical information, while a long TRF may decrease ViT's performance and computational efficiency caused by the unrelated contents in videos and the quadratic complexity of the attention mechanism. To tackle this issue, we introduce Adaptive Memory Mechanism (AMM) that enables ViT to adjust its TRF dynamically in response to the video's dynamic contents. Instead of discarding Key-Value (KV) Cache from the earliest inference when the settings limit is reached, our approach uses a Memory Bank (MB) to retain the most important embeddings from the Key-Value Cache that would otherwise be discarded in memory-augmented methods. The selection is based on the attention score calculated between the Class Token (CLS) in current iteration and the KV Cache in previous iterations. We demonstrate that Adaptive Memory Vision Transformer (AMViT) outperforms existing methods across a diverse array of tasks (action recognition, action anticipation, and action detection).

Decision: reject
Reviewer scores: 5, 5, 5, 3, 5

Reviews and discussion:
<thread>
creation date:
1729721965702

thread_type:
official_review

<thread_object>
creator: reviewer_j6ew

<content>
summary:
This paper introduces an Adaptive Memory Mechanism (AMM) to improve Vision Transformers (ViT) for long-form video understanding. AMM dynamically adjusts the Temporal Receptive Field (TRF) based on video content, overcoming limitations of fixed TRF approaches that either lose key information or increase computational costs. Experiments show that AMViT, integrating AMM, outperforms existing models like MeMViT in tasks such as action recognition, anticipation, and detection, while reducing computational overhead, validated on datasets like AVA and Epic-Kitchens.

strengths:
1. Long-form video understanding is an important task, and efficiency is indeed a crucial metric in this context.

2. The proposed method can reduce both training and inference costs.

3. Introducing a memory bank to handle long sequence inputs is intuitive and reasonable.

weaknesses:
1. (important) The number of benchmarks (only 2) and baselines (also only 2) compared seems somewhat limited. Adding more experiments would make the paper more convincing.

2. (important) Although the authors emphasize that the new architecture is designed for long-form video, this aspect is not discussed in the experimental section. Are the benchmarks presented in the paper truly for long videos, and what is the average input length? It would have been better if the authors had conducted more detailed evaluations on benchmarks like MovieChat-1K [1] or LongVideoBench [2].

3. The writing and figures in the paper need improvement, especially regarding the notation for memory. There are too many subscripts and superscripts, along with the extensive use of qkv notations, which made it take me three times longer to understand the entire paper. 

[1] Song, Enxin, et al. "Moviechat: From dense token to sparse memory for long video understanding." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Wu, Haoning, et al. "Longvideobench: A benchmark for long-context interleaved video-language understanding." arXiv preprint arXiv:2407.15754 (2024).

questions:
Please revise the Weaknesses section point by point. This is a paper with great potential. If the authors can provide additional responses to certain issues, discuss related work more thoroughly, and include more experiments and observations, I would be very happy to raise my score.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730644416065

thread_type:
official_review

<thread_object>
creator: reviewer_a8ag

<content>
summary:
The paper proposes an Adaptive Memory Mechanism (AMM) for Vision Transformer (ViT) in long-form video understanding. It addresses the issue of selecting an optimal Temporal Receptive Field by allowing ViT to adjust TRF dynamically. Instead of directly discarding early Key-Value cache, AMM uses a Memory Bank to retain important embeddings from the Key-Value cache based on attention scores. Experiments on AVA and Epic-Kitchens show the advantages of AMM in action recognition, anticipation, and detection tasks.

strengths:
1.Long-form video understanding is an important research task, and the author has provided a reasonable solution.

2.The paper is well-written, making it easy to read.

weaknesses:
1.The novelty of memory bank is limited. Many studies have explored how to utilize memory to retain important historical information and how to dynamically update memory. For example, Xmem[1] prioritizes retaining the most frequently used candidates. MA-LLM[2] and MovieChat[3] merge the two most similar candidates based on similarity once the memory bank capacity is exceeded. The innovations and advantages of the memory bank proposed in this paper compared to these methods are unclear.

2.The fairness of the experiment is in question. When comparing with the baseline model MeMViT, the authors replaced the backbone of MeMViT from MViT to UMT. This seems to have led to a decline in the performance of the baseline model. For example, in the EPIC-KITCHEN-100 action recognition task, the performance reported in the original paper on MeMViT was 48.4%, while the performance presented in this paper is 43.03%. The authors should maintain the same settings as MeMViT for the experiments to make the results more credible.

3.The performance improvement is limited. Compared to the baseline model MeMViT, the performance improvement is less than 1% in all experiments.

4.Lacks of comparison with the latest methods. This article only presents comparisons with ViT and MeMViT. Some recent methods are missing, such as MAT[4] and MC-ViT[5].

5.Lacks of necessary ablation studies. (2) This paper uses an input-aware selective module to prevent redundant embeddings from being retained, and uses a memory bank to retain useful embeddings. However, there are no ablation experiments to demonstrate the effectiveness of these two components individually. (2) The lack of ablation experiments on the memory bank update method. For example, comparing the update of the memory bank using attention score of class tokens proposed in this paper with previous methods (see weakness 1) and First-In-First-Out (FIFO).

[1] XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model, ECCV 2022
[2] MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding, CVPR 2024
[3] MovieChat: From Dense Token to Sparse Memory for Long Video Understanding, CVPR2024
[4] Memory-and-Anticipation Transformer for Online Action Understanding, ICCV 2023
[5] Memory Consolidation Enables Long-Context Video Understanding, arxiv 2024

questions:
When comparing with MeMViT, your model uses the memory bank and the selected Q-V cache, while MeMViT only uses Q-V cache. Have you ensured that the number of embeddings in both model is consistent? Specifically, does the size of the memory bank plus the size of the selected Q-V cache match the size of the unselected Q-V cache?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730696005681

thread_type:
official_review

<thread_object>
creator: reviewer_cdcg

<content>
summary:
This paper aims to enhance ViT for long-term video understanding. The authors design a memory bank to store historical information and develop input-aware adaptive memory selection to retrieve the relevant information to assist long-term analysis. The experiments show that the architecture demonstrates satisfactory performance with high efficiency.

strengths:
1. The analysis of the limited temporal receptive field in long-term video understanding makes sense, and the motivation is clear.
2. The method is simple and intuitive.

weaknesses:
1. The experiments are limited. Only AVA and Epic-Kitchens are reported. Results on more video datasets are required to verify the effectiveness of the adaptive memory design. Besides, the performance improvements are marginal.
2. The memory bank is recurrently updated by adaptive selection. Is it possible that in a long video, the content in the middle of the video is not closely related to the beginning, and only relevant content appears towards the end? However, during the memory bank update process, the tokens of the earlier video content were already discarded.

questions:
1. Does the KV Cache in this paper retain the gradient?
2. This paper focuses on a pure vision model with enhanced memory design. However, the ViT-only architecture is capable of a limited range of video-related tasks. Is it possible to integrate it with video-language models to achieve wider range of video tasks to exert more impact on the community?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730697220907

thread_type:
official_review

<thread_object>
creator: reviewer_trqz

<content>
summary:
This paper presents an adaptive memory method to improve the existing memory-augmented methods for long-form video understanding. The method is based on MeMViT but makes the memory bank adaptive to support the adaptive temporal receptive field. The experiments are conducted on Ava and Epic-Kitchens dataset with the comparison with ViT and MeMViT.

strengths:
* Long-form video understanding is an important video research topic and the idea of using an adaptive memory bank sounds reasonable and promising. 
* Compared to MeMViT, the results show consistent improvements though some datasets only have marginal gain.

weaknesses:
* One of the main motivations of the paper is to retain embeddings instead of discarding memory when the memory limit is reached. However, based on the experiments, it's unclear if the effective receptive field of AMViT is indeed larger than MeMViT through the proposed adaptive memory module. Are they still using the same memory bank size?
* In the model section, the paper presents two new modules, including Input-aware selective module (ISM) and Adaptive Memory mechanism(AMM). However, there are no ablations to validate the individual effectiveness of these modules.
* How do we select parameters for MeMViT? Some parameters for MeMViT (Table 6) are not defined, e.g, memory bank size. Is it the same as AMViT? Given the authors are reproducing MeMViT with a different backbone, how the results compare to the original paper.
* In Table 1, it's unclear why all the three methods are having the same FLOPs and parameters given MeMviT and AMViT has additional memory bank modules. It's also better to conduct run-time comparison.
* The experiments are also missing a system-level comparison with the current SOTA results on the benchmarks.

questions:
Please see weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730720321195

thread_type:
official_review

<thread_object>
creator: reviewer_ghkc

<content>
summary:
The paper addresses a solution for better long-form video understanding using a method named Adaptive Memory Mechanism (AMM). This method enables the Vision Transformer (ViT) to adjust its temporal receptive field dynamically depending on the input video. A memory bank is utilized to save the most important Key-Value when temporally processing the videos. The proposed method is tested on AVA and Epic-Kitchens datasets for action detection, recognition, and anticipation tasks. Experiment results show performance improvement to the ViT baselines without additional cost.

strengths:
1. The method have better performance than baselines without additional cost.

weaknesses:
1. The paper lacks SoTA comparisons. Is the task different from common action recognition and action detection? Multiple methods such as VideoMAE, Omnivore, or MMT have been tested on these datasets. It would be helpful if the authors could explain the difference between previous SoTAs with the proposed method, for example in parameter count or GFLOP difference.
2. The improvement to ViT and MeMVit baselines is marginal.
3. There is no difference in the FLOPs and Param(M) numbers compared to the baselines. Can the authors explain further the efficiency advantage achieved by the proposed method?

questions:
1. Will there be a significant performance difference if the model is not pre-trained with UMT?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732542275066

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
SoTA Comparison

comment:
As suggested by all the reviewers, we have added back the SOTA methods suggested by the reviewers and those we further identified to provide a more comprehensive performance comparison. Most of them are the results reported in their original papers, and some are reproduced by us for fair comparison. (* represent the results that are reproduced by ourselves.)

Table 1: Performance Comparison. (Dataset: Epic-kitchen-100; Evaluation Task: Action Anticipation)
| Model         | Pre-train Dataset |Top-1 Acc (%)  |
| ------------- |:-----------------:|:-------------:|
| RU-LSTM [1]   | IN-1K             |13.3           |
| AVT [2]       | IN-1K             |13.6           |
| DCR [3]       | IN-1K             |14.6           |
| TeSTra[4]     | IN-1K             |17.0           |
| MAT [5]       | IN-1K             |18.8           |
| ViT-B*         | K710              |19.3           |
| MeMViT-B*      | K710              |19.5           |
| AMViT-B(Our)  | K710              |19.8           |
| AMViT-L(Our)  | K710              |22.6           |

Table 2: Performance Comparison. (Dataset: Epic-kitchen-100; Evaluation Task: Action Recognition)
| Model         | Pre-train Dataset |Top-1 Acc (%)  |
| ------------- |:-----------------:|:-------------:|
| TSN [6]       | IN-1K             |20.54          |
| LSTA [7]      | IN-1K             |30.33          |
| VNMCE [8]     | IN-1K             |29.00          |
| RU-LSTM [1]   | IN-1K             |33.06          |
| ViT-B*         | K710              |37.12          |
| MeMViT-B*      | K710              |37.87          |
| AMViT-B(Our)  | K710              |38.60          |
| AMViT-L(Our)  | K710              |43.15          |

Table 3: Performance Comparison. (Dataset: AVA; Evaluation Task: Action Detection)
| Model         | Pre-train Dataset |Param(M)|mAP (%)  |
| ------------- |:-----------------:|:------:|:------------:|
| SlowFast[9]   | K600              |59      |27.5          |
| X3D-XL [10]   | K600              |11      |27.4          |
| MViT [11]     | K700              |51      |31.8          |
| ViT-B*         | K710              |86      |28.59         |
| MeMViT-B*      | K710              |86      |29.17         |
| AMViT-B(Our)  | K710              |86      |30.07         |
| AMViT-L(Our)  | K710              | 307    |35.98         |

Table 4: Performance Comparison. (Dataset: Diving48 [14]; Evaluation Task: Action Recognition)
| Model         |Param(M)|Top-1 Acc (%))  |
| ------------- |:------:|:------------:|
|TimeSformer [12]|121    |74.9          |
| MC-ViT [11]*    |86     |74.5          |
| ViT-B*         |86      |75.1         |
| MeMViT-B*      |86      |76.2         |
| AMViT-B(Our)  |86      |77.6         |

**Reference:**
[1] What Would You Expect? Anticipating Egocentric Actions with Rolling-Unrolling LSTMs and Modality Attention. [ICCV’2019]
[2] Anticipative Video Transformer. [ICCV’2021]
[3] Learning to Anticipate Future with Dynamic Context Removal. [CVPR’2022]
[4] Real-time Online Video Detection with Temporal Smoothing Transformers. [ECCV’2022]
[5] Memory-and-Anticipation Transformer for Online Action Understanding. [ICCV’2023]
[6] Temporal Segment Networks for Action Recognition in Videos. [TPAMI’2019]
[7] LSTA: Long Short-Term Attention for Egocentric Action Recognition. [CVPR’2019]
[8] Leveraging Uncertainty to Rethink Loss Functions and Evaluation Measures for Egocentric Action Anticipation. [ECCV’2018]
[9] SlowFast Networks for Video Recognition. [ICCV’2019]
[10] X3D: Expanding Architectures for Efficient Video Recognition. [CVPR’2019]
[11] Improved Multiscale Vision Transformers for Classification and Detection. [CVPR’2022]
[12] Is Space-time Attention All You Need for Video Understanding? [ICML’2021]
[13] Memory Consolidation Enables Long-Context Video Understanding. [ICML’2024]
[14] RESOUND: Towards Action Recognition without Representation Bias. [ECCV’2018]
</content>
</thread_object>

</thread>
<thread>
creation date:
1732542320652

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Module Ablation

comment:
Table 5 Ablation study based on the Epic-Kitchen dataset and the evaluation task of action recognition. In the experiment, a 12-layer ViT-B is adopted for all the methods. Settings for each transformer are reported. Note that the number of compressed KV pairs obtained via CNN per video segment is 197. The accuracy reached the maximum when the selection size is 60 where the total number of augmented compressed KV pairs is 170 as compared to 394 for MeMViT.
| Model              | Memory length (m) | Total # of compressed <br> KV pairs (m*197) | Memory bank size (L) | Selection size (k) |Total # of selected <br> KV pairs (m*k)|Total # of augmented <br> KV pairs (L+m*k) | Acc.   |
|---------|------------|--------------|-------------|--------------|-------------|--------------------|--------|
| ViT-B              | 0            | N/A               | N/A            | N/A          | N/A                   |N/A           | 37.12  |
| MeMViT-B       | 1         | 197               | N/A        | N/A          | 197         |197       | 37.96  |
| MeMViT-B        | 2       | 394        | N/A            | N/A         | 394           |394         | 37.87  |
| AMViT-B (no Memory Bank) | 2    | 394           | 0         | 50                | 100        |100          | 38.12  |
| AMViT-B        | 1             | 197              | 50         | 50         | 50         |100         | 38.18  |
| AMViT-B            | 2         | 394              | 50        | 30         | 60         |110         | 38.43  |
| AMViT-B            | 2         | 394     | 50        | 50       | 100         |150            | 38.45  |
| AMViT-B            | 2        | 394    | 50         | 60         | 120         |170         | 38.60  |
| AMViT-B            | 2           | 394        | 50      | 70          | 140         |190          | 38.51  |
</content>
</thread_object>

</thread>


Metareview:
metareview:
(a) Scientific Claims and Findings

The paper introduces an Adaptive Memory Mechanism (AMM) for Vision Transformers (ViT) aimed at improving long-form video understanding. AMM allows ViT to dynamically adjust its Temporal Receptive Field (TRF) based on video content, utilizing a memory bank to retain important Key-Value pairs. The method is evaluated on AVA and Epic-Kitchens datasets, demonstrating performance improvements over ViT baselines without additional computational cost.

(b) Strengths

Reviewer gHkC highlights that the method improves performance over baselines without extra computational cost. TRqz and j6eW note that the approach is relevant to the important task of long-form video understanding and is efficient in reducing both training and inference costs. CdCg and a8AG appreciate the clarity and readability of the paper, with CdCg also noting the simplicity and intuitiveness of the method.

(c) Weaknesses

A major concern, as pointed out by reviewers gHkC, TRqz, and a8AG, is the lack of comparisons with state-of-the-art methods, which raises questions about the method's relative performance. TRqz and CdCg also mention that the performance improvements are marginal. The experiments are limited to two datasets, as noted by CdCg and j6eW, and the paper lacks ablation studies to validate the contributions of individual modules, as highlighted by TRqz and a8AG. Additionally, a8AG questions the novelty of the memory bank, and there are concerns about the fairness of the experimental setup, particularly in comparisons with MeMViT.

(d) Decision Reasons 

The AC aligns with the reject recommendation of all reviewers. The decision to reject the paper is primarily due to the lack of comprehensive comparisons with state-of-the-art methods and limited experimental validation, as noted by reviewers gHkC, TRqz, and j6eW. The marginal performance gains over existing methods, as mentioned by TRqz and CdCg, do not provide sufficient justification for acceptance. Furthermore, the paper's contributions are not deemed novel or significant enough compared to existing work, as pointed out by a8AG. Overall, while the paper addresses an important problem, it requires more robust experiments, comparisons, and analyses to strengthen its contributions.