Kronecker Mask and Interpretive Prompts are Language-Action Video Learners

Authors: Yang JingYi, Zitong YU, Nixiuming, He Jia, Hui Li
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Action Recognition, Video Recognition, Spatiotemporal Modeling

Abstract:
Contrastive language-image pretraining (CLIP) has significantly advanced image-based vision learning. A pressing topic subsequently arises: how can we effectively adapt CLIP to the video domain? Recent studies have focused on adjusting either the textual or visual branch of CLIP for action recognition. However, we argue that adaptations of both branches are crucial. In this paper, we propose a **C**ontrastive **L**anguage-**A**ction **V**ideo Learn**er** (**CLAVER**), designed to shift CLIP's focus from the alignment of static visual objects and concrete nouns to the alignment of dynamic action behaviors and abstract verbs. Specifically, we introduce a novel Kronecker mask attention for temporal modeling. Our tailored Kronecker mask offers three benefits 1) it expands the temporal receptive field for each token, 2) it serves as an effective spatiotemporal heterogeneity inductive bias, mitigating the issue of spatiotemporal homogenization, and 3) it can be seamlessly plugged into transformer-based models. Regarding the textual branch, we leverage large language models to generate diverse, sentence-level and semantically rich interpretive prompts of actions, which shift the model's focus towards the verb comprehension. Extensive experiments on various benchmarks and learning scenarios demonstrate the superiority and generality of our approach. The code will be available soon.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1730432220464

thread_type:
official_review

<thread_object>
creator: reviewer_scy3

<content>
summary:
This paper proposes a Contrastive Language-Action Video Learner (CLAVER) to efficiently adapt both the visual and textual branches of the CLIP for improving video action recognition. CLAVER introduces the Kronecker mask for temporal modeling to capture the long-range and wide-range dependencies among frames. Moreover, CLAVER resorts to LLMs to generate interpretive prompts of actions to facilitate the alignment of action behaviors and verbs. Experimental results on multiple benchmarks, such as Kinetics-400 and Kinetics-600, show the superiority of the proposed method.

strengths:
1 The proposed method incorporating the Kronecker mask and interpretive action prompts is demonstrated to be effective in supervised, few-shot, and zero-shot settings.

2 The visualization results sufficiently show that the proposed method captures better spatial-temporal attention as well as focuses more on verbs.

3 The proof of KMCTA can guarantee full rank is provided in detail in the Appendix. 

4 The paper is well-written with figures and tables nicely presented.

weaknesses:
Overall, I think this paper introduces an effective and generalizable mechanism for improving video action recognition. The proposed method is novel and supported by extensive experimental results in both the main text and Supp. I only have some small concerns as follows:

1 Since the LLM tends to introduce some task-agnostic information in its output, will this noise impact the actual action recognition performance? I would like to see more analysis and examples here. Moreover, the proposed interpretive prompts look like a plug-and-play module. When it is combined with other existing models, will it contribute to performance improvement consistently? 

2 Can the authors give a complete analysis with examples to show when the proposed method may fail? This will help better understanding and deeper analysis of the proposed method.  

3 The font size in Figure 6 is too small for reading. The authors should modify it to give a better presentation. Moreover, Table 14 in Supp has the wrong format.

questions:
See details in the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730648321325

thread_type:
official_review

<thread_object>
creator: reviewer_2oca

<content>
summary:
The paper introduces a novel framework called Contrastive Language-Action Video Learner (CLAVER), aimed at enhancing the adaptation of CLIP for video understanding, particularly in action recognition tasks. This research addresses a critical gap in existing methodologies, which often focus on modifying either the textual or visual components of CLIP in isolation. The authors argue that a dual adaptation of both branches is essential for effective video analysis. The key contributions are twofold: (1) Kronecker Mask Attention that significantly improves temporal modeling in video data (2) The authors utilize large language models to generate rich, sentence-level prompts that focus on action verbs, steering the model's learning towards a deeper understanding of dynamic behaviors rather than static objects. The extensive experiments across multiple benchmarks, showcasing the effectiveness and robustness of CLAVER compared to existing methods.

strengths:
* The paper is well-written and easy to understand. The figure illustration and captions are informative.
* The proposed Kronecker mask temporal attention and Kronecker mask causal temporal attention schemes seem interesting. The authors have proved its effectiveness for temporal modeling
* Experimental results on four widely used benchmarks show that the proposed model could achieve superior performance with the existing contrastive baselines.

weaknesses:
* The performance gain is marginal. The advantages on multiple benchmarks compared to the current state-of-the-art methods do not show a significant improvement, which somewhat undermines the effectiveness of this approach. It remains unclear whether the enhancements stem from the hyperparameter settings or other tricks.
* Can the authors provide details on how much additional computational cost and efficiency are incurred by adding the extra Kronecker mask temporal attention compared to the original CLIP method?

questions:
Refer to weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730937725717

thread_type:
official_review

<thread_object>
creator: reviewer_rydq

<content>
summary:
The author proposes CLAVER: a Contrastive Language-Action Video Learner, to shift CLIP’s focus for video action recognition tasks from the alignment of static visual objects and nouns to dynamic action behaviors and abstract verbs. They introduce CLAVER with generalizable Kronecker Mask Temporal Attention (KMTA) to expand the temporal receptive field for each token and serve as an effective spatiotemporal heterogeneity inductive bias, mitigating spatiotemporal homogenization. In addition, they utilize large language models to generate interpretive prompts of actions, which shift the model’s focus towards verb comprehension. They conduct extensive experiments on well-known action recognition datasets like Kinetics-400, Kinetics-600, HMDB-51, and UCF-101 demonstrating its competitive performance across models in full-shot, few-shot, and zero-shot settings.

strengths:
•	Innovative Kronecker Mask Attention: The Kronecker mask temporal attention and Kronecker mask causal temporal are innovative and effective for temporal modeling. Compared with previous commonly used approaches such as joint attention, spatial attention, pipeline temporal attention, etc., it allows each patch at timestamp t to interact with all other patches and expands the temporal receptive field width of each token. It also alleviates the impact of spatiotemporal homogenization.

•	Leveraging large language models to create diverse, sentence-level prompts for action verbs is an effective approach. The author focuses on action decomposition, synonym conversion, which is an inspiring idea of “facilitating the alignment of action behaviors and verbs” by decomposing complex actions into basic and conveying the same core concept with varied expressions. 

•	This paper is well-written and provides a clear overview of CLAVER, with Kronecker Mask Temporal Attention and interpretive prompts. It conducts extensive experiments across several commonly used benchmarks with previous state of the art models as comparison and shows competitive performance, especially in few-shot and zero-shot settings, and provides comprehensive ablation studies to clearly claim the strength of CLAVER by parts.

weaknesses:
•	Insufficient performance difference between KMTA and KMCTA. KMCTA aims to alleviate the low-rank bottleneck and the author claims that it is important to avoid the low-rank bottleneck to improve the representation power of transformer architecture by giving a formal proof. In the latter ablation studies, the author also successfully shows that KMCTA is profoundly affected by both PreTE and PostTE shuffling, which suggests that KMCTA possesses varying degrees of ability in mitigating spatiotemporal homogenization. However, based on the experiment results in Table. 1 and Table. 7, the difference between KMTA and KMCTA is marginal and we also lack experiments on zero-shot, few-shot settings for CLAVER with KMCTA, which I didn’t see how KMCTA solved the bottleneck according to the current experiments setting.

•	The pipeline of interpretive prompts. The author introduces an interesting pipeline of interpretive prompts by performing action decomposition, synonym conversion, and involving body parts. Then, they provide the prompt to LLaMA-3 for text completion. However, based on the illustration in Figure 4, and the textual description, I feel confused about how to make up these three steps from the left side of the figure to the right side of the figure into format prompts to LLaMA in order to perform text completion.

•	Qualitative analysis for word importance needs further proof. Based on the visualization for CLIP, X-CLIP in Figure. 5, while the author make a claim that CLIP tends to focus on nouns, whereas CLAVER prefers verbs, it’s hard for me to recognize the difference between CLIP and CLAVER’s visualization. As a result, the effectiveness of interpretive prompts for noun concept to verb concept transition might need further explanation.

questions:
In addition to the questions raised in the weakness section, there are a few minor questions.

•	In Figure 4, the transaction from the left side to the right side might need further explanation. In Figure 5, the example might not fully explain the claim for word importance, please consider making another effective example. 

•	In Table 1, we can see the most powerful model for CLAVER is the CLAVER-L/14(KMT/KMCT). However, when conducting later experiments on Kinetic-600, and other few-shot, zero-shot experiments, the author uses CLAVER-B/16 (KMT) but not the strongest model in Table.1 even if it can follow the same setting. I’m wondering the reason why the author didn’t use their best model for all experiments.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731214487274

thread_type:
official_review

<thread_object>
creator: reviewer_rnm4

<content>
summary:
This paper focuses on video action recognition tasks and presents CLAVER to improve CLIP by aligning video representations with verb understanding. The authors design the Kronecker mask attention for improved temporal modeling and interpretive prompts generated by large language models to enhance verb comprehension. The authors demonstrate that CLAVER shows competitive performance on benchmarks such as Kinetics-400, Kinetics-600, HMDB-51, and UCF-101.

strengths:
1. The paper observes two challenges when applying CLIP to video action classification: How to perform effective temporal modeling and how to design suitable text descriptions for verb understanding. The authors then propose the Kronecker mask temporal attention (KMTA) and Kronecker mask causal temporal attention (KMCTA) to solve 1) and interpretive prompts to solve 2).
2. The paper conducts experiments on multiple video action classification datasets under supervised, few-shot, and zero-shot settings. The results show improved performance over baseline models and other state-of-the-art approaches.
3. The paper includes thorough ablation studies to isolate the contribution of each component, such as KMTA and interpretive prompts, providing evidence for their effectiveness.

weaknesses:
1. My major concern about this work is its novelty. The proposed Kronecker Mask Attention seems to be a simple combination of pipeline temporal attention and joint attention by attending to all patches in other frames. I expect the authors to provide more insightful analysis and discussion about the proposed method and the differences from previous works.
2. The computational cost introduced by the Kronecker mask attention and the interpretive prompts could be further detailed, especially regarding training and inference time compared to other baselines.

questions:
See weakness section
</content>
</thread_object>

</thread>
<thread>
creation date:
1732115917509

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary: Reply to all reviewers

comment:
We are deeply grateful to all the reviewers for their thoughtful consideration and constructive suggestions of our paper. We also appreciate their recognition of the novelty and the fluent writing. Based on the initial reviews, we have summarized their questions and provided corresponding responses, which mainly cover the following aspects:

+ **Detailed Comparison of Computation Cost and Efficiency** (proposed by reviewers rNM4 and 2oCA). According to the reviewer's request, we have listed a comprehensive comparison table.

+ **The Pipeline and Potential Effects of Interpretive Prompts** (noticed by reviewers RydQ and sCy3). We have provided detailed explanations and analysis under the comments of the relevant reviewers.

+ **Insightful Motivation and Analysis, Effectiveness of Our Spatiotemporal Modeling Approach** (highlighted by reviewer rNM4 and 2oCA). Please refer to the corresponding responses, as well as our emphasis on contributions below.

+ **More Representative Visualization of Conversion of Word Importance from Nouns to Verbs** (pointed by reviewer RydQ). We have provided Fig. 15 in the appendix of the rebuttal revision.

Again, we want to emphasize that this paper is dedicated to achieving three primary contributions:

+ Exploring **a General Spatiotemporal Modeling Approach** (**Kronecker Mask Attention**) and **Revealing the Intrinsic Correlations of Commonly Used Spatiotemporal Modeling Methods** (as mentioned in Sec. 3.1 and Fig. 2, most common spatiotemporal attention can be uniformly classified as Kronecker Mask Attention). We propose the Kronecker mask temporal attention (KMTA) for temporal modeling, aiming to capture the long-range and wide-range dependencies among frames with spatiotemporal heterogeneity. And further improve KMTA to Kronecker mask causal temporal attention (KMCTA) to alleviate the low-rank bottleneck of KMTA. Through multiple visualizations in Fig. 6-8, 12-14, we illustrate the interpretability of spatiotemporal modeling.

+ Proposing **a Potential Universal Tool for Understanding Abstract Verbs** (**Interpretive Prompts**), which encompasses action decomposition, synonym conversion, and collaboration with body parts. By doing so, it promotes the model's attention and understanding of action concepts, as demonstrated through visualizations in Fig. 5, 15. Additionally, it also contributes to zero-shot generalization and helps with the action recognition/understanding in open scenarios.

+ Integrate the above two techniques to efficiently **Shifting the Alignment in CLIP (VLMs) from Visual Objects and Nouns to Action Behaviors and Verbs**, tapping into the potential of VLMs in video understanding.

Furthermore, in accordance with the reviewers' suggestions, we have made several modifications to the article in the rebuttal revision.  Specifically, we update Fig. 4 and Fig. 6, as well as Tab. 8 and Tab. 14. Additionally, we provide more representative visualizations of the word importance in Fig. 15 and the rank of KMTA, KMCTA and joint attention with respect to various patch size and frame length in Fig. 16. The modified parts are marked in $\textcolor{blue}{\mathrm{blue}}$ font.

We are looking forward to having discussions with the reviewers in the next few days, focusing on the parts that might still be ambiguous in the rebuttal and other relevant issues. We sincerely welcome them to join in the discussion.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The submission addresses the problem of adapting contrastive language-*image* pretrained models (CLIP) to video domains. It introduces "Contrastive Language-Action Video Learner" (CLAVER), which adapts not only the visual branch (via "Kronecker Mask Attention" for temporal modeling) but also the text branch (via LLM-generated prompts that focus on verbs) of the CLIP model. After rebuttal, all four reviewers recommend acceptance of the submission, with one accept (8) and three borderline accepts (6). Most of the concerns were addressed by the authors' rebuttal, with minor remaining concerns on the novelty of the introduced "Kronecker Mask Attention" (rNM4). Reviewer RydQ had concerns regarding KMCTA, which the AC believes has been mostly addressed by the rebuttal. Overall, the AC agrees with the reviewers and believes the submission is ready to be published at ICLR 2025.