Kronecker Mask and Interpretive Prompts are Language-Action Video Learners
Authors: Yang JingYi
Zitong YU
Nixiuming
He Jia
Hui Li
Venue: ICLR 2025 Poster (2025)
Keywords: Action Recognition
Video Recognition
Spatiotemporal Modeling
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Contrastive language-image pretraining (CLIP) has significantly advanced image-based vision learning. A pressing topic subsequently arises: how can we effectively adapt CLIP to the video domain? Recent studies have focused on adjusting either the textual or visual branch of CLIP for action recognition. However, we argue that adaptations of both branches are crucial. In this paper, we propose a **C**ontrastive **L**anguage-**A**ction **V**ideo Learn**er** (**CLAVER**), designed to shift CLIP's focus from the alignment of static visual objects and concrete nouns to the alignment of dynamic action behaviors and abstract verbs. Specifically, we introduce a novel Kronecker mask attention for temporal modeling. Our tailored Kronecker mask offers three benefits 1) it expands the temporal receptive field for each token, 2) it serves as an effective spatiotemporal heterogeneity inductive bias, mitigating the issue of spatiotemporal homogenization, and 3) it can be seamlessly plugged into transformer-based models. Regarding the textual branch, we leverage large language models to generate diverse, sentence-level and semantically rich interpretive prompts of actions, which shift the model's focus towards the verb comprehension. Extensive experiments on various benchmarks and learning scenarios demonstrate the superiority and generality of our approach. The code will be available soon.

Published as a conference paper at ICLR 2025
KRONECKER MASK AND INTERPRETIVE PROMPTS ARE LANGUAGE-ACTION VIDEO LEARNERS
Jingyi Yang'', Zitong Yu Xiuming Ni', Jia He Hui Lift Uniiersiity of Science and Technology of China, Great Bay University, 'Dongguan Key Laboratory for Intelligence and Information Technology, + *Anhui Tsinglink Information Technology Co.,Ltd.
yangjingy eargyi@@mill ustc.edul en, yuzitongegbu. edu en, (nixm, jia eetsinggini con, mythleel ustc. edu, on
ABSTRACT
Contrastive language-image pretraining (CLIP) has significantly advanced image- based vision learning. A pressing topic subsequently arises: how can we effec- tively adapt CLIP to the video domain? Recent studies have focused on adjust- ing either the textual or visual branch of CLIP for action recognition. However, we argue that adaptations of both branches are crucial. In this paper, we pro- pose CLAVER: Contrastive Language-Action Video Learner, designed to shift CLIP's focus from the alignment of static visual objects and concrete nouns to the alignment of dynamic action behaviors and abstract verbs. Specifically, we introduce a novel Kronecker mask attention for temporal modeling. Our tailored Kronecker mask offers three benefits 1) it expands the temporal receptive field for each token, 2) it serves as an effective spatiotemporal heterogeneity induc- tive bias, mitigating the issue of spatiotemporal homogenization, and it can be seamlessly plugged into transformer- based models. Regarding the textual branch, we leverage large language models to generate diverse, sentence-level and seman- tically rich interpretive prompts of actions, which shift the model's focus towards the verb comprehension. Extensive experiments on various benchmarks and learn- ing scenarios demonstrate the superiority and generality of our approach. Code is available https///hhhbb.oomyyyddc/iiii
INTRODUCTION
Video action recognition has long been representative topic in video understanding. Over the past decade, there has been continuous pursuit of learning spatiotemporal representations, giving rise to diverse architectures, such as traditional two-stream networks [Simonyan & Zisserman (2014);
Wang et al. (2016); Zhou et al. (2018); Karpathy et al. (2014); Xie et al. (2024), 3D convolutional neural networks Carreira & Zisserman (017)); Feichtenhofer (2020); Feichtenhofer et al. (2019)
Hara et al. (2017); ; Qiu et al. (2017); | Tran et al. (2015) 2018); | Wang et al. (2018); Xie et al. 2018
and Video Vision Transformers Arnb et al. (2021):
1 Bertasius et al. (2021); Fan et al. (2021) 1);
eet al. ) Patrick et al. (2021) Zhao et al. (2022) Li et al. (2022a); : Yan et al. (2022). Recently, there has been increasing interest in leveraging risual-language models (VLMs) like CLIP Radford al. (2021), Florence Yuan et al. (2021), and ALIGN JiLGGNiaae al. (2021) for various video tasks, owing to the superior generalization abilities of these models. Several studies Wang et al. 20 Lin et al. (202); Ni et al. (2022) y Ju et al. (2022); RRahhed et al. (2023): Tu (2023); Chen et al. (2023) have devoted to adapt the CLIP for video action recognition, but they often focus on adjusting a single branch. According to predecessor studies, transferring CLIP from the image domain to the video domain involves two key considerations: 1) how to perform effective temporal modeling. 2) how to design suitable text descriptions for verb understanding that align with rich text semantics in the VLM's pre-training dataset. We argue that addressing both issues simultancously is crucial.
"qqual Contribution
1 Corresponniig Author

---

Published as a conference paper at ICLR 2025
Rlumplor Conrge
Moen Poolling
Temporal (KMT) Transfounce
Sinilarity
X
Keunecker Mark
Lx
Tempural Irandarmer
FBN
Traporal
Enbedilia
Test Fnceder
Enosder
Pasiiinal
Enbebliig
Kroneeker Attration
LLAMAJ
Norte
Johiv Integyeclie Prong!
An overview of CLAVER. (Right) Image encoder and KMT transformer are assembled as a video encoder. (Left) How to get the interpretive prompts for actions.
Figure 1:
In addressing the issue 1), several studies Wang et al. (2021; Ju et al. (2022): Chen et al. (023))
Rasheed et al.] (2023) implement straightforward and simple strategies such as mean pooling or employing temporal attention among
ID-temporal convolution across the temporal dimension,
class tokens. X-CLIP Ni et al. (2022) and CLIP-ViP Xue et al. (2022) introduce extra tokens for cross-frame communication. Alternatively, some studies Lin al. (2022); Tuu et al. (2023) engineer tailored modules. In our work, we aim to elucidate the distinctions and intrinsic correlations between space and time, as well as design more general spatiotemporal modeling approaches.
Regarding the issue 2), some studies Hendricks & Nematzadeh (20TT) Thrush et al. (2022) indicate that VLMs tend to focus on the correspondence between visual objects and nouns rather than action behaviors and verbs. We consider the essential gap is that visual objects are static and presented in lower dimensions, and nouns are concrete and easily understandable. However, action behaviors are dynamic that presented in higher dimensions, and verbs are intricate and abstract. Several existing methods[Ni et al. (2022); Lin et al. (2022); Rasheed et al. (2023); Tu et al. (2023) use verbs or phrases as direct text descriptions. ActionCLIP Wang et al. (2021) integrates prompt templates to expand verbs or phrases into sentences. Ju et al. Du et al. (2022) propose trainable continuous prompts to construct virtual prompt templates. However, these methods do not address aforementioned issues in essence. Alternatively VoC[Momeni et al.] (2023) and MAXI et al. (2023) consider leveraging large language models (LLMs) to provide positive and negative text samples for contrastive learning or multiple instance learning, while ASU Chen et al. (2023) presents the concept of semantic units to supplement the semantic information of action labels.
To address these issues, we propose Contrastive Language-Action Video Learner (CLAVER, Fig. D to efficiently adapt the CLIP for video action recognition. Specifically, for the issue 1), we first obtain the frame-level visual representation from the image encoder, then apply tailored Kronecker mask for temporal modeling with a wider temporal receptive field to establish long-range and wide- range dependencies among frames, while mitigating spatiotemporal homogenization. Additionally, we reveal the intrinsic correlations between space and time from the perspective of Kronecker mask attention. Regarding the issue 2), we leverage LLMs to effectively generate diverse, sentence-level. and semantically rich interpretations of actions, augmenting text descriptions during training and testing. This approach allows the text descriptions to be presented in a more flexible, sentence-level form during inference. In summary, our main contributions are four-fold:
We propose the Contrastive Language-Actinn Action Video Learner (CLAVER) to adapt both the visual and textual branches, efficiently shifting the alignment in CLIP from visual objects and nouns to action behaviors and verbs.
We propose the Kronecker mask temporal attention and Kronecker mask causal temporal attention for temporal modeling, aiming to capture the long-range and wide-range depen- dencies among frames with spatiotemporal heterogeneity.
We introduce interpretive prompts of actions to facilitate the alignment of action behaviors and verbs, thereby improving zero-shot and few-shot generalization capabilities.
Extensive qualitative and quantitative experiments demonstrate the effectiveness of CLAVER. Our method achieves superior or competitive performance on Kinetics-400 and Kinetics- 600 under sulyy-sueer scenario, and on HMDB-51 and UCF-101 under zero- shot, shw-shot scenarios.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Video Recognition, Among early video recognition methods, 3D convolution is widely employed
Qiu et al. (2017); Tran et al. 2015 2018); Xie et al. (2018); Feichtenhofer et al. (2019); 1; Feichten-
hofer 2200)) Yu et al. (2021) Some studies Qiu et al. (2017): Tran al. (2018 10 Xie et al. (2018)
propose to factorize convolutional operations across spatial and temporal dimensions, while others
design the specific temporal modules to embed them into 2D CNNs [L al. (2020b); Lin et al.
; Liu et al. (.02T). Over the past few years, there has been an influx of mranfformrrbbseee
video works Arnab al. (2021): Neimark et al. (20T); Bertasius et al. (2021):
> Fan al. (2021); )
Liu ct al. (2022) Yan et al. (2022):
; Li (2022a), demonstrating promising performance. For ex- ample, some methods Arnab et al. (2021); Neimark et al. (202T); Girdhar & Grauman (202T) adopt fusion. Alteraatvely, another family employ a factorized attention structure. Such as TimeSformer [Bertasius et al. (202I) ViViT Armab et al. (2021). and ATA Zhao et al. (2022) which proposes sligneen--gudeed tepmoral attention. While Video Swin Transformer Liu et al. (2022) employs 3D window attention. The representative atten- tion calculation forms these works can be summarized into joint attention and factorized (divided) spatiotemporal attention. In this paper, to make minimal modifications to the original structure, we adopt factorized encoder structure for video stream.
factorized encoder structure for spatial
Visual-language Representation Learning. Visual-textual multi multi-mmddlityy is a hot topic in recent years. Several studies based on masked image modeling (MIM) have achieved commendable per- formance Li et al. (2020a); Lu et al. (2019); Su et al. (2019); : Tan & Bansal (2019). There are also efforts focused on video-language representation learning Miech et al. (2019); Sun et al. (2019a b): Zhu & Yang (2020); Xu et al. 2..I). Concurrently, contrastive language-imas
pretraining Rad- ford et al. (2021): Jia et al. (2021): Yuan al. (2021) achieved remarkable progress, particularly in demonstrating impressive zero-shot generalization capacities. CLIP Radford et al. (202T) is one of the most representative works, with numerous follow-up studies have explored to adapt for downstream tasks. For example. object detection, semantic segmentation, video retrieval and cap- tioning, etc.Gu et al. (202T); /Vinker et al. (2022): Li et al. (2022b): [uo et al. (2022); Xu et al. 2022)) Additionally, there are also many applications in the video action recognition Wang et al. 2021 ; Ni et al. (2022); Lin et al. )2022) Ju et al. (2022); Pan et al. (2022); Rasheed et al. (2023): Chen al. (2023); Tu et al. (2023): ; Lin et al. 2023); Momeni al. (2023); 3 Yang al. 2023
Rasheed et al. 2023) aim to minimize modifications to original models and facilitate efficient transfer, while Chen Ju et al. Ju et al. (022) suggest optimizing few prompt
For instance, ViFiCIIPP Rasheed
vectors for adapting CLIP to various video understanding tasks. X-CLIP et al. (022) proposes an efficient cross-frame attention module. ILA Tu al. (2023) designs implicit mask-based align- ment to align features of two adjacent frames and EVL Lin et al. (002)) proposes a image encoder and video decoder structure. Regrading to the language branch, most previous works directly use verbs or phrases that lack rich semantics which overlook the importance of semantics. With the advancement of large language models like the GPT-3 Brown al. (2020), PaLM Chowdhery et al. 2023), LLaMAs Touvron et al. (2023a b): Abhimanyu Dubey et al. (2024) and ChatGPT 2024). LLMs can replace manual labor Chen & Huang (021); y; Qian et al. (2022) and automati- cally generate texts that meet human expectations to benefit visual-textune learning. For example, LaCLIP Fan et al. (2024) employs LLMs to rewrite text descriptions associated with each image for text augmentation. VFC Momeni et al. (2023) and MAXI Lin et al. (202)) leverage LLMs to generate positive and negative texts with diversity for Language-vddes learning.
METHODOLOGY
In Sec. 3.1 we overview our proposed contrastive language-action video learner architecture. Then, we elaborate on the detail of the Kronecker mask attention in Sec. 3.2 Finally, we present the technique details of action interpretive prompt in Sec. 3.3
3.1
OVERVIEW
Our contrastive language-action video leamner architecture is illustrated in Fig. I We utilize video encoder to obtain video representations, comprising two transformers- based components: an image encoder (ViT) from CLIP, Kronecker mask temporal transformer. The text encoder
3

---

Published as a conference paper at ICLR 2025
Jocal Park
Aputa Pask
Reppora Park
Kronecker Mank Affcctionn
Spatial Affentiien
Tobro-ooo-aaly Tengural Attention
Spetial Krvnecket Mask Tempprrll Miteatien
farrs Fromer
1
Spatial Teuporal Attentive
W ?
Meor PLATR
Sparial Alignneet
Attretion
Kranecker
Temperal
Pipellnc Temporaal Attcation
Figure 2: (Left) Red indicates the currently focal patch, green patches are visible in spatial attention, orange patches are visible in temporal attention, purple patches visible in joint attention. (Right) Kronecker mask attention: Several attentions can be seen as employing tailored Kronecker masks for joint attention.
aims to align representations with the video representations. Concretely, given video clip V [Vo, ... Ut -1- E RT* C [cp cm. . CM-1 E MxN, Cm E EN,, where H. W are the number of frames, height, width, respectively, M is the number of diverse text descriptions (share the same central concept) for an action category, N is the max sequence length. We feed texts C into the text encoder foe. (-) to obtain text representations C [Co. Cas-1] For the video stream, firstly, we input the video clip to the image encoder fo, (-) to obtain frame-level representations I.
E R and corresponding text descriptions
I. So, (PE(v:) + cPOs), C (C)
where PE(-) the patch embedding. Each frame is split into number of patches, P is the patch size, oPos is the absolute positional embedding, PE(U)) + eos
# W patches, L is the
[vr,o+ed + PE((),)) - pos ..... PE(U).)) +ej .... LE( +e 200 = [u. Zt,: .. L.L 2t,0 is the class token. I, = fe, (2..leeeTee)))) = [L.0 I
I.
Then, we add absolute temporal embedding e to the L, T and feed them into the Kronecker mask temporal transformer for (-). Finally, by selecting the class token from each frame and aver- aging them, we obtain video representation v with the same dimension as Cm, m. E M.
V V (HLKET eemm)
v= Avg (oo( Ve,oleET).
(2)
where V = for (L)TTT + etem) = fer + etem (eT) = for (oooo + e
Lo.L +
e J. ... 1,, + e efom II,L + e, dem) ".... [ + TT-1" IT- + ffmil)) [Vo...... Vo.rl.... [V..,.... Vr,th,--. ..... IVr-1,0,... Vr-1,t]], Avg(-) is the average pool- ing function. Our optimization goal is to maximize the cosine similarity between video v and its corresponding texts cm C representations:
(v, Cm)
sim(v, Cm)
(3)
|
|m
3.2 KRONECKER MASK ATTENTION
For an image E #xxx3 x3 it is first split into patches and then flattened into token sequence after patch embedding. The resulting feature shape is (L, D). dimension. This process is standard operation in VIT Dosovisski et al. (2020), denoted as Spatial Attention (SA):
where D is the hidden
LLLDD) = Softmax Softmax(QQr..)))) (LD,
where Q, K, represent the query, key, value matrices, respectively.
For video E XTxxxxx3 T is the number of frames, previous studies Arna et al. 2022 Berta- sius et al. (202T); Neimark et al.] (202I): Guo et al. (202T); Tong et al. (2022); Feichtenhoferr et al. 2022 typically employ either joint attention Bertasius et al. (2021 Feichtenhofer et al. (2022) or
4

---

Published as a conference paper at ICLR 2025
factorized (divided) attention Bertaasuus al. (2021); Arnab et al. (2021). Joint attention flattens video into a longer token sequence, resulting in a feature shape of (T x D). This token interac-
tion mode is illustrated in Fig. 2 Left (a). Joint attention encounters spatiotemporal homogenization
issues, where random shuffling of tokens not affecting (or slightly) the final pooling result.
In contrast, factorized attention factorize joint attention into spatial attention and temporal attention to avoid spatiotemporal homogenization. They utilize the same spatial attention as Eqn. 4 while the temporal attention varies. Two common temporal attentions are pipeline temporal attention Arnab
et al. (202T): Bertasius et al. (202T) and class-token-only temporal attention Arnab et al. (2021);
Ni et al. ( Wang al. (021). Their feature shapes are (L, S.,TDD D) and (T, D). respectively.
Pipeline temporal attention has a limited temporal receptive field, which is limited to fixed time
pipeline (tube), hence the term "pipeline temporal attention", as shown in Fig. 2 Left (c). It has limited scope of capture dynamic information since objects of interest do not always appear in the same 2D location across frames. Although ATA Zhao et al. (2022) utilizes alignment techniques to bend the time pipeline to capture dynamic objects, it still has a limited temporal receptive field, in Fig. 2 Left (d). Similarly, class-token-only temporal attention retains only the class token time pipeline and discard others, which both face limited receptive field and may discard lots of poten- tially valuable information. is worth noting that in pipeline temporal attention, mean pooling is necessary across all tokens, otherwise, it is equivalent
-ons-y temporal attention.
To address aforementioned drawbacks, we propose Kronecker Mask Temporal Attention (KMTA)) Specifically, we allow each patch (token) at timestamp & can interact with all other patches (tokens), excluding those sharing the same timestamp f, as illustrated in Fig. 2 Left (b). Compared to pipeline temporal attention, KMTA expands the temporal receptive field width of each token. KMTA can be achieved through joint attention incorporated a Kronecker mask, as shown in Fig. 2 Right (left down). Additionally, KMTA alleviates the impact of spatiotemporal homogenization due to the presence of the Kronecker mask. The trick for obtaining the Kronecker mask is Kronecker product 8::
an B
d12 B 422
ain B,
021
B
B
ag B
Amxn X B Bpxq
(5)
am
am B
amn B mpxng
Eqn. is the definition of @ where A E Rmxx, B E Rexq, Thus, it is referred to as the Kronecker
mask. Kronecker Mask Temporal Attention (KMTA) can be formulated
M(XXL,TLL) M(r,r @ (J(L,)) -I,,,)))
(6)
(TT = Softmax Q(TxL.D) K (TXLLDI/VVD + MITXLTXX)
(7)
where I I(.-) is an identity matrix, -(-.)) ) is an all-ones matrix, l1==-inf means replacing 1 in the matrix with negative infinity inf), and M M(TXLTTXL) is the Kronecker mask. A transformer equipped with KMTA referred to as a Kronecker Mask Temporal (KMT) transformer.
In fact that both the spatial and temporal attention bining tailored Kronecker mask with joint attention. Therefore, we collectively refer to them as Kronecker Mask Attention, depicted in Fig. 2 (Right). The Kronecker mask serves as prior spatiotemporal heterogeneity inductive bias. Spatial attention allows intra- firrafrrame interactions but blocks inter-frame interactions, while Kronecker mask temporal attention allows fater-frame inter- actions but blocks intra-framm interactions, exhibiting spatiotemporal structural complementarity.
mentioned above can be derived by com-
Moreover, we design another type of Kronecker mask for temporal model- ing, known as Kronecker mask causal temporal attention (KMCTA) that aims to alleviate the low-rank bottleneck, as shown in Fig. The mask M Lt) of KMCTA can be formulated as:
Canral Teaparal Attetios
45
Figure 3: Kronecker mask causal temporal attention.
(8) where U U(---) is an upper triangular matrix, and all elements the upper triangle are 1. A trans- former equipped with KMCTA is referred to KMCT transformer. KMCTA ensures the causality
M M(TXL,TXL) = (Tr,r) @ (J(L,)) - (L,,)) + (U(r,z) ((r,))) & J J(,,))i==-iuf
5

---

Published as a conference paper at ICLR 2025
of time (in the time dimension, KMTA is bidirectional, while KMCTA is unidirectional). In addition, KMCTA always has a full-rank attention matrix, whereas KMTA and joint attention can not guaran- tee this property. The proof is detailed in Appendix[A] [A] Some studies Bhojanapalli et al. (2020); Han et al. 2 indicate that it is important to avoid the low-rank bottleneck to improve the representa- tion power of transformer architecture. This problem may become more knotty when transitioning images to videos.
3.3 INTERPRETIVE PROMPT
To address the issue 2) in Sec. I we prompt LLMs to generate interpretive texts that align the text semantics of video-text pairs with the rich text semantics in VLM's pre-training dataset. This design aids the model in understanding ab- stract verbs. This approach, termed interpre- tive prompt, designed from the following aspects: 1) Action decomposition: Providing detailed de- scriptions of actions by decomposing complex actions into simpler, more basic ones. This clarifies the include relationship between complex and basic actions, and help distinguish similar actions, as not all action concepts are of equal status. For example, the action "playing basketball' may consist of 'running', 'jumping', and 'shooting'. 'Run- ning', 'jumping,, and 'hand movements' are considered as basic actions, while 'playing basketball' a complex action. Similarly, 'dribbling' may also involves 'running', and 'hand movements' be- sides, it is also a subset of playing basketball'. Action decomposition enhances the separability of action concepts in semantic space, helping models understand the relationships between actions. 2) Synonym conversion: Generating synonyms for verbs and phrases that convey the same core concept but with more varied expressions. This improves zero-shot generalization as an action concept may have multiple expressions. When encountering similar action descriptions in the unseen domain, the model demonstrates stronger generalization robustness. 3) Involving body parts: Describing actions based on possible body parts involved, helping the model to localize the region where action occurs.
Pormed Frverpl -Chnmwesh + -napploo 1. Ycche Cacrpy Lakgy
Decripdoo afacils desepallio
Draghh (NN Jog Amm
LLaM
Blannnr L.M Interaciio
Intorpestic Freeuph ceroopaddnng Chanpe
Figure 4: The Interpretive Prompt scheme.
We leverage ChatGPT O (2024) and LLaMA-3 Abhimanyu Dubey et al. (2024) to automati- cally generate action interpretations. Initially, ask the knowledgeable ChatGPT OpenAl| (2024) to provide several examples of text descriptions that align with our expect (based on the aforemen- tioned aspects). Subsequently, we provide the format prompt to LLaMA-3 AAbhimanyu Dubey al. (204) for text completion, as illustrated in Fig. 4 For example, "Generate a detailed text descrip- tion corresponding to the video class..." " -Command, accompanied by a few examples like, "abseil- ing + Abseiling combines several actions..." " -Examples, and the action concept that LLM needs to interpret-Aee -Action Concept. We feed the format prompt to LLaMA-3 multiple times to obtain di- verse interpretive prompts of actions. For a given action category, the texts used during training and inference include the original verbs or phrases, template filling prefixes or suffixes, and interpretive prompts. Our interpretive prompts provide texts at the sentence, phrase, and word-levels, enhancing the flexibility of text usage during the inference. More technique details refer to Appendix F
3.4 TRAINING AND INFERENCE
In each training step, a batch of B videos is sampled, and M represents the total number of text descriptions of each verb or phrase. There are a total of K action categories. The training loss is formulated as follows:
1 (ub(Mr N
Σ log exp(sim (', c/m/T)
L -=
Σ
(9)
Σx exp(sim (iisiiivv.,, ct))7))
where, is the temperature parameter, } indicates that we sample subset of all text descrip- tions for a single training step. v² represents the video belonging to the i-th category. ef,, denotes the descriptions corresponding to the i-th category. During inference, we take the sum of the similarities between each v and all c,,, as the final similarity score S:
M
exp(sim(y'sc c/T)T)) Dx exp(sim(v',, c 3/T)
S =
log K
(10)
m
6

---

Published as a conference paper at ICLR 2025
Table 1: Comparison with state-of-the-art on Kinetics-400. * indicates pretraining with a video-text collection. The FLOPs per view of each method is reported. Red represents optimal performance. Bold represents optimal performance among CLIP-based methods at the same model scale.
Top-1
Top-5
Without Language
Unifammer-B ctal
IN-IE
32
83.0
95.4
4x3
259
TimeSformert
IN-21K
80.7
94.7
2380
2031
Purick
Ziao
81.9
95.3
Swar (261
[Eleea 2013
IN-21K
84,9
96.7
10x3
2107
MVTTV-1I. (3)
IN-21k
86.1
97.0
5x.5
2826
2021
J00M
32
84.9
95.8
4x3
8316
TokenLezrner Em 2390 sereet
2021
JFT300M
85.4
96.3
4x3
4076
CoVell Zhang
JFT3B
87.2
Language
MIV. atal 2003
89.1
98.2
4x3
37NS
X-C Ierel
CLIPZROM
H0.1
04.8
430
25
X-CLIP CLIP-B/V213T Do NUNC
CLIP-400M
16
81.0
95.1
4x3
49
ILA. Tu Br 202
CLIP-400M
80.6
94.9
4333
40
ILA-B/30 1LA-B/30110011
CLIP-400M
16
81.8
05.4
4x4
75
CLAVER-RA22
CLIP-ROONT
K1.5
CLAVER-B32 (KMCT)
CLIP-400M
95.5
4x3
CLAVER-B/32 (KMCT)
CLIP 400M
16
82.6
95.9
4x3
64
Actio CLIF-RI Wang al. 20211
CLIF-ROOM
15
82.6
96.2
10x3
Ad at 2022
CLIP
16
76.9
93.5
X-CLIP INTEA RUAN
CLIP 400M
96.5
92
CLIP-400
W/L-B/IO LI
EVL-B/IOLI LII
188
MIRCLIP BIIGEEIISSEE
2023
CLIP-40OM
16
83.9
96.3
4x3
181
ASIL-R/I6 CCTherrannn DIZA)
CLIP-JOMM
84.1
96.1
4x3
146
ASU-B/IH BUU-BINNCIIEES COIERRI 10M67
CLIP-10OM
18
84,8
96.7
4x3
238
TLA-B/I6
ATM
CLIP-400M
83.4
96.3
4x3
150
CLIP 400-4
85.0
91.0
302
ALT-RVIS
CLIP-400M
85.5
1706
84.3
96.1
122
CLAVER-B/IE (KNF))
CIJP-400M
85.9
97.3
4x3
241
CLAVER-B/16 (KMCT)
CLIP-400M
84.1
96.2
4x3
122
CLAVER-B/16 (KMCT) (KMCT)
CLIP-100
16
86.0
97.2
4x3
241
X-CLIRDTINN NE N091
CLIF-ROM
87.0
97.7
4x3
420
X-CLIP LICTOOE
2022
CLIP
87.6
91.5
1870
EVL-U/14 Lanceral
CLIP-400M
2022
EVL-L/4
87.7
18196
ASU--114
87.8
97.8
660
ASU-L ELLTESSSONN CIREE SIR
CLIP 400M
15
88.3
98.0
4x0
3084
ILA-L OUA MUZAS
CLIP-JOIM
87.6
97.88
43.
1647
ILA-L wow.1aaaaaaaaa Mann 202
CLIP-S0UM
16
88.1
97.88
436
31330
ALT-R/IA Chen
CLIP-40M
16
87.8
97.7
3x1
4947
CLAVER-LIR
CLIPEZON
CLAVER-L/14130
CLIP-100M
CLAVER-LA14 (KMCT)
87.9
97.7
558
CLAVER- (KMC))
CLIP
15
88.9
98.0
4x3
2488
EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
Architectures and hy pyerpaaraeeees. We employ CLIP-B/32, CLIP-B/16 CLIP-L/14 as our back- bones and derive corresponding variants: CLAVER-B/32 CLAVER-B/II CLAVER-L/14, respec- tively. The frame length settings include and 16. For all CLAVER variants, the number layers in the KMT/KMCT transformer is equal to one-third of the number of layers in the image encoder. For example, if the image encoder has 12 or 24 layers, then the KMT/KMCT transformer employs 4 8 layers. The detailed hyperparameter settings are provided in Appendix D
Datasets and metrics. We evaluate the performance of our method on four benchmarks: Kinetics- 400 Kay et al. (017), Kientics-600(Carrein Carreira al. (2018), UCF-101 (Soomro al. (2012), HMDB-51 ICF-100/Soomro We report the 1 and Top-5 accuracy as evaluation metrics.
Kuchne et al.
2011
4.2 COMPARISON RESULTS
Fully-supervised Experiments. We conducted fully supervised experiments on Kinetics-400 and Kinetics-600, respectively. In Tab. D we employ three variant models, CLAVER-B/32. CLAVER- B/16, and CLAVER-L/14, and sample 8 or 16 frames f with a sparse sampling for each model. employ KMTTKMMC., respectiely. CLAVER-B/ B/16R///11/// (KMT/KMCT) and CLAVER-B/16gs: (KMT/KMCT) surpass several methods Bertasius et al. (202T); Patrick et al. (2021): Zhao et al.
2022)
Li et al. (002a)) pretrained ImageNeet11/211 1k/21k Deng al. (2009) with shorter frames.
7

---

Published as a conference paper at ICLR 2025
Table 2: Comparison with state-of-the-ar on Table Few-shot performance on HMDB-51 Kinetics-600.
and UCF-101.
NVVI. Se Kil1
Modbad
Pisirala
Tope1
833
1ap-3
16.3
Virm
5:11
Mithed
Swas L
JFE 30084
833
96.3
4-13
17.
3335
16.2
3110
153
564
033
03.6
16.4
15.6
18.
3F03300
Colar Zarteeu
JFT-38
87.9
4-3
Actin
580
35.0
78.0
53.
850
913
MIN. peEEn
WTST
816
96.3
4-3
51.5
64.1
14.
540
923
96.5
ASU-STTI CasTrril
X-CLIP IP
CLAVE AANIIRTTTTAATT
CLPP 401M
0L804005
400M
853
85.7
V73
4x3
693
585 619
02.1
(83 13.5
66.8
16.5
813
865
908
463
961
V6A
Table 4: sero on HMDB-51 and UCF-101.
Table 5: Zero-shot on Kinetics-600,
UCP-101
MIL etal 2016
19.7
1 1.6
158
Method
Top-1
ASR Wint & Chan 20
218
= 0.9
24
1.0
DEVISLI
ZSBCOU CebcoctQueete Ofinet de 2017
22.6 1.2
15.1
UR 7R77har aiTh)
344+16 1.6
17.5+ 1.6
AL.R eran P0n30
48.2
OCN
2.
3.22
ESZSL
Kollen Fa &
2015
22. 1.2
48.3
0.8
Bel PURU
48
DEA 2ram e 201
23.6 00
49.5
04
=1
GCN oi MoNeEeee cell PUAU
22.3
49.7
0.6
ActionCC.PP
ER. ER.ZSAR Liee Hali 2021
1.1
75.1 033
X-CLIP KTere
44
72
X-C In Arr
652+004
86.1
0.8
ASE Iinueeii PUJA3
2.
75.0 2.3
6..112
87.2
0.3
MAX in 0125
11
752+009 0.9
70.9
92.1
OST
1.9
753121
(KAT)
(EAVER((NI1)
93.1
CLAVER-B/161s/ (KMT) outperforms Swin-L (3841) Liu et al. (2022) by 1.0%, and slightly lower than V-LTv2-L (3121) Li et al.] as lower resolution, shorter frames and fewer views.
CLAVER-BI BEEER-B114, y (KMT/KMCT) outperforms some methods Arnab et al. (202T); |yoo RRyoo al.. (202I);
Zhang al. (2021) pretrained on JFT-300M/JFT31. CLAVER-B/I4 (KMT) outperforms CoVeR
Zhang et al. (202T) by 1.6%, however, inferior MTV-H Yan et al. (2022), as it utilizes WTS*
which contains 70M video-text pairs with about 17B images, much larger than CLIP-400X, Com- pared to those approaches based on CLIP Radford et al. (2021), under configuration ViT-B/32s and ViT-B/3216/. CLAVER (KMT/KMCT) surpasses X-CLIP Ni et al. (2022) and ILA Tu et al. 2023). Under configuration ViT-B/16g,,16, and ViT-L /t..//144....... CLAVER (KMT/KMCT) exceeds
most methods under the same configuration. In Tab. 2 CLAVER-B/16g, (KMT) achieves higher
performance compared to ViViT-H/14x2 AArab et al. (202I). MViT-B-24 Fan et al. (2021). Our
method has lower performance than these methods Ryoo al. (2021); Zhang et al. (202T: Yan
et al. 2022); Yuan et al. )) Liu et (2022), as they use longer frames, or more data, or higher
resolutions. In addition, CLAVER B/16g/ (KMT/KMCT) outperforms ASU-B/16g/ Chen et al.
2023) and X-CLIP-B/166 65 et et al. (2022) at the same scale.
Few-shot Experiments. We pretrain CLAVER-B//UE BIVER-B/1622f (KMT) (32f means with frames) on Kinetics-400, and then perform few-shot transfer of 2, 4, 8, 16 samples on HMDB-51 and UCF-101. Tab. depicts the results few-shot experiments. CLAVER is comparable wtih ASU Chen et al. 2023 and consistently surpasses the X-CLIP|Ni et al. (2022) and MAXI Lin al. (2023) across all ranges. Additionally. CLAVER significantly outperforms other previous methods like Lin et al.
2019 Bertasius et al. (2021 Liu et al. 20022. More details about the evaluation protocols are
provided in the Appendix D
Zero-shot Experiments. We also pretrain CLAVER- B/1652 (KMT) on Kinetics400 for zero-shot. As shown in Tab. 4 on HMDB-51 Kuehne et al. (2011) and UCF-101 Soomro et al. (2012) bench- marks, CLAVER surpasses ChTn et al. 2024a) ASU Chen et al. (2023) and X-CLIP Ni al.
2022)
under the same configuration, and far outperforms other previous methods. Additionally, in Kinetics600) Carreira al. (2018) benchmark, CLAVER outperforms OST Chen et al. MAXI Lin et (2023) and all other methods. More details about the evaluation protocols
Tab.1
2024a)
are provided in the Appendix
4.3 ABLATION STUDY
Components ablation studies. We performed ablation studies to evaluate the effects of each com- ponent under the SLAVERR-B332. (KMT) configuration. The results are shown in Tab. 6 Our baseline, denoted CLIP-Mean, implements temporal mean pooling for CLIP. By equipping the CLIP with a KMT transformer at 1/3 scale, the Top-1 accuracy increases by 3.5%. We only intro- duce interpretive prompt, the performance increases by 1%. When we further incorporate both of them, CLAVER surpasses the CLIP-Mean by 4.1 %. In addition, we test the effect of the number of KMT transformer layers. With only one layer, the performance improvement is minimal. Increasing
8

---

Published as a conference paper at ICLR 2025
Table 6: Ablation of each component.
Table 7: Comparison of temporal attentions.
Components
Top-1 (%)
Temporal Modeling
Tog-1 (%)
Tup-5
Baseline (CLIP-Meun)
77.4
Biscling (Mean Pooling)
75.4
94.3
Bascline + KMTA 1 layer
78.3
78,9
94.3
Baseline KMIA 1/6 scale
79.6
Pipeline Temporal Attention
79,4
Baseline KMTA 1/3 scale
80.9
Kronecker Mask Causal Temporal Attention
Joint Attention
80.1
81.4
94,9
95.5
Baseline + Interpretive Prompt
78.4
Kronceker Temporal Atleation
Baseline + KMTA 1/3 scale + Interpretive Prompt
81.5
Table 8: The impact of patch size and frame length on joint attention (IA), KMTA and KMCTA. Conducting on HMDB-51 and UCF-101.
patch size 32 (VIT-B/32)
patch size 16 (ViT-B/16)
HMDB-51, UCF-101
JA (%)
KMTA (%)
KMCTA (%)
IA (%)
KMTA (%)
KMCTA (%)
Frume length =8
67.9 93.3
68.8, 94.1
93.9
72.2, 96.2
73.2, 96.3
72.2, 96.1
length
68.0, 93.6
69.1,
69.4, 94.8
72.1, 96.3
72.3, 96.6
72.8,
CLAPER
SCLPP
man is ab iinng. which involiess descent down vertical surface using a
cutting Watermelon
Iatermoton
involves
vertical wsing
playing Sekkettal
ilaying beskethall
CLor
man ab Bing. which involves descers dowa vertical surface using rope.
bling
buoketblll
Figure 5: Word importance of CLIP, X-CLIP, ILA and CLAVER. Darker color, higher importance.
the number of layers to 1/6 scale results in further performance gains, and at 1/3 scale, we observe
significant improvement.
Comparison of different temporal attentions and low-rank bottleneck issue. Tab. 7 compares the performance of different temporal modeling methods mentioned in Sec. 3.2 We find that class- token-only temporal attention and pipeline temporal attention have inferior performance. KMTA and KMCTA outperform joint attention. The top-1 1 (%) of KMTA slightly surpasses KMCTA by 0.1% Then, we conduct further experiments on HMDB-51 and UCF101 that observing the impact of the low-rank bottleneck issue on them. We increasing the length of token sequences by reducing patch size or increasing frame length, and observe the effects on them, as shown in Tab. g When reducing the patch size, however, do not observe the low-rank bottleneck. Joint attention, KMTA and KMCTA achieve better performance due to more fine- ime-grained features as the smaller patch size results in each token representing smaller local region. In contrast, when we increase the frame length, KMCTA's performance is optimal in most configurations, and only the performance of KM- CTA can steadily improve. The performance improvement of joint attention and KMTA limited. Meanwhile, in Tab. [ with the increase of frame length under the same backbone, the performance increase KMCTA also greater than that of KMTA. This indicates that KMCTA has a more significant advantage with longer frame length.
5
VISUALIZATION AND ANALYSIS
We employ Chefer et al. (2021) for bi-modal visualization and show the explainability of visual- textual attentions.
Word importance. We visualize word importance for CLIP, X-CLIP, ILA and CLAVER in Fig. 5 Observations indicate that CLIP tends focus on nouns, whereas CLAVER prefers verbs. Compared to previous works, CLAVER is more inclined towards verb concept, while X-CLIP Ni (202)) and ILA Tu al. (023) show a slight inclination towards nouns. These findings indicate the effectiveness of interpretive prompts for nouns concept to verbs concept transition.
Spatiotemporal homogenization study. We define spatiotemporal homogenization as phe- nomenon where random token shuffling has little impact on the semantics of the visual branch, which is illogical. aims to illustrate the interpretability behind the performance of spatiotemporal modeling. In Fig. 6 (Upper Left), we illustrate the token shuffling. We denote token shuffling before adding time embedding as PreTE shuffling, token shuffling after adding time embedding as shuffle. Fig. 6 shows changes in word importance and
PosTTE shuffling, and None represents similarity following both PreTE and PostTE shuffling. For joint attention, we observe that PreTE shuffling marginally affects similarity and word importance, while PosTTE shuffling does not affect

---

Published as a conference paper at ICLR 2025
Figure 6: Spatiotemporal Homogenization. (Upper Left) Token shuffling. (Vertical bar chart) Word importance refers to the degree of correlation between each word in sentence description and the semantics of the video content, while (Horizontal bar chart) Similarity refers to the similarity between visual and textual representations.
Speniad Hoeite
Sparia
S-U
Figure 7: Visualizing spatiotemporal attention maps of different spatio-temporal modeling.
similarity score and word importance. Regarding KMTA, both PreTE and PostTE shuffling lead to changes in the similarity and word importance. Mcanwhile, KMCTA is profoundly affected by both PreTE and PostT' shuffling, which result in lower similarity and disturbance of word importance. This phenomenon suggest that KMTA and KMCTA possess varying degrees ability in mitigating spatiotemporal homogenization and the Kronecker mask serves as a natural inductive bias for spa- tiotemporal structural heterogeneity. It also demonstrates that equipping learnable position/time encoding is not insufficient to alleviate spatiotemporal homogenization. Please refer to Appendix B for more and explanations and visualizations.
Visualization of spatiotemporal attention map. Fig. presents various spatiotemporal attention maps. X-CLIP al. (2022) and ILA Tu et (003) adopt factorized attention structures resulting in intertwined spatiotemporal attention maps. However, CLAVER employs a factorized encoder structure, allowing for the separation of spatial and temporal attention maps. Notably, class-token- only temporal attention discards other time tubes so that its temporal attention maps appear vacant. KMTA and KMCT have more reasonable spatial (focuses on the executor of the action) and temporal (focuses on areas where the action occurs) attention maps, while ILA and X-CLIP ignore some action dense areas. Moreover, joint attention and pipeline temporal attention will be attracted by some irrelevant backgrounds
CONCLUSION
In this work, present the Contrastive Language-Action Video Learner (CLAVER) shift from the aligmment of visual objects and nouns in CLIP to the alignment of action behaviors and verbs. We propose Kronecker mask temporal attention and Kronecker mask causal temporal attention for temporal modeling. Interpretive prompts are employed to transition the focus on nouns to verbs. Ex- tensive experiments under different evaluation settings demonstrate the effectiveness of our method.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGEMENT
The numerical calculations in this paper have been done on the supercomputing system in the Su- percomputing Center of University of Science and Technology of China. This work was supported by the National Science Foundation of China, under Grant No. 62171425, This work was supported by National Natural Science Foundation China under Grant 62306061. This work was supported by Guangdong Basic and Applied Basic Research Foundation (Grant No. 2023A15151400377 This work was supported by Guangdong Research Team for Communication and Sensing Integrated with Intelligent Computing (Project No. 2024KCXTD047).
REFERENCES
Abhinay Jauhri Abhimanyu Dubey, Abhinav Pandey, and et al. The Ilama 3 herd of models, 2024.
URL http://aaaii...rraa///////0000
Zeynep Akata, Florent Perronnin, Zaid Harchaoui, and Cordelia Schmid. Label-embedding for image classification. IEEE transactions on pattern analysis and machine intelligence, 38(7):
1425-1438, 2015a.
Zeynep Akata, Scott Reed, Daniel Walter, Honglak Lee, and Bernt Schiele. Evaluation of output embeddings for fine-grained image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2927-2936, 2015b.
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEF/CVF international conference on computer vision, pp. 6836-6846, 2021.
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICML, volume 2, pp. 4, 2021.
Srinadh Bhojanapalli, Chulhee Yun, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar. Low- rank bottleneck in multi-head attention models. In International conference on machine learning, pp. 864-873. PMLR, 2020.
Biagio Brattoli, Joseph Tighe, Fedor Zhdanov, Pietro Perona, and Krzysztof Chalupka. Rethinking zero-shot video classification: End-to-end training for realistic applications. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4613-4623, 2020.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL https: /ttt:::://pppaa.... ooooaiinn...... com/ htwp/rrseaacc///
video-gereeattion
s as- ioworld-ss imm Amuuatoos
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299-6308, 2017.
Joao Carreira, Eric Noland, Andras Banki-Horvath Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600. arXiv preprint arXiv: (008.004488014 01340, 2018.
Hila Chefer, Shir Gur, and Lior Wolf. Generic attention model explainability for interpreting bi- modal and encoder-decoder transformers. Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 397-406, 2021.
Shizhe Chen and Dong Huang. Elaborative rehearsal for zero-shot action recognition. Proceed- ings of the IEEF/CVF International Conference Computer Vision, pp. 13638-13647, 2021.
11

---

Published as a conference paper at ICLR 2025
Tongjia Chen, Hongshan Yu, Zhengeng Yang. Zechuan Li, Wei Sun, and Chen Chen. Ost: Refining text knowledge with optimal spatio-temporal descriptor for general video recognition. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18888- 18898, 2024a.
Yifei Chen, Dapeng Chen, Ruijin Liu, Hao Li, and Wei Peng. Video action recognition with attentive semantic units. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 10170-10180, 2023.
Yifei Chen, Dapeng Chen, Ruijin Liu, Sai Zhou, Wenyuan Xue, and Wei Peng. Align before adapt: Leveraging entity-to-region alignments for generalizable video action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18688-18698, 2024b.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240); 1-113, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1 11929, 2020.
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In Proceedings of the IEEE/CVF in- ternational conference on computer vision, pp. 6824-6835, 2021.
Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving clip training with language rewrites. Advances in Neural Information Processing Systems, 36, 2024.
Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. In Pro- ceedings of the IEEE/CVF conference on computer vision and paltern recognition, pp. 203-213, 2020.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEF/CVF international conference on computer vision, pp- 2002-6211,200
Christoph Feichtenhofer, Yanghao Li, Kaiming He, et al. Masked autoencoders as spatiotemporal learners. Advances in neural information processing systems, 35:35946-35955 2022.
Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc' Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semante embedding model. Advances neural infor- mation processing systems, 26, 2013.
Junyu Gao, Tianzhu Zhang, and Changsheng Xu. know the relationships: Zero-shot action recog- nition via two-stream graph convolutional networks and knowledge graphs. Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 8303-8311, 2019.
Pallabi Ghosh, Nirat Saini, Larry S Davis, and Abhinav Shrivastava. All about knowledge graphs for actions. arXiv preprint arXiv:2008. 12432. 2020.
Rohit Girdhar and Kristen Grauman. Anticipative video transformer. In Proceedings of the IEFF/CVF international conference computer vision, pp. 13505-13515, 2021.
Xiuye Gu, Tsung- Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.19921, 2021.
12

---

Published as a conference paper at ICLR 2025
Xudong Guo, Xun Guo, and Yan Lu. Ssan: Separable self-attention network for video representation learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 12618-12627, 2021.
Dongchen Han, Xuran Pan, Yizeng Han, Shiji Song, and Gao Huang. Flatten transformer: Vi- sion transformer using focused linear attention. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5961-5971, 2023.
Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Learning spatio-temporal features with 3d residual networks for action recognition. In Proceedings of the IEEE international conference on computer vision workshops, pp. 3154-3160, 2017.
Lisa Anne Hendricks and Aida Nematzadeh. Prohing image-language transformers for verb under- standing. arXiv preprint arXiv:2 04/1, 2021.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pp. 4904-9916 PMLR. 2021.
Chen Ju, Tengda Han, Kunhao Zheng. Ya Zhang, and Weidi Xie. Prompting visual-language models for efficient video understanding. In European Conference on Computer Vision, pp. 105-124. Springer, 2022.
Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei- Fei. Large-scale video classification with convolutional neural networks, In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 1725-1732, 2014.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv: 06950, 2017.
Hildegard Kuehne, Hucihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb; large video database for human motion recognition. In 20// International conference on computer vision, pp. 2556-2556 2563. IEEE, 2011.
Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. Unicoder-vl: universal encoder for vision and language by cross-modal pre-training. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp- 11336-11344, 2020a.
Kunchang Li, Yali Wang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uni- former: Unified transformer for efficient spatiotemporal representation learning. arXiv preprint arXiv:2201. 04676, 2022a.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Li- juan Wang. Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2022b.
Y Li, CY Wu, H Fan, K Mangalam, B Xiong, J Malik, and C Feichtenhofeer. Improved multiscale vision transformers for classification and detection. arxiv 2021. arXiv preprint arX0v:2112.01556
Yan Li, Bin Ji, Xintian Shi, Jianguo Zhang, Bin Kang, and Limin Wang. Tea: Temporal excitation and aggregation for action recognition. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 909-918, 2020b.
Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding. In Proceedings of the IEEE/CVF international conference computer vision, pp. 7083-7093, 2019.
Wei Lin, Leonid Karlinsky, Nina Shvetsova, Horst Possegger, Mateusz Kozinski, Rameswar Panda, Rogerio Feris, Hilde Kuehne, and Horst Bischof. Match, expand and improve: Unsupervised finetuning for zero-shot action recognition with language knowledge. In Proceedings of the IEFF/CVF International Conference Computer Vision, pp. 2851-2862, 2023.
13

---

Published as a conference paper at ICLR 2025
Ziyi Lin, Shijie Geng, Renrui Zhang, Peng Gao, Gerard de Melo, Xiaogang Wang, Jifeng Dai, Yu Qiao, and Hongsheng Li. Frozen clip models are efficient video learners. In European Con- ference Computer Vision, pp. 388 4044. Springer, 2022.
Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin trans- former. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3202-3211, 2022.
Zhaoyang Liu, Limin Wang, Wayne Wu, Chen Qian, and Tong Lu. Tam: Temporal adaptive module for video recognition. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 13708-13718, 2021.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolin- guistic representations for vision-and-languae tasks. Advances in neural information processing systems, 32, 2019.
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clipp: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing, 508: 293-304, 2022.
Antoine Miech, Dimitri Zhukov, Dean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howtol00m: Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF international conference on computer vision, pp.
2630-2640, 2019.
Liliane Momeni, Mathilde Caron, Arsha Nagrani, Andrew Zisserman, and Cordelia Schmid. Verbs in action: Improving verb understanding in ladeo-lagguage models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15579-15591, 2023.
Daniel Neimark, Omri Bar, Maya Zohar, and Dotan Asselmann. Video transformer network. In Proceedings the IEEE/CVF international conference on computer vision, pp. 3163-313177111 2021.
Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng. Jianlong Fu, Shiming Xiang, and Haibin Ling. Expanding language-imag pretrained models for general video recog- nition. European Conference on Computer Vision, pp. 1-18. Springer, 2022.
OpenAl. Gpt-4 technical report, https:/Aarivvogg/abb/22030887740 2024. URL hattps://chat
openai. .com
Junting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao, and Hongsheng Li. St-adapter: Parameter-efficient image-to-video transfer learning. Advances in Neural Information Processing Systems, 35:26462- 26477, 2022.
Mandela Patrick, Dylan Campbell, Yuki Asano, Ishan Misra, Florian Metze, Christoph Feichten- hofer, Andrea Vedaldi, and Joao F Henriques. Keeping your eye on the ball: Trajectory attention in video transformers. Advances in neural information processing systems, 34:12493-12506. 2021.
Yijun Qian, Lijun Yu, Wenhe Liu, and Alexander G Hauptmann. Rethinking zero-shot action recog- nition: Learning from latent atomic actions. In European Conference on Computer Vision, pp. 104-120. Springer, 2022.
Jie Qin, Li Liu, Ling Shao, Fumin Shen, Binghing Ni, Jiaxin Chen, and Yunhong Wang. Zero-shot action recognition with error-correcting output codes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2833-2842, 2017.
Zhaofan Qiu, Ting Yao, and Tao Mei. Learning spatio-temporal representation with pseudo-3d residual networks. In proceedings of the IEEE International Conference on Computer Vision, pp- 5533-5541, 2017.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. PMLR, 2021.
14

---

Published as a conference paper at ICLR 2025
Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fahad Shah- baz Khan. Fine-tuned clip models are efficient video learners. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6545-6554, 2023.
Bernardino Romera-Paredes and Philip Torr. An embarrassingly simple approach zero-shot learn ing. In International conference machine learning, pp. 2152-2161. PMLR, 2015.
Michael Ryoo, AJ Piergiovanni, Anurag Amab, Mostafa Dehghani, and Anelia Angelova. Token- learner: Adaptive space-time tokenization for videos. Advances in Neural Information Processing Systems, 11112-288699977 2021.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- fion Processing Systems, 35:36479-366 2022.
Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. Advances in neural information processing systems, 27, 2014.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 1212.0402, 2012.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VI-bert: Pre-training of generic visual-linguistic representations. arXiv preprint
2019.
Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid. Learning video representations using contrastive bidirectional transformer. arXiv preprint w:Xvv:196600743. 2019a.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: joint model for video and language representation learning. In Proceedings of the IEEE/CVF interna- tional conference on computer vision, pp. 7464-7473, 2019b.
Hao Tan and Mohit Bansal. Lxmert: Learning cross- coss-modality encoder representations from trans- formers. arXiv preprint arXiv:19065 074900 2019.
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic composi- tionality. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recogni- fion, pp. 2238-5244 2022.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data- efficient learners for self-supervised video pre-training. Advances in neural information process- ing systems, 35:10078-10093 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint 2xXi:2307.00288 2023b.
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 4489-4497, 2015.
Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 6450-6455 2018
Shuyuan Tu, Qi Dai, Zuxuan Wu, Zhi-Qi Cheng, Han Hu, and Yu- Yu-Gang Jiang. Implicit tempo- ral modeling with learnable alignment for video recognition. Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 19936-19947, 2023.
15

---

Published as a conference paper at ICLR 2025
Yael Vinker, Ehsan Pajouheshgar, Jessica Y Bo, Roman Christian Bachmann, Amit Haim Bermano, Daniel Cohen-Or, Amir Zamir, and Ariel Shamir. Clipasso: Semantically-awarr object sketching. ACM Transactions Graphics (TOG), 41(4):1-11, 2022.
Limin Wang, Yuanjun Xiong, Zhe Wang. Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In European conference on computer vision, pp. 20-36. Springer, 2016.
Mengmeng Wang, Jiazheng Xing, and Yong Liu. Actionclip: A new paradigm for video action recognition. arXiv preprint arXiv:2109.08472, 2021.
Qian Wang and Ke Chen. Alternative semantic representations for zero-shot human action recog- nition. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2017, Skopje, Macedonia, September 18 22, 2017, Proceedings, Part 10, pp- 87-102. Springer, 2017.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7794-7803, 2018.
Saining Xic, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotem- poral feature learning: Speed-accuracy trade-offs in video classification. In Proceedings of the European conference on computer vision (ECCV), pp. 305-321, 2018.
Xinyu Xie, Yawen Cui, Tao Tan, Xubin Zheng, and Zitong Yu. Fusionmamba: Dynamic feature enhancement for multimodal image fusion with mamba. Visual Intelligence, 2(37), 2024.
Hu Xu, Gargi Ghosh, Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. arXiv preprint 1rXiv:2109.144 2021.
Jiarui Xu, Shalini De Mello, Sifci Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18134-18144, 2022.
Xun Xu, Timothy M Hospedales, and Shaogang Gong. Multi-task zero-shot action recognition with prioritised data augmentation. In Computer Vision-ECCC 2016: 14th European Conference, Am- sterdam, The Netherlands, October 11-14, 2016, Proceedings, Part 11 14, pp. 343-359 Springer, 2016.
Hongwei Xue, Yuchong Sun, Bei Liu, Jianlong Fu, Ruihua Song, Houqiang Li, and Jiebo Luo. Clip-vip: Adapting pre-trained image-text model to languaae alignment. In The Eleventh International Conference Learning Representations, 2022.
Shen Yan, Xuehan Xiong, Anurag Amab, Zhichao Lu, Mi Zhang, Chen Sun, and Cordelia Schmid. Multiview transformers for video recognition. In Proceedings of the IEEE/CVF conference computer vision and patlern recognition, pp. 3333-3343, 2022.
Taojiannan Yang. Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, and Mu Li. Aim: Adapting image models for efficient video action recognition. arXiv preprint arXiv:2302.0302.. 2023.
Zitong Yu, Benjia Zhou, Jun Wan, Pichao Wang. Haoyu Chen, Xin Liu, Stan Z Li, and Guoying Zhao. Searching multi-rate and multi-modal temporal enhanced networks for gesture recognition. IEEE Transactions on Image Processing, 30:5626-5641) 2021.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao. Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111 1432, 2021.
Bowen Zhang, Jiahui Yu, Christopher Fifty, Wei Han, Andrew M Dai, Ruoming Pang, and Fei Sha. Co-training transformer with videos and images improves action recognition. arXiv preprint arXiv:2112.0711 2021.
16

---

Published as a conference paper at ICLR 2025
Li Zhang, Tao Xiang, and Shaogang Gong, Learning deep embedding model for zero-shot learning. In Proceedings of the IEEE conference on computer vision and patten recognition, pp. 2021- 2030, 2017.
Yizhou Zhao, Zhenyang Li, Xun Guo, and Yan Lu. Alignment-guided temporal attention for video action recognition. Advances in Neural Information Processing Systems, 35; 1551--37-,,,, 2022.
Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. In Proceedings of the European conference on computer vision (ECCV), pp. 803-818, 2018.
Linchao Zhu and Yi Yang. Actbert: Learning global-local video-text representations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8755,,755, 2020.
Yi Zhu, Yang Long, Yu Guan, Shawn Newsam, and Ling Shao. Towards universal representation for unseen action recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9436-9445, 2018.
17

---

Published as a conference paper at ICLR 2025
8 APPENDIX
Roadmap
In the Appendix, we present proof in A Spatiotemporal homogenization study details some additional experiment results in [ details in E analysis of synthetic video testing in F dataset details in and more visualization of spatiotemporal attention map in H
experiment setting details in D interpretive prompt technique
A
PROOF
We conclude that there is low-rank bottleneck problem for joint attention and KMTA when facing longer token sequence lengths (d & n), while KMCTA always guarantees the property of full rank under assumption. Proofs follow:
A.1 PROOF OF JOINT ATTENTION CANNOT GUARANTEE FULL RANK
Based on the Representation Theorem of Bhojanapalli et al. Bhojanapalli et al. (2020)
Theorem (Representation Theorem Bhojanapalli al. (2020)) dy de d 2 n. then given
any full row rank matrix X Ruxd and an arbitrary n. positive stochastic matrix P, subject to the sum of each row of P is equal to I, ie., P1 there always exists x d projection matrices W, and Wk such that
XW (XWx) P
Softmax
(11)
ddz
If d, d4 = =d < n. there exist X and P such that Eqn.
11 does not hold for all W and Wk.
The proof process of Theorem 1 are referred from Bhojanapalli et al. Bhojanapalli al. (2020)
By the way. in Bhojanapalli et al. |Bhojanapalli et al. (2020), they set X E Rax" However, for the convenience our subsequent proof, we transpose the and d dimensions, Xe R=dd
For d case. Since X exhibits full row rank, there exists right pseudo inverse X xT (X -1 Rdxn such that xx In. Let W. X WW and W, X W.. Then
XW (XW)) XW w
(12)
-XX (X
(13)
-XX W. (X
(14)
I, W, W. I
(15)
WW Wek
(16)
According Eqn.
we obtain that
XW (XWx)
W gk
Softmax
Softmax
(17)
Vdz
Vd
Wak
D exp(
(18)
Was
Vdk
where Dw... is
n diagonal matrix such that
(Wak)
W
(Dw )it Σ expl
(19)
Vdle
W
(exp(
1);
(20)
/d
18

---

Published as a conference paper at ICLR 2025
We can establish the desired result by showing that there always exists W gl that satisfies the following fixed point equation:
W exp
(21)
Given to construct such a W gk: we can pick an random positive diagonal matrix Do:
W oh dd
log(DpP)
(22)
Since P is positive matrix, and Do is positive diagonal matrix, such Wa always exists. Ac-
cording Eqn. 19 we can conclude
W ok dd 11)
Dw.. Diag(exp)
(23)
Diag(D,P1) = Diag(D,1)) = Do
(24)
which indicates that exp(
Web Vda
Dw, P. This completes the proof of 2 case of DoP
Theorem 1.
Regarding d < n case. Consider the special case of d 1 and n 2. There has X E R2x and
Wg. We E R1x1, Suppose
so that:
W, WT [1,0]
XW
(XWx)
Softmax
= Softmax
(25)
Vdg
Vdl
W.
0
Softmax
vdg
(26)
0
This matrix clearly cannot be used to generate P. Then we extend the above special case to general
values of n and d, (d < n). Let X = [1a,". la, 0z]" = [1mat- Os)" E R=dd, where 1a, 02 E Rd denotes the all ones, zeros column vector, and 1a denotes the dl x (n 1) all ones matrix. Then
Softmax (XW (XWx))
[mat 0z]" W, WT [1 Oa)
Softmax
(27)
ddg
ddc
1 W,W." 0n-1
Softmax
mat
ddu
mat
(28)
The basic idea of the proof remains consistency, and we can conclude the same conclusion. Accord- ing to the Theorem 1, we can not ensure the attention matrix of joint attention to be full rank.
A.2 PROOF OF KMCTA CAN GUARANTEE FULL RANK
Assumption 1 The elements of (XW,) does not exist negative infinity in normal cases,
Vdd
which means that 0 will not appear at any other position in the attention matrix Á
Softmax ((W.) + M] except for the masked position. Vdl
The softmax operation can ensure that the elements at all other positions are greater than O except for the masked position. The attention matrix Á of KMCTA is always lower triangular matrix due to the presence of the M and the diagonal elements are always greater than O, i.e., Va, > 0:
a11
0
0
a21
022
0
0
Ä
031
0:32
033
(29)
Gm1
0,2
4.3
The determinant of the Á. A
011022
Onn
0 0, because Vag 0. Thus, the attention
matrix Á of KMCTA is always reversible, i.e. always be full rank.
19

---

Published as a conference paper at ICLR 2025
A.3
PROOF OF KMTA CANNOT GUARANTEE FULL RANK
According to the Assumption 1.
We consider a special case, when 7=2 and L=2, the attention
matrix A Softmax|
(XW,
M| of KMTA can be formulated as:
Q11
013
014
011
0
11
011
0
022
023
024
0
222
Q21
031
032
433
0
0
033
011
0:13
d31
022
a22
0:2
0.18 031
0 (142_
024 032
041
042
0
0444
0
012 0441
023 042
0.4
211 0.1
411
422
411
022
(30)
where 011 + 013
+
0114
= 1,022
+
023
+ Q24
= 1, d3i + 032 + 033 = 1, 041 + 042 + 044 = 1. When
433
412
031
122
811
0.1
d23
0.42
A can not always be full rank, For example:
d11
822
611
422
70.4
0
0.4
0.27
0.4
0.4
0.2
A
(31)
0.4
0.5
0.1
0
0..
0.4
0.2]
For video, T' x L, If fix the number of attention heads, the dimension of tokens, the patch size, and the resolution of the input image, the only factor that can affect n for video is the frame length. When we pray for a longer frame length and an increase the number of tokens to improve performance, we may face the problems pointed out in Theorem 1. Because increasing the frame length will further increase n, the expressive power of self attention may encounter bottlenecks. The low-rank bottleneck is vital, because it may lead to that many rows of the attention map are seriously homogenized. As the output of self-attentin is the weighted sum of the same set of value vectors, the homogenization of attention weights inevitably leads to the resemblance among the aggregated features.
i cartoon kangaroo disco dunces
man is abseiling vie using a rope ILA
X-CLIP
X-CLIP
an on a suefboard surfing water
person cuiting natermelon with a Anife
X-CLIP
X-CLIP
We visualize the impact of token shuffling on the ILA, XCLIP. In most cases, token shuffling has limited influence on their word importance. Surprisingly, in some figures, after token shuffling, the similarity actually increases, which is illogical. This indicates that the response of ILA and XCLIP to spatiotemporal position destruction uncertain and has poor interpretability.
Figure 8:
B SPATIOTEMPORAL HOMOGENIZATION STUDY
Fig. shows that equipping learnable position/time encoding with joint attention is not insufficient to alleviate spatiotemporal homogenization. For example, (Upper Left) "a cartoon kangaroo disco
20

---

Published as a conference paper at ICLR 2025
dances'. We can observe that token shuffling has limited impact on word importance and similarity of joint attention, but it has a sigmificant impact on the word importance of KMT and both similarity and word importance of KMCT. (Upper Right) 'a man is abseiling via using rope'. (Lower Left) an otter stands on a surfboard surfing water'. (Lower Right) 'a person is cutting watermelon with a knife'. The key point is that classification accuracy does not necessarily indicate that these learnable position encoding or attention biases genuinely grasp the concept of spatiotemporal structure. Since the optimization goal of the network is usually to pursue performance, the learned positional encod- ing or attention bias serve to improve accuracy rather than understand the spatiotemporal concept. This is because their optimization guided by gradients aimed at increasing accuracy, rather than by an objective function that tells them what spatiotemporal structure is. The Kronecker mask in KMTA and KMCTA pursues not only for good performance, but also for interpretability. The Kro- necker mask act as a natural spatiotemporal heterogeneity inductive bias, KMTA/KMCTA exhibit better spatiotemporal complementarity with spatial attention. Additionally, in Fig. & we show that several tailored modules (ILA and XCLIP) also encounter spatiotemporal homogenization issues during our testing. Surprisingly. in some cases like 'a man is abseiling via using a rope,' 'an otter stands surfboard surfing water,' and 'a person is cutting watermelon with knife," the similarity actually increases after token shuffling, which is illogical.
Additionally, evaluate the performance of joint attention, Kronecker mask temporal attention (KMTA) and Kronecker mask causal temporal attention (KMCTA) on K400 et al. (2017) after introducing token shuffling. The results are shown in Tab. [ The greater performance degrada- tion observed in KMTA and KMCTA indicates that they are more sensitive to disturbances in the spatiotemporal structure, thus alleviating spatiotemporal homogenization.
Table 9: Performance of joint attention, KMTA and KMCTA while introducing token shuffling.
Token Shufllin (Yes-Y, No-N)
Top-1 (%)
Joint Attention,
80.1
Kronecker Mask Temporal Attention, N
81.5
Kronecker Mask Causal Temporal Atteation, N
81.4
Joint Allention, Y
58.9 (-21.2)
Kromecker Mask Temporal Attention, Y
43.5 (-38.0)
Kronesker Mask Causal Temporal Attention,
(-38.)))
C ADDITIONAL EXPERIMENTS
The effects of interpretive prompt. In Tab. 10 we can observe that, directly use verbs and phrases achieve the lowest performance. When employing prefix and suffix prompt templates, the perfor- mance slightly improved. When introducing action decomposition interpretive prompts, the perfor- mance significantly improved. When adding synonym conversion interpretive prompts, the perfor- mance is further increased. Last, we involve body parts interpretive prompts that the performance reaches the best.
Which branches should be finetuned? We separately freeze the parameters of the pretrained image and text encoder. From Tab. [ we conclude the following observations: 1) Freezing both image and text encoders, only tuning KMT transformer achieves the worst performance. 2). Finetuning only one of them will improve the performance. 3). Finetuning them simultancously achieves the best performance.
The Evaluation performance of zero- and few-shot experiments on Kinetics-600, HMDB-51, and UCF-101 with CLAVER-B/16 (KMCT) configuration. The comparison of KMT and KMCT as shown in Tab. 12 13 In the zero-shot scenario, the performance of KMCTA and KMTA has its own wins and losses, which is due to the lack of further fine-tuning, and the model's ability is relatively dependent on previous training. In the few-shot scenario, we can observe that in most cases, the performance of KMCTA better, which further indicates that when the scale of training data is limited, the low-rank bottleneck is more significant.
Table 10: Ablation interpretive prompt.
Table 11: Which branches should be finetuned.
Test Prompls
Top-1 (5)
Top-5 (%)
transformer
Visual
Text
Top-1 (%)
Top-5 (%)
Nomn Phrase
77.4
78.7
94.4
Action Decomposition Intespretive Prompe
Suffix-Sutfix
71.4
78.0
93.6
94.0
V
x
81.0
95.1
Synonym Conversion Interpestive Prompt
78.2
94.2
80.8
95.2
livolving Body Pars Interpretive Prompt
78.4
94.3
81.5
95.5
21

---

Published as a conference paper at ICLR 2025
Table 13: Comparison of KMT and KMCT on Table 12: Comparison of KMT and KMCT on few-shot settings. zero-shot settings.
KMI
KMCT
GLAVHI--R R (KMCT)
CLAVER Bile
Zero-duo
Kinetic-408
24.1 +.0.9
HMDE: $4.0 + 2.0
UCF-101
150 76.4
HMD38-11 UCF-10I
rew-shot
63.9
K-1
KG3
K-15
72.5
K-2
58.3
K=1
K-8
K=16
D EXPERIMENTAL SETTING DETAILS
Architectures. CLAVER-B/32 adopts ViT-B/32 (Lv=12, =A=12, d=768, y=32) and is equipped
with a KMT/KMCT transformer (Lx=2, 12 d=768, p=32). CLAVER-B/I BB16 employs VIT- B/16 (Ly=12, =2 d=768, p=16), along with the KMT/KMCT transformer (Lk=2, 12,12,, d=768, /=32). CLAVER-L/14 is equipped with ViT-L/14 (2,= N1=16, d=1024, p=14) and the KMT/KMCT transformer (Lx=4, N1,=16, d=1024, p=14). Here Ly denotes the layers of ViT, Lk denotes the layers of Kronecker mask temporal transformer, N₂, refers to the number of attention heads, d represents the embedding dimension and p is the patch size.
The experiments are conducted on 8 NVIDIA 80G A A100 GPUs. We present the training hyperparameters in Tab. 14 Additionally, the learning rate for updating the KMT/KMCT transformer (randomly initializedd parameters is set 10x higher than the learning rate for parameters of the text encoder or image encoder. Because the text/image encoder already possesses ability to extract high-quality text/image representations, conversely, KMT/KMCT transformer is trained from scratch. the experiment, we freeze the parameters of the CLIP's image encoder reduce certain computational costs, as it already has strong image feature extraction capabilities. And then we also conducted experiments on this aspect in subsequent ablation study.
Hyperparameters.
Table 14: The training hyperparameters settings of experiments.
Contig
Fully-sup
Few-shot
Zero-shot
Optimizer
AdamW
Base learning rate
12e-6
2e-6
12e-6 12c-8
Minimal learning rate
12c-8
2c-8
Weight decay
0.001
Optimizer betas
3, Bg =0.9, 0.98 128 (ViT-B) 32 (ViT-L)
Batch size
Learning rate schedule
Cosine decay
Warmup epochs
Training epochs
60 (VTT-B) 40 (ViT-L.) 80 (20 on K400) 0 (20 on K400) RandomFlip, MultiScaleCrop. ColorJitter
Augmentation
GrayScale, Label smoothing, Mixup, Cutmix
Fully-supervised experiments setting, We conduct the fully-supervised experiments on Kinetics- 400&600. During training, sparse sampling strategy is used. The number of frames is set 8 or 16. We spatially scale the shorter side of each frame to 256 and take a 224 crop center crop. Following, we adopt the multi-view inference with 3 spatial crops and 4 temporal clips.
Few-shot experiments setting. We randomly sample 2, 4, 8 and 16 videos from each class on UCF- 101 and HMDB-51 constructing the training set. For evaluation, use the first split of the test set on UCF-101 and HMDB-51.
Zero-shot experiments setting. We train CLAVER-B/16 with 32 frames on Kinetics-400. The same as, we apply the following two evaluation protocols in zero-shot experiments. 1) Evaluation for HMDB-51 and UCF-101. Following, the prediction is conducted on the three splits of the test data, and we report the average top-1 accuracy and standard deviation. 2) Evaluation for Kinetics- 600. Following, the 220 new categories outside Kinetics-400 in Kinetics-600 are used for evaluation. The evaluation is conducted three times. For each iteration, we randomly sampled 160 categories for evaluation from the 220 categories Kinetics-600.
E INTERPRETIVE PROMPT DETAILS
First, we input command prompts into the ChatGPP| (2024) chatbot API to obtain demon- stration case of a small amount (in our experiment, we set it to here) of detailed descriptions of actions.
22

---

Published as a conference paper at ICLR 2025
Question to ChatGPT:
Below, will provide you with scme action nouns. Please provide de a simple and detailed description (explanation) about the action decomposition of these action nouns. Please note that the sentence Tength of the description should not exceed words.
Answered by ChatGPT:
Certainly! Please go ahead and provide the action nouns you'd d like descriptions for, and I'11 provide concise explanations within the speci fied sentence length.
Question to ChatGPT:
abseiling.
Answered by ChatGPT:
Abseiling combines several actions to descend vertical surface with rope. Climbers secure themselves with harness utilize descender device for controlled descent. simple actions, like maintaining straight body position and requlating rope tension, form the basis. Abseiling demands proper training, safety measures, and is popular in operaticns, allowing individuals experience controlled descent various settings.
adventure sports and
Question to ChatGPT:
air drummi ng.
Answered by ChatGPT:
Air drumming is a rhythmic expressicn where individuals simulate playing drums witheut physical ruments. Simple actions, like mimicking drumming motions in the air, combine to create this imaginative and playful activity. Knthhsiastss use their hands and feet to imitate drumming patterns, syncing with music. It's a spontaneous, enjoyable gesture often done during music listening or live performances, showcasing one's s connection to the rhythm without the need for actual drums drumsticks.
Question to ChatGPT:
answering questions.
Answered by ChatGPT:
Answering questions involves providing responses queries posed by others. Simple actions like active listening, comprehensien, and concise articulation combine in this communicative process. It is fundamental. in various contexts, facilitating information exchange and problem- solving. Respondents draw their knowledge and expertise to address inquiries, contributing to effective communication and fostering understanding between individuals or groups.
Question to ChatGPT:
Below, will provide you with some action nouns or phrases. Please provide many words and phrases that share the same central concept these action nouns phrases but have more diverse expressions.
Please note that your reply should not exceed 76 words in length.
23

---

Published as a conference paper at ICLR 2025
Answered by ChatGPT:
1. Running:
Jogging, sprinting, dashing, hurrying, racing, pounding the pavement, speeding, trotting.
2. Wating: Dining, feasting, devcuring, munching, ng, nibbling, gobbli Saabling in up indulging, savoring.
3. Writing: inscribing, scrawling, scripting.
Scribbling, jotting down, composing, drafting, penning,
4. Laughing:
Chuckling, giggling, guffawing, snickering, rearing with laughter, belly laughing, cackling, chortling.
5. Reading: perusing, browsing, scanning, skimming, devouring books, absorbing, immersing oneself, delving into texts.
Question to ChatGPT:
Cutting in the kitchen.
Answered by ChatGPT:
Slicing, dicing, chopping, mincing, cleaving, carving, trimming, preparing ingredients.
Question to ChatGPT:
driving car.
Answered by ChatGPT:
Operating vehicle, maneuvering behind the wheel, navigating the road, piloting an automobile, steering, cruising, commuting ing by car, motoring.
Question to ChatGPT:
Walking With Dog.
Answered by ChatGPT:
Strolling with canine companicn, ambling with pet, premenading with pup, hiking with a furry friend, sauntering alongside a dog, wandering with a four-legged buddy, leash-walking, trotting with a pooch.
Question to ChatGPT:
Below, will provide you with seme action nouns. Please describe these actions based on their nouns and possible body parts involved.
Please note that the sentence length your response should not exceed 76 words.
Answered by ChatGPT:
1. Running: The rhythmic motion of legs propelling the body forward, feet striking the ground with each stride, muscles contracting and relaxing to generate nomentum.
2. Swimming: Arms slicing through water, legs kicking to maintain propulsion, torso rotating to streamline movement, lungs rhythmically inhaling and exhaling for oxygen.
3. Writing: Fingers gripping pen er pencil, hand moving keyboard, muscles in the forearm and wrist coordinating precise
paper
movements to form letters and words.
4. Jumping: Legs bending at the knees, feet pushing off the ground, arms swingim ng for momentum, body soaring through the air momentarily before landing.
24

---

Published as a conference paper at ICLR 2025
5. Dancing: Feet tapping, legs extending and bending, hips swaying, arms gesturing and flowing in syne with music, body moving rbythmically with expression and grace.
Question to ChatGPT:
Cutting in the kitchen.
Answered by ChatGPT:
Using a sharp knife, fingers gripping the handle, hand guiding the blade through ingredients fingers curling slightly hold the food steady, precision applied to achieve desired shapes or sizes, ensuring safety and efficiency during food preparation.
cutting beard, wrist controlling the motion,
Question to ChatGPT:
driving car.
Answered by ChatGPT:
Gripping the steering wheel, hands adjusting position, tingers pressing pedals for acceleration and braking, eyes scanning surroundings for obstacles, feet coordinating between clutch, brake, and accelerator, body positicned comfortably in the driver's seat, mind focused on navigation
and traffic signals, reacting swiftly to changing road conditions.
Question to ChatGPT:
Walking With Dog.
Answered by ChatGPT:
Leash in hand, fingers securing grip, relaxed it swings alongside the body, legs moving tandem with the dog's pace, feet stepping forward with purpose, eyes attentive to the dog's s behavior and surroundings,
occasional stops for sniffing marking, bond of companionship evident in synchroni zed movement.
Subsequently, as illustrated in Fig we input the examples generated by ChatGPT OpenAl (2024) into
Dubey al, (2024) multiple times in the format of Command Exam- ples + Action Concept, automatically generating diverse text descriptions about the Action Con- cepl through the program. When dealing with datasets containing large number of categories, this scheme can greatly save labor costs. In addition, it should be noted that LLaMA-3 offers control- lable parameters to control the randomness (r) and diversity (p) of output texts. We 7 and p to 0.90 and 0.95, respectively to ensure that there are significant variations in output content for the same input, while maintaining consistency in the central concept.
F ANALYSIS OF SYNTHETIC VIDEO TESTING
We use the original prompts of synthetic videos (several accion-related examples generated by Im- agen Saharia al. (202) and Sora Brooks al. (2024) their corresponding text descriptions, and show Top-5 most relevant texts in Fig. The results shows the robustness and generalization of CLAVER.
For cases generated by Imagen, in Fig. 9 Upper, we denote the Kinetics-400 action text set as A..
The action category "lift cup" in Fig. 9 (a) does appear in A, while, (b) "washing dishes". (c)
25

---

Published as a conference paper at ICLR 2025
for A hand fifn cup
(b) 4 roddy bear washing the dishes
(c) An untronent on riding hore
66) Shoreling soove
Rodako
(e) A corkow kamgaron Sico donces
Aodii summon unilly desea
Col 1 younng
his 206
(U enforabii hagps offer
Figure 9: Testing on synthetic videos from Imagen (Upper) and Sora (Lower). The bar chart represents the Top-5 similarity texts, orange bar denotes the original prompts of synthetic videos.
"riding horse", (d) "shoveling snow" do appear in A. All videos do not come from the real-world, which are synthetic. We observe that all the original prompts appear in the Top-5 similarity (more precisely, Top-2). Specifically, for (a), "making tea" has the highest similarity, possibly due the presence of cups and human hands in the scene. In addition, the original text has the second-highest similarity. In (b), apart from the original prompt in Top-1, other texts such as "washing dishes" convey the same meaning as the original prompts but lack a description of the subject ("teddy") of the action. "Washing hands" and "peeling potatoes" may involve actions and scenes like hand movements and sinks, thus exhibiting high similarity. For (c), although the one with the highest similarity is not "An astronaut riding a horse", "riding or walking horses" the exactly describes the action in the video, with the only difference being the lack of the subject of the action. For (d), "Shoveling snow" belongs to A, and since the original prompt of the video "Shoveling snow", its similarity is very high, exceeding 90%.
For the cases generated by Sora, as depicted in Fig. Lower, (c) "disco dances" concept is not in A but has the highest similarity. Besides, other action concepts in the Top-5 ("country line danciig","air druming". "dancing macarena", "aaaarrnaa"""mmaa")) are all dence-related. (f) is a relatively chal- lenging sample here. Although the ground truth description is located in the second, the "smoking" in Top-1 I is entirely unrelated to the content of the video, because the original prompt of the video primarily describes the scene and the action-related text is relatively short. In addition, the shot is a process from far to near, which may lead the model to arrive at an unreasonable Top-1. In (g), the Top-2 descriptions are original prompt and "reading book". both strongly related the action in video. For (h), the original prompt only rank third. however, Top-1 "surface water" and the second "water skimming" are descriptions strongly related to the action in video.
G DATASET DETAILS
Sinetics-400&600, The Kinetics dataset consists of 10-second video clips collected from YouTube. Specifically, Kinetics-400 Kay al. (2017) consists approximately 240k training videos and 20k validation videos with 400 categories, while Kinetics-600 Carreira et al. (2018) is an extension of Kinetics-400, consisting of approximately 410k training videos and 29k validation videos with 600 categories.
UCF-101. UCF-101 Soomro et al. (0122) consists of 101 action categories, over 13k clips and 27 hours of video data. The database comprises realistic user uploaded videos containing camera motion and cluttered backgrounds. The training and test data are divided into three splits.
HMDB-51. HMDB-51 Kuehne et al. (011) is a collection of realistic videos from various sources, including movies and web videos. It is composed of 6,766 video clips from 51 action categories, with each category containing least 101 clips. The dataset is divided into three splits for training and test data. In each split, each action class has 70 clips for training and 30 clips for testing.
Synthetic Videos. We select some action-related videos from the demo cases generated by Imagen Saharia et al. (2022) and Sora Brooks et al.] (2024), and employ the original prompts and their corresponding synthetic videos pair of test samples.
26

---

Published as a conference paper at ICLR 2025
H MORE ATTENTION HEAT MAP VISUALIZATION
We visualize several spatiotemporal attention map of samples from Kinetics400, Imagen, Sora.
Spatial & Kronecker Mask Temporal Attention
Focel Frame
Word Importance
a hand lifts cup
(egrril
Word Importance
sho veling SHOW
rondiing reganb
Importance
surfing on a surf board in space
Focal Frame
Word Importance
panda bear driving
Bundong poueds
Maskedl
Word Importance
teddy times
pounts
Maskeed
Word Importance
teddy washing the dishes
Masked
Figure 10: Attention map of synthetic videos from Imagen.
27

---

Published as a conference paper at ICLR 2025
Spatial & Kronecker Mask Temporal Attention
Word Importance
riding walking with horse.
Masked
Word importance
playing basketball
Masked
Word Importance
cano cing kayaking
Squttal Hompara
Figure 11: Attention map of videos from Kinetics400.
Spatial & Kronecker Mask Temporal Attention
Word Importance
down tokyo
Word Importance
stands surf board, surfing water.
cOANton rrgreis
Word Importance
young man at his 2 0 is sitting piece of cloud in the sky reading book.
Madked
Figure 12: Attention map of synthetic videos from Sora.
28

---

Published as a conference paper at ICLR 2025
the pole running then lifing the gole
the pole
CLi
mon dri bbles under his hips on the basketball court showcase his superb de b bling skills
CLAVER
man dr bbles under his hips on the baskethall court showcase his supert dr bling skills
with ropes
steep surface
iling with
stoep ice surfoot
CLIP
croSs conncy skiing involves gli ding forward tuming stppping and mance vering around absocles
involves ding freward, stopping
amund obstzeles
Figure 13: More visualization examples of transition CLIP's attention on nouns CLAVER's preference on verbs.
Patch Size 32, Franee Length =
Full Rank 400, KMT Rank 399
Fell Rank 408, KMCT Rank 400
Full Rank - 400, Joint Rank 361
Patch Size 16, Frame Length B
-111
sightt
Full Rank 1576, KMT Rank 1560
Full Rank 15%, KMCT Ronk 1576
Fall Rank 1576, Joint Rank 1378
Putch Size Frame
Pall Rank 3152, KMT Kank 2983
KMCT Kank 3151
Pall Mank 3152, Joint Kank 2381
Figure 14: The rank of Kronecker temporal attention (KMT), Kronecker causal temporal attention (KMCT) and joint attention (Joint) on different patch size and frame length. (The calculation of rank is done via forch.linalg.oo
29


Reviews:
[{"review_id": "GBrVI1QBED", "reviewer": "Reviewer_rNM4", "summary": "This paper focuses on video action recognition tasks and presents CLAVER to improve CLIP by aligning video representations with verb understanding. The authors design the Kronecker mask attention for improved temporal modeling and interpretive prompts generated by large language models to enhance verb comprehension. The authors demonstrate that CLAVER shows competitive performance on benchmarks such as Kinetics-400, Kinetics-600, HMDB-51, and UCF-101.", "questions": "See weakness section", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper observes two challenges when applying CLIP to video action classification: How to perform effective temporal modeling and how to design suitable text descriptions for verb understanding. The authors then propose the Kronecker mask temporal attention (KMTA) and Kronecker mask causal temporal attention (KMCTA) to solve 1) and interpretive prompts to solve 2).\n2. The paper conducts experiments on multiple video action classification datasets under supervised, few-shot, and zero-shot settings. The results show improved performance over baseline models and other state-of-the-art approaches.\n3. The paper includes thorough ablation studies to isolate the contribution of each component, such as KMTA and interpretive prompts, providing evidence for their effectiveness.", "weaknesses": "1. My major concern about this work is its novelty. The proposed Kronecker Mask Attention seems to be a simple combination of pipeline temporal attention and joint attention by attending to all patches in other frames. I expect the authors to provide more insightful analysis and discussion about the proposed method and the differences from previous works.\n2. The computational cost introduced by the Kronecker mask attention and the interpretive prompts could be further detailed, especially regarding training and inference time compared to other baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MBzXseU9xo", "reviewer": "Reviewer_RydQ", "summary": "The author proposes CLAVER: a Contrastive Language-Action Video Learner, to shift CLIP’s focus for video action recognition tasks from the alignment of static visual objects and nouns to dynamic action behaviors and abstract verbs. They introduce CLAVER with generalizable Kronecker Mask Temporal Attention (KMTA) to expand the temporal receptive field for each token and serve as an effective spatiotemporal heterogeneity inductive bias, mitigating spatiotemporal homogenization. In addition, they utilize large language models to generate interpretive prompts of actions, which shift the model’s focus towards verb comprehension. They conduct extensive experiments on well-known action recognition datasets like Kinetics-400, Kinetics-600, HMDB-51, and UCF-101 demonstrating its competitive performance across models in full-shot, few-shot, and zero-shot settings.", "questions": "In addition to the questions raised in the weakness section, there are a few minor questions.\n\n•\tIn Figure 4, the transaction from the left side to the right side might need further explanation. In Figure 5, the example might not fully explain the claim for word importance, please consider making another effective example. \n\n•\tIn Table 1, we can see the most powerful model for CLAVER is the CLAVER-L/14(KMT/KMCT). However, when conducting later experiments on Kinetic-600, and other few-shot, zero-shot experiments, the author uses CLAVER-B/16 (KMT) but not the strongest model in Table.1 even if it can follow the same setting. I’m wondering the reason why the author didn’t use their best model for all experiments.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "•\tInnovative Kronecker Mask Attention: The Kronecker mask temporal attention and Kronecker mask causal temporal are innovative and effective for temporal modeling. Compared with previous commonly used approaches such as joint attention, spatial attention, pipeline temporal attention, etc., it allows each patch at timestamp t to interact with all other patches and expands the temporal receptive field width of each token. It also alleviates the impact of spatiotemporal homogenization.\n\n•\tLeveraging large language models to create diverse, sentence-level prompts for action verbs is an effective approach. The author focuses on action decomposition, synonym conversion, which is an inspiring idea of “facilitating the alignment of action behaviors and verbs” by decomposing complex actions into basic and conveying the same core concept with varied expressions. \n\n•\tThis paper is well-written and provides a clear overview of CLAVER, with Kronecker Mask Temporal Attention and interpretive prompts. It conducts extensive experiments across several commonly used benchmarks with previous state of the art models as comparison and shows competitive performance, especially in few-shot and zero-shot settings, and provides comprehensive ablation studies to clearly claim the strength of CLAVER by parts.", "weaknesses": "•\tInsufficient performance difference between KMTA and KMCTA. KMCTA aims to alleviate the low-rank bottleneck and the author claims that it is important to avoid the low-rank bottleneck to improve the representation power of transformer architecture by giving a formal proof. In the latter ablation studies, the author also successfully shows that KMCTA is profoundly affected by both PreTE and PostTE shuffling, which suggests that KMCTA possesses varying degrees of ability in mitigating spatiotemporal homogenization. However, based on the experiment results in Table. 1 and Table. 7, the difference between KMTA and KMCTA is marginal and we also lack experiments on zero-shot, few-shot settings for CLAVER with KMCTA, which I didn’t see how KMCTA solved the bottleneck according to the current experiments setting.\n\n•\tThe pipeline of interpretive prompts. The author introduces an interesting pipeline of interpretive prompts by performing action decomposition, synonym conversion, and involving body parts. Then, they provide the prompt to LLaMA-3 for text completion. However, based on the illustration in Figure 4, and the textual description, I feel confused about how to make up these three steps from the left side of the figure to the right side of the figure into format prompts to LLaMA in order to perform text completion.\n\n•\tQualitative analysis for word importance needs further proof. Based on the visualization for CLIP, X-CLIP in Figure. 5, while the author make a claim that CLIP tends to focus on nouns, whereas CLAVER prefers verbs, it’s hard for me to recognize the difference between CLIP and CLAVER’s visualization. As a result, the effectiveness of interpretive prompts for noun concept to verb concept transition might need further explanation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "syOGBcLLaT", "reviewer": "Reviewer_2oCA", "summary": "The paper introduces a novel framework called Contrastive Language-Action Video Learner (CLAVER), aimed at enhancing the adaptation of CLIP for video understanding, particularly in action recognition tasks. This research addresses a critical gap in existing methodologies, which often focus on modifying either the textual or visual components of CLIP in isolation. The authors argue that a dual adaptation of both branches is essential for effective video analysis. The key contributions are twofold: (1) Kronecker Mask Attention that significantly improves temporal modeling in video data (2) The authors utilize large language models to generate rich, sentence-level prompts that focus on action verbs, steering the model's learning towards a deeper understanding of dynamic behaviors rather than static objects. The extensive experiments across multiple benchmarks, showcasing the effectiveness and robustness of CLAVER compared to existing methods.", "questions": "Refer to weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper is well-written and easy to understand. The figure illustration and captions are informative.\n* The proposed Kronecker mask temporal attention and Kronecker mask causal temporal attention schemes seem interesting. The authors have proved its effectiveness for temporal modeling\n* Experimental results on four widely used benchmarks show that the proposed model could achieve superior performance with the existing contrastive baselines.", "weaknesses": "* The performance gain is marginal. The advantages on multiple benchmarks compared to the current state-of-the-art methods do not show a significant improvement, which somewhat undermines the effectiveness of this approach. It remains unclear whether the enhancements stem from the hyperparameter settings or other tricks.\n* Can the authors provide details on how much additional computational cost and efficiency are incurred by adding the extra Kronecker mask temporal attention compared to the original CLIP method?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j9FksGltbo", "reviewer": "Reviewer_sCy3", "summary": "This paper proposes a Contrastive Language-Action Video Learner (CLAVER) to efficiently adapt both the visual and textual branches of the CLIP for improving video action recognition. CLAVER introduces the Kronecker mask for temporal modeling to capture the long-range and wide-range dependencies among frames. Moreover, CLAVER resorts to LLMs to generate interpretive prompts of actions to facilitate the alignment of action behaviors and verbs. Experimental results on multiple benchmarks, such as Kinetics-400 and Kinetics-600, show the superiority of the proposed method.", "questions": "See details in the weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1 The proposed method incorporating the Kronecker mask and interpretive action prompts is demonstrated to be effective in supervised, few-shot, and zero-shot settings.\n\n2 The visualization results sufficiently show that the proposed method captures better spatial-temporal attention as well as focuses more on verbs.\n\n3 The proof of KMCTA can guarantee full rank is provided in detail in the Appendix. \n\n4 The paper is well-written with figures and tables nicely presented.", "weaknesses": "Overall, I think this paper introduces an effective and generalizable mechanism for improving video action recognition. The proposed method is novel and supported by extensive experimental results in both the main text and Supp. I only have some small concerns as follows:\n\n1 Since the LLM tends to introduce some task-agnostic information in its output, will this noise impact the actual action recognition performance? I would like to see more analysis and examples here. Moreover, the proposed interpretive prompts look like a plug-and-play module. When it is combined with other existing models, will it contribute to performance improvement consistently? \n\n2 Can the authors give a complete analysis with examples to show when the proposed method may fail? This will help better understanding and deeper analysis of the proposed method.  \n\n3 The font size in Figure 6 is too small for reading. The authors should modify it to give a better presentation. Moreover, Table 14 in Supp has the wrong format.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)