T2L: Efficient Zero-Shot Action Recognition with Temporal Token Learning
Authors: Shahzad Ahmad
Sukalpa Chanda
Yogesh S Rawat
Venue: Accepted by TMLR ()

Abstract: Recent advancements in large-scale pre-training of visual-language models on paired image-text data have demonstrated impressive generalization capabilities for zero-shot tasks. Building on this success, efforts have been made to adapt these image-based visual-language models, such as CLIP, for videos extending their zero-shot capabilities to the video domain. While these adaptations have shown promising results, they come at a significant computational cost and struggle with effectively modeling the temporal aspects inherent to the video domain. In this study, we present Efficient Zero-Shot Action Recognition with Temporal
Token Learning(T2L), a simple and efficient adaptation of CLIP that addresses these challenges. T2L leverages Temporal Token Learning (TTL) for seamless temporal adaptation, requiring no fundamental changes to the core CLIP architecture while preserving its
remarkable generalization abilities. TTL relies on temporal feature diversity (TFD), a novel learning objective, which guides TTL to focus on capturing motion, thereby enhancing  its learning capabilities from videos. We perform extensive experiments on nine different
benchmark datasets, thoroughly evaluating T2L for zero-shot learning and base-to-novel video action recognition, and also demonstrating its potential for few-shot generalization. Impressively, with merely 5.2 million learnable parameters, T2L can be efficiently trained
on a single GPU (with 25x less learnable parameters, 3x reduction in GFLOPs, and 4x improvement in throughput when compared with prior best model), outperforming existing approaches in several evaluations.

Published in Transactions on Machine Learning Research (04/2025)
T2L: Efficient Zero-Shot Action Recognition with Temporal Token Learning
Shahzad Ahmad
MaizzaShhof..im
Deportment of Computer Science and Communiiaatinn, Østfold University College, Norway
Sukalpa Chanda
sukalpa@@ieee.org
Department of Computer Science and Communication, Østfold University College, Norway
Yogesh S. Rawat
yogesh@ucff edu
Center for Research in Computer Vision University of Central Florida
Reviewed on OpenReview: /ttps://openreeee/ net/ forum? <d- HugozpCpuU
90
HMDB
UCF-O
X-CLIP
79.1
ActionCLIP
T2L
UCF-DS
25.1
K-600
Model Accuracy
322.4
VIFi CUP P: 71-1 X
63.6
T2L
VIFI CLP
P:: 58.8
HMDB-R.
52.
N/A
63.0 59.0
66.3
K-400(B2N)
70
N/A
NO
57.M
Vanila TP: 560
N/6
65
UCF-p 35.1
66.3
60
Action CLIP
P. 67.7
63.041.55
70,0
HMDB(B2N)
13.2
85.4
55
SSv2(FS)
UCF(B2N)
125 Tunable Parameters (Millions)
100
200
UCF(FS)
92.8
67.7
SSV2(B2N)
14.8
HMDB(FS)
Figure Effectiveness of T2L: (Left) T2L outperforms existing methods on the UCF-101 dataset for zero-shot evaluation with fewer tunable parameters and GFLOPS (bubble size), and higher throughput (TP). (Right) Evaluation across 14 different benchmarks demonstrating superior or competitive performance in comparison with existing approaches.
Abstract
Recent advancements in large-scale pre-training of visual- visual-language models on paired image- text data have demonstrated impressive generalization capabilities for zero-shot tasks. Build- ing on this success, efforts have been made to adapt these image-based visual-language models, such as CLIP, for videos extending their zero-shot capabilities to the video domain. While these adaptations have shown promising results, they come at significant computational cost and struggle with effectively modeling the temporal aspects inherent the video domain. In this study, present Efficient Zero-Shot Action Recognition with Temporal Token Learning(T2L) simple and efficient adaptation of CLIP that addresses these challenges. T2L leverages Temporal Token Learning (TTL) for seamless temporal adaptation, requir- ing no fundamental changes to the core CLIP architecture while preserving its remarkable generalization abilities. TTT relies on temporal feature diversity (TFD), a novel learning objective, which guides TTL focus on capturing motion, thereby enhancing its learning capabilities from videos. We perform extensive experiments on nine different benchmark datasets, thoroughly evaluating T2I. for zero-shot learning and base-to-novel video action

---

Published in Transactions on Machine Learning Research (04/2025)
recognition, and also demonstrating its potential for few-shot generalization. Impressively, with merely 5.2 million learnable parameters, T2T. can be efficiently trained on a single GPU (with 25x less learnable perameters, 3x reduction in GFLOPs, and 4x improvement in throughput when compared with prior best model), outperforming cxisting approaches in several evaluations. Code available here httts:///itthht com/ com/Shahzadnit//22l
1 Introduction
Large-scale pre-training of vision-language (VL) models on image-text pairs has shown exceptional effectiveness across diverse downstream tasks, particularly excelling in zero-shot scenarios Radford et al. (2021) Models
like CLIP Radford et al. (2021) and ALIGN Jia et al. (2021). : trained on vast internet-soureed data, provide strong, transferable representations with impressive generalization capabilities.
However, extending such pre-training strategy to videos poses significant challenges. Unlike image-text pairs, aligned video-text data is scarce, and curating it is a challenging task Jia et al. (2021); Xu et al. (2021). Furthermore, videos inherently are more complex due to temporal dimension and entail substantial computational costs, while appearance cues can be captured efficiently through image-text pains with a much lower compute budget. Therefore, adaptation of these image-language models for video-basod tasks, while retaining their generic multimodal learned representations, is a promising research direction.
Motivated by this, recent works have adapted image-based VL models like CLIP Radford et al. video representation learning by introducing additional learnable components for spalio-tempora modeling such as cross- frass- aelf-attention layers Ju et al. (2022). textual or visual prompts Wang et al. (021))
2021 for
specialized video encoders Ni et al. (2022). More recently, Rasheed et al. (0223 proposed fine-tuning the entire
CLIP model for video tasks. While these approaches achieve promising results, they are computationally intensive and often require fine-ture fin-tuning large numbers of parameters, which can hinder generalization.
In light of these limitations, we propose T2L, an efficient adaptation of visual-language models for zero-shot action recognition. T2L introduces Temporal Token Learning coupled with simple yet effective motion constraint to address the modeling of temporal aspects in videos. T2L does not update spatial features learned during pre-training, which preserves the generalization capabilities. The proposed approach is computationally efficient and can be trained on a single GPU with minimal learnable parameters.
We provide extensive evaluations across nine different action recognition benchmark datasets (Kinetics-400, Kinetics-600, UCF--10,, HMDB-51, fomething-somethhnng UCF-101-O) for zero-shot, base-to-novel, and few-shot settings, showcasing our model's robust generalization capability, particularly in scenarios where motion plays crucial role.
UCF-101-DS, UCF--10-P,, HMDB51-P and
The main contributions of our work are as follows:
We propose efficient adaptation of image-based visual-language models for zero-shot video action recognition.
We introduce Temporal Token Learning (TTL) to model temporal aspect in videos. It effectively learns temporal dependencios across video frames with minimal learnable parameters.
We propose a simple yet effective temporal feature diversity (TFD) loss which guides TTL learning temporal behavior.
We perform exlensive evaluation across nine different action recognition datasets demonstrating its robust generalization capabilities. T2L outperforms previous best methods in several evaluations with merely 5.2 million learnable parameters.
2
Related Work
Video Action Recognition Effective video understanding requires modeling both spatial and motion cues. Vision Transformers have emerged strong alternatives to 3D CNNs, capturing long-range spatio-temporal
2

---

Published in Transactions on Machine Learning Research (04/2025)
dependencies and outperforming carlier models Feichtenhofer al. (0099) Carreira & Zisserman 2017
Wang et al. (2017) Christoph & Pinz (2016). Moving beyond traditional uni-modal approaches, recent
methods like ActionCLIP Wang et al. (2021), XCLIP Ni et al. 2022). ; and Ju et al. JJu et al. (2022) adopt a
multi-modal framework, leveraging image- based visual-language models for zero-shot video understanding.
Vision-Language Models The effectiveness of learning multi-modal representations through large-scale image-text pre-training has been well established, showing strong performance across a wide range of uni-
modal and multi-modal tasks Chen et al. 2020); Kamath et al. (2021) Li et al. (2019 2020). Vision-Language (VL) models like CLIP Radford et al. [2021] and ALIGN Jia et al. 2021 have pioneered this area, using contrastive self-supervised objectives on large-scale image-caption pairs. These models exhibit impressive transfer capabilities in downstream vision tasks, including few-shot and zero-shot recognition, detection, and segmentation. However, extending these models to the video domain is challenging due to the absence of temporal cues in their image-level pre-training. To address this, recent efforts Ju et al. (2022); Ni et al. (2022); Wang al.] (2021) have adapted CLIP for video applications by integrating additional learnable modules such sell-allemtion layers, visual textual prompls, and dedicated visual decoders- ssowing performance improvements in video tasks. Specifically, Ju et al. Ju et al. (2022) adapt CLIP by incorporating text prompts and transformer layers for temporal modcling. While these temporal adaptations enhance performance in certain scenarios, they can reduce CLIP's generalization ability, particularly in zero-shot settings. To overcome these challenges, Rasheed et al. (2023) introduce ViFi-CLIP, a fully-supervised approach that improves both generalization and performance in zero-shot video understanding.
3 Methodology
We first discuss image-basod visual language models (Soc. 3.1, then we introduce T2L (Soc. 3.2 and describe Temporal Token Learning (Sec. 3.2.1 followed by temporal feature diversity (Sec. 3.3
3.1 Background
An image- based visual-language model comprises an image encoder Emaage and a text encoder Evext jointly trained on large-scale image-text pairs by maximizing the similarity between image (ex) and text (y:) encodings from positive pairs. This is typically achieved using contrastive objective Radford et al. 2021
Lole,y) = - Σ log
exp(sle.,u)))
(1)
exp(s(e),
where s(e,, v1) is the cosine similarity between image encoding e and its corresponding class encoding Vo N denotes the total number of classes, and is temperature parameter applied to the similarity. This simple training strategy, when scaled to large datasels, has demonstrated impressive zero-shot capabilities. Once trained, these models can be applied to downstream tasks like image classification, where each target class is represented as a textual prompt using handcrafted templates such as 'this is a photo of <class name>'. Classification is performed by matching the visual encoding with the textual prompts of candidate classes. This prompt-based representation enables the model's zero-shot capability.
3.2 T2L
Our goal is to efficiently adapt image-based visual-language models pre-trained on large-scale image-text, pairs for vidco domain while preserving their zero-shot generalization capability. Since these encoders are pro- trained on image-text pair, they do not have any understanding for temporal aspects in videos. To overcome this challenge, existing methods have developed special temporal encoding blocks such self-attention layers Ju et al. (2022) or dedicated video encoder modules Ni et al. 22022), which can learn the temporal relation between frames. However, such approaches are not very efficient in capturing the temporal cues since each video frame is encoded independently. Moreover, they require a large number of trainable parameters which is an overhead on training time and computing resources.
We want to adapt visual-language to capture temporal aspect from videos while preserving the spatial learning from images with minimal modifications (without changing the already learned weights). Towards this goal,
3

---

Published in Transactions on Machine Learning Research (04/2025)
Ve R
RD
Co Mb)
Temporal Pooling
MLP
MLP
MLP
LN
LN
*
Text Encoder
(1)
(9)
(T)
Adapter d
Adapter 6
Adapter A
B
B
B
TTL block shared across
frames
Tokerize
Action description
11-1
MHA
MHA
B₁
B,
B
Sharees Weighee
LN
Stardd Weightss
LN
Shared Weights
TTL block shared across frames
Stage
a) Text Feature Extraction
Stage
Mean
Mean
Mean
Stage
init update token token
init update token token
init update
token token
Stage
Video Encoder
Skip Conmection
Temporal Taken Il Image Embedding
Text Embodding Concat
Frazen
c)TTL
Overview of the proposed method: T2L utilizes Temporal Token Learning (TTL) to efficiently learn temporal aspects, eliminating the need for the frame integration module, a bottleneck in adapting image models for video understanding. Module (a) shows the text encoder(see detail in section 3.2.3). Module (b) depicts the architecture of video encoder, where each frame parallel encoded by a vision transformer. Temporal relations between frames are learned by module Temporal Token Learning(see detail in section
Figure 2:
3.2.1 and within the vision transformer, each block is adapted using spatial adapters(see detail in section shown in (c)
3.2.2
we develop T2L, an Efficient Zero-shot video action recognition model based on CLIP Radford et al. (2021) which relies Temporal Token Learning(TTL) and novel temporal feature diversity (TFD) loss to capture motion cues from videos. We seamlessly integrate these elements into the CLIP architecture, leveraging its pre-trained capabilities. In this study, we experiment with CLIP but the proposed approach is general and should be applicable to other vision-language models.
Problem formulation Given a collection video samples, we represent a video as V f f E RTxHxxx consisting of frames represented f. each with resolution of H x W and C 3 for RGB channels. Each video is associated with a text prompt e which represents the targel action class. We want model M, which leverage visual E, and text E, encoders from image-based pre-training and provide encoding for video V and its text-pair which are similar to each other and dissimilar from prompts of other action classes.
Overview Given a video with frame sequence, we rely on the image encoder Eş for encoding each frame and the text encoder Eg to encode the textual prompt for action class. The visual encoder handles each frame, resulting in the generation of frame-lovel embedding denoted as fe(1), e(2) ...., e(T) RTxD shown in Figure 2. Here, ( signifies the embedding of the t frame within the video. The Temporal Token Learning help in capturing any relations between frames to model motion aspect in videos. The spatial features also adapted to enable effective temporal learning, These frame embedding are then combined to create holistic video-level representation V R The text encoder in image-based models also lack motion understanding therefore T2I. also adapts text encoder along with visual encoder. During training, only temporal tokens, spatial adapters and text adapters are trained keeping all the weights from CLIP frozen.
3.2.1 Temporal Token Learning
Let's consider video clip V RT composed of T frames. Following similar approach to the CLIP
image encoder Radford al. 2001)) each f-th frame is divided into non-overlapping patches (((()) J of size E PxPX Here, signifies the temporal index, and #xW where P denotes the patch size. These ps
patches, (0,00N Ji=ly are then transformed into D-dimensional patch embedding s.!0 E NDD A class token
4

---

Published in Transactions on Machine Learning Research (04/2025)
wits E ERD is prepended to xy = embedding Eps E ( are added to (1) as 20 20 + Epos, where 20 is the final input being fed to a sequence of transformer blocks at the t-th frame
(t)
[ptsi (0) [ To encode positional information, positional
(1) (1)
To facilitate cross-frame temporal learning in video, we introduce a novel concept called Temporal Token Learning.
Stage 1: Initialization of Temporal Tokens
These temporal tokens are strategically initialized at the input space of each Transformer layer, denoted as pTemp E RLXTxD where L is the mumber of layers in the CLIP Transformer. This collection of input learnable tokens is defined as pl'emp = {or E RRXD | 11= 0, 0....,
Stage 2: Linking Temporal Tokens with Frame Embeddings Each f-th frame's embedding is linked to the respective temporal token as,
N N1
This is illustrated in Figure module (c).
Stage 3: Temporal Multi-Head Attention (MHA) Block
To capture temporal dependencies effectively, introduce additional MHA operation within cach transformer layer. This operation shares the same parameters as the existing MHA layer, ensuring consistency while maintaining computational efficiency. The shared MHA mechanism is dedicated to processing only the temporal tokens, enabling effective inter-frame modeling without increasing the number of trainable parameters.
Stage 4: Processing Temporal Tokens
Temporal tokens are obtained by aggregating the average pooling of each frame token and the learnable embeddings. Subsequently, MHA is performed on all tokens for each frame independently (including the temporal token). This operation at the I-th block is expressed as pr MHA(LN((i)) where pt (n PI (2) ,.... (27), MHA represents a pre-trained and frozen atlention block at the 1-th layer derived from CLIP also shown Figureg module (c), and LN denotes layer normalization. The final learned temporal
token is concatenated with the frame embedding [=]-1_p]") [0]] to facilitate further processing.
Our Temporal Token Learning approach is novel and unique as it enables efficient temporal learning without a separate architecture. By leveraging the cxisting CLIP image MHA with self-lcarnable tokens, we simplify the design while elfectively capturing temporal dependencies, making highly efficient solution for temporal learning in video processing.
3.2.2 Spatial Adaptation
In Figure 2 module (c) depicts the transformer block structure B,. After the Multi-Head Attention (MHA)
module, the temporal token is discarded, and the frame patch token is forwarded for subsequent processing. The intogration of the Adapter from Yang et al. (2023) after the self-attention layer within this transformer block modifies the CLIP architecture. For the image encoder, the computation within a transformer block is expressed as:
[ o 1= [x
(3)
where [ concatenates the frame embeddings and temporal tokens. This output is then passed through an Adapter from Yang et al. (2023) followed by a residual block:
(1) Adapter(=!!))
20 MLP(LN(2()), ). 1 1_0.1...1.-1
(5)
Gradient Flow for Temporal Token Learning Gradient of the loss function Ciotal with respect to temporal tokens pTemp {pr E RTD
|| 1- O..,, 1} is shown in equation
6 See section A.1 in Appendix
for detail derivation).
O opmmp
ΣΣ
DErotal Op
(M (1) (
OMHA(LN().)
(6)
ap
ap!
1 1m1
5

---

Published in Transactions on Machine Learning Research (04/2025)
3.2.3 Language Adaptation
To facilitate effective temporal learning, we adapt the text encoder to incorporate motion aspects into the textual prompts. Leveraging insights from efficient fine-tuning techniques in NLP Zhang al. (2023); Zong al. (2021); Zaken al. (2021). recent advancements ZZang et al. (2023) Yang et al. (2023) have demonstrated strategies for fine-tuning without perturbing the original model weights. Specificallyy,
use
the adapters from [Zhan et al. 2023 to adapt the language model. We represent action class descriptions generated by a LLM (GPT-3.5) as textual prompts. We use a template "describe [category] as an action performed by humans' to generate descriptions. Throughout training, all transformer layers remain frozen except for the Adapters, which are updated. As shown in Figure 2 module (b), the text encoder adaptation is represented as De Cdes + MHA (Tdees),,, De = Adapter and ye = Oc + (T.((..((e)),, where Cdes is the text embedding of the class description
3.3 Learning Objective
In the context of multimodal embedding models, our learning objective is minimize the dissimilarity between video embeddings (v) and class embeddings (ye) using a Contrastive loss Radford et al. (2021) (Eq.
The contrastive loss function is vital for training, quantifying dissimilarity between predicted and ground- truth distributions to facilitate video and class embedding association. Yet, may overlook video's 's intrinsic properties. For instance, if a'appearance suffices for classification, it might prioritize motionless video embeddings. To tackle this, we introduce the temporal feature diversity loss, emphasizing intrinsic motion properties, enhancing motion features in video embeddings.
Temporal Feature Diversity A video is composed of sequence of TT frames picked at equal intervals. When there is motion in the video, generating embeddings ee,,,..,,ee for each frame results in subtle differences among these embeddings. Our goal encompasses two facets: enhancing both the diversity (variance) and the distinctiveness (local diversity) among frame embeddings. This objective entails creating embeddings in way that not only amplifies the differences between frames but also accentuates their contral variations. To achieve this objective, we introduce the Temporal Feature Diversity (TFD) loss. Let Var denote the degree of diversity among the frame embeddings, and let C represent the measure of central difference. Then Var E Raand C' E IR are defined as,
Var
I T
emean)? 2 where emean
T
where (c( -Cmeau) denotes clement-wise squaring of the vector difference (elt) - Cmean) RR
Σ
T-1
8e
where
(0+)))
(8)
2
dim(Var)
dim(o)
c= dim(V ar)
Σ
Var, dim((C)
Σ C.
(9)
Here dim(Var) and dCm(C) is same as dimension of embeddings et. Our goal is to maximize the value of L during the training process, which amalgamater both the desired diversity and central distinctiveness. This leads us to the formulation of the Temporal Feature Diversity (TFD) loss:
CTFD 5+C where 8
(10)
Here, ô is a positive constant (specifically set to 1). The computed value of the loss inversely proportional to the sum of 8 and L. This design choice emphasizes the importance of both high diversity and substantial central differences among frame embeddings. The overall learning objective is to minimize the final loss
function termed as Ltatel that linearly combines the traditional contrastive loss L. I along with LTTD and is defined as,
Crstal = AeCe(v, ve) + >2CreD
(11)
where X and X2 are corresponding weights; we use equal weights for both in all our experiments.
6

---

Published in Transactions on Machine Learning Research (04/2025)
3.4 Fine- Tuning Technique and Training Protocol
Our proposed method, Temporal Token Learning (T2L), efficiently models temporal dynamics in videos while maintaining computational efficiency. Unlike prior approaches that require extensive backbone fine-tuning T2L introduces lightweight adaptations with
(e.g,, ActionCLIP Wang et al. 2021). X-CLIP Ni et al. 2022 )
a frozen base model.
Frozen Backbone: T2L leverages the pre-trained CLIP VIT-B/16 (86.0M parameters) and the corre- sponding text encoder (63.0M 0830MM parameters), keeping their 149.0M parameters frozen during training. This strategy preserves the generalization capabilities of the base model and significantly reduces computational overhead.
Lightweight Adaptations: T2L adds only 5.205M trainable parameters: spatial adapters (3.550M), text adapters (1.581M), and temporal tokens (0.074M). This constitutes just 3.5% of the total parameters, in contrast to prior methods that require extensive fino-tuning or architectural changes.
Temporal Modeling: T2L employs TTL for capturing lemporal dependencies and TFD loss to enhance frame-level variability, achieving efficient temporal adaptation without additional temporal layers or self- attention mechanisms. Notably, TTL learns temporal consistency across all layers of the ViT-B/16 image encoder transformer, rather than relying solely the output of the last layer.
Efficiency: By fine-tuning only adapters and temporal tokens, T2T. minimizes FTOPPs and memory usage while preserving high throughput efficiency. This lightweight approach achieves superior or competitive performance on nine benchmark datasets with minimal computational overhead compared to prior methods. This design underscores the efficiency and scalability of T2I., offering an effective solution for video under- standing tasks.
4 Experiments and Results
Datasets: We evaluate our proposed method on nine different video action recognition benchmarks: Kinetics-
400 Kay (2017), Kinetics-600 Carreira al. (2018). HMDB-51 Kuehne et al. [2011). UCF-101 Soomro
et al. 22022), Something Something V2 (SSv2) Goyal et (2017) UCF-DS SShhaappp et al. 2023 UCF-101-P,
et al. (023)) and UCF- Modi et al. (202)). Kinetics-400, Kinetics-60M 600, HMDB-51, and UCF-101 are known to have some appearance biases where background can also be important for recognizing actions Choi et al. (2019). On the other hand, sometting-something-- is a more challenging dataset where temporal understanding is critical in recognizing the actions. UCF-DS, UCF-101-P, HMDB- 51-P and UCF-101-O datasets have occlusion distribution shifts due to weal-world perturbations instead of adversarial perturbations, making them useful for evaluating the robustness of models. Additional datasct details can be found in the Appendix A.5.
HMDB-51 MMDDB-51-PSchapppe
We ViT-B/ 1a11116-aasss based CLIP model visual encoder in experiments. In addition, we also assess our model's generalization using CLIP ViT-32 and CLIP ViT-14 backbones. The setup employs only 8 sparsely sampled frames per video, ensuring computational efficiency. We use Adamm optimizer with a base learning rate of 5 x 10-5 to train out models for 50 epochs with and a weight decay of 0.2. The learning rate warms up for the initial 10% of epochs and then follows a cosine schedule. Training is performed on single NVIDIA A100 80GB GPU, with batch size of 70, and maintaining an input frame resolution of 224 224 pixels.
Implementation Details:
4.1 Evaluation and comparisons
Next, we show evaluation of T2L for zero-shot, baso-to-novel, generalization capability for few-shot learning, and finally its robustness against distribution shift. Detail can be found in Appendix A.6
Zero-shot: In this setting, the proposed model trained on a source dataset, Ds (Kinetics-400), and then directly tested on downstrcam cross-datasets, specifically HMDB-51, UCF-101, and Kinetics-600, for evaluation. The source dataset, Ds, contains samples from source classes, Ys {y: /= and the evaluation is performed on the target dataset. Dr, where Ys Yr 0.

---

Published in Transactions on Machine Learning Research (04/2025)
Table Zero-shot comparison: We assess T2L against uni-modal approaches for zero-shot action recognition and image-based Vision-Language (VL) models adaptod for video action recognition. Performance is measured using top-1 accuracy. Models are trained on Kinetics-400 and tested on HMDB-51, UCF-101, and Kinetics-600 (disjoint classes not shared with Kinetics-400).
Method
Backbone
HMDB-51 51
UCF-101
K-600
GFLOPS
Uni-modal zero-shot action recognition models
ASR [Wang & Chen 2017a
BiDiLEL (Wing & Chen 2017b
21.8
21.4
ZSECOC (Qin al.
2017
Fisher Vectors Perronnin al.
2010
22.6
15.1
UR (Zhu al 2011
ResNet-2000 Xte al. 2017
24.4
17.5
E2C [Drattoli al.
2020
R|2
1]D-18 Iran
2018
32.7
48.0
ER-ZSAR (Chen Huang
2021
TSM Lin et al.
2019
35.3
51.8
Adapting pre-trained image VL models
Vanilla CLIP [Radford
2021
ViT-16
40.8
63.2
59.8
ActionCLIP [Wang et al.
2021
VIT-16
40.8
58.3
67.7
563
XCLIP (Ni et al. [2022
VIT-16
44.6
72.0
65.2
287
A5 [Ju et al. 2022
VIT-16
44.3
69.3
55.8
ViFi CLIP (Rusheed et al.
2023
VIT-16
51.3
76.8
71.2
281
T2L
VIT-16
52.9
79.1
70.1
102
T2L(VVI-32
VIT-32
50.0
77.5
67.0
31.1
(VIT-11)1
VIT-14
55.2
82.6
72.1
454.9
Table 2: Base to novel generalization: We perform comparison on four diverse datasets Kinetics-400, HMDB-51, UCF-101, and SSv2. Models are evaluated using top-1 accuracy and HM is the harmonic mean of the base and novel classes.
Method
Backbome
Kinelávs- 400
HMDB-51
ICF-101
S8v2
Base
Novel
HM
Base
Novel
HM
Buse
Novel
HM
Base
Novel
HM
Vanilla CLIP Radford et al.
2021
VIT-16
62.3
53.4
57.5
53.3
46.8
49.8
78.5
63.6
70.3
4.9
5.3
5.1
ALIionCIIP Wang et al. 2021
VIT-16
61.0
46.2
52.6
69.1
37.3
48.5
90.1
58.1
70.7
13.3
10.1
11.5
XCLIP [Ni ot al. [2022]
VIT-16
74.1
56.4
64.0
69.4
45.5
55.0
89.9
58.9
71.2
8.5
6.6
7.4
A5 [Ju et al.
2022
VIT 16
69.7
37.6
48.8
46.2
16.0
23.8
90.5
40.4
55.8
8.3
5.3
6.4
CLFF (Raslleed et al. 2023
VIT-16
76.4
61.1
67.9
73.8
53.3
61.9
92.9
67.7
78.3
16.2
12.1
13.9
(2L(VIT-16)
VIT-16
73.1
60.6
06.3
77.0
58.2
66.3
94.4
77.9
85.4
16.6
13.3
14.8
T2I
VFT-32
67.7
55.7
60.9
74.7
46.1
57.0
94.0
76.5
84.3
15.1
10.0
12.0
(VTTV1))
VFT-14
77.1
64.0
69.8
81.2
60.5
69.3
95.0
83.0
88.5
18.3
13.1
15.2
We compare T2T, with both uni-modal models, and models adapting image-based multi-modal Vision-Tanguage (VL) models (Table D.. T2L's distinguishing feature is its consistent performance, even with training only 8 frames per video. This elficiency is due to its ability to learn appearance and motion harmoniously. This reduces computational demands and makes our model lightweight for streamlined training and deployment. As shown in Table T2L outperforms existing models (with similar backbone ViT-16), achicving gains of +1.6% and +2.3% in HMDB-51 and UCF-101 respectively, whil showing competitive performance on Kinetics-600.
Basc-to-Novel: To assoss the gencralization capabilitics of T2L towards novel classes, we evaluate it in base-to- novel setting Rasheed et al. (2023). We begin with dataset Ds with labels Ys {u / which is partitioned into two categories: the base classes YB and the novel classes YN- This partition ensures that YB UYN = Ys and YB n YN = 0. The model is trained on the base classes using 16 samples per base class. The evaluation is shown in Table 2 for four diverse datasets: K-400, HMDB-51, UCF-101, and SSv2. We observe that T2T. outperforms existing models on HMDB-51, UCF-101, and SSv2 under all evaluations with competitive performance on Kinetics-400.
In the few-shot learning setting, we examine the model's capacity to learn under limited supervision. Given dataset Ds with labels Ys {ur ) we create a general K-shot datasot where K samples are randomly selected from cach category y: E Ys for training. We experiment with different values of K, specifically K = 2, 4, 8, and 16 and use the same samples as Rasheed el al. (0023). Evaluation is performed on the validation set of Ds. Table 3 shows the performance of T2L, alongside methods that adapt CLIP for vidco tasks. It's worth noting that while T2L doesn't exhibit significant performance improvements due to its relatively low number of tunable parameters, it consistently maintains good performance across different shot settings outperforming existing methods in most settings.
Generalization to few-Shot learning:
8

---

Published in Transactions on Machine Learning Research (04/2025)
Table 3: Generalization to few-shot learning: Comparison of T2L with existing approaches on HMDB-51,
UCF-101, and SSv2 Datasets across different few-shot scenarios (K 2, 4, 6, 8 shots). Performance is
evaluated using top-1 accuracy. Despite fewer tunable parameters, T2L exhibits robust generalization abilities, with improved performance across most evaluations.
Method
Backbone
HMDB-51
UCF-101
SSv2
K=2
K=4
K=8
1=
K=2
K=4
K=8
K=16
K=2
Km44
K K=8
K K=
16
Vanilla CLIP Raalordd el al.
2021
VGT-16
41.9
41.9
41.9
41.9
63.6
63.6
63.6
63.6
2.7
2.7
2.7
2.7
CLIP Wann el al. 22021
ViT 16
47.5
57.9
57.3
59.1
70.6
71.5
73.0
91.4
4.1
5.8
8.4
11.1
XCLIP [N] et al. 2022
VIT-16
53.0
57.3
62.8
62.4
71.4
79.9
83.7
91.4
3.9
4.5
6,8
10.0
A3 [Ju al. 2022
VFT-16
39.7
30.7
57.0
62.4
71.4
79.9
85.7
89.9
4.4
9.7
ViFi [Rasheed
2023
VGI-16
37.2
62.7
64.5
06.8
80.7
90.00
92.7
6.2
7.4
12.4
(VL((17-11))
VFT-16
57.3
61.1
65.4
67.7
84.7
88.3
91.3
92.8
6.8
8.7
9.6
13.2
TVL
VIT 32
32.4
56.9
60.4
63.7
79.7
83.7
87.5
89.8
5.8
6.7
8.0
12.1
(VI(VIT-
VII-14
6244
65.3
68.4
70.7
85.2
89.7
92.1
93.3
8.9
10.1
11.1
15.2
Table 4: Generalization to ont-of-distribution datasets:Porformant comparison of our model trained on the K-400 dataset and tested on extreme out-of-distribution datasets, including UCF-DS, UCF-101-P, HMDB-51-P, and UCF-101-O.
Method
Backbone
HMDB-51-P
UCF-101-P
UCF-101-DS
UCF-101-O
ActionCI.IP (Wag al. 2021
VFT-16
34.0
46.0
62.8
21.3
ViFi CLIP (Rasheed cl al. 2023
VIT-16
33.1
50.6
63.1
24.8
T2L(VTT-16)
VII-I6
35.1
52.7
63.7
25.1
(VL(VTT-14)
V4T-14
36.6
55.7
71.9
27.7
Robustness to distribution shift: We evaluat the zero-shot performance of our model trained on the K-400 dataset, on extreme out-of-distribution datasets including UCF-DS Schiappa al. (2023), UCF-100-P, HMDB-51-/Schiappaa et al. (2023), and UCF-101-CModi et al. (2024). The results are summarized in Table
where observed comparatively better performance of the model, indicating its degree of robustness.
4.2 Ablations
In this section, we delve into the impact of Temporal Token Learning, temporal feature diversity loss, and their combined effect on our model's performance.
We use a base model with just spatial and text adapters as a baseline without Temporal Token Learning and temporal feature diversity loss. We perform this ablation on Kinetics-400, HMDB-51, UCF-101, and datascts.
Somnthing-ssmotthnng Impact of Temporal Token Learning: We study the impact of Temporal Token Learning (TTL) by incorporating it into the baseline model, which is trained with spatial adapters at the image level. As shown in Tables 7 and 10 TTL consistently improves accuracy across all datasets. The improvements are particularly notable on the SSv2 dataset compared to HMDB-51 and UCF-101, highlighting TTL's 's superior temporal modeling capability, especially for datasets with complex temporal dynamics. This demonstrates TTL's effectiveness in enhancing temporal representation and overall model performance.
Impact of Temporal Feature Diversity Loss: Adding the proposed TFD as constraint during base model training consistently enhances performance, albeit to lesser extent than observed with Temporal Token Learning in Tables 7 and S This improvement is attributed to the model's limited ability to learn temporal aspects, given the absence of temporal tokens.
Impact of Temporal Feature Diversity Loss (TFD) on Video Frame Embeddings: TFD plays a crucial role in enforcing diversity between frame embeddings within video. Visualization of video frame embeddings using Uniform Manifold Approximation and Projection (UMAP) Mclnnes et al. (0188 plot provides insights into the impact of TFD, as shown in Figure 3a The UMAP plot, generated with 512-dimensional features reduced to 2 dimensions using paramcters such as n_ neighbors=15,, mán_ dist=6, m and
highlights these effects. Without TFD, frame embeddings are densely clustered, suggesting the model predominantly focuses on appearance while neglecting motion. In contrast, with TFD, frame emboddings exhibit greater variance, reflecting the model's ability to discern motion by treating frames uniquely. These visualizations emphasize TFD's effectiveness in disentangling temporal features and improving motion understanding.
9

---

Published in Transactions on Machine Learning Research (04/2025)
23
13
(a) UMAP plots for frame embeddings without (left) and with()ight)) TFD Loss
/1
(b) Capturing motion dynamics within the video
Figure 3: Subfigure 3a UMAP visualizations of frame embeddings, showcasing the effect of Temporal Feature Diversity (TFD). In the left plot, embeddings without TFD exhibit dense clustering, indicating limited feature scparation. In contrast, the right plot domonstrates embeddings with TFD, revealing enhancod variance and better-defined clusters. Each cluster represents a single video, with individual dots inside the clusters corresponding to frames, highlighting how TFD disentangles temporal features effectively
Subfigure 3b Shows attention maps demonstrating the impact of Temporal Token Learning (T2L) and Temporal Feature Diversity (TFD) loss SSv2 and UCF-101 validation sets. T2L effectively focuses moving parts and objects, encoding video-specific information, as shown in the first example (Pulling something next to something), where T2L highlights interactions and motion. The first row shows input frames, the second row shows base model ViT-16 (with spatial adapters) results, and the third row shows TTL with TDF results. The attention heatmaps further capture motion dynamics by emphasizing subtle inter-frame changes, showcasing the model's ability extract mcaningful motion information.
Additional Ablation Experiments: We performed extensive ablation experiments to evaluate the effectiveness various components of our approach. Specifically, we analyzed the performance of TTL and TFD in transitioning from base classes novel classes in Table g For the few-shot sctup, we examined the effectiveness of the LLM-generated descriptions UCF-101, HMDB-51, and SSv2 in Table g and evaluated
the effectiveness of TTL and TFD loss in Table 10
Fully-Supervieed Comparison: Table 5 provides a comparison of fully supervised results on the HMDB-51 and UCF-101 datasels. This comparison assesses the performance of T2L alongside uni-modal approaches for action recognition and image-based multi-modal Vision- Langugge (VL) models adapted for video action recognition. The evaluation metric usod top-1 accuracy. In this table, we observe the performance of T2L, a novel approach, compared the baseline method, ActionCLIP Wang al. (2021). on the HMDB-51 and UCF- 101 datasets. Both methods utilize the same backbone architecture, VTT-16, and achieve competitive results.
10

---

Published in Transactions on Machine Learning Research (04/2025)
Table 5: Pully-superoised comparison: Fully supervised results on HMDB-51 and UCF-101 datasets for T2L compared uni-modal approaches for action recognition and image-based multi-modal Vision-Language (VL) models adapted for video action recognition. Performance is measured using top-1 accuracy. Red cells indicate fully supervised evaluation trained respective datascts HMDB-51 and UCF-101, while blue cells indicate zero-shot evaluation. In the third row, zero-shot evaluation occurs when the model is trained on HMDB-51 for HMDB-P and on UCF-101 for zero-shot evaluation for UCF-101-P, UCF-101-O, and UCF-101-DS. f: Model trained on HMDB-51/UCF-101, $: zero-shot evaluation, model trained on Kinetics-400.
Method ALIP Wang ot al. 2021
Backbone
HMDB-51
UCF-101
HMDB-51-P
UCF-101-P
UCF-101-0
UCF-101-DS
WIT-16
76.2
97.1
ActionCLIP (aang sl. 2021
VIT-16
40.8
63.2
T2L
VIT-16
80.9
95.1
55.6
63.4
30.1
50.6
T2L1
VIT-16
52.9
79.1
35.1
52.7
25.1
63.7
Table Zero-shot ablations on LLM description: Impact of LLM-generated action class descriptions as prompts on Zero-shot and Base-to-Novel settings.
LLM-D
HMDB-61
UCF-101
K-600
K-400
HMDB--1
UCF-101
Zero-shot
Base
Novel
HM
Base
Novel
HM
Base
Novel
IIM
52.0
77.8
67.2
72.8
59.1
65.2
75.2
53.0
63.55
94.1
75.5
83.7
52.9
79.1
70.1
73.1
60.6
66.3
77.0
58.2
66.3
94.4
77.9
85.4
Table 7: Zero-shot ablation on TTL and TFD: We evaluate the effectiveness of Temporal Token Learning (TTL) and temporal feature diversity loss(TFD) while using Zero shot setup train on Kinetics-400 tested on UCF-101, HMDB-51 and Kinetics-600.
TFD
HMDB-51
UCF-101
K-600
50.0
75.9
65.5
X
51.2
78.1
07.5
layer only
52.9
79.1
70.1
Table 8: Base to Novel ablation on TTL and TFD: We evaluate the effectiveness of Temporal Token TTLL) and temporal feature diversity loss (TFD) using the base-to-novel setup on Kinetics-400, UCF-101, HMDB-51, and SSv2, using all samples from the base classes.
Method
Kinetics-400
HMDB-51
UCF-101
SSv2 Novel
TTL
TFD
Base
Novel
HM
Base
Novel
HM
Base
Novel
HM
Base
HM
71.3
92.1
72.8
81.3
8.1
9.5
69.0
56.1
61.8
72.6
53.3
61.4
92.5
13.4
81.8
13.2
10.3
11.5
69.7
58.0
63.3
74.0
55.2
63.2
93.7
75.4
83.5
15.3
11.5
13.1
73.1
60.6
66.3
77.0
58.2
66.3
94.4
77.9
85.4
16.6
13.3
14.8
Table 9: Few shot Ablation LLM: Impact of LLM LLM- generated action class descriptions as prompts on
Few-shot setting.
LLM-decription
HMDB-51
UCF-101
S8v2
K-2
K
8-8
K-16 16
K-2
K
K-8
K-16
K-2
K-4
K-8
K-16
16
X
57.0
59.1
65.1
65.2
81.8
86.6
88.8
90.2
6.1
8.0
9.3
11.9
57.3
65.4
84.7
Specifically, T2L achieves a top-1 accuracy of 80.9% on HMDB-51 and 95.1% on UCF-101, outperforming ActionCLIP's accuracy of 76.2% on HMDB-51.
Finally, when we train a model jointly with both Temporal Token Learning and temporal feature diversity loss, we observe significant improvement across all datasets; the improvement significantly better case of SSv2 dataset as compared with HMDB-51 and UCF-101. This further strengthens our claim that Temporal Token Learning and temporal feature diversity loss can help in better temporal modeling in videos.
Joint training:
4.3 Discussion and analysis
UCF-101 and SSv2 as extremes: UCF-101 and SSv2 represent the opposite ends of the spectrum in action recognition datasets. UCF-101 achieves high performance without. explicit motion learning, leveraging
11

---

Published in Transactions on Machine Learning Research (04/2025)
Table 10: Few shot ablation on TTL and TFD: We evaluate the effectiveness of Adaptive Temporal Token Tuning (TTL) and temporal feature diversity loss(TFD) while using Few-shot setup on UCF-101, HMDB-51 and SSv2.
Method
HMDB--51
UCF-10I
SSv2
TTL
TFD
K=2
K=4
K=8
K=16
K=2
K=4
K=8
K=16
K=2
4
K 8
K 16
x
53.7
57.9
61.7
63.6
81.2
85.5
86.6
89.8
5.7
6.7
7.9
10.8
x
56.0
58.5
64.7
65.7
82.1
84.6
88.5
90.3
5.9
7.7
8.7
12.0
x
56.7
60.3
63.5
G6.7
83.3
86.
89.9
91.6
6.6
8.1
9.0
12.7
65.4
88.3
91.3
92.8
8.7
13.2
200
150
863
200
400
522
Action CLIP
150
131
150
131 124
300
*X CLIP
-VIFI CLIP
287
100
100
200
T2L
58
5-2
Tunable Params(M)]
GFLOPs
Total Parame(Mi)
Troughpull image per sec)1
Figure 4: Comparison of T2L with ActionCLIP Wang et al. (2021), XCLIP Ni et al. 2002), and ViFi CLIP Rasheed et al. (2023). Throughput per view (TP) is measured on single A100 GPU. T2L shows superior efficiency in GFLOPs, throughput, and parameter count, highlighting its computational advantages.
pre-trained CLIP weights that emphasize appearance. However, when Temporal Token Learning and temporal feature diversity loss are introduced, the attention shifts towards motion, as shown in the right part of Figure 3b The baseline CLIP with adapters focuses on appearance and background, while T2L tracks the motion region. In contrast, SSv2, with its high reliance on motion cues, benefits significantly in our 'base to novel' experiments. Attention visualizations in Figure 3b highlight CLIP's emphasis on background, while T2L learns complex relative motion features.
Effect of action class description: Table 6 and Table 9 provides a detailed comparison of the model's performance with and without L.L.-generated descriptions on two datasets: UCF-101 and HMDB-51. Our findings reveal that incorporating LLM-gencrated action class descriptions lcads to improved text embodding generalization and enhanced model accuracy.
We analyze the computational complexity (see Figure 4 and observe that existing approaches show lower throughput due to added vidco-specifie Icarnable components. In contrast, T2L eflectively addresses this challenge by maintaining the base model parameters frozen, including 86.0M parameters for the CLIP ViT-B/1K 16 Image Encoder and 63.0M parameters for the CLIP Text Encoder, resulting in a total of 149.0M frozen parameters. T2L adds only 5.2M trainable parameters, consisting of 0.07M for Temporal Tokens, 1.5M for Text Adapters, and 3.5M for Spatial Visual Adapters. This efficient design, with just 3.5% additional parameters over the base model, achieves higher throughput. efficiency while maintaining comparable FLOPs to previous approaches, showcasing its ability to perform effective temporal modeling with minimal computational overhead.
Efficiency Analysis:
5 Conclusion
In this work, we propose T2L, an efficient model for zero-shot video action recognition. T2L can effectively adapt image-based visual-language models to learn temporal aspect in videos while preserving its generalization capability. It is based on Temporal Token Learning which requires very few learnable parameters making the adaptation efficient. We also propose a novel temporal feature diversity loss which enhances the temporal modeling capability of T2T.. With limited number of learnable parameters, T2L can be trained on a single GPU with siguificantly reduced computational resources, yet consistently achieving superior performance across different evaluations.
Limitations: In this work, we focused mainly on action recognition. However, learning the temporal aspects in videos is crucial for all video-based tasks. Exploring the benefits of the proposed approach for other video tasks, such as video summarizing, object tracking, and video segnentation, would be interesting direction for future research.
12

---

Published in Transactions on Machine Learning Research (04/2025)
References
Biagio Brattoli, Joseph Tighe, Fedor Zhdanov, Pietro Perona, and Krzysztof Chalupka. Rethinking zero-shot video classification: End-to-end training for realistic applications. In Proceedings of the IEEE/ /VF Conference on Computer Vision and Pottern Recognition, pp. 4613-4623, 2020.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299 6308, 2017.
Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600. arXiv preprint 1r0i 01340, 2018.
Shizhe Chen and Dong Huang. Elaborative rehearsal for zero-shot action recognition. In Proceedings of the 2021.
IEEE/CVE /CFF International Conference on Computer Vision pp.
Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European conference on computer vision, pp. 104-120. Springer, 2020.
Jinwoo Choi, Chen Gao, Joseph CE Messou, and Jin-Bin Huang. Why can't dance in the mall? learning mitigate scene bias in action recognition. Advances in Neural Information Processing Systems, 32, 2019.
R Christoph and Feichtenhofer Axel Pinz. Spatiotemporal residual networks for video action recognition. Advances in neural informalion processing systems, 2, 2016.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CV international conference on computer vision, pp. 6202-6211, 2019.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The' something something' video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pp. 1842-5850 2017.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Ducrig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pp. 4904-4916. PMLR, 2021.
Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. Prompting visual-language models for efficient video understanding. In European Conference on Computer Vision, pp. 105 124. Springer, 2022.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Modtr-modulatedd detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF International Conference on Compuler Vision, pp. 1780 1790, 2021.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloc Hillier, Sudheendra Vijayanarasimman, Fabio Viola, Tim Green, Trevor Back, Paul Natsey, et. al. The kinetics human action video dataset. arXiv preprint (tfe::1705.6690 2017.
Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: a large vidco database for human motion recognition. In 2011 International conference on computer vision, pp. 2556 2563. IEEE, 2011.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsich, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arxi 1908. 100.009557, 2019.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pro-training for vision-language tasks. In Computer Vision-ECOS 2020: 16th European Conference, Glasgow, UK August 23-28, 2020, Proceedings, Part XXX 16, pp. 121-137. Springer, 2020.
13

---

Published in Transactions on Machine Learning Research (04/2025)
Ji Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding In Proceedings of the IEEE/CVE International Conference on Computer Vision (TCCV), October 2019.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.034 2018.
Rajat Modi, Vibhav Vineet, and Yogesh Rawat. On occlusions in video action detection: Benchmark datasets and training recipes. Advances in Neural Information Processing Systems, 36, 2024.
Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, and Haibin Ling. Expanding languago-image protrained models for general vidco recognition. In European Conference on Compuler Vision, pp. 1-18. Springer, 2022.
Florent Perronnin, Jorge Sánchez, and Thomas Mensink. Improving the fisher kernel for large-scale image classification. In Computer Vision-ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, Seplember 5-11, 2010, Proceedings, Part IV 11, pp. 143 156. Springer, 2010.
Jie Qin, Li Liu, Ling Shao, Fumin Shen, Bingbing Ni, Jiaxin Chen, and Yunhong Wang. Zero-shot action recognition with error-correcting output codes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2833 2842, 2017.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Fine-tuned clip models are elficient. video learners. In Proceedings of the TEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6545 6554, 2023.
Madeline C Schiappa, Naman Biyani, Prudvi Kamtam, Shruti Vyas, Hamid Palangi, Vibhav Vincet, and Yogesh Rawat. Large-scale robustness analysis of video action recognition models. In The IEEE/CVE Conference on Computer Vision and Pottern Recognition, 2023.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A datasot of 101 human actions classes from videos in the wild. arXiv preprint arXiw:1212.0400 2012.
Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri, A closer look spatiotemporal convolutions for action recognition. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pp. 6450 6459, 2018.
Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In European conference on compuler vision, pp. 20-36. 36. Springer, 2016.
Mengmeng Wang, Jiazheng Xing, and Yong Liu. Actionclip: A new paradigm for vidco action recognition. arXiv preprinl arXiv:2109.08. 08472, 2021.
Qian Wang and Ke Chen. Alternative semantic representations for zero-shot human action recognition. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2017, Skopje, Macedonia, September 18 22, 2017, Proceedings, Purt I 10, pp. 87 102. Springer, 2017a.
Qian Wang and Ke Chen. Zero-shot visual recognition via bidirectional latent embedding. International Journal of Computer Vision, 124:356 383, 2017b.
Yunbo Wang. Mingsheng Long, Jianmin Wang, and Philip Yu. Spatiotemporal pyramid network for vidco action recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 1529 1539, 2017.
14

---

Published in Transactions on Machine Learning Research (04/2025)
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEF conference on compuler vision and pollern recognilion, pp. 1492 1500, 2017.
Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Vidcoclip: Contrastive pre-training for zcro-shot video-text understanding. In Proceedings of the 2021 Conference on Empirical Melhods in Natural Language Processing, pp. 6787- 6800, 2021.
Taojiannan Yang, Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, and Mu Li. Aim: Adapting image models for efficient video action recognition. arXic preprint X020:2302.00324, 2023.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficien tne-tuning for transformer-based masked languago-models. arXiv preprint SSS::2106.11999 2021.
Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning, arXiv preprint arxiv:2308. 03303, 2023.
Yi Zhu, Yang Long, Yu Guan, Shawn Newsam, and Ling Shao. Towards universal representation for unscen action recognition. In Proceedings of the IEEE conference on compuler vision and pallern recognition, pp.
2018.
Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli. Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference natural language processing (volume 1: long papers). In Proceediiggs of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2021.
15

---

Published in Transactions on Machine Learning Research (04/2025)
A
Appendix
In this appendix section we present materials that enhances the understanding of our main paper through additional details and in-depth qualitative analysis. This appendix is structured as follows:
1. Loss gradient flow for Temporal Token
Class Description Table: A comprehensive table detailing the descriptions various action classes, enhancing the clarity of our dataset.
3. Dataset Details: Detailed information about the datasets used in our study, shedding light on their characteristics and muances.
4. Evaluation Protocol: A thorough explanation of our evaluation protocols, offering insights into our methodology and ensuring transparency in our assessments.
Out-of-Distribution Results: Further analysis of out-of-distribution results, demonstrating the robustness of our model unconventional scenarios.
A.1 Loss gradient flow for Temporal Token
To derive the gradient of the loss function Lrotat with respect pTemp (pr R [ --= 0, ...., . L 1} each layer of the transformer, we need to follow these steps:
Let's start with the key equations:
N+1
D (0)
Pi
1 N
(12)
P MHA(LN(())).
(13)
(8)
+ MHA(LN((Z( (4) (()
(14)
Gradient of the loss function Liotal with respect to temporal tokens pTemp = {pr E RRTD | 11 0,..., L 1} is given
Orotal @pTemp
L-1 T
ihiotal
(15)
apl")
1=0
from Eq. 12 13 and 14 we chain these gradients together using the chain rule to compute the gradient of the loss function Leotal with respect to p
Ohroal
Ghroal
Op
(t)
ap
(1)
api
(16)
Op:
(t)
Op:"
(t)
ap, (t)
op (t)
0pi
from Eq. 12 we can write
ap 1
(17)
0p
from Eq.
13 we write
0pl (0)
aMHA(LN
(18)
07
0p (0)
from Eq. 14 we can write
Op
OMHA(LN (1)
(2 ((D
(19)
0p(()
0p(()
16

---

Published in Transactions on Machine Learning Research (04/2025)
then,
itotal
errotal 0p(()
(1+ OMHA(IN( (9)
()
aMHA(LN((.))
(20)
0p
0pl")
0pl
Thus, the combined expression for the total gradient is:
itootal
L-1
iLioial (1 I+ Upl
(MHA(LN(( ( D))
OMHA(IN(())
(21)
@pTemp
0pl (1)
Op( (4)
1 1=1
A.2 Class Description Table
In our pursuit of enhanced generalization, we utilized GPT-3.5 to generate action class names, aiming for a broader and more adaptable understanding of actions. To provide a comprehensive overview of our approach, we present a curated selection of UCF class names and their corresponding descriptions in Table 11 This table encapsulates the amalgamation of linguistic and visual comprehension, illustrating how LLM-enhanced model interprets and defines various actions within the UCF-101 dataset.
Table 11: Description of selected UCF action classes generated by GPT-3.5.
Action Class name ApplyEyeMakeup
Example of GPT-3.5 Description
ApplyEyeMakeup is an action performed by humans to enhance their eyes and create a more dramatic look. It involves using a variety of makeup products such as eyeshadows, eyeliners, and mascaras, as well as blending and contouring techniques to create a desired effect.
Basketball
Basketball is a sport played by two teams of five players on a rectangular court. The objective is to shoot ball through hoop 18 inches in diameter and 10 feet high mounted to backboard at each end. The game is played by bouncing the ball on hard court surface.
Pizza'l PizzaTossiig
Pizza tossing is an art form of spimning and stretching dough to create a thin and crispy pizza. crust. It involves the use of hands, wrists, and arms to shape the dough into round shape before it is placed in the oven. The goal is to achieve a consistent thickness and texture.
Typing
Typing is an action performed by humans using a keyboard to input text into computer or other device. It is a skill that requires practice and repetition to become proficient. Typing can be used to create documents, send emails, type code, or enter data into databases.
A.3 Computational Efficiency and Performance Comparision
provide a comprehensive evaluation of the efficiency and effectiveness of different design choices, we present comparative analysis of FLOPs, throughput, trainable paramcters, and accuracy in the tables below. This analysis compares three configurations: (1) Temporal Adapters, (2) Cross-Attention Fusion, and (3) Temporal
Tokens (TTL).
The results in Tables 12 and 13 provide key insights into the trade-offs between computational efficiency and
model performance for different temporal modeling approaches:
Computational Efficiency: The T2L with Temporal Tokens (TTL) method achieves the lowest FLOPs (101.91 GFLOPs) while maintaining the highest throughput (322.0). Temporal Adapters introduce higher computational cost. (107.49 GFLOPs) with lower throughput (264.4).
Trainable Parameters: TTL requires significantly fewer trainable parameters (0.074M) compared to Temporal Adapters (3.5M) and Cross- Attention Fusion (18.9M), demonstrating its efficiency.
17

---

Published in Transactions on Machine Learning Research (04/2025)
Table 12: Comparison of FLOPs, throughput, and trainable parameters for different temporal modeling
approaches. FLOPs are reported in GFLOPs.
Method
Image Model
Text Model
Fusion Model
Total
Throughput
Trainable Params (M)
FLOPs (GFLOPs)
T2L with Temporal Adaateess
101.31
6.17
N/A
107.49
26144
3.5
T2L with Proees-Atenttion Pusion
95.74
6.17
0.10
102.01
290.6
18.9
Temporal Tokens (TTL)
95.74
6.17
N/A
101.91
322.0
0.074
Table 13: Comparison of accuracy across different temporal modeling approaches.
Method
Accuracy (%)
Novel Accuracy (%)
Harmonic Mean (%)
T2L with Temporal Adapters
92.1
74.4
82.3
T2L with Cross-Attention Fusion
93.8
76.3
84.1
T2L with Temporal Tokens (TTL)
94.4
77.9
85.4
Performance Comparison: While the primary focus of our paper is not detailed baso-to-novel evaluation here, this comparison is provided to illustrate the effectiveness of different temporal modeling choices. T2L with Temporal Tokens (TTL) consistently achieves the highest accuracy among the three approaches, with 94.4% overall accuracy.
Effectiveness of Temporal Learning: Compared to Temporal Adapters and Cross-Attentian Fusion, TTL provides the best balance of efficiency and accuracy. The significant reduction in trainable parameters and FLOPs, without sacrificing performance, demonstrates the effectiveness of the proposed method.
These findings validate that T2L with Temporal Tokens (TTL) offers the most efficient and effective approach among the compared temporal modeling methods.
A.4 Impact of Discarding Temporal Tokens
To evaluate the necessity of retaining temporal tokens, compare model performance when these tokens are kept vs. discarded.
Motivation: Temporal tokens encode global temporal relationships across frames using Multi-Head Atten- tion (MHA) without adding significant, parameter overhead. However, once they transfer temporal information to frame embeddings, retaining them may introduce redundancy.
Efficiency Considcration: Discarding temporal tokens optimizes computation while preserving accuracy, allowing the model to focus refined frame-level embeddings.
Table 14: Comparison of retaining vs. discarding temporal tokens.
Method
Accuracy (%)
Novel Accuracy (%)
Harmonic Mean (%)
T2L (Discard Temporal Tokens)
94.4
77.9
85.4
T2L. (Retain Temporal Tokens)
94.2
77.5
85.0
Table 14 shows that retaining temporal tokens provides **no significant * The harmonic mean remains nearly unchanged, validating our choice to discard them after their role in encoding temporal dependencies is fulfilled.
Findings:
18

---

Published in Transactions on Machine Learning Research (04/2025)
These results reinforce our design decision, demonstrating that temporal tokens efficiently propagate informa- tion before being removed. Future work could explore retaining CLS-style tokens for long-range temporal aggregation.
A.5 Dataset Details
Our analysis is conducted on five well-established action recognition datasets: Kinetics-400 and Kinetics- 600 Carreira et al. (2018): The Kinelics-400 dalasel comprises 400 human action classes, represented by video clips sourced from various YouTube videos, each lasting around 10 seconds. includes approximately 240,000 training videos and 20,000 validation vidcos. Kinctics-600 extends Kinctics-400, covering 600 action calegories, with about 410,000 training video clips and 29,000 validation video clips.
HMDB-51 (Kuehne et al., 2011): The HMDB-51 dataset contains 71,000 realistic videos collected from various sources, spanning 51 action categorics. The standard split includes 3,570 training samples and 1,530 validation samples. These sets are further divided into three splits, each containing training clips and validation clips for each action category.
UCF-101 (Soomro et al., 2012): UCF-101 comprises 13,000 realistic videos sourced from YouTube, encompassing 101 action categories, including five action types: human-object interaction, body-motion, human-human interaction, playing instrumental music, and sports. The standard split involves training on 9,537 videos and evaluation on 3,783 videos, distributed across three splits.
Something Something V2 (SSv2) (Goyal et al., 2017): The SSv2 dataset is an extensive collection of video clips depicting humans performing actions with everyday objects, spanning 174 action categorics. This dataset focuses on recognizing fine-grained actions, such as covering something with something or uncovering something, making it more temporally biased compared to other datasets. The standard split consists of 168,913 training videos and 24,777 validation videos. Our reported performance metric is top-1 accuracy over the validation split.
Out-of-Distribution Datasets: Out-of-distribution datasets, such as UCF-DS Schiappa et al. (2023) UCF-101-P, HMDB-51-P Schiappa et al. (2023), and UCF-101-O Modi et al. (024)), play a crucial role in evaluating the robustness of action recognition models. These datasets focus on understanding how distribution shifts caused by real-world perturbations affect model performance, rather than solely considering adversarial manipulations. To achieve this goal, these datascts introduce four benchmark datasets, including HMDB51-P and UCF101-P, which encompass diverse range of 90 perturbations. Additionally, UCF101-DS provides a platform for validating these findings by offering a dataset with realistic distribution shifts. Furthermore, UCF-101-O consists benchmark datasets specifically designed to incorporate synthetically controlled static/ dynamic occlusions. Collectively, these initiatives contribute to enhancing our understanding of model performance in challenging real-world scenarios.
A.6 Evaluation Protocols
In our analysis, we explore four distinct experimental scenarios: zero-shot, base-to-novel generalization, few-shot, and fully-supervised settings. Across these scenarios, employ sparse sampling approach Wang el al. (2016), capturing 8 frames consistently. Each sampled frame is resized, with the shorter side set to 256 pixels, and centered to create a 224-pixel square crop.
A.6.1 Zero-Shot Setting:
In the zero-shot setting, models trained on the Kinetics-400 dataset undergo evaluation on three different. cross-datasets: HMDB-51, UCF-101, and Kinetics-600, For HMDB-51 and UCF-101, the methods are assessed across their respective three validation splits, and the top-1 average accuracy is reported. Regarding Kinetics-600, we assess the performance on 220 categories that do not overlap with Kinetics-400, reporting 1 accuracy. In this setting, single-view inference using frames is applied.
19

---

Published in Transactions on Machine Learning Research (04/2025)
099
Figure T2L's attention map visualizations extreme out-of-distribation examples from the UCF-DS dataset Schiappa et al. )023). The model demonstrates remarkable generalizability unconventional actions like "Underwater Haircut" (top left), "Fencing in Crowd" right), "Hammering by Unimal"(bottnn left., and "Underwater Archery (bottom right).
A.6.2 Base-to-Nowe Setting:
For a comprehensive assessment of the generalization capabilities of various approaches, we adopt the base-to- novel generalization setting Rashoed et al. (2023) for vidco action recognition tasks. Here, model is initially trained on a set of base (seen) classes in a few-shot. manner and subsequently evaluated on a set of novel (unseen) classes. We conduct a thorough generalization analysis across four datasets: Kinetics-400, HMDB-51, UCF-101, and SSv2. The datasot employs three training splits, classifying the total categorics into two equal halves. The most frequently occurring classes constitute the base classes, while the rarely occurring categories are designated the novel classes. This setup employs frames and follows single-view inference.
A.6.3 Few-Shot Setting:
The few-shot setting involves creating a general K-shot split, with K samples used accordance with splits from Rasheed al. (2023). Specifically, we experiment with 2, 4, 8, and 16 shots three datasets: HMDB-51, UCF-101, and SSv2. The models are assessed on the first. validation split for HMDB-51 and UCF-101, and the full validation split, even cemporally-challenging datasets like SSv2.
A.7 Out-of-Distribution Results
In our exploration of extreme sitooof-dssributtin examples, turned our attention to the UCF-DS dataset Schiappa et al. (2023). collection of videos that push the boundaries of conventional contexts. These unconventional scenarios serve as litmus tests for a model's adaptability and robustncss. In the figure peculiar world of "Underwater Haircut", T2L's allention hones in on the person's 's head region, showcasing its ability to interpret unique actions. Venturing further, the model navigates the complex dynamics of "Fencing in Crowd", deftly capturing the intricate hand motions amidst the chaos. Delving into more extraordinary territories, "Hammering by Animal" presents scenario where an animal performs an unexpected action. T2L astutely focuses the animal's hand motion, demonstrating its capacity to understand unconventional
20

---

Published in Transactions on Machine Learning Research (04/2025)
movements Finally, in the realm of "Underwater Archery" the model's attention locks onto the action area with precision, highlighting its exceptional adaptability. These insightful visualizations underscore T2T's extraordinary ability to navigate the unknown, positioning it not just as a tool for everyday scenarios but as a robust solution even in the face of the extraordinary.
21


Reviews:
[{"review_id": "tj3fwH1Zsu", "reviewer": "Reviewer_ziAr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* This paper tackles an interesting (though not a novel) problem of adapting CLIP-style models for video action recognition tasks: how to continuously adapt CLIP models for video tasks efficiently. \n* The paper is well written and the datasets, fine-tuning, and other details are clearly explained.\n\nWeakness:\n- Novelty: The proposed method is an adaption of existing methods that adds modality/domain specific learnable tokens (aka registers) to learn modality/domain specific details (e.g., Llava, Qwen-VL, and many more). While the paper provides interesting (but limited) ablations, the novelty is limited from a technical perspective.\n* Method design choices are not evaluated thoroughly. A comprehensive evaluation of each design decision would help in better understanding\n    * In 3.2.1 (Stage 3), it is mentioned that MHA block is duplicated. However, it is not clear why this duplication is necessary? Are there other alternatives to duplication (e.g., cross-attention instead of self-attention OR light-weight adaptors)?\n    * Section 3.2.1 introduces temporal tokens, which are subsequently discarded in Section 3.2.2. The reasoning behind discarding these tokens is unclear. Typically, special tokens like these are added to capture global information from other tokens, as seen in ViT's use of CLS tokens for image-level embeddings. Discarding the temporal token implies that the global information it encodes is not useful. It would be beneficial to explain why other tokens are not discarded and to provide thorough ablations to support these design choices\n    - The same template per capability is used for generating a text prompt. Would using diverse text prompts be beneficial instead of the same template per capability?\n\nSuggestions:\n- The citation format in many places is wrong. Instead of “Authors (Year)”, it should be “(Authors; year)”. It would be great if authors can thoroughly check it.\n- Section 3.2 “Since these encoders are pre-trained on image-text pair, they do not have any understanding for motion aspect in videos”. Maybe change “motion aspect” to “temporal aspects” for clarity.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the efficient adaption of contrastive image-text models to videos. Specifically, the paper introduces temporal token learning and temporal feature diversity to improve the accuracy on various standard video action recognition datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Thorough ablations for different design choices.\n* The citation formatting issues should be addressed as per the [TMLR guidelines](https://jmlr.org/tmlr/author-guide.html).\n\n\"\"\"When the authors or the publication are\nincluded in the sentence, the citation should not be in parenthesis, using \\verb|\\citet{}| (as\nin ``See \\citet{Hinton06} for more information.''). Otherwise, the citation\nshould be in parenthesis using \\verb|\\citep{}| (as in ``Deep learning shows promise to make progress\ntowards AI~\\citep{Bengio+chapter2007}.'').\"\"\"", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MvpgIH8KZQ", "reviewer": "Reviewer_kyPs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The new adaptation method is extremely efficient compared to previous approaches. The key idea is to learn a single aggregation token for each layer and share across input frames and freeze the visual features. This idea is really nice and effective. In practice, Figure 1 shows that T2L adapts a CLIP model using ~5M tunable parameters, a throughput of 322, and obtain a better performance than SOTA concurrent approach ViFi CLIP that adapts the same CLIP model using ~125M tunable parameters and a throughput of 71.\n\n- The temporal diversity loss is novel and a very nice idea. More generally I believe that adding constraints like this in the loss function gives more capacity to the network compared to hard coding constraints using architectural components.\n\n- The evaluation is very thorough, both in terms of data diversity with for example SSv2 which is motion-based as opposed to UCF101 which is more appearance-based, and in terms of tasks. The model adapts very well to new tasks, to other distributions or with very few samples. T2L demonstrates SOTA results on a large variety of these tasks compared to many other concurrent work.\n\n\n\nWeaknesses:\n\n- The main concurrent work of this paper is ViFi CLIP that is very similar but differs in the adaptation method. The performance of T2L is not consistently better than ViFi. For example ViFi demonstrates superior performance on Kinetics. The overall performance is on par or superior in average, but it is important to make sure that there is not a trade-off with this light-weight approach gaining compute efficiency while losing in performance. It would be interesting to have an analysis on where T2L fails compared to ViFi.\n\nThe main weakness of this paper is the presentation that needs some additional work, in particular putting attention to some details: \n\n- There is a section 3.2.1 but no section 3.2.2 ?\n- Figure 4 caption describes 3 rows per sample but only 2 are presented.\n- The figure titles are inconsistent, sometimes bold, sometimes italic. Please harmonize everything.\n- The main Figure (2) is too busy. The elements are not correctly aligned and some text take too much space making the Figure unclear. For example it is unclear what “Init token” “update token” refer to.\n- Table 10 is missing bold numbers.\n- There are many missing spaces, for example in abstract “Temporal Token Learning”(T2L)”. At beginning of page 6: “shown in equation 7(See section A.1”.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a new and very efficient adaptation method, called T2L, of image-pretrained models to video. In particular CLIP is used as the image backbone and is adapted using a temporal token learning approach and a temporal diversity loss, with very few parameters, on the Kinetics dataset. The obtained model is evaluated on a wide range of video benchmarks, from zero-shot action recognition to few-shot learning, base-to-novel classes recognition and out-of-distribution generalization.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Requested changes: Fix the presentation.\n\nQuestions:\n- What is the training fps ?\n- Do you have the results for fully supervised on Kinetics 400 and SSv2 ? How does it compare to ViFi ?\n- Have you considered the attentive probing evaluation of V-JEPA ?\n- “even with training on only 8 frames per video.” Don’t you think this tells more about the benchmarks than the model ? For example if the fps is 1 it is unclear whether the model truly understands motion or just high level temporal relations between actions.\n\nConclusion:\nThis paper presents really cool findings in video adaptation of image-based models, a model can be adapted and perform well on video tasks with extremely light-weight adaptation. If the presentation is improved I will support acceptance for this paper.", "broader_impact_concerns": "Ethical considerations are not discussed, maybe add a small paragraph on the ethical implication of using video data.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xuGDAZUgmL", "reviewer": "Reviewer_Wmpv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The paper introduces an adaptation technique and a loss to efficiently adapt a pre-trained image-text dual encoder architecture to video inputs.\n- Compared to previous approaches, the method is more data and compute efficient, while achieving state-of-the-art results.\n- The authors tested the method extensively on a wide range of datasets and settings, consolidating the method's capabilities.\n\n**Weaknesses**\n\n- The method section is hard to read, explaining poorly some steps and containing potential superfluous information. For instance, the overview paragraph is difficult for a first-time reader, and some equations may be unnecessary, e.g., Eq. 3, Eq. 8. Another example is the first paragraph of Sec. 3.2, where the authors explain ViT patches and class tokens. Moreover, Fig. 2 is very complex and does not simplify the understanding of the method.\n- Potentially missing baselines, i.e., CLIP fine-tuned with prompt tuning/adapters on the task and using the average representation of the frames for classification.\n- Missing ablation on the effect of the spatial adapter and the parameter used to combine the losses. Moreover, comparing against the zero-shot pre-trained model without fine-tuning would help assess the performance boost.\n- The t-SNE visualization is unreliable in understanding the effectiveness of the TFD loss due to its non-determinism. It may be better to use UMAP instead.\n- The architectures are reported unconventionally, making it more complex to understand the actual number of parameters, i.e., it is unclear if ViT-16/32/14 are all ViT-B models or include ViT-L or others.\n- The difference between Fig. 3b and Fig. 4 is unclear, i.e., why does the paper need both?\n- It may be useful to discuss more in-depth the tuning techniques and the additional layers introduced by the competitors in the related work or experimental protocol.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces T2L (Temporal Token Learning), an efficient approach for adapting image-based visual-language models like CLIP for video action recognition. The authors introduce temporal token learning and a temporal feature diversity loss to tune a pre-trained CLIP model for the task. The method surpasses or is competitive with state-of-the-art models that require more training time and data; it is tested on multiple settings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Update the method section and figure to make it simpler to understand for a reader, removing unnecessary information and emphasizing the technical components of the work. The figure is especially complex to understand.\n- Add missing baseline considering CLIP fine-tuned with prompt learning/adapters at image-level and then tested by taking the average video representation for classification. This would help in understanding how techniques that tune only a small set of samples compare with the proposed approach.\n- Add missing ablation study on the effect of the spatial adapter and the model performance when trained with different interpolation weights for the loss.\n- Replace the t-SNE generation with a UMAP, adding more details on the visualization, i.e., evaluation setup.\n- Why are Fig. 3b and Fig. 4 required and presented in two different parts?\n- Can the authors provide a longer description of how their model differs from previous approaches in terms of fine-tuning technique and training protocol? Moreover, it would be helpful to break down the number of parameters for each model, including the proposed architecture, e.g., report the base size + added parameters for each technique.\n- Add missing details for some ablation/experiment, e.g., the \"base model\" used for comparison in Fig. 4, the backbones with their full name (and potentially pre-train model, e.g., CLIP ViT-B/32 pre-trained on LAION2B).", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall, the reviewers are positive, with one quite enthusiastic about the technical contribution while another is lukewarm. The idea of temporal tokens is interesting and quite timely. It allows for accuracy improvements while maintaining a modest computational cost, which is a great benefit. A question to me is how are these temporal tokens extended for a diverse set of video durations or even streaming videos. Moreover, how would the method perform with videos where actions and events happen at varying paces? That said, the fact that the method consistently outperforms the baselines shows that it deserves to be published and communicated with a broader audience.
Decision: Accept (poster)