Storyboard-guided Alignment for Fine-grained Video Action Recognition
Authors: Enqi Liu
Liyuan Pan
Yan Yang
Yiran Zhong
Zhijing Wu
Xinxiao Wu
Liu Liu
Venue: NeurIPS 2025 poster ()
Keywords: Storyboard
Fine-grained
Atomic action.
Primary area: general_machine_learning

Abstract: Fine-grained video action recognition can be formulated as a video–text matching problem. Previous approaches primarily rely on global video semantics to consolidate video embeddings, often leading to misaligned video–text pairs due to inaccurate atomic-level action understanding. This inaccuracy arises due to i) videos with distinct global semantics may share similar atomic actions or visual appearances, and ii) atomic actions can be momentary, gradual, or not directly aligned with overarching video semantics. Inspired by storyboarding, where a script is segmented into individual shots, we propose a multi-granularity framework, SFAR. SFAR generates fine-grained descriptions of common atomic actions for each global semantic using a large language model. Unlike existing works that refine global semantics with auxiliary video frames, SFAR introduces a filtering metric to ensure correspondence between the descriptions and the global semantics, eliminating the need for direct video involvement and thereby enabling more nuanced recognition of subtle actions. By leveraging both global semantics and fine-grained descriptions, our SFAR effectively identifies prominent frames within videos, thereby improving the accuracy of embedding aggregation. Extensive experiments on various video action recognition datasets demonstrate the competitive performance of our SFAR in supervised, few-shot, and zero-shot settings.

Storyboard-guide Alignment for Fine-grained Video Action Recognition
Enqi Liu', Liyuan Pan 12, Yan Yang', Yiran Zhong', Zhijing Wu', Xinxiao Wu', Liu Liu
'Beijing Institute of Technology, Beijing, China
Yangtze Delta Region Academy of Beijing Institute of Technology, Jiaxing, China $BDSI, Australian National University, Canberra, Australia
'OpeNNLPLab, Shanghai, China
'Huwwei, Beijing, China
(enqi.liu, liyuan. pan, zhijingwu, vuxinxiao]@bit..de
yan. yangyang edu. au, zhongyirandgmail com, 1iuliu33@huawei.cmm com
Abstract
Fine-grained video action recognition can be formulated as a video-text match- ing problem. Previous approaches primarily rely on global video semantics to consolidate video embeddings, often leading misaligned video-text pairs due to inaccurate atomic-level action understanding. This inaccuracy arises due to i) videos with distinct global semantics may share similar atomic actions or visual appearances, and ii) atomic actions can be momentary, gradual, or not directly aligned with overarching video semantics. Inspired by storyboarding, where a script is segmented into individual shots, we propose multi- qultt--raanularity frame- work, SFAR. SFAR generates fine-grained descriptions of common atomic actions for each global semantic using large language model. Unlike existing works that refine global semantics with auxiliary video frames, SFAR introduces a filtering metric ensure correspondence between the descriptions and the global seman- ties, eliminating need for direct video involvement and thereby enabling more nuanced recognition of subtle actions. By leveraging both global semantics and fine-grained descriptions, our SFAR effectively identifies prominent frames within videos. thereby improving the accuracy of embedding aggregation. Extensive ex- periments on various video action recognition datasets demonstrate the competitive performance of SFAR in supervised, few-shot, and zero-shot settings.
Introduction
Fine-grained video action recognition has garnered increasing attention due to its broad applicability in areas such as sports analyties
[54], human- compute interaction [13, surveillance 44 and video understanding [12]. In contrast to standard action recognition, fine-grained video action recognition necessitates more detailed understanding of actions with similar appearances, demanding greater precision in capturing prominent frames with subtle actions.
The advent of large language models has revealed that robust multimodal encoder like CLIP 30 consolidate significantly more potent learned embeddings compared to manually crafted embeddings for action recognition [39] 21 29 23 28 51 [5] [ 47]. These methods involve using CLIP's visual encoder to extract video embeddings and its textual encoder to extract text embeddings. The video and text embeddings are then aligned for video classification. However, the potential of the textual encoder has not been fully explored.
"Corresponinng author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Making pizza Baking cookics
To effectively integrate textual in- formation, several works 48 26
1, 24 refine global semantics by leveraging LLM MMs or additional lexicons to generate class-specific atomic action de- scriptions. These auxiliary de- scriptions are selected based on their alignment with each video frame, which leads to two rigid assumptions: i) each global video semantic must correspond to identical atomic actions, and ii) all atomic actions within a video should be closely related to its global semantics. How- ever, these assumptions are fre- quently violated, leading to inac- curate embeddings. For instance, in Fig. [ (a), the videos 'Bak- ing cookies' and 'Making pizza' are ambiguous with each other, featuring overlapping atomic ac- tions. In Fig. [ (b), the atomic action 'Stand' can be irrelevant to the global video semantics "Swing legs'. Moreover, the non- uniform distribution nature of atomic actions has been overlooked previously, i.e., the duration of each atomic action can vary. Based this, using the global video semantics directly for aligning videos and text prompts can lead to misunderstandinng due to their granularity discrepancy. In this paper, we address these issues and propose galtiigranullrity framework, SFAR. The video semantics into fine-gran fine-grained sub-texts' and 'a video action into multiple atomic actions'. This allows both coarse and fine-grained identifications of prominent frames in videos.
Ambiguous actions
Long jump
43% Crouch, swing arms
22% Extend legs
35%
time
Land with bent knees
Swing lags
lime
Inrelevant Trames
Swinging legs backward Swinging legs forward
(b) Non-un uniform actions
Figure Example videos with ambiguous or non-uniform actions. Each row displays sample frames of a video with its class name. (a) Ambiguous actions. Both 'Baking cookies' and 'Making pizza' are actions performed in the kitchen with similar visual appearances and share alomic actions, e.g., the 'rolling dough'. (b) Non- cor-uniffrm actions. Multiple atomic actions distribute unevenly within a video to form an action, e.g.. the video 'Long jump' is composed of 43% of 'Crouch, swing arms' and 22% of "Extend legs'. Moreover, some atomic actions may be irrelevant to the global class, e.g., 34% of the frames show 'Stand', unrelated 'Swwig Legs'. The axis indicates percentage of frames per atomic action.
of our SFAR is decomposing 'the global
Inspired by the concept of storyboarding, which breaks down a script into individual shots, we enhance the global semantics (text) by generating detailed descriptions using pre-trained large language model (LLM). These fine-grained descriptions, i.e., sub-texts, capture common atomic actions depicted in videos, requiring only their class names and utilizing designed question prompts, eliminating the need for direct video involvement. To fit the flcxibility of various pre-trained LLM and question prompt formats, design text prompt perplexity metric to measure the diversity among sub-texts and the similarity between the sub-text and the global text for filtering sub-texts. provides an effective schema for selecting sub-texts to train the video action recognition model.
We then use global texts and sub-texts to coarsely and fine- grainriin compute a video embedding for videos with ambiguous or non-uniform actions. Specifically, we augment the global text with sub-texts the embedding space of CLIP by abounding described video actions with sub-texts, which also decreases the granularity of the global text. This augmented global text is used to weight video frames in embedding space and compute a coarse video embedding. We illustrate network attention of the augmented global text in Fig. 2 (c) and sub-texts in Fig. 2 (d). Compared to the previous method constrained by granularity differences (Fig. (b)), our augmented global text and sub-texts direct network attention more comprehensively toward regions with actions.
We then fuse the coarse and fine- fine-grained video embedding to compute video embedding that improves the classification performance on ambiguous and non-uniform videos. Heatmaps of our network are in Fig. 2 (e), and improvements over videos with different ambiguous and non- uniform scores are in Fig. 2 (f). We rigorously validate our approach across different scenarios, including supervised, few-shot, and zero-shot video action recognition. Our method delivers top-notch performance in all these scenarios, showcasing the effectiveness of our framework.
Our main contributions are:

---

Making sandwich
video using Increase Top-1 Acc (%)
Increase Top-1 Ace (%) 202g
1ea
Pullups
Increme
Incrcase
Shearing shoop
Increase
Increase
10
10
Ambiguous Score
Non-unifoem Score
(a) Frame
(b) Single
(c) Coarse
(d) Fine
(c) Coarse-Fine
(f) Performance
Figure 2: Examples that illustrate the ineffectiveness of directly aligning video frames and global text, from (a-e) heatmaps and classification performance improvements. (a) Video frames with their class nome annotated at the left. Heatmaps the single granularity-bon difference between the video frame and global text. (c-e) Heatmups using coarse video embedding. finiind video embedding, and the fusion of the two. (f) We show the performance distributions (Top-1 Accuracy) over ambiguous scores and non-unin con-uniform scores of videos for (b-e) the whole set. The first row is the absolute performance for (b), and the last three our supplementary materials for details the computation of the anbiguous score and the non non-uniform score, as well as the accuracy improvements on the Kinetics dataset presented Tab.
(single) method (48), which limited by the granularity
are improvements of (c). (d), and (e) concerning (b). Please refer
A multi-gre granularity framework for transferring CLIP trained image-text pair to video action recognition.
A schema for automatically decomposing video action into common atomic actions to provide fine-grained knowledge of the video action to CLIP.
A coarse and fine-ge mee-grained video embedding module for videos with ambiguous and non- uniform actions.
2 Related Works
We review video action recognition methods focusing visual and visual-text representation learning.
Visual Representation Learning. Action recognition requires accurately capturing temporal seman- tic variations. Numerous studies [59]5]]
35| 39 [5] have thus focused on visual representation learning. Early works studied the joint learning of spatial and temporal features
52 42,
various architectures [26] 23, 22,22,55 32]. Benefiting from large-scale visual and visual-text pre-training, recent methods leverage the strong spatial features learned in pre-training and focus on fine-tuning the trained model to capture the temporal semantics of a video. However, these methods do not explicitly address the fundamental challenge of recognizing video actions from ambiguous and non-uniform videos. This paper proposes decomposing video actions into atomic actions to improve recognition of challenging videos.
have been developed overcome these limitations by identifying video frames that strongly align with the text prompts of video actions for video action recognition in vision-language models, e.g., CLIP. One remarkable work is BIKE 48), which weights the video frames based on their alignment with the text prompts to compute a video embedding for action recognition. However, these methods are limited by the granularity difference between the video frames and the text prompts, where the text prompt is global context for the video. Unlike previous methods focused on generative modeling (49 25 57]. where LLMs generate captions or texts conditioned on video or frame embeddings, our approach adopts discriminative modeling approach. We generate sub-texis from the global text to describe atomic actions video frames. The sub-texts are then used to construct coarse and fine- ene-ggained video embeddings, which improves the recognition of fine-grained actions.
Visual-text Representation Learning, Several methods 8 40

---

Coarse Video Embedding
Coarse Importance
Texual Encoder
Global Text Prompt
1
T
Tc
Tc
Encoder
Coarse Aggrepaation
Global Text Angmentation
Sub-text Generation
(1)
[ F.
Fusion
(S 1
Output Video Embedding
fine
Fine ineeggraied Aggregation [oftto
Texual Encoder
Sub-text Prompt
[vela
(S 1.8
(Seahe
Fine-grained Importance
vvi ( J[=
Fine-graimed Video Embedding
Figure 3: An overview of our framework for video action recognition. We extend CLIP for classifying the video (I with L frames by compusing a video embedding from frame embedding {vi 1 in three steps. (i) We decompose the global text prompt Te that describes the class semantic of action into descriptions of atomic action (i.e. sub-text prompts) by using a pre-trained large language model. The global text prompi To and sub-text prompts are then embedded by the textual encoder of CLIP for extracting embeddings of T. and (Sc (fi) A coarse video embedding is extracted by augmenting the global text embedding T, with the sub-text embedding (Sc.)fe1. calculating coarse importance of the frame embedding {vi*)] augmented glohal text embedding Te, and using the importance (m) to aggregate coarse video emhedding o from frame embedding {vrh)/2 (iil) Similar to the coarse video embedding. we get a fine seee-grained video embedding the by calculating fine-graine importance {aj"=))_ of the frame embedding { with sub-text embedding (Se,) for aggregating frume embedding {vi-t Ji=1- The course and fine graiied embeddings o coune and fine are fused to form the final video embedding o for action recognition.
Ji=1 with the
3 Methodology
Preliminary. CLIP (30.. visual-language pre-training method, consists of visual and textual encoder. It learns joint embedding space by maximizing similarities between aligned image- text pairs and minimizing them for misaligned ones. Given an image I and global text prompts (To)== of C' classes formatted as [a photo of a class] that globally describe class semantics, where class is the class name, CLIP performs zero-shot classification by extracting the visual class embedding vels from I and text class embeddings (tel)e] from [T The image I is then classified into the class c' with the maximum cosine similarity, i.e. arg max sim(v tel)).
Overview. To extend CLIP for classi- fying the video {L with T. frames, we adaptively compute a coarse and fine-grained video embedding. Two multi-granularity embeddings capture global semantic and atomic seman- tics of ambiguous and non non-uniform ac- tions in the video, for computing the cosine similarity with the text class embedding. There are three key steps: (i) Sub-text generation, decomposing each global text prompt T. into sequence of N atomic action text descriptions (S (i.e., sub-text prompts), by using a pre-trained large language model; (ii) Coarse video embedding, embedding each frame I, into visual class embedding veth that is frame embedding. Then, by using the global text prompt augmented with sub-text prompts to identify salient video frames CLIP embedding space, we aggregate the frame embedding (vi ] into coarse video embeddings coarse (iii) graeneg video embedding, finding video frames with atomic actions by using sub-text prompts and CLIP for computing a fine-grained video embedding o fine We fuse the coarse and fine-grained video embedding into video embedding o to compute
Sprringtoou diving
LLaVa1.6 GPT-40
Preparing
2. Takeoff
3. Rotation
4. Entry the water
1. Prepare diving
2. Jump off the board
3. mid-air
4. Dive into the water
Figure 4: Example sub-texts {Sc. generated from GPT-4o and I.L.Val.6, with N in the example.
iv

---

(n) Earing cake
(b) Tasting food
(c) Dining
(d) Eating spaghetti
Eating burger
Single
Coarse vidco embedding
30
285
25%
30
225
20%
207%
231
20/3
17%5
10
10
3%
(4) (5)
(c)
(d)
(6)
(4)
(6)
(4)
Figure 5: An example illustrating the benefits of using our coarse video embedding for ambiguous actions. We show a video in the first now, with the ground truth action labeled as (a) on the left. Its ambiguous actions annotated with (b), (c), (d) and (e) the bottom of the video. The action probability from the single granularity based method (48) and our coarse video embedding are the next row. Compared with the single granularity bassed method that identifies the video as (b) "Tasting food', we obtain a higher probability of the correct class (a) 'Eating cake'.
cosine similarities with text class embeddings t.th for classifying the video. The overview of our method is in Fig. B
3.1 Sub-text Generation
A global text prompt Te is description of video action. It overlooks that an action is usually formed by performing sequence atomic actions, like a video performing the action. Directly aligning video frame embedding the global text prompt T. introduces the gramularity differences and decreases the video classification performance. To minimize human burden and bias, we propose pipeline that automatically generates and selects sub-texts based on the global text (class name).
We leverage pre-trained large language model LLM(.) to decompose global text prompt T. into N potentially atomic actions {Sc.u J LLM(P.), where P. is the prompt for describing the action class e and directing the pre-trained large language model find potential atomic action descriptions, sub-texts ) Refer to Fig. for an example.
However, the existence of various pre-trained large language models and the flexibility in designing Pe make selecting meaningful sub-text set challenging. We assume that an optimal sub-text set { for video classification should be sufficiently related with the global text prompt T. while each sub-text should be diverse from the others. To measure the similarities and diversity, we propose text prompt perplexity metric select the sub-text set with the highest text prompt perplexity score.
In Fig. we show ambiguous action probability computed from a single granularity-basee method 48 and coarse video embedding. Our method uses sub-text augment the global text help the model to identify different atomic actions in ambiguous actions, and improve the accuracy of recognizing ambiguous actions.
With the class embedding the global text T, and sub-texts { from the CLIP textual encoder, tc and sels we define the text prompt perplexity score TPP. for tels and slls as below
(-
TPP. exp
Σ
log (a(0em)
N
sim thth
scls
+1
den
(2)
2
8cp
1
Σ
N
sim cls
c,n sc.')) +
N
='=Ln

---

Drinking
9
24%
300%
241
2,355
50%
26%
115
and jork
Clean
13%
5%%
Figure 6: Examples of coarse and fine-grainet importance probability for videas with non-uniform actions. The fine-grained importance probability ah allocate larger scores than the coarse importance prohability af to prominent frames. For example, the illustrated fourth frame the second row, 'lowering barbell back down the ground', represents an atomic action of "Clean and jerk', and dj Auz 23% has higher probability than di 9%.
where the sim(.;)) calculates the cosine similarities of inputs. We employ de,n to quantify the similarity between a global text and its corresponding sub-texts, a higher value dan indicates greater similarity. Meanwhile, denn is used to evaluate the similarity among sub-texts themselves, where a higher value of óc. indicates greater diversity across the sub-texts. Here, ar-) and B(-) are linear scaling functions for the scores. Please refer to Fig, [br the correlations between the video classification performance and the text prompt perplexity score TPP.-
3.2 Coarse Video Embedding
For identifying different atomic actions in ambiguous videos, we augment the global text T. with 1 sub-texts [S to adaptively aggregate the video frames [m) in CLIP embedding space. As the class name in global text is usually phrase, and sub-text Sa,, describes the atomic action with multiple words, we keep the word embeddings to benefit from the rich semantics of each word embedding for video frame aggregations. We denote the matrix form of the word embeddings for global text and sub-text as T. and Sc. and augment T. by the cross- attention mechanism,
Q W K = Sa,a W
Ven = Sc,a W
(4)
K. K Ve = [(Ven)
(5)
T Attention Qe: Ke V.) T
(6)
where W9, W and W are matrices for projecting the query Qu. key K and value Ven- Here, [(Ka and [(Ve,n 1.N [ concatenates all keys and values for Qe-
Using the augmented global text embeddings Te, we find the salient video frame embedding vi for aggregating coarse video embedding. To find the coarse- coarse-grained importance score a qgnare of each frame embedding, we calculate the overall normalized similarity between each word embedding and the video frame embedding by
exp (sim(te, v(l))
EET. wー exp (sim(fe, vgb))
We then compute the coarse video embedding o course with summing over the video frame embedding vith weighted by a crarse
coarse
cl coarse
a
(8)
3.3 Fine-grained Video Embedding
While the video coarse embedding coarse captures salient semantics for ambiguous actions, it may overlook atomic actions the video that align with the expected action class e due to the non-uniform property of video actions. To address the limitation, we use the word embedding of the sub-texts to find atomic actions among the video frame embeddings {v(k)] [= to compute the fine-grained video embedding.
vi

---

Table 1: Comparison result on the Kinetics-400 dataset. We report the GFLOPs in the inference phase. "Views" indicates number of temporal clips and spatial crops used in inference (# temporal clip x spatial crop). The magnitude is Million (10") for number of model parameters (Param). We achieve the highest top-1 and top-3 accuracy by employing only an 8- 8-frame input evaluated with single view, outperforming past methods with more model parometers, imput frames, and views. We highlight the hest top-1 and top-5 accuracy in bold.
Method
Venue
Input
Pre-training
Top-1(%)
Top-5(%)
Views
GFLOPs
Param
Methods with large-scale visual pre-training
MVENetg-
AAAI'21
24x224
ImtgeNet-IK
79.1
93.8
188x30
VIVIT B/16x2 2
ICCV'21
32x2242
JFT-300M
80.0
4x1
455.2
151,9
ST-Adapter VII-B/I6
28
NeurlPS'22 22
8x224
WIT-400M
82.0
95.7
3x1
455
128.8
EVL. VIT-B/16 21
ECCV
8x224
WIT-400M
82.9
3x1
444
177.7
MTVVB
CVPR'22
32x224
JFT-300M
81.8
95.0
384x12
310
VideoSwin-1 24
CVPR'22
32x224
ImageNat-2IK
82.7
95.5
4x3
282x12
88.1
ActionCLIP VIT B/16
INNLS'23
16x224
WIT-400M
82.6
96.2
10x3
282x30
141.7
ATM VTT-B/16
ICCV'23
8x224
WIT-400M
82.8
95.6
DIST ViT-B/16
29
ICCV'23
8x224
WIT-400M
83.6
3x1
163x3
105
AIM ViT-B/16
15
ICLR'23
8x2242
WIT-400M
83,9
96.3
3xl
202x3
97
ILA-ViT-B/16
ICCV'23
8x2242
WIT-400M
149x12
Methods with large-scole VistollTeNT pre-fraining
X-CLIP VTT-B/16
26
ECCVV22
8x224
WIT-40DM
82.3
1xl
145x1
VideoPPropt ViT-B/16
ECCV'22
16x2242
WIT-400M
76.9
93.5
SIF VIT B/16 40
ACMMM'23
8x224
WIT-400M
77.4
93.6
4x3
1136x12
143.9
Vita-CLIP V/T-B/16
4B
CVPR'23 23
8X224
WIT-400M
80.5
96.0
1xl
97x1
187.9
Viaa-CIP VIT-B/16
CVPR'22
WIT-400M
81.8
96.0
97x12
BIKE R/T-
4S
CVPR'23
8x224
WIT-400M
83.2
1xl
124.1
BIKE VIT-B/16
48
CVPR'23
8x224
WIT-400M
83.9
4x3
124.1
M VTT-B/16
AAAI'24
8x224
WIT-400M
83.4
96.3
4x3
214x3
ALT ViT-B/16
CVPR'24
8x224
WIT-400M
83.6
95.9
3xl
328
134.4
OST VIT B/16
CVPR'2 24
8x2242
WIT-400M
82.0
Ixl
Ours VIT-B/16
NeurIPS'25 25
8x2242
WTT-400M
84.5
96.7
1x1
90.2x1
126.1
We compute a fine-grained importance score afte for each video frame embedding vi as the maximum average similarity between vith and Se,. and normalize by a SoftMax function,
exp (max, SS.. Σ Se,n &c,nsim(SE,as v(l))
fine dj
2p=1 exp maxx | Σ Se,n E Se,nsim(Ss,,. vgla)
where [Sc. is the number of words in the sub-text, and Se, Se, is word embedding for the sub-text. Similar to Eq. (8) weight the video frame embedding vfth with finee, and get the fine-grained video embedding o fine by
fine
aith
(10)
We compare the coarse scores qoouse and fine-graing scores of Fig. and find that a fine can fine- the-grainedly allocate larger scores than af craare to prominent frames that humans perform the action peak the non- non-uniform videos.
3.4 Loss
We fuse the coarse video embedding and fine-grained video embedding as a video embedding with two feedforward layers FFN (-) and FFN (-), projecting the coarse video embedding and fine-grained video embedding to the semantic of class
o FFN cearse
cranse
+ FFN (o fine
(11)
and compute the cosine similarity with the text class embedding tels by sim(te,
We optimize our network by maximizing the similarity Yo,e between the b-th video embedding in a batch and text embedding of its ground truth class c and minimizing the similarity between other video and text class embeddings { (Jh.c)e=1,eppe". Following 27). we use the InfoNCE loss,
B
exp(yb.er)
LT2
log
(12)
B
kh
Σ =="= exp(yb",e0)
b=
E'
vii

---

Table 2: Comparative experiments conducted Charades (33 HMDB-51 19) and UCF-101 34, while
ablation studies are performed Kinetics-400 dataset 17y. report accuracy (%) for a single 8-frame clip with a spatial resolution of 224x224, unless otherwise specified. The "VZ" column denotes if the method specifically designed for zero-shot action recognition or adapted from CLIP.
Method
Frames
mAP
Method
VZ UCF-101
HMDB-51
Backbume
FVE FVP: (%) Top-5(
STM
MultiScale TRN
58
16
35.3
25.2
ER 55
E2E 3
51.8
29.8
35.3
VideoPrompi
L5]
79.3
76.9
93.5
95.1
SlowFast+NL
ActionCLIP
X3D-XLAB
39
10
16+64
32
42.5
44.3
BIKE 48
Vita-LIP
20
16
58.7
75.0
78.4
48.6
55.6
ATM 46
82.9
82.8
95.6
96.4
BIKE (88
BIKE 48
83.2
96.0
Ours
16
51.1
(b) Comparisons of zero-shot ac-
Ours
79.0
56.6
(c) Generalization of fine-- tree-grained video embedding (FVE)
83,8
96.5
(a) Comparisons for multi-label ac-
tion recognition the HMDB-51
fion recognition the Charades
19 and UCF-101 34 datasets.
state-of-the-ar methods using the Kinetics-400 dataset (17).
dataset (33]
Generator
TPP
Top- %))
Top-5(%) (%)
Coarse VE
Fine-grained VE
Top-1(%)
Top-5(%)
LLaVA-L6-34b
54.4
84.35
96.5
Baseline (CLIP)
GPT-Davinci
54.7
84.35
96.6
Baseline (Temporal)
80.3
95.0
GPT-3.5
60.4
84,47
96.7
/
82.7
96.2
83.5
96.3
GPT-4o
60.8
84.52
96.7
84.5
96.7
(d) Comparison of sub-texts generated from different LLMs on Kinetics-400 A higher Text Prompt Per- plexity (TPP) score indicates better sub-texts.
(e) Ablations of coarse and fine grained video embed- ding (VE) on Kinetics-400. Using fine-tued CLIP and CLIP model with temporal layers as baselines.
1
B
exp(yw
CVT
log
(13)
B
kbb
>c= exp(ytr.c)
b=1
b'
L Lrzv XLvzT,
(14)
where the number batches, k, find index of the video that has the same class with the b-th video, [k]] is its size, and À is hyperparameter.
4 Experiment
4.1 Experimental Setup
Our proposed sub-text set and code are given in our project page
Datasets. We experiment across four extensively recognized video benchmarks: Kinetics-400 Charades 333, UCF-101 [34, and HMDB-51 [19] datasets.
17
Supervised Learning. Our model is implemented using the PyTorch framework. We train our network with batch size 256 for 30 epochs using the AdamW optimizer. The learning rate set to 5 x 10-5, and we use the cosine annealing strategy with 5 warm-up epochs. We follow 48 for data augmentation in training.
Zero-shot Learning. We evaluate our model, pre-trained on Kinetics-400 17 using the UCF- 101 34 and HMDB-51 [19]
26 to experiment with different shot settings, selecting 2, 4, 8, and 16 examples per human action category for training. Training for epochs the Kinetics-400 dataset, and then use the same settings to train on other datasets for 10 epochs. Importantly, we do model pre-trained on Kinetics-400 for few-shot learning other datasets.
Few-shot Learning. We follow
Evaluation Metrics. We evaluate our model with top-1 top-5 accuracy on the single-label datasets. On multi-label Charades, follow [4] to report mean average precision (mAP).
viii

---

UCF-101 34 and Kinetics- 400 datasets (17) with state-of-the of-tte--a---reeee art action recognition methods. utilize ViT-B/16 as the backbone and use 8 frames for training/validatinn. All performances are reported as top-1 accuracy ( (%) in the few- sew-shot setting using single view inference with spotial size of 224 x 224, where all models are directly fine tuee-tned from CLIP. "Avg." refers the average performance across all datasets.
Table 3: Comparisons on few-shot action recognition across the HMDB-51 199
HMDB-51
UCF-101
Kinetics-4000
All
Method
K=2
K=4
K=8
K=16
K=2
K=4
K=8
K=16
K=2
K=2
K=4 K=8 K=16
Avg.
Vanilla CLIP
30
41.9
41.9
41.9
41.9
63.6
63.6
63.6
63.6
57.2
57.2
57.2
57.2
54.2
ActionCLIP
339
47.5
57.9
57.3
59.1
70.6
71.5
73.0
91.4
61.0
63.0
64.8
68.5
65.5
VideoPrompt
15
39.7
50.7
56.0
62.4
71.4
79.9
85.7
89.9
67.0
X-CLIP 26
53.0
57.3
62.8
64.0
76.4
83.4
88.3
91.4
56.8
60.7
62.3
64.6
68.4
ViFi-CLIP [3
57.2
62.7
64.5
66.8
80.7
85.1
90.0
92.7
37.1
42.8
49.1
55.5
65.4
TC-CLIP
18
57.3
62.3
67.3
68.6
85.9
89.9
92.5
94.6
58.5
61.9
65.5
69.9
72.9
OST
59.1
62.9
64.9
68.2
82.5
87.5
91.7
93.9
44.0
48.2
52.5
56.5
67.6
Ours
64.0
66.1
68.7
70.7
92.1
93.0
94.3
94.7
73.9
74.8
75.5
76.2
78.7
4.2
Main Results
Action Recognition. We compare network against state-of-the-art methods that utilize large-scale visual pre-training and visual-text pre- pre-training on the Kinetics-400 dataset in Tab. [ All methods use the ViT-B/16 backbone. Using fewer input frames and views during testing, our method that employs only an 8-frame input evaluated with single view achieves the highest top-1 and top-5 accuracy. For example, the second-best method of large-scale visual-text pre-training uses temporal clips spatial crops of a video but achieves top-1 accuracy 0.5% lower than ours.
To further demonstrate the effectiveness of our framework, we conduct experiments on challenging scenarios from Kinetics-400 based on the ambiguous and non-uniform score. We select the top 10% of actions with the highest scores, representing the most difficult categories to classify. The results are shown in Tab. where we achieve an average performance 2.6% higher than the second-best.
Multi-Label Action Recognition. We evaluate our method for multi-label action recognition on the Charades dataset in Tab. 2a The mAPs of state-of-the-ar methods are reported, and we follow 9 to use 16 frames. Our method finds 51.1 mAP, achieving 0.7 mAP more than the second-best
Zero-shot Action Recognition. Our method is trained with supervision from texts and be used for zero-shot action recognition. We compare our network with methods that adapt from CLIP, which is pre-trained on images, and with zero-shot methods developed for videos on the UCF-101 and HMDB-51 datasets in Tab. 26 All methods use a single view during testing. Our method demonstrates superior generalization capabilities in zero-shot action recognition. Specifically, versus the latest zero-shot methods developed for video (38), our zero-shot performance on the HMDB51 dataset is 9.5% higher than 1..1%. Further highlighting the versatility of our pipeline.
Few-shot Action Recognition, We explore 2, 4, 8, and 16-shot action recognition on the HMDB-51 19). UCF-101 34 and Kinetics-400 datasets (17) in Tab. B] With limited amount of videos, our approach that decomposes a video action into atomic actions exhibits the highest performance. Specifically, the average performance of our approach across all shots is 78.7%, which is 5.8% higher than the second best method (TC-CLIP, with 72.9%).
4.3 Ablation Studies and Analysis
We perform ablation studies with the Kinetics-400 dataset to examine our approach.
Number of Sub-texts. Fig. 7a illustrates the number N of sub-texts used in our model. As the N increases from 2 to 5, the recognition accuracy of our method improves. However, we observe that the performance gain is minimal when increasing the number of sub-texts from 4 to 5. Considering both computational complexity and recognition accuracy, we opted for 4 sub-texts in our experiments.
Correlation of TPP and Performance. In Fig. 7b 11 sub-text groups are generated, and show the relationship between TPP and the action recognition performance, where TPP is
ix

---

Table 4: Comparison of Top-1 accuracies some representative actions with N=2 and N=5 sub-actions. The Difference column shows the change Top-1 accuracy from N=2 N=5.
Action
N=2
N=5
Difference
Action
N=2
N=5
Difference
Dribbling basketball
89.2
94.6
+5,4
Arm wrestling
100.0
100.0
0.0
Kicking soccer ball
71.0
76.3
+5.3
Christmas tree
100.0
100.0
0.0
Lunge
75.6
82.2
+6.6
Brushing teeth
89.5
89.5
0.0
High kick
35.1
40.5
+5.4
Slicing onion
98.6
97.4
-1.2
Clay pottery making
92.1
94.7
+2.6
Clapping
47.2
50.0
-2.8
computed the average of TPP: across all ac- tion categories. We observe a positive correla- tion, as shown by the fitted dashed line, with an r2 value of 0.79, demonstrating the effective- ness of our TPP method in selecting sub-texts without the need for computationally intensive experiments.
Top- Ace (%
84.6
Top-1 Acc (%)
84.6
84.5
84.5
84.4
84.4
2
3
25 35
Number of sub-texts
TPP
Comparison of Sub-text Generators. Tab. 2d
(a) The impact of numbers compares sub-texts generated by GPT-Davinci of sub-texts (N).
(b) Study the relationship between TPP and Top-1.
[4. GPT-3.5, GPT-4o, and LLava-v1.6-344 (22).
GPT-4o achieveshe highest TPP (60.8), along
Figure 7: Analysis of sub-texts.
with the best Top-1 (84.52%) and Top-5 (96.7%) accuracies. Both LLaVA-1.6-34B and GPT-Davinci
show similar performance, with minor differences in accuracy and TPP. These results indicate that GPT-4o provides the most effective sub-texts for action recognition.
Effectiveness of Components. We study the effective usage of our coarse video embedding and fine-grained video embedding Tab. 2e A CLIP model fine-tuned on the Kinetics-400 dataset to establish performance baseline. Following the common practice of existing works [48]. we build a baseline (Temporal) using 6-layer Transformer encoder with position embedding for sequence features. Individually, both the coarse and fine-grained video embeddings enhance performance relative to the baseline; together, they achieve consistently superior results. For example, using all embeddings, achieve top-1 accuracy of 84.5%, which is 4.6% higher than the baseline.
Different sub-actions. We previously observed that the accuracy of different action categories changes with respect to the number of sub-actions (N). Taking N = 2 and N = 5 as examples, the accuracy differences across representative actions are shown in Tab. 4 The results indicate that complex, multi-stage action categories (e.g.. 'Dribbling basketball' and 'Clay pottery making') benefit from large N, whereas repetitive or simple motions such as 'Clapping's and 'Slicing onion' get better accuracies with small N. We had investigated overall performance gains by using class-aware N, and only found marginal improvement over fixed N=5, 0.5% and 0.2% for Top-1 and Top-5, respectively. Given the computational cost of involving class-aware value of N for each action, choose to use fixed N for efficiency.
Generalization. To validate the generalization of our components of fine- the-grained video embedding structure, we apply them to state-of-the action recognition methods, shown in Tab. [c] Specifi- cally, we study VideoPrompt [15), ATM [46] and BIKE [48]. The results indicate that we enhance the accuracy these state-of-the-arr methods, such as VideoPrompt, which shows a 2.4% increase in accuracy on the Kinetics-400 dataset.
More. We provide additional implementation details, comparisons, and ablation studies in the supplement, e.g., analyses of few-shot, zero-shot, and supervised learning on other backbones.
5 Conclusion
In this paper, we propose framework to transfer CLIP trained on image-tex. pairs to video action recognition. Similar to how a video forms a video action by performing sequence of atomic actions, our key insight is to decompose a video action into atomic action descriptions using pre-trained LLM. We then select these atomic action descriptions with a proposed metric. The global and atomic action descriptions are used identify salient video frames from ambiguous and non-uniform videos for action recognition. Experiments on standard benchmark datasets demonstrate that our method significantly outperforms previous works in supervised, few-shot, and zero-shot settings.
X

---

Acknowledgments and Disclosure of Funding
This work was supported by the National Natural Science Foundation of China under grant No. 62302045, and the Beijing Institute of Technology Special-Zone.
References
[1] Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404. 1401:249..4,,,, 2024.
[2] Anurag Amab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: vidco vision transformer. In Proceedings of the IEEE/CVF international conference on computer vision,
pages 6836 6846, 2021.
[3] Biagio Brattoli, Joseph Tighe, Fedor Zhdanov, Pictro Perona, and Krzysztof Chalupka. Rethinking zero- shot video classification: End-to-end training for realistic applications. In Proceedings of the IEEL/CVF Conference on Computer Vision and Patlern Recognition, pages 4613-4623,, 2020.
[4] Tom B Brown. Language models are few-shot learners. arXiv preprint ArXiv:2005, 14165, 2020.
[5] Shizhe Chen and Dong Huang. Elaborative rehearsal for zero-shot action recognition. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 1 13638-13647, 2021.
[6] Tongjia Chen, Hongshan Yu, Zhengeng Yang. Zechuan Li, Wei Sun, and Chen Chen. Ost: Refining text. knowledge with optimal spatio-temporal descriptor for general video recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 18888-18898, 2024.
[7] Yifei Chen, Dapeng Chen, Ruijin Liu, Sai Zhou, Wenyuan Xue, and Wei Peng. Align before adapt: Leveraging totity-to-region alignments for generalizable video action recognition. In Proceedings of the IEFF/CVP Conference Computer Vision and Pattern Recognition, pages 1 18688-18698, 2024.
[8] Yuxiao Chen, Jianbo Yuan, Yu Tian, Shijie Geng, Xinyu Li, Ding Zhou, Dimitris N Metaxas, and Hongxia Yang. Revisiting multimodal representation in contrastive learning: from patch and token embeddings to finite discrete tokens. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 15095-15104 2023.
[9] Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. In Proceedings of the JEEF/CVF conference on computer vision and pattern recognition, pages 203-213, 2020.
[10] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings the IEEE/CVF international conference on computer vision, pages 6202-
6211, 2019.
[11] Zhili Feng, Anna Bair, and J Zico Kolter. Text descriptions compressive and invariant representations visual learning. Transactions on Machine Learning Research, 2023.
[12] Yudong Han, Qingpei Guo, Liyuan Pan, Liu Liu, Yu Guan, and Ming Yang. Dynfocus: Dynamic cooperative network empowers ilms with video understanding. Proceedings of the Computer Vision and Pattern Recognition Conference, pages 8512-8522, 2025.
[13] Mohamed Hassan, Partha Ghosh, Joachim Tesch, Dimitrios Tzionas, and Michael Black. Populating 3d scenes by learning human-scene interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19708-14718, 2021.
[14] Boyuan Jiang, MengMeng Wang, Weihao Gan, Wei Wu, and Junjic Yan. Stm: Spatiotemporal and motion encoding for action recognition. Proceedings of the IEEE/CVF international conference on computer vision, pages 2000-2009, 2019.
[15] Chen Ju. Tengda Han, Kunhao Zheng, Ya Zhang. and Weidi Xie. Prompting lisual-language models for efficient video understanding. In European Conference on Computer Vision, pages 105-124. Springer, 2022.
[16] Chen Ju, Zeqian Li, Peisen Zhao, Ya Zhang, Xiaopeng Zhang, Qi Tian, Yanfeng Wang, and Weidi Xie. Multi-modal prompting for low-shot temporal action localization. arXiv preprint arXiv:2333. /732, 2023.
[17] Will Kay, Jono Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv: 7005: 06950, 2017.
[18] Minji Kim, Dongyoon Han, Taekyung Kim, and Bohyung Han. Leveraging temporal contextualization for video action recognition. In European Conference on Computer Vision, pages 74-91. Springer, 2024.
[19] Hildegard Kuchne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdh: a large video database for human motion recognition. In 2011 International conference on computer vision, pages
2556-2563. IEEE, 2011.
[20] Chung- Chung-Ching Lin, Kevin Lin, Lijuan Wang. Zicheng Liu, and Linjie Li. Cross moosl representation learning for sero-shot action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19978-19988 2022.
xi

---

[21] Ziyi Lin, Shijie Geng. Renrui Zhang, Peng Gao, Gerard de Melo, Xiaogang Wang, Jifeng Dai, Yu Qiao. and Hongsheng Li. Frozen clip models are efficient video learners. In European Conference on Computer Vision, pages 388-404, Springer, 2022.
[22] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lee. Improved baselines with visual instruction tuning, 2023.
[23] Ze Liu, Jia Ning. Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 3202-3211, 2022.
[24] Chaofan Ma. Yang Yuhuan, Chen Ju, Fei Zhang, Ya Zhang. and Yanfeng Wang. Attrseg: open- vocabulary semantic segmentation via attribute ceaippooiiioongggegatiinn Advances neural information processing systems, 36: 100258-12200 2023.
[25] Himangi Mittal, Nakul Agarwal, Shao- Lo, and Kwonjoon Lee. Can't make an omelette without breaking some eggs: Plausible action anticipation using large video-language models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 18580-18590, 2024.
[26] Bolin Ni, Houwen Peng. Minghao Chen, Songyang Zhang. Gaofeng Meng, Jianlong Fu, Shiming Xiang. and Haibin Ling. Expanding language- image pretrained models for general video recognition. In European Conference on Computer Vision, pages 1-18. Springer, 2022.
[27] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint 0rXiv:1807.0377
2018 [28] Junting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao, and Hongsheng Li. St-adapter: Parameter-efficient image-to-video transfer learning. Advances in Neural Information Processing Systems, 35:26462-26477,, 2022.
[29] Zhiwu Qing. Shiwei Zhang, Ziyuan Huang, Yingya Zhang, Changxin Gao, Deli Zhao, and Nong Sang. Disentangling spatial and temporal learning for efficient image-to-video transfer Iearning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13934 13934-13944, 2023.
[30] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[31] Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Fine-tuned clip models are efficient video learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6545 6554,,,5,222 2023.
[32] Michael Ryoo, Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. Tokenlcamer: Adaptive space-time tokenization for vidcos. Advances in Neural Information Processing Systems, 34: 1777- 2021.
[33] Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in homes: Crowdsourcing data collection for activity understanding. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 1-14, 2016, Proceedings, Part I 14,
pages 510-526 Springer, 2016.
[34] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. A dataset of 101 human action classes from videos in the wild. Center for Researck in Computer Vision, 2(11):1-7, 2012.
[35] Anirudh Thatipelli, Sanath Narayan, Salman Khan, Rao Muhammad Anwer, Fahad Shahbaz Khan, and Bernard Ghanem. Spatio-temporal relation modeling for sew-shot action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19958-19967, 2022.
[36] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307. 09288, 2023.
[37] Shuyuan Tu, Qi Dai, Zuxuan Wu, Zhi-Qi Cheng, Han Hu, and Yu-Gang Jiang. Implicit temporal modeling with learnable alignment for video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19936-19947, 2023.
[38] Mengmeng Wang, Jiazheng Xing, Boyuan Jiang, Jun Chen, Jianbiao Mei, Xingxing Zuo, Guang Dai. Jingdong Wang. and Yong Liu. A multimodal, multi-task adapting framework for video action recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 5517-5525, 2024.
[39] Mengmeng Wang, Jiazheng Xing, Jianbiao Mei, Yong Liu, and Yunliang Jiang. Actionelip: Adapting language-image pretrained models for video action recognition. IEFF Transactions on Neural Networks and Learning Systems, 2023.
[40] Qiang Wang, Junlong Du, Ke Yan, and Shouhong Ding, Seeing in flowing: Adapting clip for action recognition with motion prompts learning. In Proceedings of the 31 ACM International Conference on Multimedia, pages 5339-5347, 2023.
[41] Xiang Wang. Shiwei Zhang, Jun Cen, Changxin Gao, Yingya Zhang, Deli Zhao, and Nong Sang. Clip- guided prototype modulating for few-shot action recognition. International Journal of Computer Vision, 132(6):1899 1912, 2024.
xii

---

[42] Zhengwei Wang. Qi She. and Aljosa Smolic. Action-net: Multipath excitation for action recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13214-13223,
2021.
[43] Syed Talal Wasim, Muzammal Naseer. Salman Khan, Fahad Shahbaz. Khan, and Mubarak Shah. Vita-clip: Video and text adaptive clip via multimodal prompting. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 55034--30044 2023.
[44] Shaokai Wu and Fengyu Yang. Boosting detection in crowd analysis via underutilized output features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15609-
15618, 2023.
[45] Wenhao Wu, Dongliang He, Tianwei Lin, Fu Li, Chuang Gan, and Errui Ding. Myfnet: Multi-view fusion network for efficient video recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 2943-2993022 2951, 2021.
[46] Wenhao Wu, Yuxin Song. Zhun Sun, Jingdong Wang, Chang Xu, and Wanli Ouyang. What can simple arithmetic operations do for temporal modeling? In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2023.
[47] Wenhao Wu, Zhun Sun, and Wanli Ouyang. Revisiting classifier: Transferring vision-language models for video recognition. In Proceedings of the AAAI conference artificial intelligence, volume 37, pages 2847-2855, 2023.
[48] Wenhao Wu, Xiaohan Wang, Haipeng Luo, Jingdong Wang, Yi Yang, and Wanli Ouyang. Bidirectional cross- modal knowledge exploration for video recognition with pre-trained vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6620-6630.
2023.
[49] Wangmeng Xiang, Chao Li, Yuxuan Zhou, Biao Wang, and Lei Zhang. Generative action description prompts for skeleton-based action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10276-10285, 2023.
[50] Shen Yan, Xuehan Xiong, Anurag Arnab, Zhichao Lu, Mi Zhang, Chen Sun, and Cordelia Schmid. Multiview transformers for video recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3333-3343, 2022.
[51] Hao Yang, Liyuan Pan, Yan Yang, Richard Hartley, and Miaomiao Liu. Ldp: Language-driven dual-pixel image defocus deblurring network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24078 24087, 2024.
[52] Jiewen Yang, Xingbo Dong, Liujun Liu, Chao Zhang. Jiajun Shen, and Dahai Yu. Recurring the transformer for video action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 14063 14073, 2022.
[53] Taojiannan Yang, Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, and Mu Li. AIM: Adapting image models for efficient video action recognition. In The Eleventh International Conference on Learning Representations. 2023.
[54] Zonglin Yang. Yan Yang. Yuheng Shi, Hao Yang, Ruikun Zhang, Liu Liu, Xinxiao Wu, and Liyuan Pan. Event-based few- fine-grained human action recognition. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pages 519 526. IEEE, 2024.
[55] Ruikun Zhang, Zhiyuan Yang. and Liyuan Pan. Dehazemamba: large multi-modal model guided single image dehazing via mamba. Visual Intelligence, 3(1):11, 2025.
[56] Wei Zhang. Chaoqun Wan, Tongliang Liu, Xinmei Tian, Xu Shen, and Jieping Ye. Enhanced motion-text alignment for image-to- toiiccoo transfer learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition. pages
2024.
[57] Yue Zhao, Ishan Misra, Philipp Krähenbühl, and Rohit Girdhar. Learning video representations from large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6586-6597, 2023.
[58] Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos.
In Proceedings of the European conference on computer vision (ECCV), pages 803-818, 2018.
[59] Xiaoyu Zhu, Po- Yao Huang, Junwei Liang, Celso M de Melo, and Alexander G Hauptmann. Stmt: teatial-temporal mesh transformer for mocap-base based action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1526-1536, 2023.
Xiii

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The paper has a clear abstract and introduction section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses the limitations the future work section included the supplementary material.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
xiv

---

Justification:
This work does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We state experimental details in Experiment section in the main paper and Supplementary Material A. We state which datasets used and provide references.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
XV

---

Answer:
[Yes]
Justification:
reproducible, they the paper is accepted, the code and data will released.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
public/guides. htte///ooaaaaaiiinnnonooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
/ //nip. httpcc/cpplllggguuueee
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We state all experimental details in Experiment section in the main paper and Supplementary Material A.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
xvi

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https:/ hat::://neuriiee httpr//pp.../pblllceccccccceeeee
Answer: [Yes]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinfornation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
Xvii

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ com/datasete has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
xviii

---

Researchers should communicate the details of the dataset/codeemmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
xix

---

Supplementary Material
We provide content omitted from the main paper due to space limitations. Specifically, the sup- plementary material is organized into three sections: (i) additional details for experiments and implementations, (ii) comparisons with other state-of ofte-of-theear methods in various settings, and (iii) additional ablation studies to analyze our model designs.
A
Additional Experimental Setup
A.1
Datasets
We describe the dataset statistics of the Kinetics-400 [7]. HMDB-51 [9). UCF-101 [ and Charades (33] datasets. The Kinetics-400 dataset is curated from YouTube, spans 400 action classes, and contains 240,000 training videos and 20,000 validation videos. The duration of each video 10 seconds. The HMDB-51 dataset consists of 6,766 videos categorized into 51 action classes, with 3,570 videos used for training, and 1,530 videos used for testing. Each video is sourced and truncated from movies, online videos, and public databases. The UCF- dataset collected 9,537 training videos and 3,783 testing videos from YouTube, across 101 action categories. The Charades dataset comprises 9,848 video clips that span 157 action categories. Each video is recorded in home environment, performed by an actor, and includes multiple human actions with temporal action annotations.
Table The hyperparameters of our model for supervised and few-shot learning.
Fully-sup
Few-shot
Optimisation
Optimizer
AdamW (0.9, 0.999)
Optimizer betas
Batch size
256
256
Learning rate schedule
cosine 5
Linear warmup epochs
Base learning rate
5e-6 30
5e-6
Epochs
2,10
Data augmentation RandomFlip
0.5 RandomSizedCrop 224(Default) 0.2
Resize
Crop size
GrayScale
Other regularisation
Weight decay
0.02
A.2 Ambiguous Score vs. Non-uniform Score
We describe computations the ambiguous score and non non-uniform score for Fig. in the main paper. Our method does not require ground truth action labels of videos, and only uses the CLIP, for the calculation.
We measure the ambiguous scores of videos by their class similarity measured with the textual encoder of CLIP. Benefiting from training on large-scale image and text pairs, the embedding space of CLIP is compression of the world that captures the similarity between action classes. Following the global text prompt from the main paper, action classes are extracted into [tel) Tc=1 with the textual encoder of CLIP. The similarity dc.e' between action class and prediction is calculated and normalized,
Ambiguous Score.
sim(tek cc tsh)) +
Oe,e
(15)
XX

---

Table 6: The question prompts for GPT-Davinci (9) GPT-3.5, GPT-4o, LLaMA2 [36. Phi-3 M. and LLava- vl.6-34b 122). Each row a question prompt. All LLMs use the four prompts to generate sub-texts for selection.
[This action a sequence of four steps. For each step, describe it in detail with 'This action is' followed by a specific description of that step. Ensure each step is distinct and crucial to the action.]
(This action is defined by four unique movements. Start each movement with 'This action is and emphasize the diversity and significance of each movement. Conclude with a brief summary that ties all movements together.]
[This action composed of four critical phases. Each phase should start with 'This action is' and focus different aspects of the action. Highlight the importance of each phase and avoid repetition.]
[This action is best understood by breaking it down into four concise descriptions. Begin each description with 'This action is' and make sure each one covers a different perspective of the action. End with a sunmarizing statement.]
Table 7: The system prompt. We the system prompt for GPT-Davvnci 4. GPT-3.5, GPT-4o, LLMM2 36), Phi-3 [ and LLava-vl.6-44k [2)
[You are an expert assistant in action recognition. Respond to the user's input by providing accurate, concise, and informative descriptions of the actions in structured manner.]
For each action class c, we select the top-10 most similar other actions, and get the ambiguous scores ASe by averaging the similarity scores,
AS.
Jc,''
(16)
d'Etop-10,
where top-10, find index of the top-10 similarity action class for the action class c. In Fig. 2 of the main paper, we normalize the similarity score AS. from 1 (least ambiguous) to 10 (most ambiguous) for visualization.
Ambiguous action classes usually share different atomic actions described by the sub-texts. Thus, using the global text to identify prominent frames corresponding to the video class semantic causes ambiguity. While avoiding granularity discrepancy between the global text and video frame (Sec. Intro), we aim to decrease the granularity of the global text by augmenting it with sub-texts for ambiguous video recognition. On the VřT-B/16 backbone, we found that augmented global text achieved an accuracy of 84.5% on the Ixl view, which 0.5% improvement over the 84.0% of global text, when the fine-graine branch is included.
Non-uniform Score. We measure the non-uniform score of video by calculating the agreements between predictions of each frame. Leveraging a CLIP, we obtain predictions of each frame and use the global text format and textual encoder of CLIP extract embedding [t for all frames. The agreements between frame 1 and frame I' are measured by similarity between the embedding, and are normalized,
sim tlls good
tels (ned yr) +1
01,1'
(17)
2
The non-uniform score NUS the video is an average of the agreements between frames,
NUS = Σ Σ orr
(18)
1-1 I'=.''
In Fig. of the main paper, we normalize the non-uniform score from 1 (uniform) to (non-uniform) for visualization.
xxi

---

Table 8: Performance comparison of course, fine-grained, and fused video embeddings over single- granularity method, across different levels of ambiguous and non con-uniform scores, on Kinetics-400 dataset. "Ambi" and "Non" represent ambiguous score and non-uniform score, respectively. All numbers are Top-1 accuracy (%))
Degree
Method
Type
2
3
4
5
7
8
9
10
Ambi
93.0
90.3
88.3
84.0
85.7
82.7
83.0
76.8
77.2
69.3
Single
Non
86.5
84.9
84.6
85.4
83.9
82.2
80.8
81.9
80.3
80.1
Ambi
94.1
91.5
89.8
86.5
87.4
84.8
84.2
78.9
79.4
70.8
Coarse
Non
91.1
88.7
87.8
90.7
86.7
85.0
83.6
84.7
83.1
82.9
Ambi
94.2
91.1
89.5
86.4
87.3
84.7
84.1
78.6
79.1
71.8
Fine
Non
90.0
87.8
86.6
87.2
86.3
83,9
82.9
84.0
82.4
82.2
Ambi
94.8
91.6
90.3
87.2
87.6
85.6
84.7
79.3
80.6
72.1
Fusion
Non
92.9
89.7
89.7
88.9
87.7
85.1
83.7
84.8
83.2
83.0
We construct two challenging subsets of the Kinetics-400 dataset based on the ambiguous score and non-uniform score. For each criterion, we select the top 10% of actions with the high- est scores, representing the most difficult cate- gories to classify. We compare the performance of our proposed method against two representa- tive single-gen graell-gaaularrty methods ATM [46] and BIKE [8). The results the two challenging subsets are presented Tab.
Table 9: Comparison of our method, BIKE (48) and ATM [40] on two challenging subsets of Kinetics-400: "Ambi Set" and "Non Set", contuining the top 10% most ambiguous and non uniform actions, respectively.
Method
Ambi Set
Non Set
ATM VIT-B/16 46
70.2
79.8
BIKE ViT-B/16 448
69.3
80.1
Ours VIT-B/16
72.2
83.0
Fig. (f) presents bar chart illustrating the accuracy improvements of our method's coarse video embedding, fine-grained video embedding, and their fusion, compared to the single-granularity method. The results presented in Tab. 8 are reported across different levels of ambiguous score and non-uniform score, showcasing the effectiveness of our method on the Kinetics-400 dataset.
A.3 Implementation Details
We CLIP trained from [30] our paper. During training, the textual encoder of CLIP is frozen. We summarize the optimization and data augmentation details of our method for supervised learning
Table 10: Comparisons zero-shot action recognition on the UCF-101 34 and HMDR-51 [9 datasets.
"VZ" column denotes the method developed for zeru-shot action recognition or adapted from CLIP. Our
model is bused on VIT-L/14.
Method
VZ
UCF-101
HMDB-51
44.1
29.8
ER 5
51.8
35.3
ResT[20
58.7
41.1
X-CLIP 26
72.0
44.6
DIST 29
72.3
55,4
Vita-CLIP
4
75.0
48.6
M2-CLIP
38
78.7
47.1
X
BIKE ViT-L
86.6
61.4
Tixt4Vis
47
85.8
58.1
Ours ViT-L/14
87.3
61.9
xxii

---

Table 11: Comparisons results the Kinetics-400 dataset. We report the FLOPss in inference phase. "Views" indicates temporal clip spatial crop. The magnitude Million (10%) (110))ooo for parameters (Param). methods are based on VIT-L.
Method
Venue
Input
Pre-training
Top-1(5)
Top-S( %)
Views
GFLOPs
Param
VIVIT-L/16X2
ICCV'21
32x320'
ImageNet-2IK
81.3
94.7
4x3
3992x12
310.8
VIVIT-116622
ICCV'21
32x320
JFT-300M
83.5
4x3
VideoSwin-L
23
CVPR'22
32x3842
ImageNet-21K
84,9
96.7
10x5
2107x50
200.0
ST-Adapter 1/14
28
NeurlPS'22
32x224
WTT-400M
87.2
97.6
3x1
8248
EVI VIT-1/14
21
ECCV122
32x224
WTT-400M
87.3
3x1
80N8
BIKE VIT-L/14
CVPR'23
8x224
WIT-400M
86.5
1xl
307
AIM VIT-L/114
S3
ICLR'23
8x224
WIT-400M
86.8
97.2
3x1
2802x1
341
ATM VTT-1/14
46
ICCV'23
Xx224
WTT-400M
87.3
97.4
4x3
421x12
MoTED VIT-L/14
[5G
CVPR'24
8x224
WIT-400M
87.4
97.8
8670
349
Ours ViT-L/14
25urIPS'25
8x2244
W0T-400M
87.6
97.8
1x1
416
312
Table 12: Comparisons on few-shot action recognition across the HMDB-51 51 [19]. UCF-101 34 and Kinetics
400 datasets (17).
Method
Shot
HMDB-51
UCF-101
Kinetics-400
VideoSwin
23
2
20.9
53.3
VideoPrompt ViT-B/16 15
5
56.6
79.5
58.5
BIKE ViT-L/14 48
2
73.5
96.1
75.7
BIKE ViT-L/14 48
5
77.7
96.5
78.2
OST ViT-B/16
66
2
64.8
90.3
Ours ViT-L/14
74.4
96.5
76.5
Ours ViT-L/14
5
78.1
96.9
79.1
and few-short learning in Tab. 5 For calculating TPP, the scaling functions are a(x) = -1² 2 + and (xx) = x, where x the input variable. In sub-text generation, we carefully design a list of question prompts for LLMs and select sub-texts with our TPP. We consider six LLMs: GPT-Davinci (. GPT-3.5, GPT-4o, LLava-vl .6-34b [22, LLaMA2 (36), and Phi-3 [. The question prompts and the system prompt for each LLM are presented in Tab, and Tab. respectively.
B Additional Experimental Results
We compare with state-of-the-are methods with ViT-L/14 backbone. Following the setting of the main paper, the results of zero-shot action recognition on the UCF-101 (34] and HMDB-51 [19
datasets, supervised learning the Kinetics-400 (17) dataset are in Tab. TO Tab. In all settings, our method consistently exhibits the highest performance.
As shown in Tab. B compared the impact of the generative capa- bilities of other large language mod- els (LLMs) on recognition accuracy, with corresponding GLops, process- ing time, and accuracy. The results show that even the slowest generation speed within 10 seconds of each ac- tion. Furthermore, LLMs with strong generative capabilities have minimal impact on our method.
Table 13: Performance comparison of Phi-3, LLaVA, LLaMA2, GPT-3.5, and GPT-4o in terms of GLOPs, processing time, and accuracy.
Model
GLOPs
Time(s)
Accuracy
Phi-3-148 14b
636
7.5
84.50
(LLVA-1.6-344
8.25e4
5.8
84.35
LLaMA2-7b
814
2.0
84.47
GPT-3.5
1.1
84.47
GPT-4o
4.6
84.52
The common practice [40] followed
to only add FFN layers the video encoder, which projects the video embedding the CLIP feature space. is unnecessary to employ an additional FFN layer for the text embedding, as the accuracy remains 84.5% with without
We explored the use visual context generating sub-texts and identified two major limitations of this approach. i) Generating sub-texts for each video sample is computationally expensive, while
xxiii

---

Table 14: Sub-text descriptions for the action "Situp" across multiple video samples.
Action: Situp Description
Description 2
Description
Description
Video1
The individual is lying on a mat a gym, beginning a sit-up with arms extended overhead.
They perform The exercise is fluid motion, rais-
The environment suggests well- equipped gym, ap- propriate for vari- ous fitness activi- ties.
conducted in a controlled manner, focusing on core strength and stabil- ity.
ing their torso to- wards their legs while keeping their feet flat.
Video2
An individual is seen lying on the grass performing sit-up exercise in outdoor setting.
The exercise involves raising the upper body towards the knees, engaging the core muscles.
The person is The green, open wearing casual workout attire,
space offers nat- ural and refreshing environment for fitness routine.
suitable for out- door physical activities.
Video3
Two individuals are in gym; one is performing an exercise on the floor while the other is walking around, possibly coaching.
The environment indicates focus on personal train- ing, as evidenced by the sign in the background.
The person on the floor appears be
The overall scene suggests casual yet focused atmo- sphere, typical of fitmess or personal
doing sit-ups or a similar core exer- cise, suggesting a targeted workout session.
training session.
class-level generation requires choosing representative video, risking inconsistency and limited coverage. ii) For complex or ambiguous actions, visual context often introduces noise and spurious details, weakening frame alignment and degrading performance. To illustrate, take 'Situp' as example, we randomly selected three sample videos and uniformly sampled eight frames from each. These were provided as input to GPT-4o to generate visually informed sub-texts shown in Tab. [4] These sub-texts introduce scene-specific elements (e.g., attire, environment, presence of others) that do not directly contribute to the representation of atomic actions and confound alignment.
(a) Drinking
GPT3.5
LLaMA2
Phi-3
Selecting
Holding a
Pick up the glass.
Pouriag drink.
Bringing the drink mouth.
glass your mouth.
Reising the drink the lipe.
Sipping the drink.
Allow the liquid to flow into your mourt.
4. Swallowing the drink.
4. Swallowing the drink.
4. Swallow the liguid.
(b) Eating chips
GPT3.5
ILaMA2
Phi-3
Selecting chips.
Pirking up chip.
Picking up chip.
2. Opening hag.
Bringing the chip the mouth.
Bringing the chip to your mooth.
Picking chips fingers.
Biing
Biting
4. Putting the mouth chewing.
Chewing chip.
4. Chewing and swallowing chip.
Figure 8: Example sub-texts generated from GPT-3.5, LLaMA2 (36), and Phi-3 [Z].
C Additional Ablation Studies
LLMss Selection. We ablate the LLM used for sub-text generation in Tab. 7 We experiment with GPT-Davinci 4, GPT-3.5, GPT-4o, and LLava-vl..C 16--44b [22]. Sub-text groups generated from GPT-4o have the highest TPP and action recognition performance.
xxiv

---

Visualization. Additional example sub-texts generated by GPT-3.5, LLaMA2 [36], and Phi-3 []] are presented in Fig. 8 Furthermore, We provide more visualizations than Fig. of the main paper in Fig. 9 and Fig. on Kinetics-400
17 UCF-101
and HMDB-51
T9 datasets.
Linitation and Future Work Fine-grained action recognition is challenging. especially in complex scenarios, with no satisfactory solution even from large vision-language models like GPT-4v. Our method mitigates this by generating sub-texts once for each global action, but future work could explore more robust approaches.
Broader Impacts Our work advances the study of fine grained action recognition methods and holds promise for a wide range of applications, including sports analytics, human-compute computer interaction, surveillance, video understanding, and related areas.
XXV

---

Making tea
Streching arm
Hulahooping
Swinglegs
Washdish
Shoot
basketball
(a) Frame
(b) Single
(c) Coarse
(d) Fine
(e) Coarse+Fine
Figure 9: Examples that illustrate the ineffectiveness of directly aligning video frames and global text. from (a-e) heatmaps. (a) Video frames from the Kinetics-400 dataset [17] with their class name annotated at the left. (b) Heatmaps of the single granularity based (single) method (48), which are limited by the granularity difference between the video frame and global (c-e) Heatmaps using coarse video embedding, fine- tra-iga video embedding, and the fusion of the two.
Xxvi

---

Brush Teeth
Biking
Lunges
Walk
Situp
Punch
(a) Frame
(b) Single
(c) Coarse
(d) Fine
(e) Coarse+Fine
Figure 10: Examples that illustrate the ineffectiveness of directly aligning video frames and global text, from (a-e) heatmaps. (a) Video frames from the UCF- 101 dataset (34] and HMDB-51 dataset [19] with their class name annotated the left. (b) Heatmaps of single gromularity-ban based (single) method (48), which are limited by the granularity difference between the video frame and global text. (c-e) Heatmaps using coarse video embedding, braiiedd video embedding, and the fusion of the two.
xxvii


Reviews:
[{"review_id": "GUPFotjM6e", "reviewer": "Reviewer_LH8r", "summary": "This paper introduces Storyboard-guided Fine-grained Action Recognition (SFAR), a multi-granularity framework for fine-grained video action recognition. \nThe key innovation is using LLMs to decompose class-level “global” action labels into multiple atomic sub-actions. Then, these sub-actions are used as fine-grained prompts to identify and aggregate class-discriminative frames using CLIP’s multimodal embedding space. Finally, the method fuses both coarse and fine-grained video embeddings for classification. \nEmpirical results on various datasets demonstrate improvements over state-of-the-art in supervised, few-shot, and zero-shot settings.", "questions": "1. Regarding the fixed number of sub-actions (N): In your experiments where you varied the fixed number of sub-actions (N), did you observe that the accuracy for different action categories changed differently? In other words, are there specific action classes that benefit from a larger or smaller N, and how does this variation affect overall recognition performance?\n\n2. Lack of Sequential Diversity in Sub-action Decomposition: Since the method produces only a single sub-action sequence per action, how does it handle cases where the same coarse action can be performed with different valid temporal orderings?\n\n3. Absence of Temporal Modeling: Given that your framework does not explicitly model temporal dependencies, how does it capture critical temporal dynamics, especially for actions where the order of sub-actions is essential?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1.\tClear Motivation. Clearly articulates the limitations of prior CLIP-based and vision-language models in capturing atomic-level video actions, with strong illustrative examples for both ambiguous and non-uniform action distributions.\n\n2.\tMulti-Granularity Embedding: Presents a novel framework for fusing coarse (global augmented with sub-texts) and fine-grained (atomic sub-text-aligned) video embeddings. The network design matches with the motivation, targeting the granularity mismatch problem.\n\nWeakness:\n\n1. Fixed Number of Sub-action for various coarse action: The method sets a fixed number of sub-actions for all action classes, regardless of action complexity or video duration. This one-size-fits-all approach is suboptimal, as some simple actions require only a single atomic step, while complex actions in longer videos may need a finer and variable decomposition.\n\n2. Lack of Sequential Diversity in Sub-action Decomposition: The approach generates a single, fixed sub-action sequence per coarse action, overlooking that the same action can correspond to multiple valid temporal orderings of sub-actions. This limits the model's ability to accommodate intra-class variability in action execution.\n\n3. Absence of Temporal Modeling: The framework does not explicitly model temporal relationships or sequence dependencies between video frames. Treating videos as unordered frame sets may miss important temporal dynamics necessary for accurate action recognition.", "was_revised": true, "final_justification": "Thanks for the author's response, it resolved my almost concerns, I will keep my positive rating for \"Borderline accept\".\nFor the fixed number N of sub-actions ablations, it is better to be considered in the final version, as it is an important discussion for the reproducibility and motivation for the design.", "rebuttal": "We appreciate your insightful comments and your recognition of our novel multi-granularity framework. We address your concerns point by point below.\n> **Q1: Fixed Number of Sub-Actions (N).**\n\n**A1.** We previously observed that the accuracy of different action categories changes with respect to the number of sub-actions (N). However, we set a fixed N, rather than using a class-aware N. The reason is given below.\n\nTaking N = 2 and N = 5 as examples, the accuracy differences across representative actions are given below.\n\n| Action                   | N=2  | N=5  | Difference | Action                          | N=2  | N=5  | Difference |\n|--------------------------|------|------|------------|----------------------------------|------|------|------------|\n| Dribbling basketball     | 89.2 | 94.6 | +5.4       | Arm wrestling                   | 100.0| 100.0| 0.0        |\n| Kicking soccer ball      | 71.0 | 76.3 | +5.3       | Decorating the Christmas tree   | 100.0| 100.0| 0.0        |\n| Lunge                    | 75.6 | 82.2 | +6.6       | Brushing teeth                  | 89.5 | 89.5 | 0.0        |\n| High kick                | 35.1 | 40.5 | +5.4       | Slicing onion                   | 98.6 | 97.4 | -1.2       |\n| Clay pottery making      | 92.1 | 94.7 | +2.6       | Clapping                        | 47.2 | 50.0 | -2.8       |\n\nThe results indicate that complex, multi-stage action categories (e.g., 'Dribbling basketball' and 'Clay pottery making') benefit from large N, whereas repetitive or simple motions such as 'Clapping' and 'Slicing onion' get better accuracies with small N.\n\nWe had investigated overall performance gains by using class-aware N, and only found marginal improvement over fixed N=5, 0.5\\% and 0.2\\% for Top-1 and Top-5, respectively.\n\nGiven the computational cost of involving a class-aware value of N for each action, we choose to use fixed N for efficiency. \n\n> **Q2: Sequential Diversity in Sub-Action Decomposition.**\n\n**A2.** While our method adopts a fixed sub-action sequence for each coarse action, it does not rely on this ordering for accurate recognition. In fact, our model is designed to remain robust to different valid temporal arrangements of sub-actions through the use of temporally agnostic attention mechanisms.\n\nSpecifically, the fine-grained importance score $a\\_{\\text{l}}^\\text{fine}$ is computed independently for each frame $\\mathbf{v}\\_{\\text{l}}^\\text{cls}$ by measuring its maximum average similarity with all sub-texts $\\mathbf{S}\\_{\\text{c}, \\text{n}}$. This design allows the model to capture fine-grained relevance without relying on a fixed temporal sequence of atomic actions. Similarly, the coarse-grained importance score $a\\_{\\text{l}}^\\text{coarse}$ leverages the augmented global text embedding $\\hat{\\mathbf{T}}\\_{\\text{c}}$, which is constructed to be invariant to sub-action order. Together, these components ensure that both local and global relevance signals are captured without enforcing a fixed temporal sequence.\n\nTo validate this, we randomly shuffled the order of the sub-texts and observed that the top-5 accuracy remained unchanged, while the top-1 accuracy decreased marginally by only 0.01.\n\n> **Q3: Absence of Temporal Modeling.**\n\n**A3.** To address the absence of explicit temporal modeling, we construct a baseline (Temporal) using a 6-layer Transformer encoder to capture sequence dependencies among video frames (Line 300, main paper). To effectively model critical temporal dynamics, especially for actions reliant on sub-action ordering, we compute fine-grained and coarse-grained importance scores $a\\_{\\text{l}}^\\text{fine}$ and $a\\_{\\text{l}}^\\text{coarse}$, respectively, which are adaptively aligned with sub-action semantics. These scores modulate the corresponding frame embeddings $\\mathbf{v}\\_{\\text{l}}^\\text{cls}$ without disrupting the video’s inherent temporal structure. The resulting weighted embeddings are then aggregated and fused via two feedforward layers (Eq. 13), enabling the model to integrate multi-granularity information and produce a temporally adaptive video-level representation."}, {"review_id": "oVqJCxx7T4", "reviewer": "Reviewer_VmYv", "summary": "The paper proposes SFAR, a multi-granularity framework for fine-grained video action recognition. By generating atomic-level sub-texts using large language models, it improves video-text alignment and achieves strong performance across supervised, zero-shot, and few-shot settings.", "questions": "See weakness.", "limitations": "The authors mentioned the limitations in the supplementary material.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n1. The paper is generally well-written and easy to follow. \n2. The issue of granularity mismatch between videos and texts is well recognized and analyzed. The proposed SFAR seems to be effective on this issue.\n3. Extensive experiments across multiple benchmarks (supervised, few-shot, and zero-shot) show strong quantitative results.\n\n**Weaknesses**\n1. The fine-grained video embedding relies on sub-texts generated by an LLM based solely on the class name, without incorporating any visual context. However, these sub-texts may miss essential atomic actions or include redundant or vague descriptions, which can limit the effectiveness of frame-level alignment.\n\n2. Both the coarse and fine-grained embeddings are derived from the same frame-level features. While they are guided by different text features, they may capture overlapping information, resulting in redundant representations. This raises concerns about whether the added architectural complexity of dual embeddings is always justified by meaningful performance gains.\n\n3. The proposed sub-text generation strategy is well-motivated; however, including more detailed ablations—such as comparisons with random sub-texts or alternative prompt templates—would better highlight the effectiveness of the TPP-based selection process.", "was_revised": true, "final_justification": "My concerns have been addressed. I keep my rating at \"Borderline accept\"", "rebuttal": "Thank you for your insightful comments and for acknowledging that our paper is generally well-written and easy to follow. Below, we address your concerns point by point.\n\n> **W1: LLM-Generated Sub-texts Without Visual Context.**\n\n**A1.** We did explore the use of visual context previously and found two major limitations for this solution. \n\n1. Generating sub-texts for each individual video sample entails significant computational and time costs. Alternatively, generating sub-texts at the class level introduces the challenge of selecting a representative video, which can affect consistency and coverage across diverse instances. \n\n2. For ambiguous, non-uniform, or contextually intricate actions, incorporating visual context yields sub-texts dominated by noisy or spurious information. This undermines frame-level alignment and degrades model performance.\n\nTo illustrate, take 'Situp' as an example, we randomly selected three sample videos and uniformly sampled eight frames from each. These were provided as input to GPT-4o to generate visually informed sub-texts:\n\n|      'Situp'   |    |      |     |        |\n|--------------------------|------|------|------|------|\n| Video1 |He individual is lying on a mat in a gym, beginning a sit-up with arms extended overhead.|They perform a fluid motion, raising their torso towards their legs while keeping their feet flat.|The exercise is conducted in a controlled manner, focusing on core strength and stability.|The environment suggests a well-equipped gym, appropriate for various fitness activities.|\n| Video2 |An individual is seen lying on the grass performing a sit-up exercise in an outdoor setting.|The exercise involves raising the upper body towards the knees, engaging the core muscles.|The person is wearing casual workout attire, suitable for outdoor physical activities.|The green, open space offers a natural and refreshing environment for a fitness routine.|\n| Video3 |Two individuals are in a gym; one is performing an exercise on the floor while the other is walking around, possibly coaching.|The environment indicates a focus on personal training, as evidenced by the sign in the background.|The person on the floor appears to be doing sit-ups or a similar core exercise, suggesting a targeted workout session.|The overall scene suggests a casual yet focused atmosphere, typical of a fitness or personal training session.| \n\nThese sub-texts introduce scene-specific elements (e.g., attire, environment, presence of others) that do not directly contribute to the representation of atomic actions and can confound alignment.\n\nIn contrast, our proposed sub-text generation method is guided by the Text Prompt Perplexity (TPP) score, which promotes relevance by emphasizing informative atomic actions. To further minimize ambiguity and redundancy, we employ an attention-based alignment mechanism (Eqs. (6–8) and (11)) that assigns lower weights to less informative content, yielding sub-texts that are both precise and semantically aligned with frame-level dynamics.\n\n> **W2: Dual Embeddings.**\n\n**A2.** We demonstrate the complementary nature of the coarse and fine-grained embeddings through corresponding heatmaps. As visualized in Fig. 2, the attention maps for coarse, fine, and their combination show distinct focus regions across three examples: 'Making a sandwich', 'Pull up', and 'Shearing sheep'. \nFor instance, in 'Shearing sheep', the coarse branch (Fig. 2c) primarily attends to the head of the sheep and the area where shearing occurs on the right, while the fine branch (Fig. 2d) highlights the body of the sheep, the shearing tool, and the wool falling to the ground. The combined attention (Fig. 2e) effectively integrates both perspectives.\n\nThese visualizations confirm that coarse and fine embeddings capture different aspects of the video, thus mitigating redundancy and supporting the effectiveness of the dual-embedding design. Quantitative results are shown in Tab. 2e.\n\n> **W3: Ablations of Alternative Prompt Templates.**\n\n**A3.** Please kindly refer to Fig. 7b of the main paper, where we present the relationship between TPP scores and the corresponding action recognition accuracy of sub-texts generated using alternative prompt templates. Representative examples of these templates are provided in Tab. 5 of the supplementary material. These results clearly demonstrate that prompts yielding higher TPP scores also lead to superior recognition performance, validating the effectiveness of our TPP-based selection strategy. We will highlight these comparisons more explicitly in the revised version for added clarity."}, {"review_id": "EPGUSxqTwD", "reviewer": "Reviewer_DdM6", "summary": "This paper proposes a multi-granularity framework SFAR for fine-grained video action recognition to further explore the potential of text encoders in adapting to the confusion and unevenness of action atomic. The core theory lies in decomposing action text into a sub-text set with LLM at a fine-grained level, and using this text set to enhance visual features at multiple granularities (i.e., coarse level and fine-grain level), and finally fusing the features to predict action categories. Rich experiments on multiple benchmarks and multiple tasks have demonstrated the effectiveness of the proposed method.", "questions": "Although this article has designed an effective framework for fine-grained video action recognition, it suffers from deficiencies (See Weaknesses) in written expression and theoretical analysis, etc., my current rating is 3.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths：\n1. The author conducted in-depth research on the characteristics of fine-grained actions and analyzed the corresponding issues. The proposal of sub-text generation for fine-grained video action recognition is insightful.\n2. The design of the text prompt perplexity score seems effective and offers a new approach for the correlation analysis between the text generated by LLMs and video information.\n3. The ablation experiment design is diverse, and extensive and effective validation has been conducted for the core components in the text.\n\nWeaknesses:\n1. I find the writing of this paper flawed, as the authors did not carefully check and optimize the content before submission. (1) The figure captions are overly verbose (even containing extensive symbolic formulas), making it difficult for readers to grasp the key theoretical points. (2) There is redundancy between the textual descriptions and equations in the theoretical presentation, such as in Lines 140-144 and Lines 193-197. (3) The use of embedded formatting for multiple figures creates a sense of disorganization for readers.\n2. The limitations of the text prompt perplexity score. The author did not provide specific analysis and explanations for the design of the text prompt perplexity score. For example, what were the design motivations for each factor in Equation 3? What is the theoretical impact of the values on the quality of text features? How was TTP specifically utilized during the enhancement process?\n3. The generated sub-text remains in the form of verb-noun. However, the encoder of CLIP focuses on static images. How can SFAR ensure the adaptability of the generated text to the dynamic video information?", "was_revised": true, "final_justification": "In general, this paper is valuable and promotes the research in this field. Considering the positive opinions of other reviewers, I choose to raise my rating to Borderline accept.", "rebuttal": "Thank you for your valuable comments and for recognizing that our ablation experiment design is diverse. Below, we address your concerns point by point.\n\n> **W1: Content Optimization.**\n\n**A1.** (1) We intentionally designed captions to be comprehensive and self-contained, ensuring symbolic formulas and context are immediately accessible without requiring readers to cross-reference text sections; (2) Thanks for the catch and fixed; (3) We will try to leave more room for figures.\n\n> **W2: Motivations of Text Prompt Perplexity Score.**\n\n**A2.** The Text Prompt Perplexity (TPP) score (Eq. (3) in the main paper) is designed with two key factors: similarity with global text and diversity among sub-texts. \n\\begin{align}\n    &\\text{TPP}\\_\\text{c} = \\exp \\bigg( - \\frac{1}{N} \\sum\\_{\\text{n}=1}^{N} \\log \\big( \\alpha (\\sigma\\_{\\text{c},\\text{n}} ) \\beta(\\delta\\_{\\text{c},\\text{n}}) \\big) \\bigg) \\ .\n    \\tag{3}\n\\end{align}\nFirst, we employ $\\sigma\\_{\\text{c},\\text{n}}$ to quantify the similarity between a global text and its corresponding sub-texts, a higher value of $\\sigma\\_{\\text{c},\\text{n}}$ indicates greater similarity. Next, $\\delta\\_{\\text{c},\\text{n}}$ is used to evaluate the similarity among sub-texts themselves, where a higher value of $\\delta\\_{\\text{c},\\text{n}}$ indicates greater diversity across the sub-texts. The linear scaling functions $\\alpha(\\cdot)$ and $\\beta(\\cdot)$ ensure that generated sub-texts with higher similarity and diversity scores receive higher TPP scores.\n\n> **W3: Video Alignment.**\n\n**A3.** Although the sub-texts generated by SFAR adopt a verb-noun structure, they are not bound to static interpretations of a single frame. As described in Eq. (11), SFAR computes similarities between each sub-text and individual frames within the CLIP embedding space, enabling the identification of frame-level visual evidence corresponding to atomic actions. These interactions are then aggregated through temporal pooling (Eq. (12)), which captures inter-frame dependencies and facilitates the flow of contextual information over time. Additionally, SFAR incorporates a 6-layer Transformer module within its visual encoder to explicitly model the temporal progression of dynamic content. This further enriches the representation of motion and continuity across the video sequence.\n\nThrough these mechanisms, SFAR effectively ensures the adaptability of the generated text to the dynamic video information."}, {"review_id": "a3BDAe83iw", "reviewer": "Reviewer_ys6V", "summary": "Fine-grained video understanding is a challenging problem. The authors aim to address this issue through a multi-granularity framework. Specifically, they propose storyboard-guided fine-grained video understanding  (SFAR), which generates  fine-grained (sub-text)  detailed descriptions of common atomic actions corresponding to each global semantic, leveraging large language models. Experiments were conducted on various video action recognition datasets to demonstrate the performance of the approach under different settings, including supervised, few-shot, and zero-shot scenarios.", "questions": "1). The main issue with the paper is that the overall contribution is incremental due to limited technical novelty. Although the authors clearly specify their primary contributions compared to existing approaches, the emphasis on the novelty of their work remains insufficient.\n\n2). In the Global Text Augmentation module, both fine-grained (sub-text) and global context are utilized. However, the authors refer to this as global text augmentation. Could you clarify why it is named “global” when it also incorporates fine-grained elements?\n\n3). The formulation of Fine-grained Importance and Coarse-grained Importance is not clear. This part requires a more detailed analysis and thorough discussion, especially since it represents the main contribution of the paper.\n\n4). Some sections of the writing need significant improvement to provide clearer explanations. For example, abbreviations should be introduced before they are used. The captions of the figures are too long and need to be clearer and more focused. Detailed information from the captions can be included in the body or explanation section instead.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The proposed approach addresses fine-grained video understanding through a multi-granularity framework. Specifically, the authors employ a multimodal approach that integrates coarse-grained and fine-grained representative feature learning. The problem formulation of this work is interesting, it aims to learn and understand fine-grained video or action recognition by utilizing both coarse-grained and fine-grained multimodal features. In the textual semantic branch, they use global semantic learning to capture coarse-grained features. Similarly, to learn fine-grained action features, the authors generate sub-action texts. These learned features are optimized jointly with the visual branch, encompassing both coarse-grained and fine-grained action features.\n\nHowever, the novelty of the proposed approach is limited. Similar methods have been explored previously, which reduces the overall contribution of this work. For a top-tier conference like NeurIP, a more significant novel contribution is required.", "was_revised": true, "final_justification": "Considering the clarifications provided by the authors during the rebuttal period, some of my concerns and questions have been addressed. Thus, I am inclined to increase my score to borderline accept.", "rebuttal": "Thank you for your comments and for recognizing that the problem formulation of this work is interesting. Below, we address your concerns point by point.\n\n> **Q1: Novelty.**\n\n**A1.** Please refer to L8–14 of the abstract and L31-64 of the introduction section. 'We propose a multi-granularity framework, SFAR. SFAR generates fine-grained descriptions of common atomic actions for each global semantic using a large language model. Unlike existing works that refine global semantics with auxiliary video frames, SFAR introduces a filtering metric to ensure correspondence between the descriptions and the global semantics, eliminating the need for direct video involvement and thereby enabling more nuanced recognition of subtle actions.'\n\n> **Q2: Why Is It Named “global”?**\n\n**A2.** As detailed in Eq. (8) of the main paper, the augmented global text embedding $\\hat{\\mathbf{T}}\\_{\\text{c}}$ is constructed via a residual connection based on the original global text $\\mathbf{T}\\_{\\text{c}}$. Compared to the fine-grained video embedding module, the coarse video embedding emphasizes holistic video semantics. Therefore, we use the term “global” to reflect the module’s target.\n\n> **Q3: Formulation of Coarse-Grained Importance and Fine-Grained Importance.**\n\n**A3.** For coarse-grained importance, we compute scores $a\\_{\\text{l}}^\\text{coarse}$ (Eq. (9)) using the augmented global text embedding $\\hat{\\mathbf{T}}\\_{\\text{c}}$ (Eqs. (6)–(8)) and video frame embedding $\\mathbf{v}\\_{\\text{l}}^\\text{cls}$, as described in L184–189.\n\nFor fine-grained importance score $a\\_{\\text{l}}^\\text{fine}$, as defined in L211–212 and Eq. (11), we measure the maximum average similarity between $\\mathbf{v}\\_{\\text{l}}^\\text{cls}$ and sub-texts $\\mathbf{S}\\_{\\text{c}, \\text{n}}$, followed by SoftMax normalization.\n\n> **Q4: Organisation Issue.**\n\n**A4.** Our intention with the detailed figure captions was to ensure that readers could understand the core methodology and concepts even when viewing the figures independently. We will improve conciseness as suggested."}]

Author rebuttal:

Meta-review: This paper introduces a multi-granularity framework for fine-grained video understanding. Fine-grained descriptions of common atomic actions are generated using LLMs and employed as prompts to identify and aggregate class-discriminative frames within CLIP’s multimodal embedding space. The framework demonstrates strong performance across supervised, zero-shot, and few-shot settings.

All reviewers found the problem well-motivated and the proposed design intuitive. Reviewer concerns were adequately addressed in the rebuttal, and thus the AC concurs with the reviewers and recommends acceptance of the paper. As noted by Reviewer LH8r, the ablation on the fixed number N of sub-actions should be included in the final version, as it is an important point for both reproducibility and the motivation of the design.
Decision: Accept (poster)