ActAlign: Zero-Shot Fine-Grained Video Classification via Language-Guided Sequence Alignment
Authors: Amir Aghdam
Vincent Tao Hu
Björn Ommer
Venue: Accepted by TMLR ()

Abstract: We address the task of zero-shot video classification for extremely fine-grained actions (e.g., Windmill Dunk in basketball), where no video examples or temporal annotations are available for unseen classes. While image–language models (e.g., CLIP, SigLIP) show strong open-set recognition, they lack temporal modeling needed for video understanding. We propose ActAlign, a truly zero-shot, training-free method that formulates video classification as a sequence alignment problem, preserving the generalization strength of pretrained image–language models. For each class, a large language model (LLM) generates an ordered sequence of sub-actions, which we align with video frames using Dynamic Time Warping (DTW) in a shared embedding space. Without any video–text supervision or fine-tuning, ActAlign achieves 30.4% accuracy on ActionAtlas—the most diverse benchmark of fine-grained actions across multiple sports—where human performance is only 61.6%. ActAlign outperforms billion-parameter video–language models while using $\sim 8\times$ fewer parameters. Our approach is model-agnostic and domain-general, demonstrating that structured language priors combined with classical alignment methods can unlock the open-set recognition potential of image–language models for fine-grained video understanding.

Published in Transactions on Machine Learning Research (10/2025)
ActAlign: Zero-Shot Fine-Grained Video Classification via Language-Guided Sequence Alignment
Amir Aghdam
amir.aghdamm@ttemple.co
Department of Computer Science, Temple University Philadelphia, PA, USA
Vincent Tao Hu
taohu6200gmall.com
Comp Vis @ LMU Munich, Munich Center for Machine Learning Munich, Germany
Bjorn Ommer
bommer de
Comp Vis & LMU Munich, Munich Center for Machine Learning Munich, Germany
Reviewed on OpenReview: https: openrreveee. net/ forum? id- Nuzz44MTGG
Abstract
We address the task of zero-shot video classification for extremely fine-grained actions (e.g., Windmill Dunk in basketball), where no video examples or temporal annotations are avail- able for unseen classes. While image-language models (e.g., CLIP, SigLIP) show strong open-set recognition, they lack temporal modeling nceded for vidco understanding. We pro- pose ActAlign, Iruly zero-shol, training-free method that formulates video classification as a sequence alignment problem, preserving the generalization strength of pretrained im- age-language models. For each class, large language model (LLM) generates ordered sequence of sub-actions, which we align with video frames using Dynamic Time Warping (DTW) in a shared embedding space. Without any video-text supervision fine-tuning, ActAlign achieves 30.4% accuracy on ActionAtlas the most diverse benchmark of fine- grained actions across multiple where human performance is only 61.6%. ActAlign outperforms billion-parameter video Ladeo-language models while using ~8x fewer parame- ters. Our approach is model-agnostic and Jomain--eneral, demonstrating that structured language priors combined with classical aligmment methods can unlock the open-set recog- nition potential of lmnguala models for fine-grained video understanding. Code Link: https:/ bonsss//mminn lamir-aghdamggithiee ao/aaataligggg////////
1 Introduction
Understanding fine-grained human activities in video such as distinguishing a hook shot from layup in basketball, or recognizing tactical formations in football requiress parsing subtle, temporally ordered visual cues across frames. These fine-grained actions unfold in structured sequences of sub-actions and are often ncarly indistinguishable from one another in appearance. In contrast to general activitics like swimming, which can often be inferred from a single frame showing person in water, fine-grained recognition demands attention to temporally extended object interactions, spatial relations, and high-level intent. As such, models must not only understand what is present in a video but also when and how key sub-actions occur. This requires accurately aligning the temporal progression of sub-actions with each fine-grained action to ensure a correct. prediction, as shown in Figure I While their high-level label is barely recognized by action recognition models (e.g. such as Roundhouse kick in Kickbozing).
At the same time, contrastive languuee models such CLIP Radford al. 2021 and SigLIP Zhai
al. 2023 have demonstrated impressive open-set fine-grained recognition in image domains by training

---

Published in Transactions on Machine Learning Research (10/2025)
Image- Image-Text Embedding Space
Positioning near basket
Crashing from perimeter
Leaping secure rebound
Putback
Dunk in one motion
Dunk
Mid-air reach and redirection
x Tip-in
Timing jump ball hits rim
Landing
off-balance
Figure 1: ActAlign improves zero-shot fine- grained action recognition by modeling them as structured lan- guage sequences. By aligning sub-action descriptions with video frames (green red paths), we achieve more accurate predictions without requiring any video-text training data.
on massive image-tex: text pairs. These models learn a shared image- text latent space, which allows both visual inputs and natural language descriptions to be embedded for direct comparison without task-specific supervision. This enables zcro-shot classification using natural language prompts and has been widcly adopted for downstream recognition lasks. However, exlending these capabilities video understanding introduces new challenges and requires temporal modeling. Existing methods that adapt CLIP-style models
to video recognition either average frame-level features Rasheed et al. (2023); Zohra et al. (0255) ignoring
temporal structure- or fine-tune on target datasets Wang et al. [2022] Ni et al. 2022 Kim et al. (2024)
Wang et al. 2024a), sacrificing generalization and open-set recognition.
In both cases, the fine-grained
temporal semantics actions are lost or diluted.
Zhang 2023). VideoChat Li et al. (2023) mPL.UG-Ow Ye et al. (023)) Owen2-VT, Wang al. (2024b), and et al.
Recent video language architectures and instruction-tune tuned LLM-based systems such as
JaepSeck-JanusPro Chen al. (2025 cnable open-ended, dialog-style vidco understanding through heavy
instruction luning, but they are not tailored for fine-grained video recognition.
Meanwhile, textual grounding (image text aligmment) remains central challenge in interactive video- language models, especially for open-set and finc-grained video recognition. Dynamic Time Warping (DTW) Vintsyuk (1968), classical algorithm for aligning lemporally mismatched sequences, has seen re- newed interest through differentiable variants Dogan al. (2018); Chang et al. (2019) designed for supervised image-text temporal alignment. Yct, these methods rely either annotated transcripts example support videos originally proposed for supervised training, making them impractical for zero-shot recognition. Likewise, approaches using part-level or attribute-level supervision Wu et al. (2023 Zhu et al. (2024) offer fine-grained cues but lack the ability to model the temporal structure between language-defined actions and visual content.
In this work, we introduce ActAlign, a novel framework that brings the open-set generalization power of image text models video classification of extremely fine-gruined actions through language-guided sub- action alignment in truly zero-shot setting. Rather than tuning model for specific domain collapsing the video into static representation, ActAlign operates in training- training-free free setting: for each unseen action class, we use large language model (LLM) to generate a structured sequence of temporal sub-actions that icalantically define the class. Then, using the pretrained SigLIP model Zhai el al. (2023) to extract frame- wise visual and sub-action features, we align the frame sequence with the LLM LL.M-generated sub-action script via Dynamic Time Warping (DTW) (sce Figure This allows us to compute soft alignment score between different action classes that respects both content and temporal ordering, enabling fine- fine-grained classification in truly zero-shot manner.
2

---

Published in Transactions on Machine Learning Research (10/2025)
Our contributions as follows:
We introduce a novel approach for zero-shot fine-grained video recognition that models each action as a general, structured temporal sequence of sub-actions derived solely from action names without access to vidcos or transcripts.
We propose ActAlign, domain- gennai, model-agnostic and zero-shot framework that applies the open-set generalization strength of image-text models to the challenging task of fine-grained and zero-shot video classification by reformulating the task to sequence matching without requiring any video-texs supervision.
We demonstrate that ActAlign surpasses zero-shot and CLIP-based model baselines, outperforming even billion-paree parian-p video-language models on the most challenging and diverse benchmark to our knowledge, ActionAtlas- where human-level accuracy caps at 61.64%.
2 Related Work
2.1 Action Recognition
Action recognition has been extensively studied, with early work leveraging hand-crafted features and classic
classifiers Dalal & Triggs (2005 : Jain et al. (2013) Wang & Schmid (2013). The field has since evolved to deep
learning approaches Carreira & Zisserman [2017]; Simonyan Zisserman (2014;; Tran et al. (2018): Wang
al.
(016). including CNNs, RNNs, and
more recently, Transformer-bast
models Arnab et al. 2021)
Bertasius al. (2021); Kim al.
2025 Yoshida et al. 2025), which model spatio-temporal dynamics
more effectively. Fino-grained action recognition remains challenging due to subtle inter-class variations over time. Recent methods lackle this under supervised Leong el. al. (2022) Sun et al. (2022) semi-supervisel Li
et al. (022c); Dave et al. (2025), and few-shot Tang et al. (2023): Hong et al. (2021): Wang et al. (2021a)
scttings to mitigate annotation costs. Yet, they remain reliant on some level of fine-grained labeling and are constrained to specific domains.
2.2
Vision-Language Models
2.2.1
Image-Language Models
Foundational Iange-aanguge models such as CLIP Radford et al. (2021). SigLIP Zhai et al. (0223) and
ALIGN Jia et al. 22021 learn joint image text embeddings from large-scale image caption pairs. Such
modeling enables strong open-set recognition without task-specific supervision. These models are widely
adopted as pretrained backbones for downstream tasks, including visual question answering Li et al. [221]
Tsimpoukelli et al. (2021]: Li ct al. (202bb); Steiner et al. 2024h. image captioning and generation Mokady et al. [2021] : Wang el. al. (2021b), and lew- Tew-/zero-she /zero-hhot classification Zhou el al. (202 0; Khattak al. (2025)
However, they lack temporal modeling, which limits their open-set recognition capability video inputs.
2.2.2 Video-Language Models
Early efforts in video modeling focused on self-supervised pretraining. Models like ACtBERT Zhu Yang 20200 and VideoBERT /Sn et al. (2019) applied masked language modeling video frames. This enabled better transfer to Mideo-language tasks through finetuning paired video-ter text data Miech et al. (2220)
Lei et al. 2021); Xu et al. (2021): Feichtenhofer et al.
2002 Li et al. (022a)). Recent progress in LLM
reasoning has driven their integration with visual encoders, enabling open-ended vidco understanding via user prompts. These models Ye el. al. (2023): Li et al. (2023): Zhang et al. (2024); Wang el al. (002bb). such as Video-LLaMA Zhang et al. (2023), combine pretrained visual backbones with chat-centric LLMs generate spatio-temporal reasoning in conversational settings. While these models excel at open-ended question answering, they require extensive instruction tuning and are not optimized for fine- ane-grained video recognition.
3

---

Published in Transactions on Machine Learning Research (10/2025)
2.3 Sequence Alignment
Sequence aligument is a long-slanding problem in fields such as biology and signal processing. A widely used technique for aligning temporally misaligned signals is Dynamic Time Warping (DTW Berndt & Clifford
19940: Koogh & Pazzani (2000); Rath & Manmathal (2003). which computes the optimal warping path
between two ordered sequences. Early work extended DTW to video tasksHuang et al. (2016); Richard
ett al. (2018). For instance, a seminal approach Bojanowski al. (2015) proposed aligning videos to ordered
scripts by enforcing the temporal order of events. With the rise of deep learning, DTW has been adapted
into differentiable forms Dogan et al. (2018); Chang al. (2019). enabling greater flexibility and improved learning capacily. A notable example is OTAM Ca et al. 22020, which leverages DTW for representation learning in few-shot video classification by aligning each video to its support set. However, these methods often assume access to ground-truth transcripts for each video or rely on task-specific supervision, such as paired video- rideo-text exemplars.
Our Work In contrast to prior methods that discard temporal structure, rely task-specific supervision, or require support data for alignment, Alin introduces a language-guided sequence aligument framework for zero-shot fine-grained video classification.
3 Method
3.1 Problem Definition
Let D = (Vi.8)); denote a datasct of N vidcos, where each vidco Vi is associated with a ground-truth fine-grained class label yi drawn from a set M candidate classes y CI,...;CM CM. Each video V/ consists of sequence of T; frames as defined in Eq. 1
V (VI VER"xxxx
(1)
where H and W denote the frame height and width. In our zero-shot setting, no video examples of the target classes y are used for training or tuming; only high-level action names Cj are provided. The goal is to construct function f: V y + R that effectively maps the sequence of video frames into their correct action class y. The predicted class label i for video V, is given by Eq, Z
0 = arg max J(Vi, Cm) E
(2)
To leverage semantic priors from LLMs, we automatically decompose each class label cm into an ordered, variablo-length sequence Kn, textual sub-actions, as definod in Eq.
Sm [sm,,1 8m,2 5m,km
(3)
where 8m,k a concise natural-language description of the k-th step in executing action m.
3.2
Our Method
We define fV1, cm) as the alignment score between the visual frame embeddings {v() and the sub-action sequence Sm: computed via Dynamic Time Warping (DTW). This alignment performed in the image- text embedding space, without requiring any fine-tuning or vidco examples from the target label set. Figure 2 illustrates the pipeline of our proposed approach.
3.3 Preliminary Sub-action Generation by LLM
In domains requiring fine-grained distinctions- such as differentiating tactical plays in sports the high- level action class name Gm often lacks sufficient discrininatory power. To address this, we define a mapping

---

Published in Transactions on Machine Learning Research (10/2025)
1, Subaction Generation
Method Overview
3. Class Prediction
Dasa nottaa Toctica
Imtagn Encodar
Hick
onte
Aimp shar
Dunk
LLM
Simfurity Hinds
Hook
Jamp
Dusk
Test Encedor
1/Picalve Bell
2/Photand jump
tworboo
Peplederpwent
Fictand
Shoct stapes
Oveep-r
2 Outter nwoitor
Fasokaydann
Normelized
Dysamic Tima Warping
Jump shot
Recalie
Laad
Alggment
forshhlydaan
fvales, Book
Hook
Jump shat
Ehooe apen
Dunk
Figure 2: Our ActA Method Overview. (1) Sub-action Generation: Given fine-grained actions (e.g. Basketball Tacties), we prompt an LLM decompose each action (e.g. Hookshot, JumpShot, Dunk) into a temporal sequence of sub-actions. (2) Temporal Alignment: Video frames encoded by frozen pretrained vision encoder and smoothed via moving-average filter. Simultancously, each sub-action is encoded by the text encoder. We compute cosine-similarity matrix between frame and sub-action embeddings,, then apply Dynamic Time Warping (DTW) to find the optimal alignment path and normalized alignment score. (3) Class Prediction: We repeat this process for cach candidato action m, compare normalized alignment scores mi and select the action sequence with the highest score as the final prediction.
p: S, where S denotes the space of ordered sub-action scripts. For each class Cm E y we generate a soquence Sm = 18m,1,..... Km. = P(cm), where each 8m,k describes trackable and temporally ordered sub-action.
We instantiate P using GPT-4o via carefully constructed natural language prompts. Given a set candidate classes fci....,Cal the LLM is instructed to:
Decompose each 6m into variable- or fixed-length sequence of semantically coherent sub-actions.
Return the list [sm,1, Km, in a consistent, structured format.
By leveraging the LLM's extensive prior knowledge, we obtain sub-action sequences without any manual labels or video supervision. These sequences serve as semantic reference signals for temporal alignment.
Importance of Context in Aub-ontions. Terse deseriptions (e.g., "drive forward") are often ambiguous and may refer to multiple sports, which hampers DTW alignment due poor textual grounding. In contrast. context-rich prompts (c.g., "player drives forward to the rim") produce more discriminative sub-actions that are reliably grounded with visual evidence.
Context Augmentation We enrich each sub-action by embedding it into a contextualized template: This is a video of doing <action name> in <sport domain> with <sub-action>. This substantially im- proves semantic clarity and further disambiguates sub-actions. The sport domain automatically provided in the ActionAtlas metadata (and could also be inferred from the action name itself).
3.4 Visual and Semantic Feature Encoding
Once each class label cm is docomposed into its sub-action sequence Sm, we project both video frames and sub-actions into shared d-dimensional embedding space using the pre-trained SigLIP image-text model, which is recognized for its strong zero-shot recognition performance.
5

---

Published in Transactions on Machine Learning Research (10/2025)
Visual Embeddings Let 6, Rxxxx F denote the vision encoder. For video V; {[v')) compute frame-level embeddings:
It=1:
= E R., t= , Tz
(4)
Stacking yields Z, = [z] : E RRxx:
Semantic Embeddings Let Φi - Rd be the text encoder. For each class cm and its sub-action sequence Sm, we embed each step:
Um (5(mm))) E RR, k 1, I...., Kms
(5)
stitching into Un [um, Um, K Rdxkm
Shared Latent Space
We normalize all emboddings that the similarity
2 Umnk
sim Um,kk)
(6)
|2/| |um,kl
is a cosine similarity measure between frame v and sub-action Sm.k of class Cm (see Eq. 6 This cross- modality similarity forms the basis for alignment in the next step.
3.5 Dynamic Time Warping
After feature encoding, each video yields visual embedding sequence Zi and ordered sequences of sub-action embeddings Un for each class Cm. We treat Um as the reference semantic signal and Z; as the query visual signal. The Um could also be viewed as a prototype sequence for class Cm-
Signal Smoothing Real-world footage often contains abrupt scene changes or irrelevant frames (e.g., replays, advertisements) that introduce noise into Zí To mitigate this, we apply simple, parameter-free 1D moving-average filter of width across the temporal dimension:
w/22
W
w
with boundary conditions handled via zero padding. The kernel width controls the trade-off between noise reduction and temporal resolution. The smoothing effectively aggregates short-term temporal context, dampening isolated spikes while preserving the overall action dynamics. As introduces no learnable parameters or additional training, remains lightweight and fully compatible with the training-free sctting.
Affinity Matrix Construction Let the smoothed visual embeddings for video V, be 7, = _ x
and the sub-action sequence embeddings for class Cm be Um [mm. Km. We first compute the cosine similarity matrix:
A (m.)) kit
(um,k %)), A (m,,) E 1m x7;
(8)
where each (.-) the inner product of 22-normalized vectors, yielding values [-1,1]. Following the SigLIP prediction approach, we then apply a sigmoid function a--) to transform these values into affinity scores in [0, 1]:
(m..)) = (a A
(m.))) + B),
(9)
where a. are pre-learned scaling parameters as part of SigLIP's pretraining recipe. The resulting At.m) is used the input affinity matrix for DTW alignment.

---

Published in Transactions on Machine Learning Research (10/2025)
DTW Alignment and Scoring Given the affinity matrix A(m,)) for class Cm and video Vi (defined in Eq.
seek warping path W(m,)) = ((114)).... (kz, (t)) that maximizes cumulative similarity under
monotonicity and continuity constraints:
W(mm) arg max (KLI)EW
Σ
A(maa))
s.t. W is valid warping path between [1, Km and [1,
(10)
We solve this using dynamic programming by selecting the path with highest alignment (i.e. similarity value) at each step:
DA.t A(raa DR DR-11-1h-
(11)
with the base case Do.- D.,0 = co. The final alignment score is maXk, Da,t, and backtracking recovers the optimal warping path (m,,)) for sub-action sequence of class and video V,.
Prediction We compute the raw aligmment score, as defined in Eq. the sum of the similarity values of the optimal warping path W(m,)) obtained in Eq. 10
Kim
Σ
(m.i) k,
(12)
(k.))e((m.))
(m.,i) is the affinity matrix introduced in Eq. To mitigate the bias toward longer warping paths (which can accumulate higher raw scores), we normalize 7i by the path length, resulting in the average alignment score:
8.m
[W(m)))]
Ya
(13)
where (m,,)) is the number of matched frame sub-action pairs. Since the similarity values in (..m) lie in [0, 1] (due to the sigmoid in Eq. the normalized alignment score 9i.m also lics in the range [0. 1].
Finally, we predict the class whose sub-action sequence best aligns-on on average-with the observed video frames. This is done by selecting the class with the highest normalized alignment score:
0i = arg max fimm
(14)
Con E)
4 Experiment
4.1 Experimental Setup
Dataset Our approach is datasct- and domain-agnostic, as it relics solely on language-driven sub-action generation from LLMs and cross-modal alignment. without lask-specific supervision. To rigorously evaluate its generality, we adopt ActionAtlas (alehi et al. (224 to our knowledge, the most diverse and challenging benchmark for fino-grained action recognition across various domains. For each vidco V,, we retain its multiple-choice candidate set (c. }. and replace each class label with an LLM-generated sub-action sequence. (Dataset statistics is provided in the Appendix.)

---

Published in Transactions on Machine Learning Research (10/2025)
Method
# *aara
Top-1 (%) +
Top-2 (%) t
Top-3 (%) t
Random (10 Trials)
20.81
42.04
62.50
Human Evaluation (Oracle)
61.64
mPLUG- PLUG-OWw--E Video Ye et
2023
7B
19.49
VidcoChat2 Li ct al. (2023)
7B
21.27
VideoLLaaMA Zhang et al. 20233
8B
22.71
IL Video Zhang al.
2024
7B
22.90
Qwen2 VL Wang et al. (2024b]
7B
30.24
X-CLIP-L/ 14-16F Ni et al.
[2022)
0.6B
16.26
33.74
49.89
Zhai et al.
20230
(mean-pool)
0.9
22.94
42.20
63.70
+ DTW Alignment (Ours)
0.93
25.72
45.99
66.26
ActAlign (Ours)
0.9B
30.40 + 0.11
53.01
70.27
Table I: Zero-shot classification results on ActionAtlas. Our method achieves state-of-the-at art Top-1, Top-2, and Top-3 accuracy, outperforming all baselines and billion-part iillion-prramttr lango-la models without any video- text supervision. These results highlight the open-set recognition capability of image-lext models and the effectiveness of structured sub-action alignment over flat representations such mean-pooling. We use context-rich prompting strategy.
Bosetti et al. 2025 report Top-k accuracy for Evaluation Metrics Following Chen & Huang E 21;
2021
N Σ
Top-k N I raank (0:) S k)
where II is the indicator function and rank;(f;) is the position of the ground-truth label in the descending list of scores f ),,, This accounts for typical cases where fine-grained actions are semantically similar and alignment scores are closely clustered, allowing improvement to be captured within a narrowed candidate set.
Experimental Detail We use SigLIP-so-00 so-10- (patch size 14, d = 384, 878M parametors). We apply a moving-average smoothing with a fixed window size of w = 30 frames (Is @30 fps) to reduce transient noise and emphasize consistent motion patterns. All experiments run on a single NVIDIA RTX A5000 GPU (25 GB). Inference consists of embedding extraction followed by DTW alignment over N class candidates, each with M sub-actions, and T video frames caanes-yiclding O(NMT) time. Following the zero-shot protocol, no example videos or sub-action sequence from these classes are used for any training tuning. We only use candidate action class names to prompt for sub-action generation.
4.2 Experimental Result
Zero-Shot Comparisons We begin with a random choice baseline and a SigLIP zero-shot baseline using mean-pooled frame embeddings. Following ViFi-CLIP Rasheed et al. 2023). each video V, is encoded as z= Et= z/, where mean-pooled embeddings z is compared via cosine similarity to each name embedding dr(c,).
We further compare against open-source langu-language models, including entries from the ActionAtlas leaderboard and fine-tuned CLIP variants. Despite using video text supervision, our method outperforms the SigLIP baseline by -7% Top-1 and +11% Top-2 accuracy, surpassing all baselines and billion billion-parameter models with ~88 fewer parameters. These larger models are often optimized for interactive tasks through extensive instruction tuning, rather than for classifying fine-grained actions. In contrast, image language models are trained on large-scale image- text. pairs, making them particularly well-suited for open-set recog- nition when paired with temporal alignment mechanisms.
As shown in Table these gains stem from our subaction-level alignment, which enables more discriminative and interpretable classification than global frame pooling method.
8

---

Published in Transactions on Machine Learning Research (10/2025)
Configuration
Top-1 (%) f
Top-2 (%) t
Top-3 (%) 1
SigL.IP Zhai al. (2023)
(mean-pool)
22.94
42.20
63.70
DTW Aligument
25.72
45.99
66.26
Context Augmentation
30.07
52.67
70.49
Signal Smoothing
30.29
53.01
70.27
Table 2: Ablation results of various design under context-rich prompting. DTW alignment introduces temporal matching, context augmentation reduces sub-action ambiguity, and signal smoothing mitigates framo-level noise.
Configuration
Top-1 (%) t Top-2 (%) f
Top-3 (%) f
SigLIP (mean-pool)
22.94
42.20
63.70
SigLIP (mean-pous pool) w/ Context Augmentation
28.51
50.89
70.38
SigLIP (mean-pool) w/ Bag-of- Words
29.06
51.34
69.49
AcIAlign (ours)
30.29
53.01
70.27
Table 3:
Comparison with basclines under context-rich prompting. Context augmentation enriches
action names (as opposed to sub-actions in Table 2 with domain names, while bag-of-words mean-pools sub-action descriptions into unified representation.
Ablation Studies We ablate each component of ActAlign on ActionAtlas (Table - starting from mean-pooled SigLIP baseline. Adding DTW alignment introduces temporal structure and yields consistent gains. Context angmentation injerting domain context (e.g., "Sprint start" - "Sprint start in basket- producss the largest boost by resolving semantic ambiguity. Signal smoothing offers a modest but complementary improvement by roducing frame-level noise and clarifying action boundarics.
Baseline Comparisons To further evaluate the generalization power of SigLIP without any temporal information, we evaluate SigLIP performance by the similar context. augmentation technique used in framework and with mean-pooling the bag of sub-actions and comparing the mean-pooled representations with video representation (See Table 3 In all of these baselines, video frame representations are mean- pooled. We further investigate classification performance under randomized and reversed sub-action orders at Table
Sub-action Baselines To evaluate the impact of sub-actions on aligmment, we test DTW alignment performance under two perturbations: randomizing sub-actions and reversing their order on short-fixed sub-actions. Results are shown in Table g
We observe that the upper bound on performance is closely tied the specificity and coherence of LLM- generated sub-action sequences.
Prompt Variations We evaluate two prompt strategics for generating sub-action scripts using GPT-4o, keeping all other components fixed:
Configuration
Top-1 (%) 1
Top-2 (%) 1
Top-3 (%) 1
Reversed Order
26.39
47.66
67.04
Randomized Order (5 Trials)
26.63 + 0.14
47.05 = 0.29
67.00 + 0.25
Normal
27.06
46.88
66.82
Table 4: Comparison with sub-action baselines. We evaluate classification performance by randomizing and reversing the order of short-fized sub-actions due their larger sequence size.

---

Published in Transactions on Machine Learning Research (10/2025)
Prompt
Description
Top-1 (%) 1
Short-fixed 2-word, fixed 10 sub-actions 27.06
Context-rich context-rich, variable-length 30.29
Table 5: Impact of prompt strategy. Context-rich prompting improves zero-shot classification perfor- mance by producing more specific and informative sub-actions.
w/o Context Augmentation
w/ Context Augmentation
Figure 3: t-SNE visualization of sub-action embeddings. Each color corresponds to sport domain.
Augmenting sub-actions with context yields more discriminative clusters and improves textual grounding.
Short-Fixed: Prompts GPT-4o to generate exactly 10 terse (2 3 word) sub-actions per class using a fixed structure.
Produces variable-length, context-rich sub-action scripts incorporating domain- specific cues (e.g., "Wrestler", "Rider").
Context-Rich:
Table M shows the performance of each strategy. The context-rich prompt with domain-specific con- text achieve the highest accuracy. contrast, short-fixed prompts-lacking sufficient semantic speci- fäity-perform worst. These results highlight that reducing ambiguity in sub-action descriptions directly improves alignment quality and classification performance.
Sub-action Embedding Structure Figure g shows 2D i-SNE projections of sub-action embeddings with and without context augmentation under context-rich prompts. Adding sppaif-specific cues (<sub-action> in <sport name>) results in tighter, more coherent clusters-indicating better semantic structure and sep- aration.
Sub-action Sequence Examples Table D shows LLM-generated sub-action sequences for two Figure skaling lactics under our context-rich prompting. The scripts highlight ordered, salient sleps enabling precise temporal matching.
Alignment Heatmaps and Paths Figure g visualizes the cosine similarity matrix and DTW path for correctly classified action (right) and an incorrect candidate (left). The correct sequence yields high- similarity regions with monotonic path. In contrast, the incorrect script shows sparse similarity regions, with the DTW path forced to follow the single most similar alignment trace.
Signal Smoothing. Figure B compares similarity matrices before and after applying a moving-average filter. Without smoothing, rapid scene changes introduce noise and scattered peaks. Smoothing yields cleaner similarity surfaces with clearer action boundaries.
10

---

Published in Transactions on Machine Learning Research (10/2025)
Sub-action Script
Biellmann Spin
1. Begins upright spin on one foot with arms extended and free leg behind, 2. Gradually pulls free log upward behind the back using both hands, Raises the free leg above head level while arching the back dramatically, 4. Grasps the blade of the free skate with both hands overhead, 5. Extends spinning leg vertically while maintaining centered spin on skating foot, 6, Maintains high-speed rotation with body in extreme vertical split position
Flying Camel Spin
1. Skater glides forward with arms extended and knees bent in preparation, 2. Performs powerful jump the toe pick while swinging free leg upward, Rotates mid-air with body extended horizontally like a 'T' shape, 4. Lands on one foot directly into a camel spin position with torso parallel ice, 5. Extends free leg backward and arms outward while spinning on the skating leg, 6. Maintains fast, centered rotation in the horizontal camel position
Table 6: LLM-generated sub-action scripts for figure skating tacties. Shown for the Biellmann Spin and Flying Camel Spin examples in Figure | these sequences are generated using context-rich prompting and provide semantically detailed, temporally ordered steps for alignment in our zcro-shot framework.
(a) Video: Performing flying camel spin in figure skating.
Suboction Indox Indox
CTM
Frame
(c) Flying camel spin Figure 4: DTW alignment paths for incorrect prediction (left) versus correct classification (right). The correct class exhibits clearer segmentation and higher alignment quality. The sub-action scripts are provided in Table
(b) Biellmann spin
Failure Case Analysis
We obscrve two primary failure modes in framework:
Vague LLM-generated steps (c.g., "move to position") result in sparse similarity matrices, weakening DTW's ability to discriminate between candidates. As shown in Figure context-rich prompts produce clearer alignment. regions than short-fixed ones.
Ambiguous sub-actions:
Global alignment bias: Vanilla DTW enforces full-sequence alignment, which is suboptimal when actions start mid-clip or exhibit temporal shifts. Without a local alignment mechanism, early trailing sub-actions can introduce noise. Figure shows temporal shift case that does not affect prediction here but may cause errors in more competitive settings.
These limitations underscore the importance of precise sub-action design and motivate future improvements in alignment robustness and prototype refinement.
11

---

Published in Transactions on Machine Learning Research (10/2025)
Subictio mndo
Subsccenn
Frame
w/O smoothing
(b) w/ smoothing
Figure 5: Signal smoothing reduces high-frequency noise and enhances transition between sub-actions.
Similarity matrices before and after applying a moving-avert filter (w = 30).
(a) Video: Varial kickflip underflip body varial in Skateboarding.
Indox
Suboctoon
Supertion
Frame Irdix
Frame Index
(b) Short-fixed (x, 0.35).
(c) Context-rich (V, 0.81).
Figure Failure case due vague sub-actions for the action Varial kickflip underflip body varial. The normalized aligument score & reflects overall sequence similarity. similarity map and incorrect prediction. (c) The Context-rich prompt produces clearer alignment and correct classification. See Appendix for the detailed scripts.
(b) The Short-fixed prompt yields a noisy
5 Conclusion
We show that contrastive image language models establish a surprisingly strong baseline for zero-shot fine- grained video classification, when used with simple mean-pooling. To fully leverage this capability in extremely fine-grained settings, propose ActAlign, novel zero-shot, domainn-eneral,, and model- agnostic framework that revisits the classic Dynamic Time Warping (DTW) algorithm to cast vidco classi- fication as a sequence-aligumenn problem. By aligning video frames with -LM-ggneraeed sub-action scripts, ActAlign introduces temporal structure into contrastive models without requiring any video video-text training or fine-tuning. Evaluated on the highly challenging and diverse ActionAtlas Salchi et al. (2024) benchmark, our method achieves state-of-the-art performance, outperforming both CLIP-style baselines and billion- parameter video language models.
12

---

Published in Transactions on Machine Learning Research (10/2025)
References
Shahzad Ahmad, Sukalpa Chanda, and Yogesh S Rawat. Ez-clip: Efficient zeroshot vidco action recognition, 2024. URL htttps:/aarxivorr///ltlt
08010
Amurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 6836-6846, October 2021.
Donald J. Berndt and James Clifford. Using dynamic time warping to find patterns in time series. In Proceedings of the 3rd International Conference on Knowledge Discovery and Data Mining, AAAIWS'94, pp. AAAT Press, 1994.
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video under- standing? In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 813 824. PMLR, 18-24 Jul 2021. URL
mlr. aeress/v139/bertasiii
html
Piotr Bojanowski, Rémi Lajugie, Edouard Grave, Francis Bach, Ivan Laptev, Jean Ponce, and Cordelia Schmid. Weakly-supervised alignment of video with text. In TEEE International Conference on Compuler Vision (ICCV), 2015.
Massimo Bosctti, Shibingfeng Zhang, Bendetta Liberatori, Giacomo Zara, Elisa Ricci, and Paolo Rota. Text- enhanced vero-shol action recognition: A training-free approach. In Apostolos Antonacopoulos, Subhasis Chaudhuri, Rama Chellappa, Cheng-Lin Liu, Saumik Bhattacharya, and Umapada Pal (eds.), Pattern Recognition, pp. 327 342, Cham, 2025. Springer Nature Switzerland. ISBN 978-3-031-78354-8.
Kaidi Cao, Jingwei Ji, Zhangjie Cao, Chien-Y Yi Chang, and Juan Carlos Niebles. Few-shot video classification via temporal aligmment. In IEFE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEFE Conference on Compuler Vision and Pallern Recogrilion, pp. 6299 6308, 2017.
Chien-Y Yi Chang, Ae Huang, Yanan Sui, Li Fei-Fei, and Juan Carlos Niebles. D3tw: Discriminative differ- entiable dynamic time warping for weakly supervised action alignment and segmentation. In IEEE/CVF Conference on Compuler Vision and Pullern Recognition (CVPR), 2019. 2019.
Shizhe Chen and Dong Huang. Elaborative rehearsal for zero-shot action recognition. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 13638-13647, October 2021.
Xiaokang Chen, Zhiyu Wu, Xingehao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus pro: Unified multimodal understanding and generation with data and model scaling. anXiv preprint ar Xro::2501. 1.001.1775 17811, 2025.
N. Dalal and B. Triggs. Histograms of oriented gradients for human detection. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'03) '05), volume 1, pp. 886-893 vol. 1, 2005. doi: 10.1 CVWpCV22..000.117
Ishan Rajendrakumar Dave, Mamshad Nayeem Rizve, and Mubarak Shah. Finepseudo: Improving pscudo-labelling through temporal-alignablity for semi-supervised fine-grained action recognition. In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Giil Varol (eds.), Computer Vision FCCV 2024, pp. 389-408, Cham, 2025. Springer Nature Switzerland. ISBN 978-3-033--33442-3
Pelin Dogan, Luke Noumann, Mohit Tyyer, Kathloen McKeown, and Dusty Summers-Stap A neural multi- sequence alignment technique (neumatch). In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
13

---

Published in Transactions on Machine Learning Research (10/2025)
Christoph Feichtenhofer, Haoqi Fan, and Yanghao Li Kaiming He. Masked autoencoders spatiotemporal learners. In Proceedingg of the 36th International Conference on Newral Informalion Processing Systems, NIPS '2, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088.
James Hong, Matthew Fisher, Michaël Gharbi, and Kayvon Fatahalian. Video pose distillation for few- shot, fine-grained sports action recognition. In Proceedings of the IEEE/CVF International Conference on Compuler Vision (ICCV), pp. 9254 -9263, October 2021.
De-An Huang, Jonathan Krause, Li Fei-Fei, and Juan Carlos Niebles. Connectionist temporal modeling for weakly supervised action labeling. In Europeon Conference on Computer Vision (ECCV), 2016.
Mihir Jain, Hervé Jégou, and Patrick Bouthemy. Better exploiting motion for better action recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp.
2013.
Chao Jia, Yin Yang, Ye Xia, Yi Chen, Zarana Parekh, Hieu Pham, Quoe V Le, Tom Duerig, and Orien Song, Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning (ICML), 2021.
QI Ju et al. Prompting visual-language models for efficient. video fine-tuning, In International Conference on Learning Representations (ICLR) Workshop, 2022.
Eamonn J. Keogh and Michael J. Pazzani. Scaling up dynamic time warping for datamining applications. In Proceedings of the Sizth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '00, pp. 285 289, New York, NY, USA, 2000. Association for Computing Machinery. ISBN 1581132336, doi: (0.1145/347090.34
URL https://doi. 10. 1tt..//.//....//470000000100 11t.111447799..37777777
Muhammad Uzair Khattak, Muhammad Ferjad Naeem, Muzammal Naseer, Luc Van Gool, and Federico Tombari. Learning to prompt with text only supervision for vision-language models. Proceedings of the AAAT Conference on Artificial Intelligence, (9(4):4230-4238) Apr. 2025. doi: 10. (0.160//ani..v3344..2....
URL https://ojs.aant aaai. oot..orgggooooo. phtp/AAA//artiill/vveessss
Jihwan Kim, Miso Lee, Cheol-Ho Cho, Jibyun Lee, and Jae-Pil Heo. Prediction-feedlbcck detr for temporal action detection. Proceedings of the AAAI Conference on Artificial Intelligence, 326444426-44274, Apr.
2025. doi: 10.1609 att.aaai.v.......... v3914.32448. 32448
CRL https://ojs. aaai wwt..orrggiiiniiieeeeee http://wex..ppp/AA///att.ta pttp://AA//arrrice///eesss
Minji Kim, Dongyoon Han, Tackyung Kim, and Bohyung Han. Leveraging temporal contextualization for video action recognition. In Computer Vision ECCV 2024: 18th European Conference, Milan, Italy,
September 29 October 4, 2024, Proceedings, Part XXI, 74 91, Berlin, Heidelberg, 2024. Springer- Verlag. ISBN 778-3-031-72663-7 doi:
3. URL https:///o1..or//1111000 org//0.
978-3-031-72644447
Jie Lei, Licheng Li, Chitta Baral, and Mohit. Bansal. Less is more: Clipbert for video-and-language learning via sparse sampling. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Mei Chee Leong, Haosong Zhang, Huibin Tan, Liyuan Li, and Joo Hwee Lim. Combined enn transformer encoder for enhanced fine-grained human action recognition. ArXiv, abs/ 2bs/2208.01897, 2022. URL https:
//api www.nticschoooioc.teoe.i.t org/ wttg/CorpusID:::. :D:22880022222
Dongxu Li, Junnan Li, Hongdong Li, Juan Carlos Niebles, and Steven C.H. Hoi. Align and prompt: Video- and-language pre-training with entity prompts. In Proceedings of the TEEE/CVF Conference on Compuler June 2022a.
Vision and Pattern Recogrition (CVPR), pp.
Junnan Li, Jason Baldridge, and Steven CH Hoi. Align before fuse: Vision and language represcntation learning with momentum distillation. Advonces in Neural Information Processing Systems, 34:9694-9705, 2021.
14

---

Published in Transactions on Machine Learning Research (10/2025)
Junnan Li, Dongxu Li, Caiming Xiong, and Steven CH Hoi. Blip:
Bootstrapping language-image pre- training for unified vision-language understanding and generation. In International Conference on Machine Learning (ICML), 2022b.
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding, arXiv preprint arXiv:2305.062: 2023.
Zhi Li, Lu He, and Huijuan Xu. Weakly-supervised temporal action detection for fine-grained videos with hi- erarchical atomic actions. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision ECCV 2022, pp. 567 584, Cham, 2022c. Springer Nature Switzerland. ISBN 778-3-031-20080-9
Ziyi Lin, Shijie Geng, Renrui Zhang, Peng Gao, Gerard de Melo, Xiaogang Wang, Jifeng Dai, and Hongsheng Li. Frozen clip models are efficient video learners. In European Conference on Computer Vision (ECCV), 2022.
Mushui Liu, Bozheng Li, and Yunlong Yu. Omniclip: Adapting clip for video recognition with spatial- temporal omni-scale feature learning, 2024. URL http://aarxiv..ogaabb///////
06158
Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Tvan Laptev, Josef Sivic, and Andrew Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), 2020.
Ron Mokady, Amir Hertz, and Amit Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111.0979,, 2021.
Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Zhang Yu, Dong Huang, Henry Rowley, Jianlong Fu, and Haoyang Liu. Expanding language-image pretrained models for general video recoguition. European Conference on Computer Vision (ECCV), 2022.
Thinh Phan, Khoa Vo, Duy Le, Gianfranco Doretto, Donald Adjeroh, and Ngan Le. Zeetad: Adapting pretrained vision- language model for zero-shot end-to-end temporal action detection. In Proceedings of the IEFE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 7046-7055, Jamuary 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML), 2021.
Hanoona Rasheed, Muhammad Uzair Khatlak, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Fine-tuned clip models are efficient video learners. In IEFE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
T.M. Rath and R. Manmatha. Word image matching using dynamic time warping. In 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings., volume 2, pp. 11 11, 2003. doi: 10.1109/CVV S.VPR.2003.21113
Alexander Richard, Hildegard Kuehne, and Juergen Gall. Action sets: Weakly supervised action segmenta- tion without ordering constraints. In IEEE/C' /CVF Conference on Compuler Vision and Pollern Recogration (CVPR), 2018
Mohammadreza Salehi, Jae Sung Park, Aditya Kusupati, Ranjay Krishna, Yejin Choi, Hannaneh Hajishirzi, and Ali Farhadi. Actionatlas: A videoqa benchmark for camain-speccilliees action recognition. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2024. URL https: //openeviie http::enwwweeooooiiiiddddcooiiie
Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in vidcos. In Proceedings of the 28th International Conference on Neural Information Processing Systems Volume 1, N4PS'14, pp. 568 576, Cambridge, MA, USA, 2014. MIT Press.
15

---

Published in Transactions on Machine Learning Research (10/2025)
Andreas Steiner, André Susano Pinto, Michael Tschannen, Daniel Keysers, Xiao Wang, Yonatan Bitton, Alexey Gritsenko, Matthias Minderer, Anthony Sherbondy, Shangbang Long, Siyang Qin, Reeve Ingle, Emanuele Bugliarello, Sahar Kazemzadeh, Thomas Mesnard, Ibrahim Alabdulmohsin, Lucas Beyer, and Xiaohua Zhai. Paligemma 2: A family of versatile vlms for transfer, 2024. URL bttps://arxiv..o/ org/
0bs/2442.03552
Baoli Sun, Xinchen Ye, Tiantian Yan, Zhihui Wang, Haojie Li, and Zhiyong Wang. Fine-grained action recognition with robust motion representation decoupling and concentration. In Proceedings of the 30th ACM International Conference on Multimedia, MM '22, pp. 4779 4788, New York, NY, USA, 2022.
Association for Computing Machinery. ISBN 9781450392037. doi: 0..1145/3503161 3548046. URL https:
//doi.org/10 10ng//1..00
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
Hao Tang, Jun Liu, Shuanglin Yan, Rui Yan, Zechao Li, and Jinhui Tang. M3net: Multi-view en- coding, matching, and fusion for few-shot fine-grained action recognition. In Proceedings of the 31sl ACM International Conference on Multimedia, MM '23, pp. Association for Computing Machinery. ISBN 9798400701085. doi: (1145//581173.361222) URL
New York, NY, USA, 2023.
https://doi.ors 10. /ttp://...orr111111/////1/111111
Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look spatiotemporal convolutions for action recognition. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Frozen: A modular framework for vision-language modeling. arXiv preprint arXie:2106. 13884, 2021.
T. K. Vintsyuk. Speech discrimination by dynamic programming. Kibernetika (Cyberneties), 4(1):81 88, 1968. doi: 2.1007/BF01074755
Heng Wang and Cordelia Schmid. Action recognition with improved trajectories. In 2013 IEEE International Conference on Computer Vision, pp. 3551-3558, 2013. doi: 10.1109/ICCV 11111//11CV.2211441111
Jiahao Wang, Yunhong Wang, Sheng Liu, and Annan Li. Few-shot fine-grained action recognition via bidirectional attention and contrastive meta-learning In Proceedings of the 29th ACM International Conference on Multimedia, MM '21, pp. 582-591, New York, NY, USA, 2021a. Association for Computing Machinery. ISBN 9781450386517. doi: 10. 10.1145/34740553377222
URL https://doi. org/10..1110 10g//0.1145/
3474085 3475216
Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016, pp. 20-36, Cham, 2016. Springer International Publishing. ISBN 978-3-319-46488--8
Mengmeng Wang, Jiazheng Xing, and Yong Liu. Actionclip: A new paradigm for video action recognition. In Proceedings of the IEEE/CV /CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Mengmeng Wang, Jiazheng Xing, Boyuan Jiang, Jun Chen, Jianbiao Mei, Xingxing Zuo, Guang Dai, Jing- dong Wang, and Yong Liu. A multimodal, multi-task adapting framework for video action recogni- tion. Proceedings of the AAAT Conference on Artificial Intelligence, 38(6):5517-5522 Mar. 2024a. doi: 10. aaai..33i....... 28361. URL https://ojs. aaai www.oorgg/./.....//eoeeeeeeee httpa///ppp//..aaaaaaaat////////
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution, September 2024b. URL https://arxvv.orr/abb//////// 12191
16

---

Published in Transactions on Machine Learning Research (10/2025)
Zihang Wang, Jiasen Li, Olga Kovaleva, Zihang Dai, Yulia Yu, Yinfei Li, and Wei Ma. Simvim: Simple visual language model pretraining with weak supervision. arxiv preprint or ar::22108. 10904,0004400 2021b.
Wenhao Wu, Xiaohan Wang, Haipeng Luo, Deng Cai, and Xuming He. Bike: Bidirectional cross-modal knowledge exploration for video recognition with pre-trained vision-language models. In IEFE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), 2023.
Hu Xu, Gargi Ghosh, Po- Yao Huang, Prahal Arora, Masoumeh Aminzadeh, Christoph Feichtenhofer, Florian Metze, and Luke Zettlemoyer. VLM: Task-agnostic vidco-language model pre-training for video understanding. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the Association for Computational Linguistics: ACI-IICNLP 2021, pp. 4227-4239, Online, Au- gust 2021. Association for Computational Linguisties. doi: (411653/v11/200 findiing--acl.370 URL https://aclanthe 2ttp://oooogor../222.... wtt.://.....iiiinneeeeeeeeeeeee. Findings-acl.370
Deming Ye, Junyang Lin, Shuai Li, Wenxin Zhang, Donglin Yu, Jiaming Liu, Yulei Niu, Wengang Zhou, Houqiang Li, Dawei Qi, et al. mplug-owl: Modularization empowers large language models with multi- modality. arXiv preprint 0rXiv:2304.44178 2023.
Shuhei M. Yoshida, Takashi Shibata, Makoto Terao, Takayuki Okatani, and Masashi Sugiyama. Action- agnostic point-level supervision for temporal action detection. Proceedings of the AAAI Conference on Artificial Intelligence, 39(9):9571-957) Apr. 2025. doi:
URL https://ojs.
maai.orga indexidexxxx. http://pppppa/aarrrrrr///r//////
Yating Yu, Congqi Cao, Yueran Zhang, Qinyi L.v, Lingtong Min, and Yanning Zhang. Building a multi-modal spatiotemporal expert for zero-shot action recognition with clip. Proceedings of the AAAI Conference on Artificial Intelligence, (119))9689-99999 Apr. 2025. doi:
URL https://ojs.
aaai org/ index. http:////////attttt/////////////
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 11941- 11952, 2023. URL htt.tttt::t//t/tttttt https://api. www.seeiacaccclaaanoooroo otg.aaaaocccccol.ooooorrrrraaaaa
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tumed audio-visual language model for video understanding. are preprint arXiv:2306. 02858, 2023.
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Vidco instruction tuning with synthetic data. arXiv preprint 2xXiv:2410002716 2024.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. arXiv preprint arXiv:2109. 01134, 2021.
Auzheng Zhu, Jun Chen, Lingxi Xie, Xiaopeng Zhang, Qi Tian, and Maojun Feng. Part-aware unified representation of language and skeleton for zero-shot action recognition. ar preprint ar arXxw:2308... 15868, 2024.
Linchao Zhu and Yi Yang. Actbert: Learning global-local video-text representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), June 2020.
Fatimah Zohra, Chen Zhao, Shuming Liu, and Bernard Ghanem. Effectiveness of max-pooling for fine- tuning clip on videos. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR) Workshops, pp. 3291-3293903 -300, June 2025.
17


Reviews:
[{"review_id": "r4dxWDtLA6", "reviewer": "Reviewer_UiFG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper makes two primary contributions. \n\nFirst, it introduces ActAlign, a novel framework that casts zero-shot fine-grained video classification as a sequence alignment problem, aligning LLM-generated sub-action scripts with video frames through Dynamic Time Warping. This reformulation enables a training-free, domain-general approach that preserves the open-set generalization of pretrained image-language models. \n\nSecond, the paper demonstrates strong empirical results on the ActionAtlas benchmark, showing that ActAlign significantly outperforms billion-parameter video-language models while using far fewer parameters, with ablation studies confirming the impact of temporal alignment and context-rich prompting.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper’s main claims are generally well supported and accurate. The central idea of casting zero-shot fine-grained video classification as a sequence alignment problem using LLM-generated sub-actions and DTW is clearly implemented and validated. \n\nExperimental results on the ActionAtlas benchmark back up the performance claims, showing significant gains over both CLIP-style baselines and billion-parameter video-language models, and the ablation studies confirm the contributions of temporal alignment and context-rich prompting. \n\nThe claims of interpretability and parameter efficiency are also substantiated through visualizations and comparisons. Overall, within the scope presented, the claims are credible and well supported.", "audience_yn": "Yes", "audience_explanation": "Yes, this paper would likely interest TMLR’s audience. It tackles a rlevant problem and does so with a simple but clever idea: using LLM-generated sub-actions and DTW to add temporal structure to image-language models. The approach is lightweight, training-free, and interpretable, which makes it appealing compared to heavy video-language models.", "requested_changes": "The paper looks well executes and complete. However, some minor changes that could strengthen the study include:\n\n1. Adding validation on an additional dataset beyond ActionAtlas to better support the claim of domain generality. Even a smaller benchmark (e.g., Something-Something, Kinetics subset) would strengthen the paper.\n\n2. Provide a scalability discussion or analysis regarding runtime and memory cost when scaling to longer videos or larger class vocabularies, given the $O(NMT)$ complexity of DTW alignment.\n\n3. Include more details and examples of the LLM-generated sub-action prompts and outputs (e.g., in the appendix). This would improve reproducibility and help readers understand how much performance depends on prompt design.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Rzjb2SWZQ2", "reviewer": "Reviewer_Nrv2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "this paper proposes a new method ActAlign which is a training-free, zero-shot framework that re-purposes fine-grained video recognition as sequence alignment between an LLM-generated \"expansion\" or script of temporally ordered sub-actions (outputted only from the action name used in teh prompt) and framewise visual embeddings. It uses a pretrained image–text model (SigLIP) to embed frames and sub-actions and applies Dynamic Time Warping to score alignments—requiring no video–text supervision and remaining domain-general/model-agnostic. The empirical results are interesting and the ablation studies seem adequate.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The empirical evidence is clear. The authors have agreed to release the code too.", "audience_yn": "Yes", "audience_explanation": "The paper is about multimodal embeddings and zeroshot video/text alignment for automatic subaction annotation; beyond the main algorithm even the techniques are interesting and should interest the community.", "requested_changes": "This is not my field of research, and the experimental results are pleasantly suprising to me. My main concern (not really a concern, but more of a surprise) is how the embeddings generated through SigLIP (\\phi_v, \\phi_t) somehow embed the video and the text in the same domain so as that the simple dot product works. I understand the semantic expansion of a subclass into temporal subactions by LLMs intuitively, and I can see how a pre-trained LLM can handle this easily (thereby allowing the “zero-shot” no-explicit labelling effort). But is there a reason that the embeddings of vector and text outputted by SigLIP should be able to embed them in the “same” semantic space ? Is this something known from before ?\n\nDid the authors think of any ablation experiments for different sequence of subactions generated by the same LLM through repeated or slightly different prompts? Would the (difference in) lengths of these subaction-sequences have an impact? I realize the authors have mentioned ablation studies and in light of expanded contextual information the impact of initial sequences that were generated may not be big. \n\nThe manuscript could benefit from additional details such as SigLIP and why it should embed video frames and text in the same frame? Similarly additional details on how signal smoothing was done, and how contextual expansion was done (e.g. adding the name of the sport) without human intervention (since the paper specifically touts ‘zero-shot’, I would imagine there is no human labeling required for this)\n\nPlease proof read, there are a few lurking typos/consistency issues:\n\n-\t “While, their high-level label is barely recognized by actionrecognition models (e.g. such as Roundhouse kick in Kickboxing).”\n-\tSigLIP vs SigLip,and other model names such as simvlm in the references should be SimVLM\n-\t similarly subaction vs sub-action, modeling vs modelling \n-\tA(i, m)  vs A(m,i) in eq 9 vs 12", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bzvwNtHBQc", "reviewer": "Reviewer_pgw3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes ActAlign, a training‑free, zero‑shot method for fine‑grained video classification that casts recognition as language‑guided sequence alignment. For each candidate class, a LLM decomposes the class name into an ordered list of sub‑actions; frames and sub‑actions are embedded with a frozen image–text model (SigLIP), and the method aligns the two sequences using Dynamic Time Warping (DTW). The final score is the mean similarity along the optimal warping path, and the predicted class is the one with the highest normalized alignment score. On ActionAtlas, ActAlign reports 30.29 Top‑1 vs 22.94 for SigLIP mean‑pooling and competitive/better results than several 7–8B video‑language models while using 0.9B image–text parameters. Figures on pp. 2 and 5 visualize the idea and pipeline; Tables 2–3 ablate DTW, context augmentation, and smoothing\n\nPro:\n- Casting recognition as alignment in a pretrained image–text space is elegant and easy to reproduce. No video–text training or tuning is required\n- The alignment path and sub‑action scripts make predictions inspectable in a way most VL baselines are not\n- The qualitative heatmaps (Figures 5–6) help diagnose behavior\n\nCon:\n- The strongest competing VLM (Qwen2‑VL) is essentially tied (30.24 vs 30.29 Top‑1), and several VLM numbers are reported only for Top‑1 (Table 1). It’s not fully clear that prompting/decoding for these large models is optimized under the multiple‑choice setting used here\n- The largest gain in Table 2 comes from context augmentation of the text (30.07 vs 25.72 after DTW). It is hard to isolate how much improvement is due to adding richer text vs DTW ordering\n- Eq. (9) applies a sigmoid with “learned” $\\alpha,\\beta$, yet the method is otherwise training‑free. It is unclear whether these are fixed from SigLIP pretraining, hand‑set, or fitted on ActionAtlas.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Table 1 (Section 4.2, p.7) shows ActAlign at 30.29 Top‑1 on ActionAtlas vs 22.94 for SigLIP mean‑pooling, with corresponding Top‑2/Top‑3 improvements. ActAlign (0.9B params) is on par with or slightly above Qwen2‑VL‑7B (30.24 Top‑1) and clearly above several other 7–8B models. Table 2 (p.8) starts from the mean‑pool baseline and adds components stepwise: +DTW (+2.8 Top‑1), +context augmentation (+4.35), +smoothing (+0.22).", "audience_yn": "Yes", "audience_explanation": "This topic is clearly interesting to the community", "requested_changes": "1. Please spell out the exact prompts and decoding for each VLM, whether the models are constrained to choose among the multiple‑choice options, how many frames/tokens were fed, and whether audio was used.\n2. The baseline Mean‑pooled SigLIP with the same context template\n3. The baseline Bag‑of‑subactions that averages sub‑action text embeddings and compares to mean‑pooled frames.\n4. Add a “randomly permuted sub‑action order” and a “reversed order” control\n5. Report variance across multiple seeds (or LLM samples) for Top‑1; include one qualitative example where different phrasings lead to different alignments", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)