Does Video-Text Pretraining Help Open-Vocabulary Online Action Detection?
Authors: Qingsong Zhao
Yi Wang
Jilan Xu
Yinan He
Zifan Song
Limin Wang
Yu Qiao
Cairong Zhao
Venue: NeurIPS 2024 poster (2024)
Keywords: Online action detection
Video understanding
Zero-shot transfer
Primary area: machine_vision

Abstract: Video understanding relies on accurate action detection for temporal analysis. However, existing mainstream methods have limitations in real-world applications due to their offline and closed-set evaluation approaches, as well as their dependence on manual annotations. To address these challenges and enable real-time action understanding in open-world scenarios, we propose OV-OAD, a zero-shot online action detector that leverages vision-language models and learns solely from text supervision. By introducing an object-centered decoder unit into a Transformer-based model, we aggregate frames with similar semantics using video-text correspondence. Extensive experiments on four action detection benchmarks demonstrate that OV-OAD outperforms other advanced zero-shot methods. Specifically, it achieves 37.5\% mean average precision on THUMOS’14 and 73.8\% calibrated average precision on TVSeries. This research establishes a robust baseline for zero-shot transfer in online action detection, enabling scalable solutions for open-world temporal understanding. The code will be available for download at \url{https://github.com/OpenGVLab/OV-OAD}.

Does Vixeo-Text Pretraining Help Open-Vocabulae Online Action Detection?
Qingsong Zhao 11. Yi Wang Jilan Xu Yinan He
Zifan Song'
Limin Wang Yu Qiao Cairong Zhao
Tongji University
2 Shanghai Al Laboratory Nanjing University
Fuuda University
{qingsongzhao, zhaocairong) {wangyi, heyinan, @pjaabb.ppp........ org. lmwang njj@gmail.... com
edu.
Abstract
Video understanding relies accurate action detection for temporal analysis. However, existing mainstream methods have limitations in real-world applications due to their offline and closed-set evaluation approaches, as well their depen- dence on manual annotations. To address these challenges and enable real-time action understanding in open-world scenarios, propose OV-OAD, zero-shot online action detector that leverages vision-language models and learns solely from text supervision. By introducing object-centered decoder unit into Transformer- based model, we aggregate frames with similar semantics using video- text correspondence. Extensive experiments four action detection benchmarks demonstrate that OV-OAD outperforms other advanced zero-shot methods. Specif- ically, it achieves 37.5% mean average precision THUMOS' 14 and 73.8% calibrated average precision on I'VSeries. This research establishes a robust base- line for zero-shot transfer in online action detection, enabling scalable solutions for open-world temporal understanding. The code will be available for download at httt:://ggtuubb..o//////////////
Introduction
Action detection is practical and demanding technique in intelligent video analysis, including anomaly detection [32] in surveillance and human coman-c interaction (29) in embodied studies. Considering high variations in possible human behaviors with dynamic scenes, action detection is significantly challenging. In this regard, most action detection approaches go offline, involving the closed-set classification and localization of actions (a few predefined categories) within the long untrimmed videos. However, real-world applications concerning real-time understanding (e.g. surveillance) require estimating the action without accessing future frames. Further, closed-set discrimination limits the applicability of action detection, and it also asks for manually annotating all action categories, especially in complex scenarios such as wide variety of actions or events in driving scenarios, which is both costly and time-consuming.
To address these challenges, formulate online action detection in open-vocabulary and transfer popular vision-language models (VLM) to tackle this problem via only paired vision-text supervision for learning. A growing number researchers have been investigating how leverage the capabilities of powerful VLM to address specific novel visual tasks of interest. For instance, existing studies [21 25 33 have explored the transfer of visual knowledge from VLM to a video understanding task to achieve zero-shot temporal action detection (ZS-TAD). Applying VLM to ZS RS-OAD is non- trivial. Plain solutions, the ZS-TAD approaches mentioned earlier, involve partitioning subset of
31 I
corresponning author, Tequal contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Closed-set online action detection
Open-vocabulary online action detection (ours)
"dtecus"
barlignosed
"o is drushing coilet"
"Yry aggs"
"chere ts person dring yoga in the viato"
Visual Encoder
Text Enooder
Visual Encoder
Text Encoder
OAD Model
OAD Model
photo of oction jacscus/fry esgs""
training
inference
training
inference
Figure I: Overview of the online action detection. Models trained on closed-set actions (e.g., discus and brush toilet) are unable to detect the novel action class (e.g., fry eggs). We train a visual-text dual-encoder on web-collected video-text pairs without using frame-scale labels. can discriminate arbitrary action classes.
base-to-novel category data from the downstream dataset and further fine-tuning the visual language model with prompt-based technique to adapt it for novel tasks. However, this poses several issues. First, sliding-window window frame sampling used in online action detection often leads to a high proportion of background frames, which contradicts the assumption of low background information for VLM training. Second, the OAD model fails to reach future frames during training, making it difficult to sample all category labels in the same batch. This is detrimental to the optimization of image-text contrast loss since the contrast loss favors the diversity samples. We experimentally explored these hypotheses in Sec. 4.1
Inspired by CLIP tection models purely through text supervision, thus avoiding the use of fine-grained temporal annota- tions. To this end, we introduce the proposed object-centered decoder unit into the Transfomrer-hh based model, enabling automatic aggregation of frames with similar semantics with textual supervision exclusively. Fig. [illusttrtess the overall framework of our method. By employing contrast loss during training on extensive video-text pairs, we enable the model to be zero-shot transferred to Online Action
given frames representation from a powerful VLM, we learn online motion de-
different action detection vocabularies. Hence, we name our model Ogpen-Vocabule Detector (OV-OAD). We pre-train OV-OAD on the video-text datasets, and manual frame-level labels are not used whatsoever. We propose three proxy tasks including alignment of current frame-text embedding, background frame mask prediction, and alignment of multi-label video-text embedding for training. The first task enables the model to prioritize discriminative information from neighboring frames. The second task enables the successful detection of complex background frames in natural videos. The third task mitigates the impact of caption noise in web videos. Our model was evaluated on four action detection benchmarks without any fine-tuning i.e., THUMOS' 14 [19], EK100 [13], FineAction [27] and TVSeries [14] in a zero-shot manner. Extensive experiments demonstrate that our model outperforms other advanced zero-shot methods. The main contributions are summarized as follows:
We investigate the critical problems of how to capitalize pre-trained visual language models for zero-shot online action detection in untrimmed videos.
We introduce a novel video-text dual-encoder architecture, namely OV-OAD, to perform open-vocabulas online action detection. Experiments on the downstream datasets show that our model successfully learns clusters of similar video frames and transfers them to multiple action semantic vocabularies in zero-shot manner.
To our knowledge, our work is the first to explore zero-shot transfer from text supervision alone to the online action detection task without relying on any precise frame-scale labels. And we have established a robust baseline for this new setting.

---

2 Related Work
Pretrained Vision-Language Model (VLM). Recently, the joint image-text learning paradigm [66] has been successfully scaled up by CLIP [4] and ALIGN [20] with the massive web data. After that, researchers have proposed many variations, including CLIP-Adapter [88] GLIP 24 and so on. One VLM's visual encoder can leverage textual descriptions to recognize objects or scenes in images when category-speciff samples are unavailable. In video domains, similar ideas have been
explored for action recognition (e.g., ActionCLIP [4)] ViFi-CLIP (55) and video understanding
(e.g., CLIPBERT 22
EffPrompt 21
is a video understanding task involving simultaneous recognition and localization of actions within an uncut video. Recently, efforts [2] 31 6 25 33 have utilized the pre- pre-trained vision-language model to give the TAD models with the capability to recognize novel action classes. For example, EffPrompt [2 proposes two-stage fine-tuning scheme for zero-shot temporal action detection (ZS-TAD) by incorporating task-specific prompt vectors. STALE [3] introduces a one-stage model to mitigate the error propagation problem encountered by EffPrompt by utilizing parallel classification and localization design. T3AL [25) presents a training- free ZS-TAD that leverages an effeettiee test-time augmentation strategy and external knowledge derived from generated subtitles. It is important to highlight that all those ZS-TAD methods adopt a Nosvel fine-tuning approach, which involves dividing the dataset categories into training and inference subsets. Due the strong diversity among the TAD datasets and the limitation of its scale size, it has been challenging to showcase the model's generalization capabilities. By contrast, we train our open-vocabulary online action detection model with large-scale video-text pairs only. During inference, the model does not require any additional fine-tuning recognize arbitrary action classes.
Zero-Shot Temporal Action Detection
Temporal action detection (TAD [6 11 48 26
Onlne Action Detection. Contrary to offline motion detection, OAD does not predict action onset timing and cannot access future visuals. Arguably, OAD emphasizes real-time response and openness of recognition over the accuracy of action classification in practice. Existing researchers [17 45 [5
38 4 often use closed-set datasets for training and testing, boosting recognition accuracy and speed on single datasets. For example, IDN 15 improves the discriminative representation of actions by selectively accumulating relevant information. OadTR [1] incorporates the fusion of current features and future frames for identifying ongoing actions. LSTR [46] captures contextual dependencies in videos. leading to improvements in action identification. E2E-LOAD [ proposes an and-to-end framework that integrates a stream buffer between the spatial and spatiotemporal modeling- MAT 38 introduces a nemooy-anticipation- based pipeline to model the entire temporal structure of video. In contrast, our work shifts the focus to enhancing the open recognition capability of OAD models. We aim leverage readily available video-text pairs zero-shot transfer visual knowledge into the OAD model, thereby improving the model's ability to handle unseen actions.
3 Methodology
Consider untrimmed video V. generate clip sequence employing a sliding window of length T on V that moves frame by frame. On the t-th slide, we get clip V (V:-. Vt-1, V1} where V, denotes t-th frame. Online action detection is predict action probability Ør in each frame V. using only past and current observations. We propose an open-vocabulary online action detection model (OV-OAD) for zero-shot online action detection with text supervision only. Our approach, illustrated in Fig. 2 consists of two primary components: a visual encoder and a text encoder. The visual encoder comprises distant neighboring frame transformer block and an action clustering one. We pre-train OV-OAD on a web-scale video-text dataset. In inference, we transfer the trained model to the zero-shot online action detection without any fine-tuning, as described in Sec. 3.3, and it can predict arbitrary action classes. We ignore the subscripts of individual images and text pairs for simplicity.
3.1 Architecture
3.1.1
Visual Encoder
The visual encoder is composed of a distant neighboring-fran transformer block (with light grey background in Fig. 2. and an action clustering block (with yellow background in Fig, 2 with an
3

---

Aug
conttrs
Object-Centric Decoder Unit
Transformer Encoder
Kask
1x K
Object-Centre Decoder
Lmaak
Addd & Norm
Feed
Transformer Encoder
Forward
Yi [-1]
ALddd Norm
MEP
ourrent
Group Embedding
Multi-Head
Slot
K
Transfomer Decoder
4X
Attention
Text Encoder
Positional Enooding
Fosiloonl
Aterrisnn Maxe
Norm
1/
Encoding
Extended Past Frames
Neighboring Frames
Multi-Head Attention
Live Streaming Video
Caption Pseudo Maskk
fou. "bockground" ......... "pick ap "gge".....n Ip..."put se into oid pon ponf
Ongoing Caption Line
Figure 2: The illustration of our OV-OAD (best viewed in color), is formulated in visual-text dual- encoder manner. Specifically, the visual encoder consists of a distant neighboring frame transformer block (9 PNNTE R. light grey backdrop) and an action clustering block AC- yellowish backdrop). The ! DNTT is built with Transformer Decoder units, which take the neighboring tokens and distant past tokens as inputs. The V Ac built with our Object-Centric Decoder and vanilla Transformer Encoder units, which take the output tokens (orange squares) and learnable group embeddings (purple gradient squares) as inputs. During testing, the OV-OAD handles each incoming video snippet online, absent future context.
object-centric decoder. For given video clip-text pair, denoted as (V', ,T,, we initially divide the video into extended past frames V E [r and neighboring ones V E Rnxd All frames y processed in the Distant Neighboring-frams TRansformer block DNNTR which comprises the transformer decoder unit (37. Then the neighboring frames V% aggregated with the information of past frames are fed into the Action Clustering block V Acc, along with learnable group embedding (G IR (x),, that aims bind the neighboring frames into clusters. The outputs of the visual encoder are defined as:
(G, V')"= = : =vv VacC( (nc((G:))]]
WONTR(VP, V%).
where the symbol o represent a function composition, g E Rexd denotes the encoded group embeddings, while yd E IRd refers to the output video frame tokens.
Distant Neighboring-Framdt Transformer. It utilizes neighboring frames as queries to extract information from frames in the distant past. It is predominantly composed of four layers of standard transformer decoder units. Each frame the video is augmented with 1-dimensional absolute positional encoding, independently applied both past and neighboring frames. A directional attention mask is incorporated only for the neighboring frames, ensuring unidirectional information flow toward the current frame. Intuitively, the current frame embedding [ with more spatial information, will be aligned with the corresponding text
video clip representation.
Action Clustering. The action clustering block namely V AC assembles frames into groups and aligns the groups to human-underssandable categories data-driven manner, only supervised by video-text pairs. It consists of three steps, i.e., the frame-to-group binding that assigns static frames with similar semantics to group, the group-to-action mapping that computes the cosine similarity between the group embeddings with the action labels, and background mask proposing that predicts set of binary masks by computing statistics from the frame-group similarity matrix A.
In its training, devise binary prediction task to cluster frames into background and non- background as well as the grouping. The background mask MB is predicted as:
KaQ ) MB = A.g.
A == softmax(
Vd

---

where A E R is derived from the attentional weights in slot-attention computation. It denotes the likelihood of each video frame being assigned into k learnable group embeddings. In Eq. (2] the group embeddings serve as the query Q, and frame tokens serve as keys and values. With the softmax operator normalizing over k, and the output of the slot-attention block A ATV V. which of shape k x d. The tensor T E R 1xx represents the d-dimensional embeddings of the video caption.
In its evaluation, similar the computation of Ma, video clip's action prediction score namely PAC E IRaxC can be calculated as follows:
P Pcc one-hot(arg max(A)) 9 T vat-
(3)
In contrast to Eq. 2 the frame-group similarity matrix A requires one-hot hard coding, while Tool E RCxd is encoded by the C' categories descriptions in the test set.
Object-Centric Decoder Unit. To group frames, we give an Object-Centrie (OC) Decoder unit employing slot attention mechanism (28) focused on the query object during cross- -toss-attentoon computation. works similarly the previously proposed GroupViT [43 and OVSegmentor 44 methods, which are specifically designed for semantic segmentation tasks.
As illustrated in Fig. 2 it requires two sets of inputs, i.e., target queries 9 consisting of fixed number of k encoded grouping embedding and n input frame tokens V% to be queried, where can be large number. Following a layer of multi-head self-attention, G is transformed to G'. G' is then employed as a query in the second layer of multi-head dot-attention, while the frame tokens V/ serve as the key and value. These two steps can be expressed as
T
9. GT vd
V/ O(G'T Vd
g' == t=soflmart
G, SlotALtn(((O'). v%) = softmar
V% (4)
where 0: kxd + Rxd denotes the dropout and norm operations.
3.1.2 Text Encoder
We adopt the pre-trained Text Transformer defined in CLIP 34 as the text encoder Vr. To bridge the gap between image-text and video-text models, we utilize the Adaptformer technique [7]. for lightweight transfer learning. We adopt the practice setting up parallel adapters in each transformer block of Tr. For the input text, we use the CLIP Tokenizer (34) to add the tokenizer start [SOT] and tokenizer end [FOT] at the beginning/end. The text embedding is computed as T Vr(()) where T E R xi represents the tokenized caption with a length l.
3.2
Video-text data for online action detection.
Web video-text datasets 8 42 are abundant in data volume, but their captions are typically pre- labeled by image-text generative models (23 9 and then filtered by humans semi-automaticallyy Consequently, these annotations are extremely noisy and include fantasy elements, despite being semantically rich and diverse. Additionally, we employ a sliding window approach to capture both the video frame and its corresponding text description during the live-streaming video. These factors introduce inherent bias in the visual and textual information a sample pair. To achieve relatively accurate textual captions of the visual information in the video, we employ multi-labed contrast learning strategy. To generate multiple captions, we employ a linguistic analysis tool (e.g., the nltk toolkit 2 when the raw labels not enough. This tool extracts verb-object phrase structures from the given descriptions, which are then utilized as keywords to create additional captions, drawing on CLIP's prompting engineering. To handle redundant captions, we choose the text associated with the highest number of frames as the global descriptor, and the remaining captions are utilized to compute the multi-label contrast loss. In the absence of any tags, we utilize the keyword "background" for sentence construction and as the global descriptor.
3.3 Optimization and Inference
We train the model through three proxy tasks, i.e., video-text alignment, current frame-text matching, and background mask prediction. The total loss is: Liotal Lcontras + aeCeument + 3. where a, B are trade-off parameters controlling the relative weight of the above cost functions. Each used loss is detailed below.
Current Frame-Text Alignment. We adopt Image-Text Contrastive loss (Lrrc) to learn whether the current frame image matches the caption. We denote the image embedding (from the distant neighboring- frame Transformer block) and the text embedding as z and 2"" respectively. Both
5

---

embeddings are projected into 512-dimensions joint feature space before calculating the matching loss. The current frame-text contrastive loss Ceurrent is:
Leurent Lire(z' 2²") 1
log exp(2! 2T/TT) + log exp(2? 2T/TT) Σ, expl2? )
(5)
exp(2 /T//T)
where 7 is a temperature parameter to scale the logits, and B denotes the batch size. Multi-Label Video-Text Alignment. We employ the multi-label video-text contrastive loss to align the visual and language representations for enhancing the textual representations of videos. The multi-label video-text matching loss Lcontras is:
ΣM earp( 2 Fm/p /T)
M log
exple 2m 1T) Σj expl(, Tm 22/TT)
Lcontras = Liro(=" 20)
log Σm Σj exp(zt 22 =T)
+
2
M
(6)
where zV is computed as the average of the output V from the visual encoder. {2 2T Tar are text embeddings, constructed in Sec. 3.2 All embeddings are mapped into 256-dimensional vectors. Refer to Eq. 5 zoo denotes a global descriptor selected from the multi-label captions.
Background Mask Proposal. Through video captions, can roughly determine which frame chunks are background and which ones correspond to actions. This prior helps the action clustering block effectively group and bind the majority of background frames. To enhance the concatenation region between the predicted background mask Mn and the caption pseudo mask Mar, we employ a per-frame binary mask loss. Following Maskformer [10], we use dice loss 33 for our mask loss, i.c., Lmask = Ldice (M,.Mrr). The binary mask MGT is derived from the neighboring frames Vi of the input to the action clustering block. It is set "1" for frames with a caption and "0" for frames without captions.
Zero-Shot Online Inference Similar to CLIP's zero-shot transfer [4),, our distant neighboring- frame Transformer block can assign the current frame image to the semantic category with the highest image-text embedding similarity. During online inference, similar to Eq. [ the action prediction score namely PANTR E IR IXC for the current frame of a video clip can be written as
PDNTR V [-1] Tout-
The action clustering block can also estimate frame action without fine-tuning. We calculate the similarity between the embedding of each frame token and the text embedding of the dataset. Then, we assign each frame token to the corresponding category with the highest similarity. This zero-shot transfer pipeline is depicted in Eq. In summary, the final action prediction score It E R IxC for video clip V can be expressed as: y -PAcl- + aPDNTR-
Experiments
Our run experiments on NVIDIA V100 x8 using Pytorch 1..0.. During both training and inference, we resample the video raw frame rate (e.g., 24/30 FPS) to 4 FPS, and resize images to 224 x 224 49 [46]. For feature extraction, we employ the CLIP model (ViT-L). Specifically, the visual encoder computes a series of image patch tokens along with one global token (aka, CLS token) for each frame, and we utilize the normalized CLS token as the output feature encoding. Unless otherwise specified, all parameters of the visual encoder in our OV-OAD model are initialized from scratch, while the text encoder is initialized with the CLIP, except for the additional Adapter parameters. We train our OV-OAD for 30 epochs with 2 warm-up epochs using the Adam optimizer with weight decay 5e - It uses a cosine schedule with a batch size of 256, and the initial learning rate is 1.6e -4 Pre-training Datasets. We use the filtered ActivityNet v1.3 (aka, ANet (3) datasets for training, which are originally collected 44M and 14950 untrimmed video-caption pairs from the web, respectively. However, the videos within the Intern Vid dataset typically have longer durations compared to ANet, and the average percentage of foreground frames with annotations on these videos is only 27.4%. We sort the ~4M videos in the Intern Vid dataset according to the number of caption annotations they contain and take the top 5000 videos (namely Intern Vid-5K) for training. For ANet, we utilize the prompting technique (following [35] 12) to convert the short action tags into sentences. And we combined its training and test sets and utilized them collectively for training. Please see Appendix A.2 for complete dataset preparation.
Viderniid110M 1OM-LO (aka, Intern Vid (22) and the
Benchmarks. We follow previous works [7 5 and evaluate our model for the zero-shot online action detection on the validation splits of the THUMOS 14 [19], TVSeries 14, EPIC-Kitchens-II 100
(aka, EK100 [3). and FineAction (27 datasets. THUMOS' 14 and TVSeries datasets comprise 20

---

Table I: Benchmark evaluation on THUMOS' 14 and
Table Base-to-novel and mally-supervised
TVSeries. "TVid" denotes Intern Vid-5K.
evaluation on THUMOS 14 dataset.
THUMOS'14
TVSeries
Train-Test Split
THUMOS
Methods
Arch
Methods
mAP (%)
cAP (%)
mAP (%)
CLIP-I
VIT/B
28.0
67.7
OadTR-D8
47.4
CLIP-I
VIT/L
29.6
69.3
LSTR
47.7
100% Seen
CLIP-II
VIT/B
29.1
69.5
MAT-D48
48.2
0% Unseen
CLIP-II
VIT/L
30.9
71.1
CLIP-1
28.0
CLIP-III III
VIT/B
29.7
71.6
OV-OADt
37.5
OV-OAD (IVid)
VIT/B
33.2
73.8
OV-OAD (ANet)
VIT/B
37.5
73.2
OadTR-DS R-
33.7
LSTR
26.9
75% Scen
Table 2: Benchmark evaluation on FineAction and
25% Unseen
MAT-D48
25.5
100.
CLIP-1
38.6
FineAction
EK100 (Verb)
OV-OAD!
44.6
Methods
Arch
OadTR-D8
9.6
mAP (%)
cAP (%)
LSTR
9.1
50% Scen
CLIP-I
VIT/B
26.5
40.1
50% Unseen
MAT-D48
7.9
CLIP-II
VIT/B
27.8
39.9
CLIP-I
28.6
OV-OAD
VIT/B
29.2
41.4
OV-OAD!
35.9
and 30 foreground action categories, respectively. EK100 and FineAction datasets comprise 97 verb classes and 106 foreground action labels, respectively. An additional background class is considered for all datasets. Turn to the Appendix A.3 for dataset details.
We evaluate metries for online motion detection based on previous studies 38
46
Specifically,
we applied per-frame mean average precision (mAP) on THUMOS' 14 [19] and FineAction 27 and
per-frame calibrated average precision (cAP) on TVSeries 14 and 1K100
13
4.1 Comparison with Existing Methods
We conducted comparison of the zero-shot online motion detection metrics between our method and other zero-shot baselines. We also explore the base-to-nove fine-tuning approach for zero-shot OAD and compare to our OV-OAD model.
Comparison with Zero-Shot Baselines. We utilize visual language models with image zero-shot capabilities (i.e., CLIP) for comparison. The inference process of online action detection involves sliding frame-by-frame sampling on an untrimmed video and subsequently predicting the action class of the last frame (aka, the current frame). We actions based on single frame image to simplify the inference. To zero-shot transfer CLIP to online action detection, We first extract the features of the frame images using its visual encoder, and then, we compute the similarity between the visual features and the text embedding of the dataset action labels. We can perform several non-parametric processes on the visual embeddings, including: 1) Averaging the visual embeddings of the neighboring frames to obtain the visual feature of the current frame, named CLIP-II: 2) Non- non-paraettric clustering of all sampled frames (e.g.. K-means algorithm), followed by averaging the visual embedding of the group to which the current frame belongs, resulting in the visual feature, named CLIP-III; 3) Directly using the visual embedding of the current frame as the visual feature, dubbed CLIP-I Table [ presents the experimental results, clearly demonstrating the superior performance of our OV-OAD over other non-parametric zero-shot methods. It is worth noting that enhancing the scale of the visual language model leads to a moderate increase in single-frams action prediction accuracy. However, this improvement is constrained, indicating that relying solely robust image discrimination is insufficient for achieving high performance. Consequently, it is essential to incorporate temporal structure information into the learning process for effective online action recognition.
the sliding length and classify the
Furthermore, as depicted in Table 2 we delve into the zero-shot performance OV-OAD on more demanding datasets. We evaluate OV-OAD's performance on the first-view shots dataset called EK 100 This dataset comprises first-vies view shots and notably deviates from the ANet data distribution. We also assess the performance of OV-OAD on the large-scale dataset, FineAction, which includes ~0,000 uneut videos categorized into 106 action classes. The outcomes indicate that OV-OAD exhibits superior generalization capabilities in online action detection when contrasted with CLIP.

---

Table 4: Ablation study on the current frame-caption contrastive loss (Lurrent) and background mask loss (Lmmas)). The baseline only uses the multi-label video clip-text contrastive loss (Crontros).
Table 5: Results of different num- ber of frame tokens about V% and V/
V/
Lcontras
Leurrent
Lemask
mAP (%)
4/
#4
#8
32.9
#16
36.1
37.1
/
36.3
#24
36.7
37.5
33.6
#28
37.1
36.9
/
37.5
#32
35.9
36.5
Table 6: Results of different num- bers of Transformer units for AC and V DNTRR.
Table 7: Results of different designs of the V DNNTR block. "TR" denotes Transformer. "OC" means Object-Centric.
The penultimate row is our proposed OV-OAD design.
V WNNTR
V VAC
mAP (%)
Block V VOTTR
Clustering Unit mAP (%)
0
9-3
31.9
n/a
OC Decoder
33.3
0
5-3
33.2
4x TR Encoder
OC Decoder
35.6
0
6-6
33.3
4x TR Encoder+ Crross Atin OC Decoder
36.5
4
6-2
37.5
4x TR Decoder
OC Decoder
37.5
4 6-0
36.9
4x TR Decoder
TR Decoder
37.0
Comparison with base-to-novel methods. We compare base-to-novel fine-tuning methods and fully-supervised transfer to online action detection on the THUMOS" 14 dataset. For base-to-novel generalization, we integrate three well-known Transformer-baaee online action detection models including OadTR [41, LSTR [46] and MAT (38] with a text encoder using the image-text contrastive loss. To ensure statistical significance, we adopted the random sampling setup and dataset partitioning method proposed by [21]. For our experiments, we employed two evaluation settings on the THU- MOS' 14 dataset, i.e., training on 75% of the action categories and testing on the remaining 25%, and training on 50% of the categories while testing on the remaining 50%. We followed the experimental finettuning setup of (35. including the Adam optimizer, the learning rate 1 and the Cosine decay function for training. For fully supervised transfer, we also train these models on 100% of the action categories with inputs of video frame features extracted by CLIP/ViT-B. We followed the training for the experiment, which remained consistent except for the different feature extractors. All experimental results are reported in Table f indicates that the model has not seen any categories and directly tests the performance of the unseen categories. The MAT-D48 indicates that MAT utilizes the ground truth future frames (48 frames in 12 seconds) during training. We observe that the recent MAT method achieves better performance in the fully-supervised setting, but its performance poor in the base-to-novel setting. One can find that our OV-OAD model outperforms the competition even without utilizing any training data. The results indicate that the base-to-a base-to-nover fine-tuning method is not suitable for direct application to the zero-shot online action detection task. The limited availability of data may be contributing factor to the unsuitability of the base-to-novel fine-tuning approach for online motion detection models.
setup in [1 46 38
4.2 Ablations
Proxy Tasks. We aim to validate the effectiveness of the three proxy tasks we introduced, namely current frame-text alignment, multi-label video-text alignment, and background mask proposal. As reported in Table 4 our baseline model employs the multi-label video-text contrastive loss Ccontras only, upon incorporating the current frame-text matching loss Courrent. we observed a substantial improvement of 3.4% in mean Average Precision on the THUMOS' 14 dataset. The performance improvement is due to the model's ability to capture spatio-temporal information from extended past frames. In addition, the prediction of the background frame mask also leads to improvements, and combining both results in optimal performance. This finding suggests that enhancing the model's capability to detect background frames is equally important.
Number of Layers and Frames. We first conduct an evaluation to assess the influence of inputting different numbers of neighboring and past frames on the model's performance. As depicted in Table 5) our OV-OAD model demonstrates flexibility with different frame choices, resulting a maximum performance variation of 1.6%. Note that, the highest mean Average Precision is achieved when

---

utilizing V 8 and V/P = 24. In addition, we investigate the effect of the number of network layers different blocks on the performance. The results are presented in Table 6 we find that increasing the number of layers for the first Transformer Encoder of our action clustering block results in notable decline in performance. Conversely, a small number of layers for the last. Transformer Encoder proves to be sufficient.
Distant Neighboring- Frame Transformer. We further investigate the design of the proposed distant neighboring-framme transformer block V WDNTE R- Unless specified otherwise, we employ 2-second neighboring frames, 6-second distant past frames, and CLIP//VT-B pre-trained features.
a) Can we remove the DNTR block? To implement this, we directly input all 8-second sampled video frames into our action clustering block, where the object-centered group module could easily cluster frame tokens that exhibit similarity. To ensure fairness, we set the number of Transformer layers in the action clustering block to be equal to the total Transformer layers in OV-OAD. As can be seen from Table (row 4 vs. row 1). OV-OAD exceeds this baseline clearly. This also demonstrates the validity of our idea of applying neighboring frames to query spatio-temporal information from distant past frames.
b) Can we remove the final Transformer encoder in V VAC block? Experiments were conducted to analyze the impact of removing the final transformer encoder on the model's performance. The result presented in the Table 7 (row 5 vs. row 4) indicates a marginal performance decrease of around 0.6% upon removing the final transformer encoder. Additionally, this action results in 15% reduction in certain training parameters, specifically in the visual encoder.
<) Can V DNTA block be learned efficiently using the Transformer encoder unit? Here, we aim to explore whether the V DNTR block can be learned efficiently using Transformer encoders only. To be specific, we combine the neighboring with distant past frames and feed them into 4-layer V DNNTR block based on a standard Transformer encoder implementation. Table 7 (row vs. row 2) illustrates that this baseline is clearly lower than our V DNTr block constituted by the Transformer decoders. Furthermore, we introduced an additional layer of cross-attention after the 4-layer Transformer encoder to create new baseline. We aim to assess the effectiveness of the "bottleneck" design of cross-attention within the V VNNTR block. Note that such an implementation also completes the process of querying discriminative information from distant past frames. Table (row 4 vs. row 3) shows that the cross-attention design does exhibit effectiveness, but it falls short of achieving top performance.
d) Ablation for the Action Clustering Block. Here, we analyze the impact of an object-centric decoder compared to standard Transformer decoder unit within the action clustering block. Both are designed to bind semantically similar static frames into a group embedding. Table 7 (row vs. row 4) demonstrate that the object-centered decoder outperforms the standard transformer decoder in performance.
On Adapting Text Encoder For our baseline approach, following the initialization of our OV- OAD's 's text encoder with the CLIP's text encoder weights, we release the weights to continue training. Then, we conduct experiments to explore the performance impact of two separate modifications: I) fix the full backbone parameters and 2) incorporate the Adapter structure. The results are depicted in T one can see that leveraging the Adapter technique leads to a substantial improvement in performance. Moreover, to achieve the best results, it is necessary to release the backbone parameter and continue training.
4.2.1 Inference Speed.
We compared the model parameters and efficiency of our OV-OAD model with other methods on single NVIDIA Tesla V100 GPU, given in Tab. | Note traditional supervised learning methods, the efficiency bottleneck of the system is primarily attributed to the optical flow computation and its feature extraction. Our OV-OAD eliminates the need for optical flow computation and the extraction of apatio-tempora features from the RGB image. The overall system achieves an impressive inference speed of 292.3 frames per second (FPS). The findings indicate that our model has the potential to be deployed on standard online video capture devices, enabling real-time action prediction capabilities. In particular, LSTR demands a larger number of input frames for optimal performance, using 520 seconds of video for inference on THUMOS' 14, while our OV-OAD utilizes only 8 seconds. This means that LSTR requires 65 times more data than OV-OAD, which likely explains why model's inference speed is six times faster. Furthermore, the primary speed bottleneck for LSTR is the extraction of optical flow (8.1 FPS), whereas for our model, it is the extraction of image features (292.3 FPS).

---

Table Efficiency comparison on parameter (M) and inference speed (FPS)
Table 8: Ablation study on Text Encoder. Pre-trained
Frames Per Second
Fixed
Adapter mAP (%)
Methods
#Param
Optical Flow
RGB Feat
Flow Feat
36.7
Model
/
35.8
OadTR
75.8M
110.0
/ 37.5
LSTR
58.0M
8.1
70.5
14.6
91.6
/
35.5
MAT
94.6M
72.6
OV-OAD
109.5M
292.3
571.4
4.3 Limitations
End-to-end online motion detection systems require simultaneous learning of spatial and temporal structures for optimal results. Our OV-OAD model utilizes the CLIP's visual encoder to extract features from pure images. This would cause the network to focus too much on modeling foreground- centered spatial information the expense of modeling spatio-temporal structure information. This does not fit the requirements of action recognition for visual representations since learned RGB features from a video commonly contain some of the temporal structural information, e.g., RGB features extracted by TSN [39] have some of the properties of optical flow features. Therefore extending OV-OAD simultaneously model scenario and temporal information and enable action recognition with an open vocabulary remains a challenging problem.
Failure Cases. Our objective is identify categories that exhibit poor recognition as well as those with high recognition rates. We provide a list of categories with the highest and lowest action recognition average precision Table Additionally, we present visual samples of these categories in Fig. 3 We find that the detection accuracy decreases in scenarios where the foreground of the action is relatively low, and multiple actions share similar backgrounds (e.g.. "ClifiDiving" and "CliffShot"). However, OV-OAD demonstrates better performance when the foreground or interacting objects are more distinct, as observed in cases like "CleanJerk" and "PoleVault" These findings suggest that future enhancements focus improving the recognition of fine-grained actions through joint modeling of opatio-temporal information.
Table 10: Action classes with the highest and lowest performance on THUMOS' 14.
Action Classes CleanJerk PoleVault GoltSwing ClifiDiving BascballPitch CricketBowling Billiards CricketShot
AP
72.38
67.88
63.56
49.08
13.17
20.27
22.65
24.38
Boschal Pitch
Cricket Bowing
Billirds
Cricket Shot
Figure 3: Failure recognition cases on THUMOS' 14, We use the red box to indicate the location of the action that is taking place.
5
Conclusion
In our study, we take the initial step towards leveraging text learning for online action detection without explicit human supervision. Our findings demonstrate that by employing OV-OAD, reproductions acquired from large-scale video-text pairs, even with noise, can be successfully transferred to online action detection zero-shot manner. Moreover, we highlight that the conventional approach of base-to-novel fine-tuning does not yield favorable results on traditional online action detection test datasets. Instead, we illustrate that similar semantic frames can be directly clustered and transferred to downstream action detection datasets using abundant textual supervision.
Acknowledgments
This work is supported by National Natural Science Fund of China (62076184, 62473286) part by Shanghai Natural Science Foundation 222RR4667000
10

---

References
[1] Wentao Bao, Qi Yu, and Yu Kong. Opental: Towards open set temporal action localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2979-2989, 2022.
[2] Steven Bird, Ewan Klein, and Edward Loper. Natural language processing with Python: analyzing text with the natural language toolkit. "'' "'Reeill Media, Inc.", 2009.
[3] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Nicbles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, pages 961 961-970, 2015.
[4] Shugiang Cao, Weixin Luo, Bairui Wang, Wei Zhang, and Lin Ma. E2e-load: end-to-end long-form online action detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages (0422-10432, 2023.
[5] Junwen Chen, Gaurav Mittal, Ye Yu, Yu Kong, and Mei Chen. Gatehub: Gated history unit with background suppression for online action detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19925-19934, 2022.
[6] Mengyuan Chen, Junyu Gao, and Changsheng Xu. Cascade evidential learning for open- worldworld weakly-supervised temporal action localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14741-14750, 2023.
[7] Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 16:6646616667, 2022.
[8] Tsai- Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang- wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin- Ying Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70m Captioning 70m videos with multiple cross-modality teachers. arXiv preprint arXiv:2402.19477 2024.
[9] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual- visual-linguistic tasks. arXiv preprint 2r3iv: 14238, 2023.
[10] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in neural information processing systems, 34: 17864 17875, 2021.
[11] Feng Cheng and Gedas Bertasius. Tallformer: Temporal action localization with a long-memorg transformer. In European Conference on Computer Vision, pages 503-521. Springer, 2022.
[12] Feng Cheng, Xizi Wang, Jie Lei, David Crandall, Mohit Bansal, and Gedas Bertasius. Vindlu: recipe for effective aadeo-and-langusse pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10739-10750, 2023.
[13] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, al. Rescaling egocentric vision: Collection, pipeline and challenges for spic-kitchenss 100. International Journal of Computer Vision. pages 1-23, 2022.
[14] Roeland De Geest, Efstratios Gavves, Amir Ghodrati, Zhenyang Li, Cees Snoek, and Tinne Tuytelaars. Online action detection. In Computer ference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pages 269-284. Springer, 2016.
2016: 14th European Con-
[15] Hyunjun Fun, Jinyoung Moon, Jongyoul Park, Chanho Jung, and Changick Kim. Learning to discriminate information for online action detection. In Proceedings of the IEEF/CVE conference computer vision and pattern recognition, pages 809-818, 2020.
11

---

[16] Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc' Aurelio Ranzato, and Tomas Mikolov. Devise: deep sisual-semantic embedding model. Advances in neural information processing systems, 26, 2013.
[17] Jiyang Gao, Zhenheng Yang. and Ram Nevatia. Red: Reinforced encoder-decoder networks for 048188 2017.
action anticipation. arXiv preprint
[18] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang. Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter Better rision-lanuuage models with feature adapters. International Journal of Computer Vision, pages 1-15, 2023.
[19] Haroon Idrees, Amir Zamir, Yu-Gang Jiang, Alex Gorban, Ivan Laptev, Rahul Sukthankar, and Mubarak Shah. The thumos challenge on action recognition for videos "in the wild". Computer Vision and Image Understanding, 155:1-23, 2017.
[20] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Lc, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904-4916. PMLR, 2021.
[21] Chen Ju, Tengda Han, Kunhao Zheng. Ya Zhang, and Weidi Xie. Prompting visual- visual-language models for efficient video understanding. In European Conference on Computer Vision, pages 105-124. Springer, 2022.
[22] Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu, Less is more: Cliphert for indo-and--laggage learning via sparse sampling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7331-7341, 2021.
[23] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-
image pre-training for unified lision-languee understanding and generation. In International conference on machine learning, pages 22888-12900 PMLR, 2022.
[24] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang. Lu Yuan, Lei Zhang, Jenq-Neng Hwang, al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10965-10975, 2022.
[25] Benedetta Liberatori, Alessandro Conti, Paolo Rota, Yiming Wang, and Elisa Ricci. Test-time zero-shot temporal action localization. arXiv preprint arXiv:2404.05422 2024.
[26] Shuming Liu, Chen-Lin Zhang, Chen Zhao, and Bernard Ghanem. End-to-end temporal action detection with 1b parameters across 1000 frames. arXiv preprint arXiv:2311.17244 2023.
[27] Yi Liu, Limin Wang, Yali Wang, Xiao Ma, and Yu Qiao. Fineaction: A fine-grained video dataset for temporal action localization. IEEE transactions on image processing, 31:6937-6950, 2022.
[28] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances in neural information processing systems, 33:1 15111115581.8,, 2020.
[29] I Scott MacKenzie. Human-computer interaction: An empirical research perspective. 2024.
[30] Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV), pages 565-571. Ieee, 2016.
[31] Sauradip Nag, Xiatian Zhu, Yi-Zhe Song, and Tao Xiang. Zero-shot temporal action detection via vision-language prompting. In European Conference on Computer Vision, pages 681-697. Springer, 2022.
[32] Rashmiranjan Nayak, Umesh Chandra Pati, and Santos Kumar Das. A comprehensive review on deep learning- methods for video anomaly detection. Image and Vision Computing, 106: 104078, 2021.
12

---

[33] Thinh Phan, Khoa Vo. Duy Le, Gianfranco Doretto, Donald Adjeroh, and Ngan Le. Zeetad: Adapting pretrained vision-language model for zero-shot end-to-end temporal action detection. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pages 7046-7055, 2024.
[34] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 7748-87658 PMLR, 2021.
[35] Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fa- had Shahbaz Khan. Fine-tuned clip models are efficient video learners, In Proceedings of the IFEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6545-6555 6554, 2023.
[36] Alexander Richard and Juergen Gall. Temporal action detection using a statistical language model. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3131-3140, 2016.
[37] Ashish Vaswani, Noam Shazeer. Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[38] Jiahao Wang, Guo Chen, Yifei Huang, Limin Wang, and Tong Lu. Memory-anddamticipatoonn transformer for online action understanding. In Proceedings of the IEEE/CVE International Conference Computer Vision, pages 13824-13835, 2023.
[39] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In European conference on computer vision, pages 20-36. Springer, 2016.
[40] Mengmeng Wang, Jiazheng Xing, Jianbiao Mei, Yong Liu, and Yunliang Jiang. Actionelip: Adapting language-image pretrained models for video action recognition. IEEE Transactions on Neural Networks and Learning Systems, 2023.
[41] Xiang Wang, Shiwei Zhang, Zhiwu Qing, Yuanjie Shao, Zhengrong Zuo, Changxin Gao, and Nong Sang. Oadtr: Online action detection with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7565-7575, 2021.
[42] Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma. Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Internvid: large-scale video-text dataset for multimodal understanding and generation. arXiv preprint mwXiv:2307.06944 2023.
[43] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18134-18144, 2022.
[44] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2955-2966. 2023.
[45] Mingze Xu, Mingfei Gao, Yi- Yi-Ting Chen, Larry S Davis, and David J Crandall. Temporal recurrent networks for online action detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5532-5541, 2019.
[46] Mingze Xu, Yuanjun Xiong, Hao Chen, Xinyu Li, Wei Xia, Zhuowen Tu, and Stefano Soatto. Long short-term transformer for online action detection. Advances in Neural Information Processing Systems, 34:1086-1099, 2021.
[47] Le Yang, Junwei Han, and Dingwen Zhang. Colar: Effective and efficient online action detection by consulting exemplars. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 3160-3169, 2022.
13

---

[48] Min Yang. Huan Gao, Ping Guo, and Limin Wang. Adapting short-term transformers for action detection untrimmed videos. arXiv preprint arXiv:2312.01897 2023.
[49] Yue Zhao and Philipp Krähenbühl. Real-time online video detection with temporal smoothing transformers, In European Conference Computer Vision, pages 485-502. Springer, 2022.
14

---

Table 11: Video- Text model evaluation on THUMOS' 14.
THUMOS' 14
Methods
Arch
MAP (%)
ViCLIP-1
ViT/B
23.0
ViCLIP-1I
ViT/B
24.1 1
ViCLIP-1
ViT/L
25.7
ViCLIP-II
ViT/L
26.3
CLIP-1
ViT/B
28.0
CLIP-II
VIT/B
29.1
OV-OAD
VIT/B
37.5
Table 12: The impact of PAC for zero-shot performance on THUMOS' 14.
Weight of PAC
THUMOS' 14 mAP (%)
1.0
37.5
0.0
34.1
A
Appendix
A.1 More Ablations
Can we use the Video-Text model visual extractor? Another promising visual extractor to consider is employing a video-text model, instead of an image-text model, utilizing sliding window approach. We opted for ViCLIP 42, a straightforward video-text baseline model, to directly assess its performance for zero-shot online motion detection on the THUMOS' 14 dataset. Following the pre- pre-training settings of 42 we employed a sliding window that samples 8 frames for input into the visual encoder, aligning with the CLIP-II methodology. Moreover, we exclusively fed the last frame of the sliding window, aligning with the CLIP-I approach. The Table TT showcases the results, indicating that employing the video-text pretraining model directly for zero-shot inference in online action detection results unsatisfactory performance. Although ViCLIP can leverage information from the entire video during pretraining, its ability to capture only very brief video frames during online motion detection inference limits its performance significantly.
Can we remove PAC? During inference, the prediction the current frame comprises two components: the action clustering block (PAc) and the distant neighboring frame Transformer block (PDNTR), as indicated Section 3.3 As illustrated in the Table 12, excluding the scores of PAC leads to significant performance drops.
A.2 Dataset Preparation
ANet. ActivityNet v1.3 is an innovative and expansive benchmark dataset designed for human activity understanding in videos. It serves as a comprehensive resource to address the challenges of recognizing and analyzing wide range of complex human activities relevant to everyday life. The main objective of ANet is to provide a diverse collection of video samples that cover broad spectrum of human activities. Currently, the dataset offers samples from 203 distinct activity classes, ensuring comprehensive representation of various actions and behaviors. On average, there are 137 untrimmed videos available per class, with each video containing an average of 1.41 instances of the corresponding activity. In total, the dataset comprises impressive 849 hours of video content.
is worth noting that ANet's annotation not performed on live frames due cost considerations. It only annotates the time points of meaningful actions present in video, and not every semantic action is annotated. Since filters the video before annotation, the duration each video is relatively short and the proportion of background frames in the video is relatively low. After our statistics, its background frames account for 35.82% of the total frames. Most of the researchers use ANet as pre-training dataset for action recognition. Instead, we deal with it as video text dataset. When
15

---

processing the ANet dataset, we employ the prompting project combined with the original action tags as keywords for sentence construction.
Intern Vid-5K. Intern Vid is a significant multimodal dataset that focuses on video-centric learning for multimodal understanding and generation tasks. It serves valuable resource for developing powerful and transferable video-text representations. The Intern Vid dataset is vast, comprising over 7 million videos with a cumulative duration of nearly 760,000 hours. Within this extensive video collection, there are 234 million video clips available, each accompanied by detailed descriptions that consist of total of 4.1 billion words. The dataset's 's multimodal nature combines visual and textual information, enabling researchers to explore the relationship between videos and their accompanying descriptions.
The annotations provided by Intern Vid for this paper offer valuable information, but they pose challenges for our pre-training process. This is because video annotations generated using large models of image-text often contain text noise and inconsistencies. Additionally, these annotations tend to be concentrated in specific segments, lacking homogeneity. These features present difficulties for online action recognition tasks. address these challenges, we conducted thorough examination of the syntax in Intern Vid's text data and identified certain speech defects in the annotations. We filtered the training samples extensively to mitigate these issues. Initially, we selected 100,000 videos by sorting them based the total number of annotated entries in each long-term video. Subsequently, we applied additional filtering based on the ratio of annotated frames to the total number of frames. Through this process, selected 5000 samples of temporal text-pair data to form the Intern Vid-5K dataset. Depending on the acceptance of the paper, we plan to release the metafile of the modified dataset, which incorporates the aforementioned alterations, to further improve the quality and consistency of the annotations.
A.3 Benchmarks
THUMOS'14. The THUMOS' 14 (THUMOS 2014) dataset is significant video dataset widely used for action detection and recognition tasks. In the THUMOS' 14 dataset, there are 220 videos in the validation set and 212 videos in the testing that have been annotated with temporal boundaries. These annotations provide precise information about the start and end times of specific actions within the videos. With its extensive video collection, class diversity, and temporal annotations, the THUMOS 14 dataset serves as valuable resource for advancing state-of-the-art in action detection research.
TVSeries. The TVSeries Dataset is a comprehensive and realistic large-scale dataset specifically designed for action detection tasks. It encompasses a total of 16 hours of video content extracted from six recent TV series. The dataset includes diverse range of scenes and contexts, offering a representative sample of real- world action scenarios. Within the TVSeries Dataset, there thirty distinct action classes defined, covering a wide spectrum human activities. Each action instance in the dataset is meticulously annotated with precise start and end times, providing valuable temporal information for action detection algorithms.
FineAction. FineAction comprises 103,000 temporal instances across 106 action categories, an- notated within 17,000 untrimmed videos. The dataset offers new opportunities and challenges for online action detection, characterized by finely defined action classes, diverse attributes, multiple instance annotations, and concurrent actions from various classes. With around 4,000 uncut videos categorized into 106 classes like "Household Activities", "Personal Care", "Socializing", "Relaxing", "Sports", and "Exercise", FineAction sets the stage for innovative research.
EPIC-KITCHENS-100. EPIC-KITCHENS- 100 (EK100) includes first-person perspective shots and significantly differs from the ANet data distribution. 1K contains 100 hours of video, 20 million frames, and 90,000 action segments across 45 environments, with narrations mapped to 97 verb classes and 300 noun classes. This dataset addresses various challenges such as action recognition, detection, and anticipation, offering a platform to assess model generalization across time and diverse contexts.
16

---

A.4 Explain the Performance Differences THUMOS' 14 and TVSeries.
The case of pre-training with ANet. The variance improvements between THUMOS' and the TV series can be attributed to the resemblances in data distribution across these two datasets and our utilization of ANet. The substantial boost of OV-OAD on THUMOS' can be attributed ANet encompassing broader array of action categories. By employing a method of close comparison in natural language, we identified similar action phrases within THUMOS" 14's 20 action categories, constituting 40% of its overall categories. In the TV series dataset, we identified 9 similar action categories of 30, equating 30% of its total categories.
Using different pre-training datasets. Similarly, OV-OAD achieves better performance on THU- MOS' due to the fact that ANet covers wider range of action categories than IVid. Using the same natural language tools, we compared the coverage ratios of ANet and IVid for THUMOS categories, which stood at 40% and 15%, respectively. To claborate, when contrasting the IVid and THUMOS datasets, we considered 500 high-frequency verbs the action categories for IVid. Then, we pinpointed 3 categories from them that were similar to THUMOS 14.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly outline the primary contributions and scope the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of our work in Sec. 4.3
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification: TODO
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have fully disclosed the datasets, models, and experimental procedures used in detail.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification: We have released our code, data, and OV-OAD models at htttss://ggthahi Sonn/DpentVLabbOV-O00
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
ssiicyolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have specified all the training and test details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The paper provides extensive information about the statistical significance of our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
20

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The experiments compute resources have been discussed in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httpppp////aarrp.cccc//////////a
Answer: [Yes]
Justification:
We fully follow the ethics guidelines of NeurlPS.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discuss both potential positive societal impacts and negative societal impacts of the work performed in Sec. 4.3
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We use publicly available datasets, benchmarks, and models for training and evaluation, free from any possible harm toward individuals or groups.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt
package should be provided. For popular datasets, paperswi thcode. cww/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: We use publicly available datasets, benchmarks, and models for training and evaluation, free from any possible harm toward individuals or groups.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: We use publicly available datasets, benchmarks, and models for training and evaluation, free from any possible harm toward individuals or groups.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "bo5QBwRktz", "reviewer": "Reviewer_1BHH", "summary": "1. The authors have proposed a new method for Online Open Vocabulary action detection by leveraging pretrained vision langugae models. \n\n2. To that end they introduce 2 main modules, a distant neighboring frame transformer and an object centric Action clustering unit.\n\n3. They train their model with three objectives on filtered versions of the Activity-Net v1.3 and Intern Vid-10M-FLT datasets.\n\n4. They evaluate the model zero shot on the validation splits of Thumos’14 and TVseries datasets and with a base-to-novel formulation on the Thumos’14 dataset. The authors show that in the case of the former their model beats naive CLIP baselines by ~2-7 % on Thumos’14 and ~2% on TVSeries dataset and in the case of the latter beat the OadTR model trained on the seen split of Thumos’14 by ~6-9%\n\n5. The authors also demonstrate superior inference speed compared to OadTR.", "questions": "see weaknesses", "limitations": "yes", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The authors have proposed a new method for Online Open Vocabulary action detection by leveraging pretrained vision langugae models. \n\n2. The authors introduce 2 main modules, a distant neighboring frame transformer and an object centric Action clustering unit.\n\n3. The authors evaluate the model zero shot on the validation splits of Thumos’14 and TVseries datasets and with a base-to-novel formulation on the Thumos’14 dataset. The authors show that in the case of the former their model beats naive CLIP baselines by ~2-7 % on Thumos’14 and ~2% on TVSeries dataset and in the case of the latter beat the OadTR model trained on the seen split of Thumos’14 by ~6-9%\n\n4. The authors also demonstrate superior inference speed compared to OadTR", "weaknesses": "1. The authors claim in lines 41 and 42 that they do not use any frame-level annotations. However the Actitvity Net dataset contains annottaions for start and end times(therefore frames) for actions. And it seems that for the L_current objective in section 3.3, the text supervision is provided to the current frame. These claims seem contradictory and more clarity about this will be better. \n\n2. In the zero shot baseline comparison the improvement in case of Thumos-14 seem to be much larger than that of TV-series. Some explanation regarding this huge disparity is neccessary. Zero shot evaluations on more datasets can help indicate the robustness of this method to different data distributions.\n\n3. In the zero shot baseline comparison, there seems to be a very large improvement for Thumos’14 in case of the ANet model compared to the InternVid model. The improvement is 2% for the latter while ~7% for the former.  Could this be attributed to similarity of actions, between thumos’14 and Anet ? Some investigation regarding this could shed light on the previous point. if that is the case then the improvement range disparity for TV-series( ~2%) could be explained.\n\n4. The authors have compared their results for the open vocab evaluation with only the OadTR model. Comparison with more/better Online Action detection models (like GATEhub, MiniROAD, LSTR, Colar) is necessary for a holistic evaluation of the proposed model.\n\n5. The ablations do not contain the different parts of the action clustering unit. Some results without the Object centric decoder are needed to justify its introduction. In table 5, there is no ablation for not using the final transformer encoder.  It should be added to justify its introduction as well.\n\n6. In Figure 2. it is not clear what is being fed from the output of the object centric decoder to the final transformer encoder. It needs to be clearly mentioned in the figure for clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. Below are our responses to your concerns:\n### Q1: The statement \"do not use any frame-level annotations\" may appear contradictory.\nThank you for highlighting this inaccurate claim. We will rephrase it as follows: \"avoid utilizing fine-grained temporal annotations, which involve humans labeling actions frame by frame to ensure no actions are missed.\"\n\nRegarding our use of ANet, the annotations consist of timestamps and corresponding actions for each temporal segment. However, the timestamps lack the ability to differentiate between consecutive frames, leading to potential mislabeling of frames with incorrect actions. Additionally, instances within ANet may sometimes omit actions or assign incorrect labels.\n\nFor example, in sample *5n7NCViB5TU.mp4* from the training set lasting 121.44 seconds and depicting discus throwing by 3 athletes (referred to as Athletes 1-3), there are discrepancies. Athlete 1's action completion was marked within the time window $[22.4, 39.5]$, while ANet indicated $[24.3, 38.1]$ as the segmentation point. Similarly, for Athlete 2, the action completion time window was $[62.5, 73.1]$, with ANet lacking a corresponding segmentation label.\n\n### Q2: Zero shot evaluations on more datasets can help indicate the robustness.\nWe validate OV-OAD's zero-shot performance on EK100, showcasing a non-trivial improvement (1.3% cAP). For further details, please refer to the response To-All-Reviewers-Q2.\n\n### Q3: Different improvement for THUMOS’14 and TVSeries. \nPlease refer to the response To-All-Reviewers-Q3 for detials. \n\n### Q4: Comparison with more online action detection models.\nPlease refer to the response To-All-Reviewers-Q1 for detials.\n\n### Q5: The ablations on the action clustering unit. \nWe will ablate different parts of action clustering block soon. \nThe reason behind using the object-centric decoder lies in its capacity to generate more semantically explicit segments compared to a standard transformer decoder block, owing to its differentiable clustering module. \nThis superiority has been validated in [41-42].\nNevertheless, we will compare it with the aforementioned rival in the following ablation.\n\n### Q6: The ablation for not using the final transformer encoder. \nRemoving the final transformer encoder (also means abandon $L_{contrast}$) would leads to training anomalies and performance drops. Without the textual guidance from $L_{contrast}$, the open-vocabulary capability is significantly compromised. As shown in Table 5 (rows 2 vs. 3). we see that reducing the number of layers in the final transformer encoder from 6 to 3 results in a performance loss.\n\n### Q7: Figure 2 needs to be clearly mentioned for clarity.\nThe object-centric decoder automatically associates $n$ input frame tokens using $k$ learnable grouping embeddings, ultimately outputting grouped tokens (${\\cal G}' \\in \\mathbb{R}^{k \\times d}$) which also can globally describe the video clip. We will label the tensor dimensions of the module's outputs in Figure 2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "meEWRlvVKO", "reviewer": "Reviewer_zhJf", "summary": "The paper introduces OV-OAD, a novel zero-shot online action detection system leveraging vision-language models for open-world temporal understanding. The authors propose a Transformer-based model with an object-centered decoder unit, trained solely on video-text pairs without manual frame-level annotations. The system is evaluated on THUMOS'14 and TVSeries benchmarks, demonstrating superior performance over existing zero-shot methods.", "questions": "1. The OadTR model was proposed in 2021, and the authors should consider using more recent OAD models such as MAT, MiniROAD, etc., as comparative benchmarks to enhance the timeliness and competitiveness of their model.\n2. The authors have employed a multitude of complex modules for Zero-shot experiments on two datasets, with improvements on the TVSeries dataset being relatively limited compared to THUMOS14. This raises doubts about whether the proposed model is robust enough to serve as a standard baseline for future research.\n3. As a pioneering work, the authors have experimented on typical action datasets like THUMOS14 and TVSeries. However, it would be beneficial to also test on atypical human action datasets, such as creating a benchmark for the HDD dataset.\n4. In the action clustering block, which serves as the query and which as the key and value between the input Group Embedding and the output tokens? According to Figure 2, it seems that the Group Embedding is the query, and the latter is the key and value, but why is the dimension of the attention weight matrix $A$ $n \\times k$ instead of $k \\times n$? (line 142)\n5. Line 154 mentions that the number of neighboring frames n can be a large value, and Table 4 shows the highest $n$ up to 8. Can the authors present experimental results for higher values of $n$?\n6. Could the authors further explain how the model achieves ultra-high inference speed with a larger scale of parameters? For example, the parameter amount is twice that of LSTR, but the inference speed is six times as fast.\n7. The paper lacks explanations for some operations, such as the operator \"$\\circ$\" in Equation 1. This may lead to misunderstandings, mistaking it for a Hadamard product rather than a composite function.\n8. It is recommended that the authors add visible variables in the figures used to illustrate the method (such as Figure 2) and improve the explanation of different parts of the chart to enhance readability.", "limitations": "In summary, the authors have examined the OAD (Online Action Detection) from an interesting perspective and proposed a novel Zero-shot model, which appears to have profound application value compared to traditional OAD methods. However, additional experiments may be necessary to substantiate the claims fully.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Innovation: The paper introduces a significant advancement in action detection by proposing a zero-shot online detection method that does not rely on manual annotations.\n2. Technical Depth: The proposed model incorporates a novel object-centered decoder unit within the Transformer framework, which is a sophisticated approach to frame aggregation.\n3. Experimental Validation:  The paper provides extensive experiments on two benchmarks, demonstrating the effectiveness of the proposed method.", "weaknesses": "1. The OadTR model was proposed in 2021, and the authors should consider using more recent OAD models such as MAT, MiniROAD, etc., as comparative benchmarks to enhance the timeliness and competitiveness of their model.\n2. The authors have employed a multitude of complex modules for Zero-shot experiments on two datasets, with improvements on the TVSeries dataset being relatively limited compared to THUMOS14. This raises doubts about whether the proposed model is robust enough to serve as a standard baseline for future research.\n3. As a pioneering work, the authors have experimented on typical action datasets like THUMOS14 and TVSeries. However, it would be beneficial to also test on atypical human action datasets, such as creating a benchmark for the HDD dataset.\n4. In the action clustering block, which serves as the query and which as the key and value between the input Group Embedding and the output tokens? According to Figure 2, it seems that the Group Embedding is the query, and the latter is the key and value, but why is the dimension of the attention weight matrix $A$ $n \\times k$ instead of $k \\times n$? (line 142)\n5. Line 154 mentions that the number of neighboring frames n can be a large value, and Table 4 shows the highest $n$ up to 8. Can the authors present experimental results for higher values of $n$?\n6. Could the authors further explain how the model achieves ultra-high inference speed with a larger scale of parameters? For example, the parameter amount is twice that of LSTR, but the inference speed is six times as fast.\n7. The paper lacks explanations for some operations, such as the operator \"$\\circ$\" in Equation 1. This may lead to misunderstandings, mistaking it for a Hadamard product rather than a composite function.\n8. It is recommended that the authors add visible variables in the figures used to illustrate the method (such as Figure 2) and improve the explanation of different parts of the chart to enhance readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. Below are our responses to your concerns:\n### Q1: The authors should consider using more recent OAD models.\nPlease refer to the response To-All-Reviewers-Q1 for detials.\n\n### Q2: The improvements on the TVSeries dataset being relatively limited compared to THUMOS'14. \nWe believe that there are two reasons why OVOAD's performance in THUMOS'14 outperforms the improvements in the TVSeries dataset. \n- First, all categories of ANet have different coverage ratios of **40%** and **30%** for the action categories of THUMOS14 and TVSeries, respectively. Similar text descriptions encountered by the text encoder during pre-training improve the accuracy of visual-text matching. Please refer to the response To-All-Reviewers-Q3 for specific calculations.\n- Second, the evaluation metrics are calculated differently for the THUMOS'14 and TVSeries datasets, the former used mAP as a metric while the latter used cAP as a metric. \n\n### Q3: It would be beneficial to test on atypical human action datasets, such as creating a benchmark for the HDD dataset.\nThank you for your advice. We have validated the zero-shot performance of OV-OAD on EK100 with non-trivial improvement (1.3% cAP). Please refer to the response To-All-Reviewers-Q2 for detials. In the final version, we will evaluate our method on those challenging online motion detection datasets, e.g., FineAction and the mentioned HDD.\n\nHDD [a] is a multimodal dataset in a driving scenario. In previous studies, e.g., OadTR [39] and Colar [45], non-visual sensors data are generally used as inputs to the model. The non-visual sensors are sourced from the vehicle's controller area network bus, encompassing critical metrics like car speed, accelerator and braking pedal positions, yaw rate, steering wheel angle, and the rotation speed of the steering wheel. In contrast, OV-OAD has only a visual feature extractor and does not directly process sensor data. Directly evaluating our current model on HDD could lead to unsatisfying performance due to the huge data distribution gap between our training (RGB videos for daily actions) and HDD. \n\n### Q4: The group embedding is the query, and the frame tokens is the key and value, but why is the dimension of the attention weight matrix $A \\in n×k$ instead of $k×n$?\nIn the action clustering block, the group embeddings serve as the query, and frame tokens serve as keys and values. As shown in Eq.(4), we define  $A = softmax(\\frac{K@Q^{\\top}}{\\sqrt{d}}), A \\in \\mathbb{R}^{n \\times k}$, with the softmax operator normalizing over $k$, and the output of the slot-attention block is $A^{\\top}V$, which is of shape $k×d$.  We will clarify the definition of $A$ in the final version.\n\n### Q5: Line 154 mentions that the number of neighboring frames $n$ can be a large value, and Table 4 shows the highest $n$ up to 8. Can the authors present experimental results for higher values of $n$?\nTraining our model with a larger $n$ than 8 (e.g. 12) needs V100×8 GPUs for ~5 days on ANet. We will add this study in the final version. We state that $n$ can be a large number relative to the number of group embeddings $k$. Generally, $n$ should be more than twice $k$. We will make these statements clear. \n\n### Q6: The parameter amount is twice that of LSTR, but the inference speed is six times as fast.\nThe inference of LSTR is much slower than ours due to two main reasons:\n- It requires more input frames than our method.\n- It relies on optic flow extraction, which is a slow process.\n\nIn particular, LSTR demands a larger number of input frames for optimal performance, using 520 seconds of video for inference on THUMOS'14, while our OV-OAD utilizes only 8 seconds. This means that LSTR requires **65** times more data than OV-OAD, which likely explains why our model's inference speed is six times faster. Furthermore, the primary speed bottleneck for LSTR is the extraction of optical flow (8.1 FPS), whereas for our model, it is the extraction of image features (292.3 FPS).\n\n### Q7: The paper lacks explanations for some operations, such as the operator \"◦\" in Equation 1. \nIn Equation 1, the symbol \"◦\" represents the function composition. We will include detailed explanations for clarity.\n\n### Q8: It is recommended that the authors add visible variables in the figures.\nWe will label the input and output tensors' names and dimensions of each module in Figure 2 for clarity.\n\n### Reference \n[a] Ramanishka, Vasili, et al. \"Toward driving scene understanding: A dataset for learning driver behavior and causal reasoning.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oE5XSBHkWz", "reviewer": "Reviewer_Boy7", "summary": "This paper addresses a challenging setting in video understanding: open-vocabulary online action detection. It leverages pre-trained visual language models with a proposed dual-encoder architecture to achieve successful zero-shot detection.", "questions": "Please see the weaknesses section.", "limitations": "Limitations have been well discussed by the authors in the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method follows a visual-text dual encoder approach and applies it novelly to zero-shot online temporal action detection, achieving state-of-the-art performance.\n2. Detailed ablation studies and analyses are included. This paper provides comprehensive ablations regarding the architecture design, loss design, and efficiency analysis, demonstrating the advantages of the proposed method over previous approaches.\n3. Clear writing. The paper is well-organized and easy to follow.", "weaknesses": "1. The choice of VLM. The method uses CLIP as the visual-text encoder. However, as discussed in the limitations section, CLIP is an image-based understanding model and lacks the capability to capture temporal context. I wonder if other VLMs, such as ActionCLIP, could mitigate these drawbacks for the proposed task.\n2. The comparison of the proposed method with previous work, such as OadTR, seems to use different visual encoders/features. Can the authors explain the fairness of such a comparison? Besides, the author should also compare the method with more recent methods, since OadTR is not state-of-the-art in recent works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. Below are our responses to your concerns:\n### Q1:  I wonder if other VLMs, such as ActionCLIP, could mitigate these drawbacks for the proposed task.\nThank you for the suggestion. We plan to enhance our model's performance in future work by incorporating improved feature extractors like ActionCLIP [38]. \n\n### Q2: The comparison of OV-OAD with OadTR, seems to use different visual encoders/features.\nThank you for pointing out the lack of description here. In line 276, we noted that both the fully supervised methods and our OV-OAD utilize the same feature extractor. Specifically, both the base-to-novel fine-tuning methods and the fully supervised methods employ the same feature extractor, namely CLIP's visual encoder ViT/B. In the final version, we will emphasize this statement at the beginning of the paragraph.\n\n### Q3: The author should also compare the method with more recent methods.\nPlease refer to the response To-All-Reviewers-Q1 for detials.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AimgMEfFCp", "reviewer": "Reviewer_wz2n", "summary": "This paper explores the problem of open vocabulary video action detection in an online setting, where the action must be detected immediately once it appears in the video stream, vs. the more common offline setting that allows examining the entire video, past and future.\n\nThe authors propose a model with two main components: a transformer decoder that cross-attends between recent and past video frames, and an action clustering block that uses slot attention to group related frames and then classify them by action.\n\nThe model is trained on combination of three tasks/losses: contrastive image-text loss between the current frame's visual embedding and text embedding (vs. text embeddings from other clips in the batch), multi-label contrastive video-text loss (for text clips described by multiple text labels), and mask loss for identifying which frames come from the background (no action present).\n\nThe model demonstrates improved performance over CLIP baselines in the novel open vocabulary online action detection setting.", "questions": "Do you see a difference between \"open vocabulary\" and \"zero shot\" action recognition? The terms seem to be used synonymously in this paper, e.g. lines 106-107.\n\nRelated, how much overlap is there between the textual descriptions in ActivityNet and InternVid  with the category label sets of THUMOS'14 and TVSeries?\n\nCan you explain \"the assumption of low background information for VLM\" from line 36 in more detail?\n\nCan you please clarify \"fails to reach future frames during training\" from line 37?\n\nWhat precisely is the circle operator in equation (1)?\n\nOn Line 175, how do you know \"when the raw labels are not enough\"?\n\nIn Table 2, OVO_{ad}^{dagger} refers to OV-OAD, correct?", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The problem setting introduced in this paper, online-streaming action recognition with an open vocabulary, is very realistic for modeling many practical real-world scenarios, e.g. home security cameras. Further, the model is impressively fast at inference (292 fps), making it quite reasonable to practically apply in this setting.\n\nThe paper includes an expansive, robust set of ablation experiments to validate the numerous model design choices.\n\nI appreciate that the paper calls out joint modeling of spatio-temporal information as a limitation and backs this up with an analysis of the model's failure cases.", "weaknesses": "Line 61 says that \"our model successfully learns clusters of similar video frames\", and I see that indirectly based on the fact that there's a clustering module that leads to improved performance. However such a claim would be stronger if supported by evidence or examples of frames being correctly clustering based on action semantics.\n\nThe Object-Centric Decoder unit seems to be confusingly-named, since it is employed \"to group frames\" (line 149), rather than to focus on specific objects appearing within a frame.\n\nIn Table 1, in the proposed setting, the model is compared only against CLIP variants. Another baseline worth exploring would be video-text models (as opposed to image-text) applied via a sliding window.\n\nThere are a few parts of the paper that could be explained more quickly, in my opinion. (See the questions below.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. Below are our responses to your concerns:\n### Q1: However such a claim would be stronger if supported by evidence or examples of frames being correctly clustering based on action semantics. \nHere, we give evidence both quantitatively and qualitatively.\n- Firstly, during inference, the prediction of the current frame comprises two components: the action clustering block ($P_{AC}$) and the distant neighboring frame Transformer block ($P_{DNTR}$), as indicated in Line 215. As illustrated in the table below, excluding the scores of $P_{AC}$ leads to significant performance drops.\n| Weight of $P_{AC}$ | THUMOS'14 (mAP%) |\n|:-----|:----:| \n| 1.0 |37.5|\n| 0.0 |34.1|\n\n- Secondly,  in the final version, we will visualize the attention matrix $A$ in the object-centric decoder. Following the computation of Eq. 3, we can obtain a representation of the grouping of the input video frames.\n\n\n### Q2: The object-centric decoder unit seems to be confusingly-named, since it is employed \"to group frames\" (line 149), rather than to focus on specific objects appearing within a frame.\nWe will clarify that in the object-centric decoder, the term \"objects\" refers to video frames.\n\n### Q3: Another baseline worth exploring would be video-text models (as opposed to image-text) applied via a sliding window.\nWe opted for ViCLIP [40], a straightforward video-text baseline model, to directly assess its performance for zero-shot online motion detection on the THUMOS'14 dataset. Following the pre-training settings of [40], we employed a sliding window that samples 8 frames for input into the visual encoder, aligning with the CLIP-II methodology. Moreover, we exclusively fed the last frame of the sliding window, aligning with the CLIP-I approach. The results presented in the table below demonstrate that utilizing the video-text pretraining model directly for zero-shot inference in online action detection yields unsatisfactory performance.\n\nViCLIP can access information from the entire video during pre-training; however, it can only capture very brief video frames during online motion detection inference. This limitation constrains its performance.\n| Methods | Arch | THUMOS'14 (mAP%) |\n|:-----|:----:|:----:|\n| ViCLIP-I  | ViT/B  |23.0|\n| ViCLIP-II  | ViT/B |24.1|\n| ViCLIP-I  | ViT/L | 25.7|\n| ViCLIP-II  | ViT/L |26.3|\n| CLIP-I     | ViT/B | 28.0|\n| CLIP-II    | ViT/B | 29.1|\n| OV-OAD | ViT/B | **37.5**|\n\n### Q4: Do you see a difference between \"open vocabulary\" and \"zero shot\" action recognition? \nThe disparity between \"open-vocabulary\" and \"zero-shot\" action recognition lies in the fact that open-vocabulary enables the recognition of any category, whereas \"zero-shot\" ensures that the testing categories were not encountered during training [a].\n\nIn Line 106-107, we label our approach as \"open-vocabulary\" since our model is designed to ideally facilitate the online detection of any actions with the assistance of a vision-language model. In this context, \"zero-shot\" signifies that our method can be directly applied for inference on downstream datasets without the need for additional fine-tuning.\n\n### Q5: How much overlap between the textual descriptions in ActivityNet and InternVid with the category label sets of THUMOS'14 and TVSeries?\nBy utilizing the method of proximate comparison in natural language, we have pinpointed 8 analogous action phrases within the 20 action categories of THUMOS'14, encompassing 40% of its total categories. Likewise, we have identified 9 similar action categories out of the 30 in TVSeries, representing 30% of its total categories. For more detailed information, please refer to the response To-All-Reviewers-Q3.\n\n### Q6: Can you explain \"the assumption of low background information for VLM\" from line 36?\nVLM are primarily designed for classification tasks rather than detection tasks. The optimization process of these models typically operates under the assumption that the background information (such as irrelevant environmental content) in the training data is less significant compared to the foreground content (task-relevant objects or regions). Essentially, these models are trained with the notion that each sample's main focus is on the foreground content, with less emphasis on the background, likely because classification tasks typically involve identifying the main objects in an image without detailed scene analysis. Consequently, VLMs may not be well-suited for detection tasks that necessitate precise differentiation between foreground and background elements. A similar perspective is echoed in STALE [29].\n\n### Q7: Can you clarify \"fails to reach future frames during training\" from line 37?\nGiven an untrimmed long video, the temporal action detection model can see the features of all frames, while the OAD model can only see the current frame and past frames in each iteration.\n\n### Q8:  What precisely is the circle operator in equation (1)?\nIn this paper, the symbol \"◦\" represent function composition. If $Ψ_{AC}$ and $Ψ_{DNTR}$ are two functions, then $Ψ_{AC}◦Ψ_{DNTR}$ denotes the composition of them, which is defined as $Ψ_{AC}◦Ψ_{DNTR}=Ψ_{AC}(Ψ_{DNTR}(x))$.\n\n### Q9:  On Line 175, how do you know \"when the raw labels are not enough\"?\nIn this context, \"raw labels\" refer to the captions associated with the selected video clips. Descriptions in large-scale video datasets often lack density. The OAD model utilizes a sliding window to extract multiple frames across the video, easily picking up segments with sparse or minimal captions.\n\n### Q10:  In Table 2, VO_{ad}^{dagger} refers to OV-OAD, correct?\nThat's a typo. OVO_{ad}^{dagger} means OV-OAD.\n\n### Reference\n[a] Hyun, Jeongseok, et al. \"Exploring Scalability of Self-Training for Open-Vocabulary Temporal Action Localization.\" arXiv preprint arXiv:2407.07024 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# To All Reviewers
We sincerely thank each reviewer for providing constructive comments for our paper, which are very helpful to improve our paper. Below, we address the general issues raised by the reviewers.
### Q1: Comparison with more online action detection models
We conducted a comparative analysis between OV-OAD and both the fully-supervised and base-to-novel fine-tuning methods, leveraging the open-source LSTR [44] and MAT [36] frameworks. For the fully-supervised training, we adhered to the experimental configurations outlined in [36, 44], with the deviation being the utilization of solely CLIP/ViT-B extracted RGB features as inputs. Key parameters remained consistent, such as training with 20 epochs, 'adam' optimizer, a learning rate of 7e-5, among others.
For the base-to-novel fine-tuning, we followed the experimental fine-tuning setup of [20, 33], including the Adam optimizer, a learning rate of 1e−3, and the Cosine decay function for training. 
| Train-Test Split  | Methods | THUMOS’14 (mAP%) |
|:--------------:|:----------:|:----------:|
| 100%-0% | OadTR-D8 | 47.4 |
| 100%-0% | LSTR | 47.7 |
| 100%-0% | MAT-D48 | **48.2**|
|100%-0%  | CLIP-I | 28.0 |
| 100%-0% | OV-OAD | 37.5 |
| 75%-25% | LSTR | 26.9 |
| 75%-25% | MAT-D48 | 25.5 |
| 75%-25% | OadTR-D8 | 33.7 |
| 75%-25% | CLIP-I | 38.6 |
| 75%-25% | OV-OAD | **44.6** |
| 50%-50% | MAT-D48 | 7.9 |
| 50%-50% | LSTR  | 9.1 |
| 50%-50% | OadTR-D48   | 9.6 |
| 50%-50% | CLIP-I | 28.6 |
| 50%-50%| OV-OAD | **35.9** |

The results are reported in the table above. The MAT-D48 indicates that MAT utilizes the ground truth of future frames (48 frames in 12 seconds) during training. We observe that the state-of-the-art MAT (ICCV23) method achieves better performance in the fully-supervised setting, but its performance is poor in the base-to-novel setting. It is evident that traditional OAD models combined with base-to-novel fine-tuning methods are not suitable for direct application to zero-shot online action detection tasks.

### Q2: Zero shot evaluations on more datasets
We evaluated OV-OAD's performance on the challenging EPIC-KITCHENS-100 (also known as EK100) dataset. This dataset comprises first-view shots and notably deviates from the ANet data distribution.
Following [a], we conducted online action detection inference on the complete test set of EK100, covering 97 verb categories.
Similar to TVSeries, we employed per-frame calibrated average precision (cAP) as the evaluation metric.
The results presented in the table below illustrate that OV-OAD enhances online action detection for first-view videos compared to CLIP.
| Methods | Arch | EK00 (cAP%) |
|:-----|:----:|:----:|
| CLIP-I | ViT/B | 40.1|
| CLIP-II | ViT/B | 39.9|
| OV-OAD | ViT/B | **41.4**|

In addition, we also chose the challenging and large OAD dataset FineAction [b], for validation. Because of resource constraints, the results are not available at the moment, and we will add a description of the testing process and experimental results of this dataset in the final version.

### Q3: Different improvement for THUMOS’14 and TVSeries. 
- Between THUMOS‘14 and TVSeries on ANet

The variance in improvements between THUMOS'14 and the TV series can be attributed to the resemblances in data distribution across these two datasets and our utilization of ANet. The substantial boost of OV-OAD on THUMOS'14 can be attributed to ANet encompassing a broader array of action categories. By employing a method of close comparison in natural language, we identified 8 similar action phrases within THUMOS'14's 20 action categories, constituting **40%** of its overall categories. In the TV series dataset, we identified 9 similar action categories out of 30, equating to **30%** of its total categories.

- Between ANet and InternVid on THUMOS‘14

Similarly, OV-OAD achieves better performance on THUMOS'14 due to the fact that ANet covers a wider range of action categories than IVid. Using the same natural language tools, we compared the coverage ratios of ANet and IVid for THUMOS categories, which stood at **40%** and **15%**, respectively. To elaborate, when contrasting the IVid and THUMOS datasets, we considered 500 high-frequency verbs as the action categories for IVid. Then, we pinpointed 3 categories from them that were similar to THUMOS'14.

### Reference
[a] Damen, Dima, et al. "Scaling egocentric vision: The epic-kitchens dataset." Proceedings of the European conference on computer vision (ECCV). 2018.

[b] Liu, Yi, et al. "Fineaction: A fine-grained video dataset for temporal action localization." IEEE transactions on image processing 31 (2022): 6937-6950.

Meta-review: The submission proposes to leverage pre-trained video-text models for the open-vocabulary temporal action detection task. The task requires _online_ detection, where an action needs to be recognized as soon as it appears in the video stream. Evaluations are performed on THUMOS'14 and TVSeries benchmarks, where the proposed OV-OAD framework outperforms CLIP-based approaches, and comprehensive ablation experiments are provided to demonstrate the effectiveness and efficiency of the proposed approach.

All reviewers actively engaged with the authors during the rebuttal discussion. The reviewers found most of their concerns to be addressed by the rebuttal and all reviewers recommended "weak accept". The AC agrees with the reviewers that the submission would be of value to the NeurIPS research community and thus recommends acceptance. The authors are expected to incorporate the reviewers' feedback and the relevant rebuttal experiments in the camera ready version.
Decision: Accept (poster)