Does Video-Text Pretraining Help Open-Vocabulary Online Action Detection?

Authors: qingsong  zhao
yi  wang
jilan  xu
yinan  he
zifan  song
limin  wang
yu  qiao
cairong  zhao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-30 08:38:35

Abstract: Video understanding relies on accurate action detection for temporal analysis. However, existing mainstream methods have limitations in real-world applications due to their offline and closed-set evaluation approaches, as well as their dependence on manual annotations. To address these challenges and enable real-time action understanding in open-world scenarios, we propose OV-OAD, a zero-shot online action detector that leverages vision-language models and learns solely from text supervision. By introducing an object-centered decoder unit into a Transformer-based model, we aggregate frames with similar semantics using video-text correspondence. Extensive experiments on four action detection benchmarks demonstrate that OV-OAD outperforms other advanced zero-shot methods. Specifically, it achieves 37.5\% mean average precision on THUMOS’14 and 73.8\% calibrated average precision on TVSeries. This research establishes a robust baseline for zero-shot transfer in online action detection, enabling scalable solutions for open-world temporal understanding. The code will be available for download at \url{https://github.com/OpenGVLab/OV-OAD}.

Main paper: 
 introduction: 
 Action detection is a practical and demanding technique in intelligent video analysis, including anomaly detection [32] in surveillance and human-computer interaction [29] in embodied studies. Considering high variations in possible human behaviors with dynamic scenes, action detection is significantly challenging. In this regard, most action detection approaches go offline, involving the closed-set classification and localization of actions (a few predefined categories) within the long untrimmed videos. However, real-world applications concerning real-time understanding (e.g. surveillance) require estimating the action without accessing future frames. Further, closed-set discrimination limits the applicability of action detection, and it also asks for manually annotating all action categories, especially in complex scenarios such as a wide variety of actions or events in driving scenarios, which is both costly and time-consuming.To address these challenges, we formulate online action detection in open-vocabulary and transfer popular vision-language models (VLM) to tackle this problem via only paired vision-text supervision for learning. A growing number of researchers have been investigating how to leverage the capabilities of powerful VLM to address specific novel visual tasks of interest. For instance, existing studies [21, 31, 1, 6, 25, 33] have explored the transfer of visual knowledge from VLM to a video understanding task to achieve zero-shot temporal action detection (ZS-TAD). Applying VLM to ZS-OAD is nontrivial. Plain solutions, as the ZS-TAD approaches mentioned earlier, involve partitioning a subset of base-to-novel category data from the downstream dataset and further fine-tuning the visual language model with a prompt-based technique to adapt it for novel tasks. However, this poses several issues. First, sliding-window frame sampling used in online action detection often leads to a high proportion of background frames, which contradicts the assumption of low background information for VLM training. Second, the OAD model fails to reach future frames during training, making it difficult to sample all category labels in the same batch. This is detrimental to the optimization of image-text contrast loss since the contrast loss favors the diversity of samples. We experimentally explored these hypotheses in Sec. 4.1.Inspired by CLIP [34] , given frames representation from a powerful VLM, we learn online motion detection models purely through text supervision, thus avoiding the use of fine-grained temporal annotations. To this end, we introduce the proposed object-centered decoder unit into the Transfomrer-based model, enabling automatic aggregation of frames with similar semantics with textual supervision exclusively. Fig. 1 illustrates the overall framework of our method. By employing contrast loss during training on extensive video-text pairs, we enable the model to be zero-shot transferred to different action detection vocabularies. Hence, we name our model Open-Vocabulary Online Action Detector (OV-OAD). We pre-train OV-OAD on the video-text datasets, and manual frame-level labels are not used whatsoever. We propose three proxy tasks including alignment of current frame-text embedding, background frame mask prediction, and alignment of multi-label video-text embedding for training. The first task enables the model to prioritize discriminative information from neighboring frames. The second task enables the successful detection of complex background frames in natural videos. The third task mitigates the impact of caption noise in web videos. Our model was evaluated on four action detection benchmarks without any fine-tuning, i.e., THUMOS'14 [19] , EK100 [13] , FineAction [27] and TVSeries [14] in a zero-shot manner. Extensive experiments demonstrate that our model outperforms other advanced zero-shot methods. The main contributions are summarized as follows:• We investigate the critical problems of how to capitalize pre-trained visual language models for zero-shot online action detection in untrimmed videos.• We introduce a novel video-text dual-encoder architecture, namely OV-OAD, to perform open-vocabulary online action detection. Experiments on the downstream datasets show that our model successfully learns clusters of similar video frames and transfers them to multiple action semantic vocabularies in a zero-shot manner.• To our knowledge, our work is the first to explore zero-shot transfer from text supervision alone to the online action detection task without relying on any precise frame-scale labels.And we have established a robust baseline for this new setting. 
related work: 
 Pretrained Vision-Language Model (VLM). Recently, the joint image-text learning paradigm [16] has been successfully scaled up by CLIP [34] and ALIGN [20] with the massive web data. After that, researchers have proposed many variations, including CLIP-Adapter [18] , GLIP [24] , and so on. One VLM's visual encoder can leverage textual descriptions to recognize objects or scenes in images when category-specific samples are unavailable. In video domains, similar ideas have been explored for action recognition (e.g., ActionCLIP [40] , ViFi-CLIP [35] ) and video understanding (e.g., CLIPBERT [22] , EffPrompt [21] ).Zero-Shot Temporal Action Detection Temporal action detection (TAD [36, 11, 48, 26] ) is a video understanding task involving simultaneous recognition and localization of actions within an uncut video. Recently, efforts [21, 31, 1, 6, 25, 33] have utilized the pre-trained vision-language model to give the TAD models with the capability to recognize novel action classes. For example, EffPrompt [21] proposes a two-stage fine-tuning scheme for zero-shot temporal action detection (ZS-TAD) by incorporating task-specific prompt vectors. STALE [31] Onlne Action Detection. Contrary to offline motion detection, OAD does not predict action onset timing and cannot access future visuals. Arguably, OAD emphasizes real-time response and openness of recognition over the accuracy of action classification in practice. Existing researchers [17, 45, 15, 41, 46, 38, 4] often use closed-set datasets for training and testing, boosting recognition accuracy and speed on single datasets. For example, IDN [15] improves the discriminative representation of actions by selectively accumulating relevant information. OadTR [41] incorporates the fusion of current features and future frames for identifying ongoing actions. LSTR [46] captures contextual dependencies in videos, leading to improvements in action identification. E2E-LOAD [4] proposes an end-to-end framework that integrates a stream buffer between the spatial and spatiotemporal modeling. MAT [38] introduces a memory-anticipation-based pipeline to model the entire temporal structure of a video. In contrast, our work shifts the focus to enhancing the open recognition capability of OAD models. We aim to leverage readily available video-text pairs to zero-shot transfer visual knowledge into the OAD model, thereby improving the model's ability to handle unseen actions. 
methodology: 
 Consider an untrimmed video V , we generate a clip sequence employing a sliding window of length τ on V that moves frame by frame. On the t-th slide, we get a clip V t = {V t-τ , . . . , V t-1 , V t } where V t denotes t-th frame. Online action detection is to predict action probability y t in each frame V t using only past and current observations. We propose an open-vocabulary online action detection model (OV-OAD) for zero-shot online action detection with text supervision only. Our approach, illustrated in Fig. 2 , consists of two primary components: a visual encoder and a text encoder. The visual encoder comprises a distant neighboring-frame transformer block and an action clustering one. We pre-train OV-OAD on a web-scale video-text dataset. In inference, we transfer the trained model to the zero-shot online action detection without any fine-tuning, as described in Sec. 3.3, and it can predict arbitrary action classes. We ignore the subscripts of individual images and text pairs for simplicity.The visual encoder is composed of a distant neighboring-frame transformer block (with a light grey background in Fig. 2 ), and an action clustering block (with yellow background in Fig. 2 ) with an Figure 2 : The illustration of our OV-OAD (best viewed in color), is formulated in a visual-text dualencoder manner. Specifically, the visual encoder consists of a distant neighboring-frame transformer block (Ψ DN T R , light grey backdrop) and an action clustering block (Ψ AC , yellowish backdrop). The Ψ DN T R is built with Transformer Decoder units, which take the neighboring tokens and distant past tokens as inputs. The Ψ AC is built with our Object-Centric Decoder and vanilla Transformer Encoder units, which take the output tokens (orange squares) and learnable group embeddings (purple gradient squares) as inputs. During testing, the OV-OAD handles each incoming video snippet online, absent future context. object-centric decoder. For a given video clip-text pair, denoted as (V t , T ), we initially divide the video into extended past frames V t P ∈ R (τ -n)×d and neighboring ones V t N ∈ R n×d . All frames V t are processed in the Distant Neighboring-frame TRansformer block Ψ DN T R , which comprises the transformer decoder unit [37] . Then the neighboring frames V t ′ N aggregated with the information of past frames are fed into the Action Clustering block Ψ AC , along with k learnable group embedding (G ∈ R k×d ), that aims to bind the neighboring frames into clusters. The outputs of the visual encoder are defined as:EQUATIONwhere the symbol • represent a function composition, G ∈ R k×d denotes the encoded group embeddings, while V t ′ ∈ R n×d refers to the output video frame tokens. Distant Neighboring-Frame Transformer. It utilizes neighboring frames as queries to extract information from frames in the distant past. It is predominantly composed of four layers of standard transformer decoder units. Each frame in the video is augmented with 1-dimensional absolute positional encoding, independently applied to both past and neighboring frames. A directional attention mask is incorporated only for the neighboring frames, ensuring unidirectional information flow toward the current frame. Intuitively, the current frame embedding V t ′ N [-1] with more spatial information, will be aligned with the corresponding text as a raw video clip representation. Action Clustering. The action clustering block namely Ψ AC assembles frames into groups and aligns the groups to human-understandable categories in a data-driven manner, only supervised by video-text pairs. It consists of three steps, i.e., the frame-to-group binding that assigns static frames with similar semantics to a group, the group-to-action mapping that computes the cosine similarity between the group embeddings with the action labels, and background mask proposing that predicts a set of binary masks by computing statistics from the frame-group similarity matrix A.In its training, we devise a binary prediction task to cluster frames into background and nonbackground as well as the grouping. The background mask M B is predicted as:EQUATIONwhere A ∈ R n×k is derived from the attentional weights in slot-attention computation. It denotes the likelihood of each video frame being assigned into k learnable group embeddings. In Eq. 2, the group embeddings serve as the query Q, and frame tokens serve as keys and values. With the softmax operator normalizing over k, and the output of the slot-attention block is A ⊤ V , which is of shape k × d. The tensor T ∈ R 1×d represents the d-dimensional embeddings of the video caption.In its evaluation, similar to the computation of M B , a video clip's action prediction score namely P AC ∈ R n×C can be calculated as follows:EQUATIONIn contrast to Eq. 2, the frame-group similarity matrix A requires one-hot hard coding, while T val ∈ R C×d is encoded by the C categories descriptions in the test set.Object-Centric Decoder Unit. To group frames, we give an Object-Centric (OC) Decoder unit employing a slot attention mechanism [28] focused on the query object during cross-attention computation. It works similarly to the previously proposed GroupViT [43] and OVSegmentor [44] methods, which are specifically designed for semantic segmentation tasks.As illustrated in Fig. 2 , it requires two sets of inputs, i.e., target queries G consisting of a fixed number of k encoded grouping embedding and n input frame tokens V t ′ N to be queried, where n can be a large number. Following a layer of multi-head self-attention, G is transformed to G ′ . G ′ is then employed as a query in the second layer of multi-head slot-attention, while the frame tokens V t ′ N serve as the key and value. These two steps can be expressed asEQUATIONwhere θ : R k×d ⇒ R k×d denotes the dropout and norm operations.We adopt the pre-trained Text Transformer defined in CLIP [34] as the text encoder Ψ T . To bridge the gap between image-text and video-text models, we utilize the Adaptformer technique [7] , for lightweight transfer learning. We adopt the practice of setting up parallel adapters in each transformer block of Ψ T . For the input text, we use the CLIP Tokenizer [34] to add the tokenizer start [SOT ] and tokenizer end [EOT ] at the beginning/end. The text embedding is computed as T = Ψ T (T ), where T ∈ R •×l represents the tokenized caption with a length of l.Web video-text datasets [8, 42] are abundant in data volume, but their captions are typically prelabeled by image-text generative models [23, 9] and then filtered by humans semi-automatically. Consequently, these annotations are extremely noisy and include fantasy elements, despite being semantically rich and diverse. Additionally, we employ a sliding window approach to capture both the video frame and its corresponding text description during the live-streaming video. These factors introduce inherent bias in the visual and textual information of a sample pair. To achieve relatively accurate textual captions of the visual information in the video, we employ a multi-label contrast learning strategy. To generate multiple captions, we employ a linguistic analysis tool (e.g., the nltk toolkit [2] ) when the raw labels are not enough. This tool extracts verb-object phrase structures from the given descriptions, which are then utilized as keywords to create additional captions, drawing on CLIP's prompting engineering. To handle redundant captions, we choose the text associated with the highest number of frames as the global descriptor, and the remaining captions are utilized to compute the multi-label contrast loss. In the absence of any tags, we utilize the keyword "background" for sentence construction and as the global descriptor.We train the model through three proxy tasks, i.e., video-text alignment, current frame-text matching, and background mask prediction. The total loss is: L total = L contras + αL current + βL mask , where α, β are trade-off parameters controlling the relative weight of the above cost functions. Each used loss is detailed below. Current Frame-Text Alignment. We adopt Image-Text Contrastive loss (L IT C ) to learn whether the current frame image matches the caption. We denote the image embedding (from the distant neighboring-frame Transformer block) and the text embedding as z I and z T , respectively. Both embeddings are projected into a 512-dimensional joint feature space before calculating the matching loss. The current frame-text contrastive loss L current is:EQUATIONwhere τ is a temperature parameter to scale the logits, and B denotes the batch size.Multi-Label Video-Text Alignment. We employ the multi-label video-text contrastive loss to align the visual and language representations for enhancing the textual representations of videos. The multi-label video-text matching loss L contras is:EQUATIONwhere z V is computed as the average of the output V t ′ from the visual encoder. {z T0 , z T1 , . . . , z T M } are text embeddings, constructed in Sec. 3.2. All embeddings are mapped into 256-dimensional vectors. Refer to Eq. 5, z T0 denotes a global descriptor selected from the multi-label captions.Background Mask Proposal. Through video captions, we can roughly determine which frame chunks are background and which ones correspond to actions. This prior helps the action clustering block effectively group and bind the majority of background frames. To enhance the concatenation region between the predicted background mask M B and the caption pseudo mask M GT , we employ a per-frame binary mask loss. Following Maskformer [10] , we use dice loss [30] for our mask loss, i.e., L mask = L dice (M B , M GT ). The binary mask M GT is derived from the neighboring frames V t N of the input to the action clustering block. It is set to "1" for frames with a caption and "0" for frames without captions. Zero-Shot Online Inference Similar to CLIP's zero-shot transfer [34] , our distant neighboringframe Transformer block can assign the current frame image to the semantic category with the highest image-text embedding similarity. During online inference, similar to Eq. 3, the action prediction score namely P DN T R ∈ R 1×C for the current frame of a video clip can be written asP DN T R = V t ′ N [-1] • T ⊤ val .The action clustering block can also estimate frame action without fine-tuning. We calculate the similarity between the embedding of each frame token and the text embedding of the dataset. Then, we assign each frame token to the corresponding category with the highest similarity. This zero-shot transfer pipeline is depicted in Eq. 3. In summary, the final action prediction score y t ∈ R 1×C for a video clip V t can be expressed as:y t = P AC [-1] + αP DN T R .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Justification: We fully follow the ethics guidelines of NeurIPS.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
experiments: 
 Our run experiments on NVIDIA V100 ×8 using Pytorch 1.11.0. During both training and inference, we resample the video raw frame rate (e.g., 24/30 FPS) to 4 FPS, and resize images to 224 × 224 [49, 46] . For feature extraction, we employ the CLIP model (ViT-L). Specifically, the visual encoder computes a series of image patch tokens along with one global token (aka, CLS token) for each frame, and we utilize the normalized CLS token as the output feature encoding. Unless otherwise specified, all parameters of the visual encoder in our OV-OAD model are initialized from scratch, while the text encoder is initialized with the CLIP, except for the additional Adapter parameters. We train our OV-OAD for 30 epochs with 2 warm-up epochs using the Adam optimizer with weight decay 5e -2 . It uses a cosine schedule with a batch size of 256, and the initial learning rate is 1.6e -4 . Pre-training Datasets. We use the filtered InternVid-10M-FLT (aka, InternVid [42] ) and the ActivityNet v1.3 (aka, ANet [3] ) datasets for training, which are originally collected ∼4M and 14950 untrimmed video-caption pairs from the web, respectively. However, the videos within the InternVid dataset typically have longer durations compared to ANet, and the average percentage of foreground frames with annotations on these videos is only 27.4%. We sort the ∼4M videos in the InternVid dataset according to the number of caption annotations they contain and take the top 5000 videos (namely InternVid-5K) for training. For ANet, we utilize the prompting technique (following [35, 12] ) to convert the short action tags into sentences. And we combined its training and test sets and utilized them collectively for training. Please see Appendix A.2 for complete dataset preparation. Benchmarks. We follow previous works [47, 5, 4 ] and evaluate our model for the zero-shot online action detection on the validation splits of the THUMOS'14 [19] , TVSeries [14] , EPIC-Kitchens-100 (aka, EK100 [13] ), and FineAction [27] datasets. THUMOS'14 and TVSeries datasets comprise 20 and 30 foreground action categories, respectively. EK100 and FineAction datasets comprise 97 verb classes and 106 foreground action labels, respectively. An additional background class is considered for all datasets. Turn to the Appendix A.3 for dataset details.We evaluate metrics for online motion detection based on previous studies [38, 46, 4] . Specifically, we applied per-frame mean average precision (mAP) on THUMOS'14 [19] and FineAction [27] , and per-frame calibrated average precision (cAP) on TVSeries [14] and EK100 [13] .We conducted a comparison of the zero-shot online motion detection metrics between our method and other zero-shot baselines. We also explore the base-to-novel fine-tuning approach for zero-shot OAD and compare it to our OV-OAD model.Comparison with Zero-Shot Baselines. We utilize visual language models with image zero-shot capabilities (i.e., CLIP) for comparison. The inference process of online action detection involves sliding frame-by-frame sampling on an untrimmed video and subsequently predicting the action class of the last frame (aka, the current frame). We can set the sliding length to 1 and classify the actions based on a single frame image to simplify the inference. To zero-shot transfer CLIP to online action detection, We first extract the features of the frame images using its visual encoder, and then, we compute the similarity between the visual features and the text embedding of the dataset action labels. We can perform several non-parametric processes on the visual embeddings, including: 1) Averaging the visual embeddings of the neighboring frames to obtain the visual feature of the current frame, named CLIP-II; 2) Non-parametric clustering of all sampled frames (e.g., K-means algorithm), followed by averaging the visual embedding of the group to which the current frame belongs, resulting in the visual feature, named CLIP-III; 3) Directly using the visual embedding of the current frame as the visual feature, dubbed as CLIP-I. Table 1 presents the experimental results, clearly demonstrating the superior performance of our OV-OAD over other non-parametric zero-shot methods. It is worth noting that enhancing the scale of the visual language model leads to a moderate increase in single-frame action prediction accuracy. However, this improvement is constrained, indicating that relying solely on robust image discrimination is insufficient for achieving high performance. Consequently, it is essential to incorporate temporal structure information into the learning process for effective online action recognition.Furthermore, as depicted in Table 2 , we delve into the zero-shot performance of OV-OAD on more demanding datasets. We evaluate OV-OAD's performance on the first-view shots dataset called EK100. This dataset comprises first-view shots and notably deviates from the ANet data distribution. We also assess the performance of OV-OAD on the large-scale dataset, FineAction, which includes ∼4,000 uncut videos categorized into 106 action classes. The outcomes indicate that OV-OAD exhibits superior generalization capabilities in online action detection when contrasted with CLIP. Comparison with base-to-novel methods. We compare base-to-novel fine-tuning methods and fully-supervised transfer to online action detection on the THUMOS'14 dataset. For base-to-novel generalization, we integrate three well-known Transformer-based online action detection models including OadTR [41] , LSTR [46] and MAT [38] with a text encoder using the image-text contrastive loss. To ensure statistical significance, we adopted the random sampling setup and dataset partitioning method proposed by [21] . For our experiments, we employed two evaluation settings on the THU-MOS'14 dataset, i.e., training on 75% of the action categories and testing on the remaining 25%, and training on 50% of the categories while testing on the remaining 50%. We followed the experimental fine-tuning setup of [35] , including the Adam optimizer, the learning rate 1e -3 , and the Cosine decay function for training. For fully supervised transfer, we also train these models on 100% of the action categories with inputs of video frame features extracted by CLIP/ViT-B. We followed the training setup in [41, 46, 38] for the experiment, which remained consistent except for the different feature extractors. All experimental results are reported in Table 3 , † indicates that the model has not seen any categories and directly tests the performance of the unseen categories. The MAT-D48 indicates that MAT utilizes the ground truth of future frames (48 frames in 12 seconds) during training. We observe that the recent MAT method achieves better performance in the fully-supervised setting, but its performance is poor in the base-to-novel setting. One can find that our OV-OAD model outperforms the competition even without utilizing any training data. The results indicate that the base-to-novel fine-tuning method is not suitable for direct application to the zero-shot online action detection task. The limited availability of data may be a contributing factor to the unsuitability of the base-to-novel fine-tuning approach for online motion detection models.Proxy Tasks. We aim to validate the effectiveness of the three proxy tasks we introduced, namely current frame-text alignment, multi-label video-text alignment, and background mask proposal. As reported in Table 4 , our baseline model employs the multi-label video-text contrastive loss L contras only, upon incorporating the current frame-text matching loss L current , we observed a substantial improvement of 3.4% in mean Average Precision on the THUMOS'14 dataset. The performance improvement is due to the model's ability to capture spatio-temporal information from extended past frames. In addition, the prediction of the background frame mask also leads to improvements, and combining both results in optimal performance. This finding suggests that enhancing the model's capability to detect background frames is equally important.Frames. We first conduct an evaluation to assess the influence of inputting different numbers of neighboring and past frames on the model's performance. As depicted in Table 5 , our OV-OAD model demonstrates flexibility with different frame choices, resulting in a maximum performance variation of 1.6%. Note that, the highest mean Average Precision is achieved when utilizing V t N = 8 and V t P = 24. In addition, we investigate the effect of the number of network layers in different blocks on the performance. The results are presented in Table 6 , we find that increasing the number of layers for the first Transformer Encoder of our action clustering block results in a notable decline in performance. Conversely, a small number of layers for the last Transformer Encoder proves to be sufficient. Distant Neighboring-Frame Transformer. We further investigate the design of the proposed distant neighboring-frame transformer block Ψ DN T R . Unless specified otherwise, we employ 2-second neighboring frames, 6-second distant past frames, and CLIP/ViT-B pre-trained features. a) Can we remove the Ψ DN T R block? To implement this, we directly input all 8-second sampled video frames into our action clustering block, where the object-centered group module could easily cluster frame tokens that exhibit similarity. To ensure fairness, we set the number of Transformer layers in the action clustering block to be equal to the total Transformer layers in OV-OAD. As can be seen from Table 7 (row 4 vs. row 1), OV-OAD exceeds this baseline clearly. This also demonstrates the validity of our idea of applying neighboring frames to query spatio-temporal information from distant past frames.b) Can we remove the final Transformer encoder in Ψ AC block? Experiments were conducted to analyze the impact of removing the final transformer encoder on the model's performance. The result presented in the Table 7 (row 5 vs. row 4) indicates a marginal performance decrease of around 0.6% upon removing the final transformer encoder. Additionally, this action results in a 15% reduction in certain training parameters, specifically in the visual encoder. c) Can Ψ DN T R block be learned efficiently using the Transformer encoder unit? Here, we aim to explore whether the Ψ DN T R block can be learned efficiently using Transformer encoders only. To be specific, we combine the neighboring with distant past frames and feed them into a 4-layer Ψ DN T R block based on a standard Transformer encoder implementation. Table 7 (row 4 vs. row 2) illustrates that this baseline is clearly lower than our Ψ DN T R block constituted by the Transformer decoders. Furthermore, we introduced an additional layer of cross-attention after the 4-layer Transformer encoder to create a new baseline. We aim to assess the effectiveness of the "bottleneck" design of cross-attention within the Ψ DN T R block. Note that such an implementation also completes the process of querying discriminative information from distant past frames. Table 7 (row 4 vs. row 3) shows that the cross-attention design does exhibit effectiveness, but it falls short of achieving top performance. d) Ablation for the Action Clustering Block. Here, we analyze the impact of an object-centric decoder compared to a standard Transformer decoder unit within the action clustering block. Both are designed to bind semantically similar static frames into a group embedding. Table 7 (row 5 vs. row 4) demonstrate that the object-centered decoder outperforms the standard transformer decoder in performance.On Adapting Text Encoder For our baseline approach, following the initialization of our OV-OAD's text encoder with the CLIP's text encoder weights, we release the weights to continue training. Then, we conduct experiments to explore the performance impact of two separate modifications: 1) fix the full backbone parameters and 2) incorporate the Adapter structure. The results are depicted in Table 8 , one can see that leveraging the Adapter technique leads to a substantial improvement in performance. Moreover, to achieve the best results, it is necessary to release the backbone parameter and continue training.We compared the model parameters and efficiency of our OV-OAD model with other methods on a single NVIDIA Tesla V100 GPU, given in Tab. 9. Note traditional supervised learning methods, the efficiency bottleneck of the system is primarily attributed to the optical flow computation and its feature extraction. Our OV-OAD eliminates the need for optical flow computation and the extraction of spatio-temporal features from the RGB image. The overall system achieves an impressive inference speed of 292.3 frames per second (FPS). The findings indicate that our model has the potential to be deployed on standard online video capture devices, enabling real-time action prediction capabilities. In particular, LSTR demands a larger number of input frames for optimal performance, using 520 seconds of video for inference on THUMOS'14, while our OV-OAD utilizes only 8 seconds. This means that LSTR requires 65 times more data than OV-OAD, which likely explains why our model's inference speed is six times faster. Furthermore, the primary speed bottleneck for LSTR is the extraction of optical flow (8.1 FPS), whereas for our model, it is the extraction of image features (292.3 FPS). End-to-end online motion detection systems require simultaneous learning of spatial and temporal structures for optimal results. Our OV-OAD model utilizes the CLIP's visual encoder to extract features from pure images. This would cause the network to focus too much on modeling foregroundcentered spatial information at the expense of modeling spatio-temporal structure information. This does not fit the requirements of action recognition for visual representations since learned RGB features from a video commonly contain some of the temporal structural information, e.g., RGB features extracted by TSN [39] have some of the properties of optical flow features. Therefore extending OV-OAD to simultaneously model scenario and temporal information and enable action recognition with an open vocabulary remains a challenging problem. Failure Cases. Our objective is to identify categories that exhibit poor recognition as well as those with high recognition rates. We provide a list of categories with the highest and lowest action recognition average precision in Table 10 . Additionally, we present visual samples of these categories in Fig. 3 . We find that the detection accuracy decreases in scenarios where the foreground of the action is relatively low, and multiple actions share similar backgrounds (e.g., "CliffDiving" and "CliffShot"). However, OV-OAD demonstrates better performance when the foreground or interacting objects are more distinct, as observed in cases like "CleanJerk" and "PoleVault". These findings suggest that future enhancements can focus on improving the recognition of fine-grained actions through joint modeling of spatio-temporal information.  
conclusion: 
 In our study, we take the initial step towards leveraging text learning for online action detection without explicit human supervision. Our findings demonstrate that by employing OV-OAD, reproductions acquired from large-scale video-text pairs, even with noise, can be successfully transferred to online action detection in a zero-shot manner. Moreover, we highlight that the conventional approach of base-to-novel fine-tuning does not yield favorable results on traditional online action detection test datasets. Instead, we illustrate that similar semantic frames can be directly clustered and transferred to downstream action detection datasets using abundant textual supervision.processing the ANet dataset, we employ the prompting project combined with the original action tags as keywords for sentence construction.InternVid-5K. InternVid is a significant multimodal dataset that focuses on video-centric learning for multimodal understanding and generation tasks. It serves as a valuable resource for developing powerful and transferable video-text representations. The InternVid dataset is vast, comprising over 7 million videos with a cumulative duration of nearly 760,000 hours. Within this extensive video collection, there are 234 million video clips available, each accompanied by detailed descriptions that consist of a total of 4.1 billion words. The dataset's multimodal nature combines visual and textual information, enabling researchers to explore the relationship between videos and their accompanying descriptions.The annotations provided by InternVid for this paper offer valuable information, but they pose challenges for our pre-training process. This is because video annotations generated using large models of image-text often contain text noise and inconsistencies. Additionally, these annotations tend to be concentrated in specific segments, lacking homogeneity. These features present difficulties for online action recognition tasks. To address these challenges, we conducted a thorough examination of the syntax in InternVid's text data and identified certain speech defects in the annotations. We filtered the training samples extensively to mitigate these issues. Initially, we selected 100,000 videos by sorting them based on the total number of annotated entries in each long-term video. Subsequently, we applied additional filtering based on the ratio of annotated frames to the total number of frames. Through this process, we selected 5000 samples of temporal text-pair data to form the InternVid-5K dataset. Depending on the acceptance of the paper, we plan to release the metafile of the modified dataset, which incorporates the aforementioned alterations, to further improve the quality and consistency of the annotations. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: The paper discuss both potential positive societal impacts and negative societal impacts of the work performed in Sec. 4.3.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We use publicly available datasets, benchmarks, and models for training and evaluation, free from any possible harm toward individuals or groups.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: THUMOS'14. The THUMOS'14 (THUMOS 2014) dataset is a significant video dataset widely used for action detection and recognition tasks. In the THUMOS'14 dataset, there are 220 videos in the validation set and 212 videos in the testing set that have been annotated with temporal boundaries. These annotations provide precise information about the start and end times of specific actions within the videos. With its extensive video collection, class diversity, and temporal annotations, the THUMOS'14 dataset serves as a valuable resource for advancing state-of-the-art in action detection research.TVSeries. The TVSeries Dataset is a comprehensive and realistic large-scale dataset specifically designed for action detection tasks. It encompasses a total of 16 hours of video content extracted from six recent TV series. The dataset includes a diverse range of scenes and contexts, offering a representative sample of real-world action scenarios. Within the TVSeries Dataset, there are thirty distinct action classes defined, covering a wide spectrum of human activities. Each action instance in the dataset is meticulously annotated with precise start and end times, providing valuable temporal information for action detection algorithms.FineAction. FineAction comprises 103,000 temporal instances across 106 action categories, annotated within 17,000 untrimmed videos. The dataset offers new opportunities and challenges for online action detection, characterized by finely defined action classes, diverse attributes, multiple instance annotations, and concurrent actions from various classes. With around 4,000 uncut videos categorized into 106 classes like "Household Activities", "Personal Care", "Socializing", "Relaxing", "Sports", and "Exercise", FineAction sets the stage for innovative research.EPIC-KITCHENS-100. EPIC-KITCHENS-100 (EK100) includes first-person perspective shots and significantly differs from the ANet data distribution. EK100 contains 100 hours of video, 20 million frames, and 90,000 action segments across 45 environments, with narrations mapped to 97 verb classes and 300 noun classes. This dataset addresses various challenges such as action recognition, detection, and anticipation, offering a platform to assess model generalization across time and diverse contexts.A.4 Explain the Performance Differences on THUMOS'14 and TVSeries.The case of pre-training with ANet. The variance in improvements between THUMOS'14 and the TV series can be attributed to the resemblances in data distribution across these two datasets and our utilization of ANet. The substantial boost of OV-OAD on THUMOS'14 can be attributed to ANet encompassing a broader array of action categories. By employing a method of close comparison in natural language, we identified 8 similar action phrases within THUMOS'14's 20 action categories, constituting 40% of its overall categories. In the TV series dataset, we identified 9 similar action categories out of 30, equating to 30% of its total categories.Using different pre-training datasets. Similarly, OV-OAD achieves better performance on THU-MOS'14 due to the fact that ANet covers a wider range of action categories than IVid. Using the same natural language tools, we compared the coverage ratios of ANet and IVid for THUMOS categories, which stood at 40% and 15%, respectively. To elaborate, when contrasting the IVid and THUMOS datasets, we considered 500 high-frequency verbs as the action categories for IVid. Then, we pinpointed 3 categories from them that were similar to THUMOS'14.NeurIPS Paper Checklist Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 


Decision: The submission proposes to leverage pre-trained video-text models for the open-vocabulary temporal action detection task. The task requires _online_ detection, where an action needs to be recognized as soon as it appears in the video stream. Evaluations are performed on THUMOS'14 and TVSeries benchmarks, where the proposed OV-OAD framework outperforms CLIP-based approaches, and comprehensive ablation experiments are provided to demonstrate the effectiveness and efficiency of the proposed approach.

All reviewers actively engaged with the authors during the rebuttal discussion. The reviewers found most of their concerns to be addressed by the rebuttal and all reviewers recommended "weak accept". The AC agrees with the reviewers that the submission would be of value to the NeurIPS research community and thus recommends acceptance. The authors are expected to incorporate the reviewers' feedback and the relevant rebuttal experiments in the camera ready version.

Reviews: review: paper_summary: paper_summary: 1. The authors have proposed a new method for Online Open Vocabulary action detection by leveraging pretrained vision langugae models. 

2. To that end they introduce 2 main modules, a distant neighboring frame transformer and an object centric Action clustering unit.

3. They train their model with three objectives on filtered versions of the Activity-Net v1.3 and Intern Vid-10M-FLT datasets.

4. They evaluate the model zero shot on the validation splits of Thumos’14 and TVseries datasets and with a base-to-novel formulation on the Thumos’14 dataset. The authors show that in the case of the former their model beats naive CLIP baselines by ~2-7 % on Thumos’14 and ~2% on TVSeries dataset and in the case of the latter beat the OadTR model trained on the seen split of Thumos’14 by ~6-9%

5. The authors also demonstrate superior inference speed compared to OadTR.
strength_weakness: strength_weakness: 1. The authors claim in lines 41 and 42 that they do not use any frame-level annotations. However the Actitvity Net dataset contains annottaions for start and end times(therefore frames) for actions. And it seems that for the L_current objective in section 3.3, the text supervision is provided to the current frame. These claims seem contradictory and more clarity about this will be better. 

2. In the zero shot baseline comparison the improvement in case of Thumos-14 seem to be much larger than that of TV-series. Some explanation regarding this huge disparity is neccessary. Zero shot evaluations on more datasets can help indicate the robustness of this method to different data distributions.

3. In the zero shot baseline comparison, there seems to be a very large improvement for Thumos’14 in case of the ANet model compared to the InternVid model. The improvement is 2% for the latter while ~7% for the former.  Could this be attributed to similarity of actions, between thumos’14 and Anet ? Some investigation regarding this could shed light on the previous point. if that is the case then the improvement range disparity for TV-series( ~2%) could be explained.

4. The authors have compared their results for the open vocab evaluation with only the OadTR model. Comparison with more/better Online Action detection models (like GATEhub, MiniROAD, LSTR, Colar) is necessary for a holistic evaluation of the proposed model.

5. The ablations do not contain the different parts of the action clustering unit. Some results without the Object centric decoder are needed to justify its introduction. In table 5, there is no ablation for not using the final transformer encoder.  It should be added to justify its introduction as well.

6. In Figure 2. it is not clear what is being fed from the output of the object centric decoder to the final transformer encoder. It needs to be clearly mentioned in the figure for clarity.
questions: questions: see weaknesses
limitations: limitations: yes
paper_summary: paper_summary: The paper introduces OV-OAD, a novel zero-shot online action detection system leveraging vision-language models for open-world temporal understanding. The authors propose a Transformer-based model with an object-centered decoder unit, trained solely on video-text pairs without manual frame-level annotations. The system is evaluated on THUMOS'14 and TVSeries benchmarks, demonstrating superior performance over existing zero-shot methods.
strength_weakness: strength_weakness: 1. The OadTR model was proposed in 2021, and the authors should consider using more recent OAD models such as MAT, MiniROAD, etc., as comparative benchmarks to enhance the timeliness and competitiveness of their model.
2. The authors have employed a multitude of complex modules for Zero-shot experiments on two datasets, with improvements on the TVSeries dataset being relatively limited compared to THUMOS14. This raises doubts about whether the proposed model is robust enough to serve as a standard baseline for future research.
3. As a pioneering work, the authors have experimented on typical action datasets like THUMOS14 and TVSeries. However, it would be beneficial to also test on atypical human action datasets, such as creating a benchmark for the HDD dataset.
4. In the action clustering block, which serves as the query and which as the key and value between the input Group Embedding and the output tokens? According to Figure 2, it seems that the Group Embedding is the query, and the latter is the key and value, but why is the dimension of the attention weight matrix $A$ $n \times k$ instead of $k \times n$? (line 142)
5. Line 154 mentions that the number of neighboring frames n can be a large value, and Table 4 shows the highest $n$ up to 8. Can the authors present experimental results for higher values of $n$?
6. Could the authors further explain how the model achieves ultra-high inference speed with a larger scale of parameters? For example, the parameter amount is twice that of LSTR, but the inference speed is six times as fast.
7. The paper lacks explanations for some operations, such as the operator "$\circ$" in Equation 1. This may lead to misunderstandings, mistaking it for a Hadamard product rather than a composite function.
8. It is recommended that the authors add visible variables in the figures used to illustrate the method (such as Figure 2) and improve the explanation of different parts of the chart to enhance readability.
questions: questions: 1. The OadTR model was proposed in 2021, and the authors should consider using more recent OAD models such as MAT, MiniROAD, etc., as comparative benchmarks to enhance the timeliness and competitiveness of their model.
2. The authors have employed a multitude of complex modules for Zero-shot experiments on two datasets, with improvements on the TVSeries dataset being relatively limited compared to THUMOS14. This raises doubts about whether the proposed model is robust enough to serve as a standard baseline for future research.
3. As a pioneering work, the authors have experimented on typical action datasets like THUMOS14 and TVSeries. However, it would be beneficial to also test on atypical human action datasets, such as creating a benchmark for the HDD dataset.
4. In the action clustering block, which serves as the query and which as the key and value between the input Group Embedding and the output tokens? According to Figure 2, it seems that the Group Embedding is the query, and the latter is the key and value, but why is the dimension of the attention weight matrix $A$ $n \times k$ instead of $k \times n$? (line 142)
5. Line 154 mentions that the number of neighboring frames n can be a large value, and Table 4 shows the highest $n$ up to 8. Can the authors present experimental results for higher values of $n$?
6. Could the authors further explain how the model achieves ultra-high inference speed with a larger scale of parameters? For example, the parameter amount is twice that of LSTR, but the inference speed is six times as fast.
7. The paper lacks explanations for some operations, such as the operator "$\circ$" in Equation 1. This may lead to misunderstandings, mistaking it for a Hadamard product rather than a composite function.
8. It is recommended that the authors add visible variables in the figures used to illustrate the method (such as Figure 2) and improve the explanation of different parts of the chart to enhance readability.
limitations: limitations: In summary, the authors have examined the OAD (Online Action Detection) from an interesting perspective and proposed a novel Zero-shot model, which appears to have profound application value compared to traditional OAD methods. However, additional experiments may be necessary to substantiate the claims fully.
paper_summary: paper_summary: This paper addresses a challenging setting in video understanding: open-vocabulary online action detection. It leverages pre-trained visual language models with a proposed dual-encoder architecture to achieve successful zero-shot detection.
strength_weakness: strength_weakness: 1. The choice of VLM. The method uses CLIP as the visual-text encoder. However, as discussed in the limitations section, CLIP is an image-based understanding model and lacks the capability to capture temporal context. I wonder if other VLMs, such as ActionCLIP, could mitigate these drawbacks for the proposed task.
2. The comparison of the proposed method with previous work, such as OadTR, seems to use different visual encoders/features. Can the authors explain the fairness of such a comparison? Besides, the author should also compare the method with more recent methods, since OadTR is not state-of-the-art in recent works.
questions: questions: Please see the weaknesses section.
limitations: limitations: Limitations have been well discussed by the authors in the paper.
paper_summary: paper_summary: This paper explores the problem of open vocabulary video action detection in an online setting, where the action must be detected immediately once it appears in the video stream, vs. the more common offline setting that allows examining the entire video, past and future.

The authors propose a model with two main components: a transformer decoder that cross-attends between recent and past video frames, and an action clustering block that uses slot attention to group related frames and then classify them by action.

The model is trained on combination of three tasks/losses: contrastive image-text loss between the current frame's visual embedding and text embedding (vs. text embeddings from other clips in the batch), multi-label contrastive video-text loss (for text clips described by multiple text labels), and mask loss for identifying which frames come from the background (no action present).

The model demonstrates improved performance over CLIP baselines in the novel open vocabulary online action detection setting.
strength_weakness: strength_weakness: Line 61 says that "our model successfully learns clusters of similar video frames", and I see that indirectly based on the fact that there's a clustering module that leads to improved performance. However such a claim would be stronger if supported by evidence or examples of frames being correctly clustering based on action semantics.

The Object-Centric Decoder unit seems to be confusingly-named, since it is employed "to group frames" (line 149), rather than to focus on specific objects appearing within a frame.

In Table 1, in the proposed setting, the model is compared only against CLIP variants. Another baseline worth exploring would be video-text models (as opposed to image-text) applied via a sliding window.

There are a few parts of the paper that could be explained more quickly, in my opinion. (See the questions below.)
questions: questions: Do you see a difference between "open vocabulary" and "zero shot" action recognition? The terms seem to be used synonymously in this paper, e.g. lines 106-107.

Related, how much overlap is there between the textual descriptions in ActivityNet and InternVid  with the category label sets of THUMOS'14 and TVSeries?

Can you explain "the assumption of low background information for VLM" from line 36 in more detail?

Can you please clarify "fails to reach future frames during training" from line 37?

What precisely is the circle operator in equation (1)?

On Line 175, how do you know "when the raw labels are not enough"?

In Table 2, OVO_{ad}^{dagger} refers to OV-OAD, correct?
limitations: limitations: yes

Comments: title: Official Comment by Authors
Official Comment by Authors
Please let us know if your concerns have been addressed
Please let us know if your concerns have been addressed
Please let us know if your concerns have been addressed
Please let us know if your concerns have been addressed
Response to Comments
Author Rebuttal by Authors
Ablation Results for Action Clustering Block
Incorporating FineAction Evaluation Results
Response to Comments
comment: Dear Reviewer Boy7,

We appreciate your increased score! **Your insights have been incredibly helpful, and we are excited to incorporate the changes based on your suggestions into our paper.**

Once again, thank you sincerely for your support and valuable feedback!

Best regards,

Authors of Paper #1178
Dear Reviewer  1BHH,

Thank you for increasing your score! Your insights have been incredibly helpful, and we are excited to incorporate the changes based on your suggestions into our paper.

Once again, thank you sincerely for your support and valuable feedback!

Best regards,

Authors of Paper #1178
Since the reviewers have addressed all my concerns and suggestions, I have now raised my rating to weak accept.
Dear Reviewer wz2n,

We wish to express our gratitude for your extensive review and positive comments. **Your comments are invaluable to us, and we are fully committed to thoughtfully incorporating your insights to enhance our paper**. As the discussion phase is nearing its end, we are warmly concerned whether our rebuttal addresses your concerns. 

**It would be appreciated if you could raise your score on our paper if we address all your concerns.** We thank you again for your effort in reviewing our paper.

Best regards,

Authors of Paper #1178
Dear Reviewer Boy7,

We wish to express our gratitude for your extensive review and positive feedback. Your feedback is invaluable to us, and **we are fully committed to thoughtfully incorporating your insights to enhance our paper.** As the discussion phase is nearing its end, we would be grateful to hear your feedback and wondered if there are any other concerns we could address, especially as mentioned in your feedback regarding weakness 1 and other reviewers' opinions.

**It would be appreciated if you could raise your score on our paper if we address your concerns.** We thank you again for your effort in reviewing our paper.

Best regards,

Authors of Paper #1178
Dear Reviewer zhJf,

We wish to express our gratitude for your extensive review and positive comments. Your comments are invaluable to us, and **we are fully committed to thoughtfully incorporating your insights to enhance our paper**. As the discussion phase is nearing its end, we would be grateful to hear your feedback and wondered if you might still have any concerns we could address.

**It would be appreciated if you could raise your score on our paper if we address all your concerns.** We thank you again for your effort in reviewing our paper.

Best regards,

Authors of Paper #1178
Dear Reviewer  1BHH,

We wish to express our gratitude for your extensive review and positive feedback. Your feedback is invaluable to us, and **we are fully committed to thoughtfully incorporating your insights to enhance our paper**. As the discussion phase is nearing its end, we are warmly concerned whether our rebuttal addresses your concerns.

**It would be appreciated if you could raise your score on our paper if we address your concerns**. We thank you again for your effort in reviewing our paper.

Best regards,

Authors of Paper #1178
Thank you for taking the time to review our response and for providing valuable feedback. We are pleased to hear that the majority of your concerns have been addressed.

Indeed, your question Q1 has inspired us significantly. In the limitation analysis of the initial draft of our paper, we made a similar intuitive assumption that better vision-language models (VLMs) could further enhance the performance of zero-shot online action detection. We are actively exploring the use of CLIP variants of VLMs (e.g., ActionCLIP) for feature extraction of video frames. However, due to resource constraints within our group, we are currently in the process of extracting features from the **14,950** videos in the ANet dataset using **ActionCLIP ViT-B/16**, which involves frame-by-frame extraction and saving the features to a storage device. We have made significant progress, with approximately **80%** of the extraction complete. We are committed to addressing all your concerns and will incorporate the aforementioned experiments and analyses into the final version of the paper.

Moving forward, our primary direction for future research is to improve the feature extraction backbone, building on the insights gained from this work.
Dear all,

We appreciate the reviewers for their valuable feedback, acknowledging that our work presents a **"novel"** method for online open vocabulary action detection (**Reviewers 1BHH, zhJf, and Boy7**), exhibits "significant advancement" with **"state-of-the-art performance"** (**Reviewers zhJf and Boy7**), and demonstrates "superior inference speed" (**Reviewers 1BHH and wz2n**). They have recognized the "extensive experiments" and **"comprehensive ablations"** provided (**Reviewers zhJf, Boy7, and wz2n**). Additionally, the paper's addressing of a "realistic problem setting" and the inclusion of a "robust set of ablation experiments" were noted positively (**Reviewer wz2n**). We are sincerely grateful to see the positive and valuable responses to our rebuttals from all reviewers, in which they claim that their concerns have been addressed, and the additional details mentioned in the response will be  **synced** into the final manuscript.

The core innovation of our OV-OAD lies in proposing an effective pre-training framework for online open vocabulary action detection, leveraging vision-language models to overcome the limitations of existing methods in real-world applications. **This is a non-trivial paradigm in the field of online action detection, and we are among the first to  propose an open vocabulary online action detection model, trained solely on video-text pairs without manual fine-grained frame-level annotations.** Our contributions can be summarized as:
- **Open-Vocabulary Online Action Detection**: Achieved by leveraging pre-trained vision-language models and training solely on video-text pairs without manual fine-grained frame-level annotations.
- **Action Clustering Block**: Designed to aggregate frames with similar semantics using video-text correspondence within a Transformer-based model.
- **Superior Model Performance**: Surpassing existing zero-shot methods on THUMOS'14, TVSeries, EK100, and FineAction benchmarks in the novel open vocabulary online action detection setting.
- **Comprehensive Evaluations**: Providing detailed ablation studies and analyses to validate the numerous model design choices and demonstrate the limitations of Base2Novel's zero-shot online motion detection approaches.

Once again, we appreciate the supportive feedback and strongly believe that these reviews have strengthened the work.

Sincerely,

Authors of Paper #1178
Thank you for carefully considering my response. I am glad you found my suggestion useful. All my concerns have been addressed.
Thank you for your detailed rebuttal. While I appreciate the effort to address my concerns, I keep my score as weak accept.
Here, we analyze the impact of an object-centered decoder compared to a standard transformer decoder unit within the action clustering block. Both are designed to bind semantically similar static frames into a group embedding. The outcomes presented below demonstrate that the object-centered decoder outperforms the standard transformer decoder in performance.
|Clustering Unit|THUMOS'14 (mAP%)|
|:-----|:-----:|
| Object-Centric Decoder | **37.5** |
| Standard Transformer Decoder | 37.0 |
Here, we assess the performance of OV-OAD on the large-scale video dataset, FineAction, which includes ~4,000 uncut videos categorized into **106** classes encompassing *Household Activities*, *Personal Care*, *Socializing*, *Relaxing*, *Sports*, and *Exercise*. In line with FineAction [b], we conducted online action detection inference on the complete validation set of FinaAction, utilizing the per-frame mean Average Precision (mAP) metric. The subsequent table displays the results, illustrating OV-OAD's notable enhancement compared to the baseline methods.
|Methods|Arch|FineAction (mAP%)|
|:----:|:----:|:----:|
|CLIP-I|ViT/B|26.5%|
|CLIP-II|ViT/B|27.8%|
|OV-OAD|ViT/B|**29.2%** ($\uparrow$ 1.4%)|

Finally, we commit to addressing all concerns and incorporating the mentioned experiments in the final manuscript if the paper is accepted.
Thank you for taking the time to review our response and for offering valuable feedback. We are glad to know that most of your concerns have been resolved.

Your feedback on Q6 was particularly insightful. We initially misunderstood your question and proceeded with the experiment as per your recommendation regarding the exclusion of the final transformer encoder. The data presented in the table below indicates a marginal performance decrease of around **0.6%** upon removing the final transformer encoder. Additionally, this action results in a **15%** reduction in certain training parameters, specifically in the visual encoder.

We are fully committed to addressing all your concerns and will incorporate the aforementioned experiments and analyses into the final version.

| $Ψ_{DNTR}$ | $Ψ_{AC}$  | THUMOS'14 (mAP%) |
|:---|:----:|:----:|
|4|6-2|37.5|
|4|6-0|36.9|
I appreciate the author's effort during the rebuttal. The author has clarified most of my concerns. Considering weakness 1 and other reviewers' opinions, I will keep my score as borderline accept.
Thank you for your detailed reply.

Most of my concerns have been addressed except for Q6 in your rebuttal where you claim "As shown in Table 5 (rows 2 vs. 3). we see that reducing the number of layers in the final transformer encoder from 6 to 3 results in a performance loss." 

The performance changes by 0.1 which is too small in my opinion to demonstrate the utility of the final transformer encoder. I suspect this is the case because the authors perform average pooling (as depicted in Figure 2.) after the encoder which makes a lot of the processing in the final encoder redundant. I would also like to point out crucially that removing the final text encoder does not mean abandoning the L_{contras} loss. I suggest that the authors remove the final encoder completely and run an experiment with the same set of objectives as before to confirm this point. This is not a major issue but i think it will improve the quality of the paper.

My rating stays unchanged.
Thank you for the additional clarification, and the analysis of the performance of a video-text baseline. I appreciate the additional inclusion of benchmarks vs. more recent techniques, and on more datasets. I believe all of my concerns were addressed.