Text-Infused Attention and Foreground-Aware Modeling for Zero-Shot Temporal Action Detection
Authors: Yearang Lee
Ho-Joong Kim
Seong-Whan Lee
Venue: NeurIPS 2024 poster (2024)
Keywords: Temporal Action Detection
Vision and Language
Zero-Shot Learning
Primary area: machine_vision

Abstract: Zero-Shot Temporal Action Detection (ZSTAD) aims to classify and localize action segments in untrimmed videos for unseen action categories. Most existing ZSTAD methods utilize a foreground-based approach, limiting the integration of text and visual features due to their reliance on pre-extracted proposals. In this paper, we introduce a cross-modal ZSTAD baseline with mutual cross-attention, integrating both text and visual information throughout the detection process. Our simple approach results in superior performance compared to previous methods. Despite this improvement, we further identify a common-action bias issue that the cross-modal baseline over-focus on common sub-actions due to a lack of ability to discriminate text-related visual parts. To address this issue, we propose Text-infused attention and Foreground-aware Action Detection (Ti-FAD), which enhances the ability to focus on text-related sub-actions and distinguish relevant action segments from the background. Our extensive experiments demonstrate that Ti-FAD outperforms the state-of-the-art methods on ZSTAD benchmarks by a large margin:  41.2\% (+ 11.0\%) on THUMOS14 and 32.0\% (+ 5.4\%) on ActivityNet v1.3. Code is available at: https://github.com/YearangLee/Ti-FAD.

Text-Infused Attention and Foreground-Aware Modeling for Zero-Shot Temporal Action Detection
Ho-Joong Kim Seong-Whan Lee" Dept. of Artificial Intelligence, Korea University, Seoul, Korea fyr_lee, jojoong_kim, au. 1eereee ac. ac.kr
Yearang Lee
Abstract
Zero-Shot Temporal Action Detection (ZSTAD) aims to classify and localize action segments in untrimmed videos for unseen action categories. Most existing ZSTAD methods utilize foreground based approach, limiting the integration text and visual features due to their reliance pre-extracted proposals. In this paper, we introduce a cross- moosll ZSTAD baseline with mutual cross-attention, integrating both text and visual information throughout the detection process. Our simple approach results in superior performance compared to previous methods. Despite this improvement, we further identify common-action bias issue that the cross-modal baseline over-focus on common sub-actions due to a lack of ability discriminate text-related visual parts. To address this issue, we propose Text-infused attention and Foreground-awan Action Detection (Ti-FAD), which enhances the ability to focus on text-related sub-actions and distinguish relevant action segments from the background. Our extensive experiments demonstrate that Ti-FAD outperforms the state-of-the-arr methods on ZSTAD benchmarks by a large margin: 41.2% (+ 11.0%) on THUMOS 14 and 32.0% (+ 5.4%) on ActivityNet v1.3. Code is available at: /https///ithub.coom http://ghhhbbbooo///eeaaaeLL//// YearangLee//T--FAD
Introduction
Temporal action detection (TAD) is fundamental task in video understanding, aiming localize and classify action instances within untrimmed videos. Conventional TAD methods are time-consuming and expensive for annotating long untrimmed videos, making them impractical for real-world scenarios. Zero-shot temporal action detection (ZSTAD) [8] 18 20 26 29 has been introduced to recognize unseen action categories that categories defined by textual descriptions. As large-scale pre-trained visual-language (ViL) models (e.g. CLIP (1 and ALIGN 6 have shown impressive performance zero-shot downstream tasks, recent works [18 20 26 have extended their application to ZSTAD.
included in the training dataset by utilizing arbitrary
Most existing ZSTAD methods 88 18 20 employ foreground- based approach that initially generates the foreground candidate proposals from the visual features and subsequently fuses them with the text features, shown in Fig. [ (a). This foreground-based approach fuses only text features with visual features the foreground, following a previous ViL model trained on an environment containing the foreground. However, addressing only foreground features limits the use of the entire information of text-visual modalities due to the following two reasons: (1) the text features are only incorporated with the foreground candidate proposals that contain the partial information of the video, and (2) the visual features are utilized for localization without incorporating the text features.
To demonstrate the importance of incorporating both text and video information throughout the entire detection process, we construct a simple cross-mi modal bascline employing the attention mechanism 255 as illustrated in Fig. (b). Even with this simple approach, we confirm that our cross-modal
*Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Embedding
Visaal Emteddiing
Viuaa Prepesal
Text
Enc
X Classification
Text
Ene
Classification
Model
Model
Video->
Enc
Localization
Euc
Lucalization
(a) Previous Approach
(b) Pross-MMdal Baseline
lump"
PPoleFand''
Running
Herioental Jump
Ruaning
Baseiio::
'LangJump'
'Highlump'
T-FAD:
'LongJump'
"Pole Vault?
(c) Seen Class
(d) Unseen Class
Figure I: Top: the structure comparison (a) the previous foreground-ben saaeround-based approach and (b) our cross-modal baseline. Bottom: the example of the common-action bias issue. The yellow section represents ground-truth segments. The blue and red lines denote the classification scores for our cross-modal baseline and Ti-FAD, respectively.
baseline achieves superior performance compared to the previous methods X However, we observe an common-action bias issue in our cross-modal baseline that the classification score tends to capture the common sub-actions within the ground truth.
18 20 (Sec. 3.2).
To illustrate this issue, we investigate classification scores of seen and unseen classes. Fig. D (d)- shows example of the common-action bias issue by comparing the classification scores that represent the similarity text and video features. As shown Fig. [c). our cross-modal baseline (blue line) achieves focusing the text-related parts but fails to clearly differentiate between two sub-actions Running and Horizontal Jump, resulting in the model focusing on common sub-actions regardless of the specific text input. As shown in Fig. (d),, for the unseen class, baseline (blue line) over-focuses on the common sub-actions (e.g., Running) that are common visual information contained in the seen ground-truie set. Consequently, our cross-modal baseline incorrectly detects the PoleVault action as HighJump. How can we enable the model to focus on the text-related discriminative sub-action (e.g., Swing Up) to alleviate the common-action bias issue?
In this paper, we propose novel Text-infused attention and Foreground- aware Action Detection (Ti-FAD) method, which captures discriminative sub-actions for accurate detection of action instances. Built upon our cross-modal baseline model, Ti-FAD introduces two key modules to solve the common action bias issue: (1) Text-infused Cross Attention (TiCA) that enables the model to focus the discriminative sub-actions that are most relevant to the text description, and (2) a foreground-awe awware head that accurately distinguishes action segments from the background. These enhancements enable Ti-FAD to improve the integration of text and visual information, leading to more precise classification and localization of action instances in untrimmed videos. As shown in Fig. (()) Ti-FAD (red line) captures the discriminative sub-action parts in the video.
In summary, main contributions
follows:
We construct a novel cross-modal baseline that integrates text and visual features throughout the entire temporal action detection process.
We propose Ti-FAD, which incorporates TICA and a foreground- aware head to focus on discriminative sub-actions, particularly in unseen action categories.
Our extensive experiments on THUMOS14 14 and ActivityNet v1.3 demonstrate that our Ti-FAD outperforms state-of-the-art methods considerable margin.

---

vid
50%-50%
75%-25%
FFN
FFN
40
37.2
X-MHA
X-MIHA
MAP (%)
30
22.1
24.1
20
10
MHSA
WI
wio Finetuning
TPT
w/ Adapter
Finctuning
F
STALE
ZEETAD
Our Baseline
vid
(a)
(b)
Figure 2: (a) Structure of our cross-modal baseline. (b) Comparison with existing methods under 50%-50% and 75%-25% settings on THUMOS14, TPT denotes text prompt tuning.
2 Related Work
perform cross-modal contrastive learning on image-text pairs, exhibiting robust performance open-set image elassification tasks. Previous works [8] [8] 20) have explored adaptation methods such as text prompt tuning [10] to apply these models for downstream tasks. In this work, we utilize the CLIP text encoder to extract text features directly, without employing any text prompt tuning.
Vision-Language Models. CLIP [2 and ALIGN 66
Temporal Action Detection. Existing TAD methods 3
have explored various approaches for localizing and classifying actions in untrimmed videos. Among these methods, have demonstrated superior performance by capturing long-length actions and simplifying the detection process. In this work, we adopt ActionFormer [27] as our baseline detector because of its superior performance and wide applicability across various datasets.
22
27
anchor-free detectors 112 22
Zero-Shot Temporal Action Detection. ZSTAD extends TAD to open-set scenarios. To employ ViL follow a foreground approach and adopt the text prompt tuning strategies. Emprommp S generates action foreground proposals via an off-the-shelf proposal detector (e.g., BMN
models for ZSTAD, the previous methods [8 18
19
20 28
I and classifies actions utilizing CLIP (2T). STALE [8] and ZEETAD (20] generate the foreground candidate proposals utilizing class-agnosee representation masking method and parallels localization and classification. Despite the advantage of not relying on an external proposal detector, both methods provide the visual features to the localiza- tion branch without fusing with textual features, and their masked foreground representation does not contain text information. Recently introduced UnLoc (26) fuses the video and text information without any pre-extracted proposals. However, the fusion stage of UnLoc proceeds with simple self-attention to align both text and video modalities. In contrast, our approach emphasizes the importance of cross-modal fusion integrating text and video information across the entire process.
3 Method
3.1 Problem Definition
Given a training set of untrimmed videos Dtrain, each video contains the set of videos features X { where T denotes snippets (a few sequences of frames). The labels of each video y= {sn, en: Cn ] contain N action instances, where sn and en represent the start and end points of each instance, and e, identifies the category of the action. For the zero-shot scenarios, the action categories in Dirain and Drest are distinct and non-overlapping, with Dtrain n Drest 2. ZSTAD focuses on classifying and localizing actions in untrimmed videos, specifically targeting unseen classes that were not present during the training phase.
3

---

3.2 Cross-Modal Adaptation Baseline
Utilizing the text-video information in the overall process is erucial in the molti- adaptation to transfer the comprehensive semantic representation to zero-shot inference [16]. Inspired by the previous work [TT] that emphasizes the necessity of enhancing text-visual fusion, we introduce cross-modal baseline that updates both text and video features in the fusion step. Specifically, 27 as an action detector.
following the previous method [26], we adopt ActionFormer
Given an untrimmed video and text, we extract video features Fild = (usa(X) E RTx with a video
encoder @via(.) (-) (e.g., I3D 22 CLIP [21), and R(2+1)D (24)), and text features P/ E CxD with text encoder, such as CLIP (21. where D denotes the dimension of the embeddings and C represents the number of classes. We utilize "classname" as the text input without any prefix contextual text prompt. Fig. 2 (a) shows the structure of our cross-modal decoder, which aggregates both text and video features by learning their joint representation. As shown in Fig. (a), both the initial video features F and text features F(0) are processed through series of L layers. Each layer consists of multi-head self-attention (MHSA), followed by a multi-head cross-attention (X-MHA), and then
refined by a feed-forward network (FFN). In MHSA for video features F(0) uid= we employ multi-scale visual features with multi-head local self-attention (7). In each layer [ = 1,.....
....... I, the features are updated through MHSA, transforming F F((-1) and E((-1) to Fx(t) vnd and ( respectively. Then, these updated features video features as the query, but both cases (video features as the query and text features as the query) are utilized. When the video features are utilized as the query and the text features as the key and value, the operation is formulated as follows:
passed through two X-MHA. For simplicity, we denote only the case with
Q(FV (((')))) Fate
F wid
X-MHA(F F'O) vid:
txx) Softmax
and
V(F
tat)
(1)
where Q(-), K()), and V(.) represent linear projections, and is the number of head dimension in X-MHA. The output features from X-MHA are denoted Fo(!) and Fv(() tat- Finally, these updated features via cross-attention are refined by an FFN, resulting in the final updated features for the
corresponding layer, denoted F(l) and F(1)- After processing through all L layers, the final
features are denoted as F(L) and P(Z)
To demonstrate the effectiveness of our cross-modal baseline, we compare it with existing foreground- based methods 18, 20 on THUMOS 14, as shown in Fig. (b). The detailed description of the experimental setup is described in Sec. 4.1 In the 50%-50% setting, our baseline achieves superior performance compared to STALE 18] which adapts text prompt tuning (TPT), and ZEETAD (00) which applies both text prompt tuning and Adapter [4) These results demonstrate that our baseline achieves higher performance without relying on text prompt tuning methods. In contrast, our baseline exhibits sub-optimal performance at the 75%% setting compared to ZEETAD (2) with Adapter
[4]. As described in Sec. | our baseline model struggles with the common-action bias issue for actions with shared common sub-actions in the untrimmed videos.
3.3 Method
In this section, we introduce the overall process of Text-infused attention and Foreground- Action Detection (Ti-FAD). We first describe Text-infused Cross Attention (TiCA) that enables the text features focus on the relevant discriminative visual features. Subsequently, we describe the foreground-awar aware head that suppresses noisy background information in the untrimmed video. Fig. shows the overview of our proposed framework.
Text-infused Cross Attention (TiCA). Our TiCA mainly aims to encourage the model to focus on the discriminative sub-actions by following steps: (1) producing the class score map using video and text features to represent text-related visual parts, (2) selecting top-k points to extract discriminative points. (3) redefining the indices by merging adjacent indices based on dynamic threshold, (4) generating Gaussian kernels based on the redefined indices to produce Salient Attentive Mask (SAM), and (5) applying the generated SAM cross-attention.
First, we utilize a temporal class score map that incorporates information from both the text and video (() by computing the similarity between the video features vid and text features F Fil)
fet after MHSA.

---

Target:
Yange
kard
End
Classification
Localization
F Foreground-Aware
Text-infused Cross Attention (TiCA)
(00)
Mask
S mazk
FEN
PEN
X-MHA
TICA
Fon Fat pm
copp
MISSA
MISA
F
Text
Video
(a)
Figure 3: (a) Overview of our proposed Ti-FAD, which includes a foreground aware head that suppresses the irrelevant background frames, leading the model to concentrate more on the foreground segments, and (b) TiCA that guides the text to focus on the discriminative visual features by employing SAM.
Then, determine the class-agnostic probability sequence by obtaining the maximum value across the class dimension C::
P. max (F p.(() T vid
(2)
tat
The agnostag probability sequence Pa = (Pt f- \T the probability of action occurrence. To produce Gaussian kernels centered where the action is probably to occur, we define Itop as center location candidates corresponding to the top-k time indices in P, follows:
provides a score for each time index, indicating
Topk {i i indices of top- K in P.).
where K is dynamically determined based on the length of the sequence T, calculated ratio r of sequence length with
r.
We redefine the indices set of center location candidates Lopk as the average value between close centers if their distance is within dynamic threshold ((T), defined as ((T) Abase T/Timit where Mhuse is the base threshold value, and Timit is the initial sequence length. Thus, the redefined discriminative indices set / is calculated as:
i+i if i j1 (TT)
v(i,)) Itopk-
otherwise
This redefined discriminative indices set is utilized to produce SAM value Smaek using the kernel function (e.g., Gaussian). To generate a Gaussian kernel, we extract E {or E /} from the updated features F=(() with a single linear layer. The sigma o, values are obtained only from corresponding indices from the redefined discriminative indices set I.
To calculate the overall temporal weight across the video sequence, we sum all weights of the Gaussian kernels as: F(t) = Eiel G(t; i, o.). Subsequently, the SAM value Smask is produced by normalization process that adjusts F(t) to ensure its maximum value across the temporal dimension scaled follows:
F(t)
Smask
ma)(F(t)) E R.T
Consequently, we directly integrate the SAM value Smask into the attention score within the multi- head cross-attention X-MHA(-) as defined in Eq. D as shown in Fig. (b). This integration dynami- cally guides the text features to focus on the most relevant visual features. In each layer I,...,
5

---

the X-MHA(-) is re-deeieed as Text-infused Cross-Attentios (TiCA) and is described as follows:
@( vid
(((F())) tat)
pn(() aid (FCAA(")) F vids
tat)
Softmax
(Smask @ 1c) V(F (6)
vd
where I E CRC denotes a vector of ones with dimensions corresponding to the class dimension. The term Smask 1c Tx denotes the element-wise multiplication of SAM with matrix of ones, broadcast across the class dimension.
Goreground-Awan Head. Our foreground-awart head aims to suppress irrelevant background information and enhance the model's focus on foreground action segments. Given the updated video feature F(L) it is fed into lightweight network with three layers of 1D convolutions after each feature level, inspired by the previous method [27]. Subsequently, the sigmoid activation function is applied to ensure that the values range from 0 to 1. This process produces two types of outputs: A E R" and A E R", representing the soft and hard sreground-awareneee scores. Depending the target values, we introduce two types of foreground-awer heads: soft foreground-awand head (S-FAD) and hard foreground-awarr head (H-FAD), both designed to predict the actionness score of each snippet in the time location.
vid
For S-FAD, the target actionness score represents the normalized distance between time step and the center of the corresponding ground-truth action segment. We define the actionness target score using a Gaussian function Φ(r, M, ar) calculated as follows:
Ysoft = min (,),; ((r; 0, o)), (c and r = |c -cl-e e/:
where and r represent the distances from time step to the start and end e points of the action, respectively. The mean μ is set to O, centering the distribution around the action center, while is standard deviation that a hyperparameter to control the spread of the Gaussian distribution.
For H-FAD, we assign for the foreground and O for the background segments. This binary setup directly reflects the presence or absence of action within each time snippet, making straightforward for the model to distinguish action segments from non-action segments. The target value Yhard is defined as follows:
Yhard [(sSige)
(8)
To optimize both aspects simultaneously, we compute the foreground-awes -ware loss Lig as follows:
Lig = aLsoft(AA softs Ysoft) + 3-Barad (A Ynard);
where both loss functions Leaft and Lhard utilize binary cross-entropy loss. Here, and B are hyper-parameters control trade-off. During inference, we multiply both scores A and Afa hard the classification scores to suppress the irrelevant information of the actions.
3.4 Training and Inference
Training. For localization, we utilize the localization head similar to the foreground-aware head, but predicts the distance (dī d; from time step / the action boundaries following the anchor- free method
[27]. We optimize the distances to the action boundaries with the DIoU loss [30].
denoted as Lioe. For classification, we compute similarity scores as followed by the standard zero-
shot settings 8 18,
20
26
29
Specifically, the updated text features (L)) are fed into simple
projection & proj (.) and the classification output is computed as: Cats (Froo(((lt)) F(L) where Cets E TTx feature snippet represents the probability of action categories at time step t. We employ the focal loss [13] to recognize action categories, denoted as L. The total objective function combined with the foreground-awand loss discussed in Sec. 33.3 is defined follows:
f= E Lets + A T. LLoe + A Lig), where T+ represents the total number of positive samples. The Xe, X1, and X, represent the hyper-paramette parameters to control trade-off.
Inference. The model infers (dī, (it.d;p((ce ((c))( at each time step l. The dē and dē are the distance from time step to the start and end points, respectively. Here, p(e)) is the action confidence score. The redundant proposals are removed by SoftNMS I to obtain the final action.

---

Table 1: Performance comparison with the state-of-the-arr methods THUMOS14 and ActivityNeet v1.3. Average mAP calculated evaluate performance at different toU thresholds: [0.3:0.1:0.7] for THUMOS14 and [0.5:0.05:0.95] for ActivityNet v1.3. represents using the prompt ensemble.
Prompt Tuning
Feature
THUMOSH4
Activity v1.3
Train Split
Model
Video
Text
0.3
0.4
0.5
0.6
0.7
Avg
0.5
0.75
0.95
Avg
CLIP-B
CLIP-B
11.2
3.2
13.0
3.7
B-I 16
CLIP-D
2
CLIP-B
21
27.2
21.3
15.3
9.7
4.8
15.7
28.0
16.4
1.2
16.0
EffPrompx
CLIP-B
1400
7.2
19.3
2.9
STALE
18
B3D
H
CLIP-B
21
38.3
30.7
21.2
13.8
7.0
22.2
32.1
20.7
5.9
20.5
Unl.ac-B
CLIP-B
50%
Unl.oc-1
76
X
CLIP-L
21
CLIP-
21
43.2
50% Unseen
UaLoc-L
26
CLIP-
21
CLIP-
21
43.7
ZEETAD
20
I3D
CLIP-B
+1
45.2
38.8
30,8
22.5
13.7
30.2
39.2
25.7
3,1
24.9
Baseline (Ours)
X
B3D H
CLIP-B
21
55.9
47.5
35.9
22.7
10.5
34.5
44.1
26.1
3.1
26.5
(Ours)
CLIP-D 21
18.2
9.4
32.1
5.2
TI-FAD (Ours)
CLIP-
21
CLIP-1
21
43.7
36.3
27.6
18.2
10.0
27.2
51.0
32.6
4.8
32.1
Ti-FAD (Ours)
130
CLIP-8
21
$7.0
43.3
33,9
21.2
41.2
50.6
32.2
5.2
32.0
Ti-AD (Ours)
13D
CLIP-L
41
56.7
50.8
42.4
32.3
20.8
40.6
50.8
32.6
5,6
32.3
B.ll LS
CLIP-B 20
CLIP-B
21
28.5
20.3
17.1
10.5
6.9
16.6
32.6
18.5
5.8
19.5
B-1
CLIP-R
33.0
11.6
5.7
38.6
20.4
2.1
EitPrompx 8
13D
CLIP-
11
39.7
31.6
23.0
14.9
7.5
23.3
37.6
22.9
3.8
23.1
STALE 16
I3D
CLIP-B
21
40.5
32.3
23.5
15.3
7.6
23.8
38.2
25.2
6.0
24.9
UaLac-B
26
CLIP-B 21
CLIP
211
40.2
UaLoc-L
26
CLIP&
CLIP-L
21
47.4
25% Unscen
75% Seen
UnLoc-L
CLIP
CLIP-L
11
48.8
ZERTAD
BD 2
CLIP-R
21
61.4
$3.9
44.7
34.5
20.5
43.2
51.0
33.4
5.9
32.5
Baseline (Ours)
X
130
CLIP-B
21
59.1
50.9
39.3
25.7
12.2
37.4
47.0
27.2
3.8
28.3
Ti-FAD (Ours)
CLIP-B 21
CLIP-B
21
47.6
40.0
30.5
19.8
10.5
29.7
53.6
34.1
5,4
33.8
Ti-FAD (Ours)
CLIP&
CLIP-L
48.3
40.9
30.8
21.1
11.7
30.6
54.5
35.2
5.3
34.5
Ti-FAD (Ours)
13D
CLIP-B
11
64.0
49.7
37.7
24.1
46.8
53.8
34.8
7.0
34.7
TiFADD (Ours)
B3D
CLIP-L
64.8
58.8
49.6
38.3
24.3
47.3
53.7
34.9
6.0
34.4
Experiments
4.1 Experiment Settings
Datasets. We conduct experiments on THUMOS14 7 and ActivityNet v1.3 [5), two benchmark datasets commonly used in ZSTAD. THUMOS14 consists of 20 sports action classes, containing 200 training and 213 testing videos. ActivityNet v1.3 contains 200 daily action classes and a total of 19,994 videos. We split the training, validation, and testing in the ratio 2:1:1 according to the standard setting [8]. For the zero-shot setting, we assume two situations as in [] 50%-50% setting (training for 50% of the action categories and testing for the remaining 50%). 75%-25% setting (training for 75% of the action categories and testing for the remaining 25%). To make it statistically robust, we utilize 10 random splits and average them in the final performance.
Evaluation Metric. We utilize mean Average Precision (mAP) as an evaluation metric. The mAP is calculated by averaging the precisions at different temporal intersection over union (IloU) thresholds. The tloU thresholds are set at intervals of 0.1 from 0.3 to 0.7 (:0.7) for THUMOS14 14, and at intervals of 0.05 from 0.5 to 0.95 ((0.5:0.05:0.955) for ActivityNet v1.3.
Implementation Details. Following the existing methods g
20 [6],, we employ two-stream I3D 2 and CLIP [1 models to extract video features. The two-stream I3I) features are concatenated for video inputs. For THUMOS14, 14, video features are extracted from frames
18
segment using a sliding window with a stride of 4. For ActivityNet v1.3, features are extracted with stride of 16 and downsampled to 128 dimensions. For the text encoder, we adopt the frozen pre-trained CLIP model (21) as the text encoder, using ViT-B/16 and ViT-L/14. Our model is trained for 20 epochs on THUMOS 14 and 15 epochs on ActivityNet v1.3 using Adam with 5 epochs of linear warmup. The initial learning rate is 0.0001, updated with MultiStepLR for THUMOS 14 and cosine annealing [17] for ActivityNet v v1.3, respectively. The hyperparameters for a, B, A and Xy are set to 1, while A is set to 0.5 for all datasets. All experiments are conducted with a single NVIDIA RTX A6000 GPU.
4.2 Main Results
compare our cross-modal baseline and Ti-FAD with state-of-the-art 7STAD methods
26 by computing the average mAP different tloUs. Table shows the performance comparison
7

---

Table 2: Analysis contributions of each component on THUMOS14.
mAP@ AVG
Row
Method
SAM
S-FAD
H-FAD
50%-50%%
75%-25%
Baseline
34.5
37.4
2
35.3
40.0
3
36.1
39.9
4
38.4
43.0
5
Ti-AD
V
35.9
40.1
6
4
4
37.4
43.2
7
40.2
45.2
8
41.2
46.8
Table 3: Analysis of X-MHA operation in TiCA module on THUMOS14.
mAP@tooU (%)
Method
50%-550%
75%-225%
0.3
0.5
0.7
Avg
0.3
0.5
0.7
Avg
Baseline
55.9
35.9
10.5
34.5
59.1
39.3
12.2
37.4
Text-to-vision
57.4
42.5
20.0
40.6
62.3
47.8
23.6
45.4
Vision-to-text (Ours)
57.0
43,3
21.2
41.2
64.0
49.7
24.1
46.8
Both
56.2
42.8
21.2
40.7
63.6
48.8
24.5
46.5
THUMOS14 and ActivityNet v1.3. Our baseline achieves superior performance for the 50%-50% setting, indicating the robustness of our baseline, as it effectively integrates text information, enabling better adaptation to unseen scenarios. However, for the 75%-25% setting, the performance of our baseline is marginally inferior ZEETAD [20]. In contrast, Ti-FAD with I3D features achieves state-of-the-ar performance for both the 50%-50% and 75%-25% settings, particularly at toU thresholds of 0.5, 0.6, and 0.7 on THUMOS14. Even with CLIP video encoder, Ti-FAD still shows competitive performance compared to the previous methods using I3D features.
4.3 Further Analysis
Component Analysis of Ti-FAD. Table shows comprehensive component analysis of our Ti-FAD on THUMOS14, 14. The table evaluates the impact of different components, including SAM, S-FAD, and H-FAD on the performance of the model. The bascline performance without any additional components is reported in Row 1. Rows 2-4 evaluate the contribution of each individual component. As shown in Table 2 we first observe that employing SAM (Row 2) the cross-modal alignment process of the bascline (Row 1) enhances the performance by +0.8% for the 50%-50% setting and +2.6% for the 75% setting, which means that applying only SAM contributes better performance. In Row and 4. we recognize that S-FAD increases the baseline performance by 1.6%, and H-FAD by 3.9% for the 50%-50% setting. These results indicate that S-FAD and H-FAD encourage background suppression effect by focusing on the class-agnostic action features in the untrimmed videos. Particularly, the 75%-25% setting, incorporating a H-FAD that emphasizes the foreground to SAM enhances the performance of the baseline from 37.4% to 45.2% (Row 7). It indicates the need to separate the foreground from the background. When incorporating all the components, our model boosts the best performance at 41.2% and 46.8% in 50%-50% and in 75%-25% settings, respectively.
Analysis of X-MHA Operation in TiCA Module. We investigate the impact of the X-MHA direction the TiCA module, shown in Table We compare the performance with three approaches: using Smask as an attention bias for eext-to-vssion cross-attention rision-to-text cross-attention, and both directions of cross-attention. To apply text-to-vision cross-attention, we transpose Smask @ 1 Eq. (6). Our model shows similar performance regardless of whether Smask is used for text-to-vision or vision-to-text cross-attention. The results indicate that the most crucial part of our TiCA SAM, and the choice between text-to-vision and vision-to-tee cross-attention is less critical.

---

Table 4: Ablation studies of our Ti-FAD on THUMOS14 in the 50%-50% setting.
(a) Kernel design of SAM.
(b) Structure SAM.
mAP@tIoU (%)
mAP@@ (lo((%)
Kernel
Type
0.3
0.5
0.7
Avg
0.3
0.5
0.7
Avg
Tophat
57.0
43.0
20.5
Sum
57.1
42.1
20.1
40.5
Cauchy
57.7
43.0
20.6
41.1
Mul
56.7
42.9
20.8
40.9
Gaussian
57.0
43.3
21.2
41.2
Wsum
58.0
42.9
20.4
41.0
Wmul
57.0
43.3
21.2
41.2
(c) Sigma (o)) of Gaussian mask.
(d) Text prompt tuning (TPT).
mAP@1loU (%)
mAP@tloU (%)
Sigma (r)
Type
0.3
0.5
0.7
Avg
0.3
0.5
0.7
Avg
0.1
56.6
43.1
21.1
40.9
EMPrommpt [
56.7
42.1
20.3
40.4
56.8
42.9
20.9
40.8
TPT (50)
[00
56.6
42.9
20.2
40.5
0= 1.0
56.8
43.1
21.2
41.1
TPT (100)
10
57.1
43.3
21.0
41.1
Learnable
57.0
43.3
21.2
41.2
w/o TPT
57.0
43.3
21,2
41.2
Ablations Kernel Design of SAM. In Table 4 (a), experiment with different kernel designs for SAM to understand their impact on performance. We test the kernels including Tophat, Cauchy, and Gaussian. The results indicate that our approach is robust the design of the kernel since Ti-FAD achieves similar performance utilizing each kernel. The Gaussian kernel consistently yields the best result with average mAp of 41.2%, thereby select the Gaussian kernel for final model.
Ablations on the Structure of SAM. We investigate the impact of different application structures for combining SAM with visual features in (b). Sum and Mul indicate the structure of reflecting the mask on the visual feature fed as input to the X-MHA operation, via element-wiss sum (Row 1) and element-wise multiplication (Row 2). Wsum and Wmul represent the application of mask to attention weights within the X-MHA operation by adding (Row 3) and multiplying (Row 4). We observe that the performance of Ti-FAD is not substantially influenced by different types of masks, with the best performance achieved by multiplying the mask by the attention weights.
A blations on Sigma of Gaussian. We analyze the performance of different values, which determine the width of the Gaussian kernel. shown in Table (c), we experiment with fixed a values of 0.1, 0.5, and 1.0, as well learnable a.. Our findings indicate that allowing to be learnable parameter yields the best performance, with an average mAP of 41.2%. A learnable a allows the model to dynamically adjust to the specific temporal characteristics of each action. During our experiments, we observe that the learnable a values converge within the range of 0.1 to
Ablations on the Strategy of Text Prompt Tuning, We investigate the effect of different text prompt tuning (TPT) strategies on the performance of Ti-FAD Table 4 (d). We compare the performance of Ti-FAD with EffPrompt S TPT (50) [IO], and TPT (100) 10 methods that add learnable embeddings of size 50 and 100, respectively, after the text embeddings on CLIP text encoder. Additionally, we evaluate the model without any TPT. The results demonstrate that Ti-FAD maintains increased performance regardless of the text prompt tuning strategy. Additionally, our model achieves the highest average mAP of 41.2% without text prompt tuning. This observation highlights the ability of Ti-FAD to effectively leverage textual information without the need for extensive prompt tuning.
Per- Per-Unseen Class AP Comparison. Fig. 4 demonstrates the per-class Average Precision (AP) comparison between our baseline model and Ti-FAD a tloU threshold of 0.5 on THUMOS 14. Ti- FAD outperforms the baseline, particularly in action categories that include common sub-actions (e.g., Running). For example, in the VoleV category, Ti-FAD achieves an AP of 77. 77.23%, significantly higher than the baseline's 59.63%. Similarly, in the LongJump category. Ti-FAD's AP is 79.58% compared to the baseline's 48.12%. 12%. These improvements highlight the ability Ti-FAD focus on discriminative sub-actions and suppress irrelevant background information.

---

79.58
AP (%)
80
57.65
62.45
61.7
63.5
61.03
60
48.12
40.86
40
20
HighJump
PoleVault
LongJump
ThrouDiscus
JarolinThrow
Buseline
Ti-FAD
Figure 4: Per-unseen class AP (%) at tloU threshold 0.5 on THUMOS14.
Sab-Actie
"Pote
Swing Up
Swing Up
Ruasing
Ruuning
Swing Up
Bevoline:
'Highlump'
'LngJump'
T--AD:
PoleVault'
'PaleVauut'
(b)
Figure 5: Visualization of the detection results on THUMOS14 in the 50%-50% setting.
4.4 Qualitative Results
Fig. shows visual examples of unseen class PoleVaul! with common sub-actions (e.g., Running). In both Fig. ((a)-b)), our baseline (blue line) over-focuses on the common sub-action (e-g.. Running), leading to incorrect detection: HighJump in (a) and Long.ump in (b). In contrast, our Ti-FAD (red line) accurately detects the action based on the visual feature the discriminative sub-action (c.g., Swing Up). By capturing the discriminative sub-action associated with the text information, Ti-FAD effectively identifies the unseen class. This ability to integrate text and visual features throughout the detection process significantly improves the model's performance in distinguishing between similar actions that share common sub-actions.
Conclusion
In this paper, we propose Text-infused and Foreground-aware Action Detection (Ti-FAD), novel approach designed to address a common-action bias issue in ZSTAD by enhancing the integration of text and visual information. Ti-FAD achieves this through Text-infused Cross Attention (TiCA), which strengthens the model's 's ability to capture text-related sub-actions and foreground -warre aware head that differentiates action segments from the background. leading an accurate detection process. Our extensive experiments show that Ti-FAD effectively addresses a common-action bias issue and achieves superior performance compared to ofate-of-the-ar ZSTAD methods on THUMOS14 and ActivityNet v1.3. Despite the significant improvements, Ti-FAD's scalability to larger datasets and real-time processing capabilities remains to fully explored. Additionally, extending our approach to incorporate other modalities, such audio or depth information, can be studied to enhance performance.
Acknowledgement. This work was supported by Institute of Information & communications Technology Planning & Evaluation (ITP) grant funded by the Korca MSVT) (No. RS-2019-1119007,, Artificial Intelligence Graduate School Program (Korea University), No. 2021-0-02066 Artificial Intelligence Innovation Hub, No. RS-2024-00455882, Research Hub Project, and RS-2024-00336673, Technology for Interactive Communication of Language Impaired Individuals).
10

---

References
[1] Navanceth Bodla, Bharat Singh, Rama Chellappa, and Larry S Davis. Soft-mms-improving object detection with one line of codc. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5561-5565 2017.
[2] Joao Carreira and Andrew 7isserman. Quo vadis, action recognition? new model and the kinetics dataset. In Proceddiggs of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 6299-6308, 2017.
[3] Guo Chen, Yin-Dong Zheng, Limin Wang, and Tong Lu. Dean: improving temporal action detection via dual context aggregation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 248 288,2577220 2022.
[4] Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adapt- former: Adapting vision transformers for scalable visual recognition. In Advances in Neural Information Processing Systems, volume 35, pages 16664-16678, 2022.
[5] Bernard Ghanem Fabian Caha Heilbron, Victor Escorcia and Juan Carlos Niebles. Activitynet: A large- scale video benchmark for human activity understanding, In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 961-970, 2015.
[6] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le. Zhen Li, and Tom Duerig. Scaling up visual and vision language representation learning with noisy text supervision. In Proceedings of the International Conference on Machine Learning, pages 4904-4916, 2021.
Sung.
[7] Y.-G. Jiang, J. Liu, A. Roshan Zamir, G. Toderici, I. Laptev, M. Shah, and R. Sukthankar. Thumos challenge: Action recognition with a large number of classes. httt::/crcvv ucf. Tct//THUMS14/ 2014.
[8] Chen Ju. Tengda Han, Kunhao Zheng, Ya Zhang. and Weidi Xie. Prompting visual-language models for efficient video understanding. In Proceedings of the European Conference on Computer Vixion, pages 105-124, 2022.
[9] Ho-Joong Kim, Jung-Ho Hong, Heejo Kong, and Scong- Whan Lec. Te-tad: Towards full end-to-end temporal action detection via time-aligned coordinate expression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18837-18846, 2024.
[10] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104 104.08691 2021.
[11] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In Advances in Neural Information Processing Systems, volume 34, pages 9694 9705, 2021.
[12] Chuming Lin, Chengming Xu, Donghao Luo, Yabiao Wang, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, and Yanwei Fu. Learning salient boundary feature for anchor-free temporal action localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3320-3329, 2021.
[13] Tianwei Lin, Xiao Liu, Xin Li, Errui Ding, and Shilei Wen. Bmn: Houndary-matching network for temporal action proposal generation. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3889-3898 2019.
[14] Tianwei Lin, Xu Zhao, Haisheng Su, Chongjing Wang, and Ming Yang. Bsn: Boundary sensitive network for temporal action proposal generation. In Proceedings of the European Conference on Computer Vision, pages 3 19, 2018
[15] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2980- 2988, 2017.
[16] Shilong Liu, Zhaoyang Zeng. Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Chunyuan Li, Jianwei Yang. Hang Su. Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprini .35449990 2023.
[17] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts, arXiv preprint 1608 03983, 2016.

---

[18] Sauradip Nag, Xiatian Zhu, Yi-Zhe Song, and Tao Xiang, Zero-shot temporal action detection via vision- language prompting. In Proceedings of the European Conference Computer Vision, pages 681-697 2022.
[19] Sayak Nag. Orpaz Goldstein, and Amit K Roy- Chywchurr. Semantics guided contrastive learning of transformers for zero-shot temporal activity detection. Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision, pages 0243-6253, 2023.
[20] Thinh Phan, Khoa Vo, Duy Le, Gianfranco Doretto, Donald Adjeroh, and Ngan Lc. Zeetad: Adapting pretrained vision- language model for zero-shot end-to- to-end temporal action detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1046-70555 2024.
[21] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference Machine Learning, pages 8748 8763, 2021.
[22] Dingfeng Shi, Yujie Zhong, Qiong Cao, Lin Ma, Jia Li, and Dacheng Tao. Tridet: detection with relative boundary modeling. In Proceedings of the IEER/CVF Conference on Computer Vision and Pattern Recognition, pages 18857-18866, 2023.
Temporal action
[23] Jing Tan, Xiaotong Zhao, Xintian Shi, Bin Kang, and Limin Wang. Pointtad: Multi-label temporal action detection with learnable query points. Advances in Neural Information Processing Systems, volume 35, pages 15268-15280, 2022.
[24] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 6450-6459, 2018
[25] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.
[26] Shen Yan, Xuehan Xiong, Arsha Nagrani, Anurag Arab, Zhonghao Wang. Weina Ge, David Ross, and Cordelia Schmid. Unloe: A unified framework for video localization tasks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13623-13633, 2023.
[27] Chen-Lin Zhang, Jianxin Wu, and Yin Li. Actionformer: Localizing moments of actions with transformers. In Proceedings of the European Conference on Computer Vision, pages 492-510, 2022.
[28] Lingling Zhang, Xiaojun Chang, Jun Liu, Minnan Luo, Zhihui Li, Lina Yao, and Alex Hauptmann. Tn- zstad: Transferable network for zero-shot temporal activity detection. IEEE Transactions on Pattern 2022.
Analysis and Machine Intelligence, 45(3):3848-322
[29] Lingling Zhang, Xiaojun Chang. Jun Liu, Minnan Luo, Sen Wang, Zongyuan Ge, and Alexander Haupt- mann. Zstad: Zero-shot temporal activity detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 879-888- 2020.
[30] Zhaohui Zheng, Ping Wang, Wei Liu, Jinze Li, Rongguang Ye, and Dongwei Ren, Distance-iou loss: Faster and better learning for bounding box regression. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 12993 13000, 2020.
12

---

Text-Infused Attention and Foreground-Aware Modeling for Zero-Shot Temporal Action Detection - Appendix
Table A: Ablation of cross cross-modal baseline THUMOS14.
mAP@uoU(%)
Type
0.3
0.5
0.7
Avg
(a)
ActionFormer W/o Cross-Modal Fusion
30.3
15.1
3.5
16.1
Cross-Modal Baseline (Self-Attn)
42.2
24.0
23.8
(c)
Cross-Modal Baseline (Cross-Attn)
55.9
35,9
10.5
34,5
Cls
Loc
Cls
Loc
ActionFormer Self-Atte
Action Former
ActionFormer
Text
Video
Text
Video
Text
Video
(a)
(b)
(c)
Figure A: Illustrations of different baseline architectures: (a) ActionFormer 27 w/o cross-modal fusion, (b)
Cross-modal baseline (self-attn), and (c) (Our baseline) Cross-mon Cross-modal baseline (cross-attn)
A Additional Experiments
Ablation of Cross-Modal Baseline, To provide fairer view of our cross-modal part. conduct comparative analysis of different baseline architectures in Table A We compare ActionFormer [27] without the cross- modal fusion and our cross-modal baselines on THUMOS14 the 50%-50% setting. This table shows that ActionFormer (27) without cross-mode moosl fusion (a) exhibits inferior performance because text vidco are not properly aligned, highlighting the importance of cross-modal fusion. Furthermore, the cross-attention baseline (c) outperforms the self-attention baseline (b). To facilitate clearer understanding of our experimental models, additionally provide illustrations of (a), (b), and (c) Fig.
Table B: Analysis X-MHA operation TiCA module on ActivityNet v1.3.
mAP@ (%)
Method
50%-550%
75%-25%
0.5
0.75
0.95
Avg
0.5
0.75
0.95
Avg
Baseline
44.1
26.1
3.1
26.6
47.0
27.2
3.8
28.3
Text-to-vision
50.8
31.9
5.6
32.0
53.9
34.5
6.3
34.3
Vision-to-text (Ours)
50.6
32.2
5.2
32.0
53.8
34,8
7.0
34.7
Both
50.9
32.1
34.4
34.2
Analysis of X-MHA Operation TiCA Module. We investigate the impact of the X-MHA direction in the TiCA module on ActivityNet v1.3, 3 as shown in Table We compare the performance with three approaches under the same experimental settings used for THUMOS14, For ActivityNet v1.3, our model also shows similar performance regardless of whether Smask is used for text-to-vision vision-to-text cross-attention.
13

---

Table C: Comparison with other methods for text prompt engineering on THUMOS14.
Basline
Ti-FAD
Text Prompt
0.3
0.5
0.7
Avg
0.5
0.7
Avg
(a) video action [clussname]"
53.8
33,9
9.7
32.8
55.6
42.6
20.9
40.4
(b) Prompt Augmentation 21
54.3
34.6
10.3
33.4
56.3
42.7
20.6
40.5
(c) Prompt Ensemble 21
53.8
34.3
10.2
33.1
56.8
43.0
20.3
40.7
(d) "(classname)"
55.9
35,9
10.5
34.5
57.0
43.3
21.2
41.2
Comparison with Other Methods for Text Prompt Engineering. We compare the performance of baseline
Ti-FAD with the most utilized types text prompts following previous works [88 g (6) 20] on THUMOS14
in Table([] (a) is used [88 g 20). (b) refers to using the 28 lemplates of the prompt, used in UnLoc [26]. (c) refers to using the average embedding vector from the 28 templates at the inference, used in UnLoc (26). The
result demonstrates that Ti-FAD shows similar performance regardless of the type of text prompt. These results
show that Ti-FAD's performance does not depend on the type of text prompt.
190
KT
70.5
7723
mii
62.26
017
AP(N)
80.2
4132
4072
5588
53.00
3457
2000
23.4)
2011
53.6
5.99
BasibltPiic
BaseebbllDunk
Biffands
CleusAndlec
ClinDivigg
CroketBowling
CrecktSled
Diving
FrishesCatil
GolfSwing
Hammerr Thres
High
Javalin Thrope
Longlump
PoleVaull
Sheiper
Succer
TennieSwing
Dincne
VolleyhattSpiking
Bascline
TiFAD
Figure B: Per-unseen class AP (%) at tluU threshold 0.5 on THUMOS14.
44
55.24
51266
56.7
68.6
$124
5121
25.19
16.00
2035
446
46.7
40.3
59A1
19.44
41.00
20.5
AP
Boach specor
Painsing
Kneding
Hating ploata
Javili thoow
Patting crntact lenser
Removiing Ice car
Doleg Knrate
Playing pine
Bathing dog
Playing lec brokey
fishing
Trimming brandes hadges
Playing beggipes
Powerbsoking
Camed ride
Chantig sink
Hals
Fining
Baseline a Ti-FA
Figure C: Per-unseen class AP (%) at tloU threshold 0.5 on Activity.Ntt v1.3.
Additional Per-Unseed Unseen Class Comparison. Fig. shows the per-unseen class Average Precision (AP) at an tloU threshold of 0.5 on THUMOS14. 14. The comparison between the baseline and Ti-FAD models illustrates the effectiveness of our approach in improving detection accuracy across various unseen classes. Ti-FAD demonstrates substantial performance improvements over the baseline model, particularly in action categories that involve common sub-actions (c.g., Running). This indicates Ti-FAD's enhanced capability capture and focus on discriminative sub-actions while effectively suppressing irrelevant background information. For instance, in the Vault category, Ti-FAD achieves an AP of 77.23%, significantly higher than the baseline's 's 59.63%. Similarly, in the Longlump category, Ti-FAD's ' AP is 79.58%, compared to the baseline's 's 48. 18... In the Diving category, Ti-FAD shows an impressive AP of 79.61% 61%,, compared to the baseline's 62.78%. For Shotput, Ti-FAD achieves an AP of 42.72%. outperforming robustness and effectiveness in dealing with unseen action categories, showcasing its ability to generalize better in zero-shot scenarios. Furthermore, provide additional per-pee-eeee per-unseen class AP results for 20 classes of ActivityNet v1.3 compared with our ross-modal baseline in Fig. C We observe that sharing common sub-acu aub-cctions
baseline's 's 30.42%. These results illustrate Ti-FAD's
14

---

between multiple classes also occurs ActivityNet v1.3. For example, actions such as Painting, Cleaning sink, Washing face, Cleaning windows, Hand car wash, Cleaning shoes, Ironing clothes, Hand washing clothes, and Washing dishes share the common sub-action part Wiping motion with hands, leading to similar issue, These results demonstrate that the common sub-acte acttiction bias issue not isolated to THUMOS14.
15

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and socictal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer (es],, [No]. or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information Not Available.
Please provide short (1 (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "I "esss] is generally preferable to "[No] ", it is perfectly acceptable answer "[No] provided a proper justification is given (c.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have described the main contributions of our paper and itemized them.
Guidelines:
The answer NA means that the abstract and introduction do include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
It is fine to include aspirational goals motivation long it is clear that these goals attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: have included the limitations the conclusion Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged create separate "Limitations" section in their paper.
16

---

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model sell-specification. asympiotic approximations only holding locally). The authors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect on the scope of the claims made, e-g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicaa,,,, the authors should discuss possible limitations of their approach to address problems of privacy and fairness,
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
Theory Assumptions and Proofs
Question: For each thcoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper does not include the theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed reproduce the main experimental results of the paper the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have included the detailed hyperparameters in the experiment setting Sec. 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access the model. general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of model checkpoint, other means that appropriate to the research performed.
17

---

While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how reproduce that algorithm.
(h) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results way to reproduce the model (e.g.. with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may he tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-soure source models, it may be that access to the model is limited in some way (e.g.. registered users), but should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code. with sufficient instructions to faithfully reproduce the main experimental results, as descrihed in supplemental material?
Answer: [No]
Justification:
We will release the code after the paper is published.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines pettpe://nips.cc/pubbb
for more details.
While we encourage the release of code and data, we understand that this might not be possible,
so "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g.. for
open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (httpa:/niipa http://iiaaaccppuiliccccco cc/public/ for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for new proposed method and baselines. If only a subset experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possihle in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We have included the experiment setting Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [No]
Justification: This paper does not contain the statistical bar. However, following previous approaches 20) we reported averaging 10 split cases in terms of mAP.
Guidelines:
18

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error hars, confidence intervals, statistical significance tests, at least for the experiments that support the main claims the paper.
The factors variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error than state they have 96% CI, if the hypothesis of Normality errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Answer:
[Yes]
Justification:
We have included the experiment setting Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well as estimate the total compute.
The paper should disclose whether full research project required more compute than experiments reported the paper (e-g., preliminary or failed experiments that didn' make into the paper).
9. Code Of Ethics
Question:
Does the research conducted the paper conform, every respect, with the NeurIPS Code
Ethics
Answer:
[Yes]
Justification:
We have ensured that our research fully complies with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., if there is special considcration due laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification:
This paper does contain socictal impact.
Guidelines:
The answer NA means that there is no socictal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does address societal impact.
19

---

Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance). fairness considerations (e.g., deploy- ment lechnologies that could make decisions that unfairly impact specific groups), privacy considcrations, and security considerations.
The conference expects that many papers will he foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example. it is legitimate to point out that an improvement in the quality of generative models could be used generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated relcase models, providing defenses in addition attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g.. pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not contain such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the all datasets and previous works.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, possible, include a URL.
The name of license (e.g., . CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperewi www.de.cccc..... com/ datasetaaattaaaaaaa has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
20

---

Justification:
We will release the code after the paper is published.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/codc/model as part of their sub- missions via structured templates. This includes details about training, license, limitations,
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, other lahor should be paid at least minimum wage country data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review,
21


Reviews:
[{"review_id": "AE4Ze1RRoh", "reviewer": "Reviewer_EXtU", "summary": "The paper provides a solution for zero-shot temporal action detection(TAD). It mainly addresses distinguishing between similar actions that share common sub-actions. The method introduces a cross-attention in the model and two actionness losses. They benchmark on the two standard TAD datasets, Thumos and ActivityNET, and outperform existing methods by a significant margin.", "questions": "Why are you using the I3D features and not the CLIP vision encoder features? \n\nAlso, since you don't fine-tune the text encoder, did you try using a larger CLIP-L? The results in UNLoc with a larger CLIP-L are better than with CLIP-B. I am curious to know if that would hold for you, given that you only use CLIP for the text part (I am not asking for an experiment on this one, just in case you have run it already). \n\nOverall, I think the results on Thumos and ANET are good, but I am concerned about why you didn't include the ActionFormer baseline. I am willing to change my opinion if my points are addressed.", "limitations": "The authors briefly mentioned the limitations in the conclusion.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1) The proposed method outperforms existing methods by a significant margin\n2) The paper mentions an interesting issue of the classes in the dataset containing a common sub-action, which could confuse the model.\n3) The paper addresses a challenging and relatively underexplored task of zero-shot TAD.", "weaknesses": "1) From my understanding, none of the previous methods deployed the ActionFormer baseline. This is a robust baseline, as you mentioned in L75. You only report the results of the cross-modal baseline, claiming that introducing the cross-modal model boosts the performance over the existing models. However, I don't see the results of using the ActionFormer detector without the baseline modifications. I think that would give a more fair view of the contributions of the cross-modal part. \n\n2) Your first contribution states the novelty of the cross-modal baseline. From what I understand from Sec.3.2, you are introducing a well-established cross-attention module into ActionFormer. That is also important to consider in 1).\n\n3) How often does the baseline model need clarification due to a common action bias? For example, if you plot the confusion matrix, would you actually see that it most often predicts the wrong class containing the common action? Furthermore, do you think this issue is specific to datasets like Thumos, where the granularity of actions is not that fine? \n\n4) [L70-71] If you are using CLIP, you must provide a specific text (prompt) for this model, even if that is just a class name. Some models, such as UnLOC, just had one prompt to provide the results of the zero-shot, which augments the class name with a prefix text. This does not seem so different from what you do. \n\n5) Please clarify more explicitly what SAM stands for (I assume it should be the salient attentive mask). The first time it is used in the text is in Figure 3, and it is already an abbreviation. \n\n6) [Table 1] Does UNLoc do prompt tuning? I don't see any mention in the paper of any learnable prompts. You also report the results of UNLoc using a simple single prompt only, so I would not say that they are doing much engineering on the prompt side for these results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **W1,W2) ActionFormer detector without cross-modal modification. I think that would give a more fair view of the contributions of the cross-modal part.**\n\nWe appreciate your in-depth comment, which allows our cross-modal part to be viewed more fairly.\nWe conduct a comparative analysis of ActionFormer without the cross-modal fusion and our cross-modal baselines on THUMOS14 in the 50%-50% setting.\nTo facilitate a clearer understanding of our experimental models, we additionally provide illustrations of (a), (b), and (c) in Fig. B of the PDF in the global response.\n\n||Model|mAP@0.3|mAP@0.5|mAP@0.7|mAP@Avg|\n|-|-|-|-|-|-|\n|(a)|ActionFormer w/o cross-modal fusion|30.3|15.1|3.5|16.1|\n|(b)|Cross-modal baseline (self-attn)|42.2|24.0|5.5|23.8|\n|(c)|**Cross-modal baseline (cross-attn)**|**55.9**|**35.9**|**10.5**|**34.5**|\n\nThis table shows that ActionFormer without cross-modal fusion (a) exhibits inferior performance because text and video are not properly aligned, highlighting the importance of cross-modal fusion.\nFurthermore, the cross-attention baseline (c) outperforms the self-attention baseline (b).\nWe will add these results and illustrations to the Appendix to provide a fairer view of our cross-modal part.\n\n> **W3) How often does the baseline need clarification due to a common action bias? If you plot the confusion matrix, would you see that it most often predicts the wrong class containing the common action?**\n\nUnlike classification tasks, zero-shot TAD involves detecting and localizing actions within continuous video streams.\nThis makes it challenging to visualize misclassifications in the form of a confusion matrix.\nInstead, we utilize Average Precision (AP) following the previous work [1], which provides a comprehensive measure of performance in detecting and classifying actions.\nTo show which action class is mostly affected by common sub-action bias, we compared the per-unseen class AP on THUMOS14 in Fig. 4 (Main paper) and A (Appendix).\nBy comparing AP between our baseline and Ti-FAD, we show that our Ti-FAD successfully addresses the common sub-action bias by improving the AP for action classes such as *LongJump*, *HighJump*, *PoleVault*, *JavelinThrow*, and *BasketballDunk* that share the common sub-action part *Running*.\nThese improvements are not only reflected in the performance metrics but also align with our expectations.\n\n> **W3) Do you think this issue is specific to datasets like Thumos, where the granularity of actions is not that fine?**\n\nWe observe that sharing common sub-actions between multiple classes also occurs in other datasets, such as ActivityNet v1.3.\nFor example, actions such as *Painting*,  *Cleaning sink*, *Washing face*, *Cleaning windows*, *Hand car wash*, *Cleaning shoes*, *Ironing clothes*, *Hand washing clothes*, and *Washing dishes* that share the common sub-action part *Wiping motion with hands*, leading to similar issue.\nFurthermore, we provide additional per-unseen class AP results for 20 classes of ActivityNet v1.3 compared with our cross-modal baseline results in the PDF of the global response.\nThe AP results for all classes will be added to the Appendix.\nThese results demonstrate that the common sub-action bias issue is not isolated to THUMOS14.\n\n> **W4) You must provide a specific text prompt. Some models, such as UnLoc, just had one prompt, which augments the class name with a prefix text.**\n\nWe simply use \"{classname}\" as the text prompt without any prefix or contextual text.\nWe will add a detailed description of the text prompt in Line 107 to provide a clear view.\nFurthermore, to show that Ti-FAD's performance does not depend on the type of text prompt, we compare the performance of our baseline and Ti-FAD with the most utilized types of text prompts following previous works [2,3,4,6] on THUMOS14 in the 50%-50% setting.\n\n||Text Prompt|Baseline mAP@0.3|mAP@0.5|mAP@0.7|mAP@Avg|Ti-FAD mAP@0.3|mAP@0.5|mAP@0.7|mAP@Avg|\n|-|-|-|-|-|-|-|-|-|-|\n|(a)|\"a video of action {classname}\"|53.8|33.9|9.7|32.8|55.6|42.6|20.9|40.4|\n|(b)|Prompt Augmentation [5]|54.3|34.6|10.3|33.4|56.3|42.7|20.6|40.5|\n|(c)|Prompt Ensemble [5]|53.8|34.3|10.2|33.1|56.8|43.0|20.3|40.7|\n|(d)|**\"{classname}\"**|**55.9**|**35.9**|**10.5**|**34.5**|**57.0**|**43.3**|**21.2**|**41.2**|\n\n(a) is used in [2,3,6].\n(b) refers to using the 28 templates of the prompt, used in UnLoc [4].\n(c) refers to using the average embedding vector from the 28 templates at the inference, used in UnLoc [4].\nThe result demonstrates that Ti-FAD shows similar performance regardless of the type of text prompt.\nWe will add this experiment to the Appendix.\n\n> **W5) Clarifying the term of SAM.**\n\nWe apologize for any confusion caused by the unclear term. We will replace \"SAM\" with \"salient attentive mask (SAM)\" in Line 158, which is used for the first time.\n\n> **W6) Does UNLoc do prompt tuning?**\n\nWe apologize for any confusion in Table 1. As you mentioned, UnLoc [4] does not use prompt tuning.\nWe have revised Table 1, including UnLoc's all reported versions.\nPlease check Table 1 in the PDF of the global response.\n\n> **Q1) Why are you using the I3D features and not the CLIP features?**\n\nIn video understanding tasks [2,3,6], using snippet-level features is generally more effective than frame-level features for capturing temporal details because addressing simultaneous spatio-temporal information is essential for video understanding.\nFor a fair comparison, we add the results using the CLIP features in Table 1.\nThe revised Table 1 shows that I3D features still outperform overall.\nEven when using CLIP as a video encoder, our Ti-FAD still shows competitive performance compared to previous methods using I3D features.\nPlease check Table 1 in the PDF of the global response.\n\n> **Q2) Did you try using a CLIP-L?**\n\nFollowing the most existing methods [2,3,6], we mainly utilize CLIP-B. For a more comprehensive comparison, we have updated the results using CLIP-L in Table 1.\nPlease check Table 1 in the PDF of the global response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ixy61S1LqI", "reviewer": "Reviewer_QEJb", "summary": "This paper deals with the problem of Zero-Shot Temporal Action Detection (ZSTAD). The authors propose a simple cross-modal ZSTAD baseline with good performance. To address the issue that the cross-modal baseline over-focus on common sub-actions, the paper further proposed a Ti-FAD module to focus on text-related visual parts. The method is evaluated on two popular datasets.", "questions": "See weakness.", "limitations": "The authors state limitations in the paper.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is generally well-written and easy to follow. The proposed method is well presented.\n2. The issue that the cross-modal baseline over-focus on common sub-actions is well recognized and analyzed. The proposed Ti-FAD module seems to be effective on this issue.\n3. The performance of the proposed method is promising.", "weaknesses": "1. The two branches in Figure 2(a) and Figure 3(a) should be cross-connected before cross-attention. Now they look like they are independent.\n\n2. The TiCA module seems a bit counter-intuitive. After obtaining the Smask, which describes which visual parts are most text-related, you didn't use it as an attention mask for cross-attention from text to vision, but instead, as a bias for vision-to-text attention. What was the rationale behind this decision? It might seem more intuitive to use the latter operation, where the Smask masks out unimportant visual segments, allowing the text to focus more on the important visual fragments. Besides, I would like to see a comparison in the experiment.\n\n3. The so-called \"Foreground-Aware\" approach refers to the model's ability to predict temporal boundaries on its own, without relying on an offline proposal generator, is that right? The design in the Foreground-Aware Head is quite common in temporal action detection methods, and I don't see anything particularly unique about it. Is this a core contribution of the article?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **W1) The two branches in Fig. 2(a) and 3(a) should be cross-connected before cross-attention.**\n\nWe apologize for any potential misleading caused by Fig. 2(a) and 3(a).\nAs shown in Eq. (1) and (6), ${F'}^{(l)}\\_{vid}$ and ${F'}^{(l)}\\_{txt}$ are used as the inputs for the cross-attention, and these notations are also depicted inside of X-MHA and TiCA in Fig. 2(a) and 3(a).\nTo enhance the clarity of our method, we have added cross-connected arrows to both figures.\nThe updated figures are included in the PDF of the global response.\nWe hope that adding the cross-connected arrows can clarify our method.\n\n\n> **W2) The TiCA module seems a bit counter-intuitive. After obtaining the $S_{mask}$, which describes which visual parts are most text-related, you didn't use it as an attention mask for cross-attention from text to vision, but instead, as a bias for vision-to-text attention. What was the rationale behind this decision? It might seem more intuitive to use the latter operation, where the $S_{mask}$ masks out unimportant visual segments, allowing the text to focus more on the important visual fragments. Besides, I would like to see a comparison in the experiment.**\n\nIn this paper, our main goal is to ensure that the cross-attention part utilizes text-related discriminative visual features, regardless of the cross-attention direction.\nWe primarily focus on how to obtain the discriminative part using the salient attentive mask (SAM, $S_{mask}$) to address text-related discriminative visual features by containing information about which temporal location is more text-related.\nThereby, the direction of the cross-attention does not change the meaning of the SAM.\nFurthermore, we believe that a video-centric approach is more beneficial than a text-centric approach because zero-shot TAD involves more extensive information from video than text (that just contains the names of actions).\nThe rationale behind this decision aligns with our experimental results comparing three approaches: using $S_{mask}$ as an attention bias for text-to-vision cross-attention, vision-to-text cross-attention, and both directions of cross-attention.\nTo apply text-to-vision cross-attention, we transpose $S\\_{mask} \\otimes \\mathbf{1}\\_{C}$ in Eq. (6).\nThe performance comparison is shown in the tables below.\n\n**50%-50% setting**\n|Method|THUMOS14 mAP@0.3|mAP@0.5|mAP@0.7|mAP@Avg|ActivityNet v1.3 mAP@0.5|mAP@0.75|mAP@0.95|mAP@Avg|\n|-|-|-|-|-|-|-|-|-|\n|Baseline|55.9|35.9|10.5|34.5|44.1|26.1|3.1|26.6|\n|Text-to-vision|**57.4**|42.5|20.0|40.6|50.8|31.9|**5.6**|32.0|\n|Vision-to-text (Ours)|57.0|**43.3**|**21.2**|**41.2**|50.6|**32.2**|5.2|32.0|\n|Both|56.2|42.8|**21.2**|40.7|**50.9**|32.1|**5.6**|**32.1**|\n\n**75%-25% setting**\n|Method|THUMOS14 mAP@0.3|mAP@0.5|mAP@0.7|mAP@Avg|ActivityNet v1.3 mAP@0.5|mAP@0.75|mAP@0.95|mAP@Avg|\n|-|-|-|-|-|-|-|-|-|\n|Baseline|59.1|39.3|12.2|37.4|47.0|27.2|3.8|28.3|\n|Text-to-vision|62.3|47.8|23.6|45.4|**53.9**|34.5|6.3|34.3|\n|Vision-to-text (Ours)|**64.0**|**49.7**|24.1|**46.8**|53.8|**34.8**|**7.0**|**34.7**|\n|Both|63.6|48.8|**24.5**|46.5|**53.9**|34.4|6.1|34.2|\n\nAs shown in the table, utilizing video features as a query (vision-to-text) slightly outperforms compared to other methods.\nAs you mentioned, text-to-vision also allows the text features to focus on the related visual features and significantly improves the performance compared to our baseline.\nOur model shows similar performance regardless of whether $S_{mask}$ is used for text-to-vision or vision-to-text cross-attention. These results indicate that the most crucial part of our TiCA is the salient attentive mask (SAM, $S_{mask}$), and the choice between text-to-vision and vision-to-text cross-attention is less critical.\nWe will revise the Ti-FAD section to emphasize the focus on $S_{mask}$ more clearly.\n\n\n> **W3) The so-called \"Foreground-Aware\" approach refers to the model's ability to predict temporal boundaries on its own, without relying on an offline proposal generator, is that right?**\n\nAs you mentioned, our model does not require an offline proposal generator. However, the \"Foreground-Aware\" approach is not related to predicting temporal boundaries. As described in Lines 173-174, the \"Foreground-Aware head\" aims for our model to focus more on foreground action segments in a class-agnostic manner.\n\n\n> **W3) The design in the Foreground-Aware Head is quite common in temporal action detection methods, and I don't see anything particularly unique about it. Is this a core contribution of the article?**\n\nOur core novelties are conducting solid cross-modal baseline, as acknowledged by Reviewer EXtU, and proposing the Ti-FAD that addresses the issue of over-focus on common sub-actions.\nIn this context, we mainly focus on capturing text-related sub-action parts using the salient attentive mask (SAM, $S_{mask}$) in TiCA and reducing the influence of unrelated parts by suppressing the background in the foreground-aware head.\nTherefore, our contribution not only lies in adopting the foreground-aware head in our Ti-FAD.\nFurthermore, while some aspects of the foreground-aware head are similar to existing methods, utilizing foreground-awareness is first applied in the zero-shot temporal action detection area.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VJDwNb8yae", "reviewer": "Reviewer_mst5", "summary": "This paper proposes a Ti-FAD framework for the zero-shot temporal action detection (ZSTAD) task, where the goal is to locate and classify unknown action classes. The proposed Ti-FAD features a mutual corss=attention integration module for detection, and leverages text-related sub-action information to mitigate the action bias problem in ZSTAD. Strong performances are achieved on THUMOS14 and ActivityNet v1.3.", "questions": "1. What are the textual prompts used for text feature extraction?\n2. What are the most important designs in the baseline to make it outperform existing methods?", "limitations": "Limitations are not discussed.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The motivation is strong and persuasive. However, the common sub-action bias may mislead the classification. This problem greatly affects the performance of visual-only and cross-modal methods.\n2. The proposed text-infused cross attention and foreground-aware head are technically sound, and demonstrate strong effectiveness in the ablation study.\n3. The paper is overall well-written and easy-to-read.\n4. The final performances are strong, achieving state-of-the-art performances on THUMOS14 and ActivityNet v1.3.", "weaknesses": "This paper is overall of good quality. I do not spot major technical weaknesses.\n\n1. What exactly is the text used for the text feature extraction? The authors only mention they use a text encoder for text feature extraction (lines 105-106) but do not mention the textual prompts.\n2. Baseline performance. The authors propose a strong cross-modal baseline in Sec 3.2, whose performance already outperforms several existing methods (Fig. 2 (b)). Can the authors provide a more detailed breakdown to identify the most effective part of the baseline?\n3. The authors only use I3D visual feature for experiments in Table 1, it would be help to evaluate with CLIP features to demonstrate the generalization ability to different features.\n4. Limitations are not discussed in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **W1,Q1) What exactly is the text used for the text feature extraction? The authors only mention they use a text encoder for text feature extraction (Lines 105-106) but do not mention the textual prompts.**\n\nWe simply use \"{class name}\" as the text prompt without any prefix or contextual text.\nWe will add a detailed description of the text prompt in Line 107 to provide a clear view.\nFurthermore, to show that Ti-FAD's performance does not depend on the type of text prompt, we compare the performance of our baseline and Ti-FAD with the most utilized types of text prompts following previous works [2,3,4,6] on THUMOS14 in the 50%-50% setting.\n\n||Text Prompt|Baseline mAP@0.3|mAP@0.5|mAP@0.7|mAP@Avg|Ti-FAD mAP@0.3|mAP@0.5|mAP@0.7|mAP@Avg|\n|-|-|-|-|-|-|-|-|-|-|\n|(a)|\"a video of action {classname}\"|53.8|33.9|9.7|32.8|55.6|42.6|20.9|40.4|\n|(b)|Prompt Augmentation [5]|54.3|34.6|10.3|33.4|56.3|42.7|20.6|40.5|\n|(c)|Prompt Ensemble [5]|53.8|34.3|10.2|33.1|56.8|43.0|20.3|40.7|\n|(d)|**\"{classname}\"**|**55.9**|**35.9**|**10.5**|**34.5**|**57.0**|**43.3**|**21.2**|**41.2**|\n\n(a) is utilized in [2,3,6].\n(b) refers to using the 28 templates of the prompt, utilized in UnLoc [4].\n(c) refers to using the average embedding vector from the 28 templates at the inference, utilized in UnLoc [4].\nThe result demonstrates that our Ti-FAD shows similar performance regardless of the type of text prompt.\nWe will add this experiment to the Appendix.\n\n> **W2,Q2) Baseline performance. The authors propose a strong cross-modal baseline in Sec 3.2, whose performance already outperforms several existing methods (Fig. 2 (b)). Can the authors provide a more detailed breakdown to identify the most effective part of the baseline?**\n\nThe most effective part of our baseline is integrating text and visual information throughout the entire detection process.\nIn Lines 31-33, most existing methods [2,3,6] utilize pre-extracted visual features, which are not integrated with textual information throughout the entire detection process. \nThis separated process limits their ability to fully leverage the contextual information between both modalities. \n\nTo demonstrate the effectiveness of cross-modal fusion in our baseline, we conduct a comparative analysis of ActionFormer without the cross-modal fusion and our cross-modal baselines.\nTo facilitate a clearer understanding of our experimental models, we additionally provide illustrations of (a), (b), and (c) in Fig. B of the PDF in the global response.\n\n||Model|mAP@0.3|mAP@0.5|mAP@0.7|mAP@Avg|\n|-|-|-|-|-|-|\n|(a)|ActionFormer w/o cross-modal fusion|30.3|15.1|3.5|16.1|\n|(b)|Cross-modal baseline (self-attn)|42.2|24.0|5.5|23.8|\n|(c)|**Cross-modal baseline (cross-attn)**|**55.9**|**35.9**|**10.5**|**34.5**|  \n\nThis table shows that ActionFormer without cross-modal fusion (a) exhibits inferior performance because text and video are not properly aligned, highlighting the importance of cross-modal fusion.\nFurthermore, the cross-attention baseline (c) outperforms the self-attention baseline (b).\nWe will add these results and illustrations to the Appendix to emphasize the contributions of our cross-modal part.\n\n> **W3) The authors only use I3D visual feature for experiments in Table 1, it would be help to evaluate with CLIP features to demonstrate the generalization ability to different features.**\n\nWe add experimental results utilizing the CLIP as a visual encoder in Table 1.\nWe have revised Table 1, including an experiment utilizing a CLIP visual encoder.\nPlease refer to Table 1 in the PDF of the global response.\n\nAs shown in the revised Table 1, the performance using I3D features still outperforms the use of CLIP features because the simultaneous utilization of spatiotemporal information is crucial in the temporal action detection area. Therefore, in the original manuscript, we primarily utilize I3D as a visual encoder, following previous works [2,3,6]. However, we agree that these experiments utilizing CLIP as a visual encoder are essential to demonstrate the generalization ability of our method with different features.\n\n> **W4) Limitations are not discussed in the paper.**\n\nWe have mentioned some limitations in Sec. 5 on lines 302-304.\nHowever, we acknowledge the need for a more detailed discussion.\nWe will extend the limitation part.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear All Reviewers,

We sincerely appreciate the reviewers' thoughtful feedback on our paper.
We are grateful for the time and effort you have taken to review our manuscript.
All constructive comments allowed us to develop our paper even further.


In this global response, we address three aspects:
1. **Our Core Novelties**
2. **Key Strengths Acknowledged by Reviewers**
3. **Additional Experiments & Revised Table and Figures (PDF)**

---

### **Our Core Novelties**

1. **Novel Cross-Modal Baseline**:
   - We introduce a solid cross-modal Zero-Shot Temporal Action Detection (ZSTAD) baseline, which shows strong performance compared to the previous methods. Our cross-modal baseline enables the detecor to capture the whole information of both video and text modalities throughout the entire detection process by eliminating the step of pre-extracting foreground candidate proposals in the previous ZSTAD methods.

2. **Identifying Key Issue**:
   - Our approach identifies the common sub-action bias issue causing confusion in our cross-modal baseline. For example, in the THUMOS14 case, action classes such as *LongJump*, *HighJump*, *PoleVault*, *JavelinThrow*, and *BasketballDunk* can share the common sub-action part *Running*.  In the ActivityNet v1.3 case, action classes such as *Painting*,  *Cleaning sink*, *Washing face*, *Cleaning windows*, *Hand car wash*, *Cleaning shoes*, *Ironing clothes*, *Hand washing clothes*, and *Washing dishes* can share the common sub-action part *Wiping motion with hands*. These common sub-actions hinder the model from learning the fine-grained alignment between video and text information, especially in a ZSTAD environment where the amount of video information is overwhelmingly larger than the text information (that just contains the name of action).
   
3. **Model Innovation**:
   - The proposed Text-infused attention and Foreground-aware Action Detection (Ti-FAD) addresses the common sub-action bias issue by (1) capturing text-related sub-action parts, and (2) distinguishing action segments from the background.
Our Ti-FAD shows promising performance on THUMOS14 and ActivityNet v1.3. 

---

### **Key Strengths Acknowledged by Reviewers**

We are encouraged by the reviewers' recognition of our approach in several key areas:
1. **Outperforming Existing Methods** (Reviewers EXtU, QEJb, and mst5):
   - Our method significantly outperforms existing methods by a significant margin.

2. **Addressing the Important Issue about Common Sub-Action Confusion** (Reviewers EXtU, QEJb, and mst5):
   - Our paper identifies and addresses the profound issue of sub-actions causing confusion in our baseline.

3. **Clarity and Readability** (Reviewers QEJb and mst5):
   - Our paper is well-written and easy to follow.

4. **Tackling an Unexplored and Challenging Task** (Reviewer EXtU):
   - Our paper addresses an unexplored and challenging zero-shot approach in the temporal action detection area.

---

### **Additional Experiments & Revised Table and Figures (PDF)**

1. **Table 1**: The revised version of Table 1 in the original manuscript. This table additionally contains three aspects: (1) Additional experiments for CLIP visual encoders, (2) Additional experiments for CLIP-L text encoder, and (3) missing part of a previous work.

2. **Figure 2(a) and Figure 3(a)** : Updated model figures for improved clarity. We have added cross-connected arrows to depict the connection of cross-attention.

3. **Figure B**: Illustration of the baseline approach to clarify the experimental setup in Reviewers EXtU's comment (W1,W2) and mst5's comment (W2,Q2). This figure demonstrates the differences in baseline architectures of the ActionFormer model without cross-modal fusion and cross-modal baselines (self-attention and cross-attention). Our response to Reviewers EXtU's comment (W1,W2) and mst5's comment (W2,Q2) contains the comparative experimental results and discussion about this part.

4. **Figure C**: Per-unseen class AP for 20 classes in ActivityNet v1.3. This figure shows the improvements of our Ti-FAD compared to our cross-modal baseline, which indicates that our Ti-FAD addresses common sub-action confusion in various datasets.

---
**References**

[1] L. Zhang et al., "ZSTAD: Zero-Shot Temporal Activity Detection," *CVPR*, 2020.

[2] S. Nag et al., "Zero-Shot Temporal Action Detection via Vision-Language Prompting," *ECCV*, 2022.

[3] C. Ju et al., "Prompting Visual-Language Models for Efficient Video Understanding," *ECCV*, 2022.

[4] S. Yan et al., "UnLoc: A Unified Framework for Video Localization Tasks," *ICCV*, 2023

[5] A. Radford et al., "Learning Transferable Visual Models from Natural Language Supervision," *ICML*, 2021.

[6] T. Phan et al., "ZEETAD: Adapting Pretrained Vision-Language Model for Zero-Shot End-to-End Temporal Action Detection," *WACV*, 2024.

---

We sincerely appreciate all constructive comments and believe that the revisions and additional experiments have strengthened our paper.

We are also committed to making our code publicly available upon acceptance of the paper, to facilitate further research in this area.

We kindly request your consideration for a positive evaluation.

Once again, we appreciate all your valuable feedback.

Meta-review: This paper presents a novel approach for Zero-Shot Temporal Action Detection (ZSTAD), which introduces a cross-modal baseline that integrates both text and visual features throughout the detection process. The paper received three reviews with unanimous recommendations of accepting the paper, mainly due to its technical novelty, strong experimental justification, clear presentation, and comprehensive experimental evaluation. The concerns raised by reviewers, especially regarding the baseline comparisons and design choices, were effectively addressed in the rebuttal. Thus, the AC suggests to accept the paper. The authors are suggested to consider the review comments to further refine the paper for preparing the final version, especially in the presentation and justification of design decisions.
Decision: Accept (poster)