Annotations Are Not All You Need: A Cross-modal Knowledge Transfer Network for Unsupervised Temporal Sentence Grounding
Authors: Xiang Fang
Daizong Liu
Wanlong Fang
Pan Zhou
Yu Cheng
Keke Tang
Kai Zou
Venue: EMNLP 2023 Findings (2023)
Keywords: Cross-modal Knowledge Transfer
Unsupervised Temporal Sentence Grounding

Abstract: This paper addresses the task of temporal sentence grounding (TSG). Although many respectable works have made decent achievements in this important topic, they severely rely on massive expensive video-query paired annotations, which require a tremendous amount of human effort to collect in real-world applications. To this end, in this paper, we target a more practical but challenging TSG setting: unsupervised temporal sentence grounding, where both paired video-query and segment boundary annotations are unavailable during the network training. Considering that some other cross-modal tasks provide many easily available yet cheap labels, we tend to collect and transfer their simple cross-modal alignment knowledge into our complex scenarios: 1) We first explore the entity-aware object-guided appearance knowledge from the paired Image-Noun task, and adapt them into each independent video frame; 2) Then, we extract the event-aware action representation from the paired Video-Verb task, and further refine the action representation into more practical but complicated real-world cases by a newly proposed copy-paste approach;  3) By modulating and transferring both appearance and action knowledge into our challenging unsupervised task, our model can directly utilize this general knowledge to correlate videos and queries, and accurately retrieve the relevant segment without training. Extensive experiments on two challenging datasets (ActivityNet Captions and Charades-STA) show our effectiveness, outperforming existing unsupervised methods and even competitively beating supervised works.

Annotations Are Not All You Need: A Cross-modal Knowledge Transfer Network for Unsupervised Temporal Sentence Grounding
Xiang Fang" Daizong Liu" Wanlong Fang" Pan Zhou' Yu Cheng" Keke Tang"
Kai Zou'
'lubei Key Laboratory of Distributed System Security, Hubei Engineering Research Center on Big Data
Security, School of Cyher Science and Engineering, Huazhong University of Seience and Technology "Peking University
Henan University "The Chinese University of Hong Kong "Guangzhou University "Protagolabs Inc.
xfang95060gmill.com dzliu@stu. pww.puu.uu..o..po. edu. cn aanlonggfag@@gmail..
panzhou@hust.edu edu.
chengyuêcse wwwk..eu.......oe hk
tangbohutbhègmall. com
Iz@protagolabe
Abstract
Querye Pazson pores water into sleess
This paper addresses the task of temporal sen- tence grounding (TSG). Although many spectable works have made decent achieve- ments in this important topic, they severely rely massive expensive video-query paired anno- tations, which require a tremendous amount of human effort to collect in real-world ap- plications. To this end, in this paper, we tar- get more practical but challenging TSG set- ting: unsupervised temporal sentence ground- ing, where both paired video-query and seg- ment boundary annotations are unavailable dur- ing the network training. Considering that some other cross-modal tasks provide many easily available yet cheap labels, we tend to collect and transfer their simple cross-modal alignment knowledge into our complex scenar- ios: 1) We first explore the entity-aware object- guided appearance knowledge from the paired Image-Noun task, and adapt them into each independent video frame; Then, extract the event-aware action representation from the paired Video- Verb task, and further refine the action representation into more practical but complicated real-world cases by a newly pro- posed copy-paste approach; 3) By modulating and transferring both appearance and action knowledge into our challenging unsupervised task, model can directly utilize this gen- eral knowledge to correlate videos and queries, and accurately retrieve the relevant segment without training. Extensive experiments two challenging datasets (ActivityNet Captions and (harades-STA) show our effectiveness, out- performing existing unsupervised methods and even competitively beating supervised works.
Ghrand Truth 6st:
1158
(a) Example of temporal sentence grounding,
(h:)
Aciioo
"dilnd"
"theree"
Appior noeds
Arthon Londellee
Vidoo
"cllak"
Video 2
Lubedad Issaee
Usery dalatons
matchigaery seen, TXUE
sourch AM
Vetsleve Eoides quecy (OuD)
wostch) dia actied ratriao
sevrth acdion
querry
Query hosed soorchiig
Valneband Intrival
(b) Illustration our proposed CMKT.
Figure 1: (a) Example of the temporal sentence ground- ing (TSG) task. (b) Illustration of our proposed CMKT network, where (bl) the appearance knowledge leam- ing module, (b2) is the action knowledge learning mod- ule, and (b3) is the knowledge transfer module, where "app" denotes "appearance". Note that there ground-truth annotations in (b3). Best viewed in color.
(Fang et al., 2022, 2023a; Liu et al., 2023c) has drawn increasing attention in recent years. TSG aims to locate temporal video segment with an activity that semantically corresponds to given sentence query. As shown in Figure I(a), only short video segment semantically matches the query, while most of the video contents are query- irrelevant. Clearly, TSG tries to break through the barrier between computer vision and natural lan- guage processing techniques for more challeng- ing cross-modal grounding (Li et al., 2023b,a, 2022; Wang and Shi, 2023; Wang et al., 202 2020c,a.b).
Most previous TSG works (Liu et al., 2023a; Fang al., 2023b: Zhang et al., 2021; Fang et al., 2020, 2021b,a; Fang and Hu, 2020: Liu al., 2021b, 2020b, 2022a, 2021a, 2022d, 2021c) are under a sully-sueervised setting, where each frame is manually annotated as query-relevant or query-
1 Introduction
As a popular yet challenging natural language pro- cessing task, temporal sentence grounding (TSG)
"Equal contributions.
Corresponding author.

---

irrelevant. Despite the decent progress, these data- hungry methods severely rely on numerous frame- query annotations, which are significantly labor- intensive and consuinngn to collect. To alle- viate annotation reliance to a certain extent, some recent works (Tan et al., 2021; Chen et al., 2019; Mithun et al., 2019; Ma et al.. 2020; Lin et al., 2020; Song et al., 2020; Zhang et al., 2020d) ex- plore weakly-supervised setting only leverage the coarse gaarse-raaieed video-query annotations instead of the fine-grained frame-query annotations. Un- fortunately, this weakly supervise setting still re- quires expensive video-query annotations and its performance validates unsatisfactory.
module, we try learn the object-guided matched region- noun information from the Image-Noun task to model the appearance information. (ii) In the action knowledge module, single-action action and multi- action branches are designed to simulate complex video scenes. In the single-action branch, we ex- tract the pure action information based on a set of labeled single-action videos. In the multi-action branch, since real-world videos often contain mul- tiple actions, we introduce clip-level copy-paste strategy to construct the action-hybrid videos adapt to more complex scenes. (iii) Finally, we re- fine both the appearance and action knowledge, and aggregate and transfer them into our complicated unsupervised TSG for inference without further training.
To this end, in this paper, we try to tackle a more practical but challenging setting for the TSG task, i.e., unsupervised TSG, which excludes both coarse-grained video-query annotations and fine-grained frame-query annotations. Therefore, how guide the unsupervised model to extract the vision-language correlations becomes a crucial problem. Luckily, unlike the expensively annotated TSG task requiring long sentences and complicated videos, some cross-modal tasks (e.g., image re- trieval and video retrieval) always provide a mas- sive number of cheap and fully-annotated datasets, i.e., matched image-noun noun and paired video-verb. Hence, we pose brand-new idea: the annotation knowledge from other cheap cross- modal annotations to the complicated and challeng- ing unsupervised TSG task? Our main motivation is that we can first learn simple image-noun and video-verb correlations to extract the visual appear- ance and action knowledge, respectively. Then, by generalizing such cross-modal knowledge and transferring into unsupervised TSG, we can not only well correlate the queries with corresponding videos without large-scale video-query pairs, but also match them with image-aware frames without frame-query pairs. Therefore, the current problem becomes how to separately collect the appearance and action knowledge and adaptively transfer the knowledge into our unsupervised TSG task.
Our main contributions summarized as fol- lows:
To the best of our knowledge, we make the first attempt to transfer the cross-modal knowl- edge from other tasks into the TSG task. With- out any TSG annotation, we can directly gen- cralize the collected pre-trained appearance and action information for precise grounding, which eliminates complex training.
We carefully design the appearance and action knowledge collection modules to learn simple knowledge from other cheap tasks. To handle the complicated vidco with multiple action contexts, we introduce a copy-paste idea to synthesize various multi-action clips, which improves the generalization ability.
transfer
Comprehensive experiments on two chal- lenging datasets (ActivityNet Captions and Charades-STA) show the effectiveness of our proposed CMKT.
2 Related Works
Most existing TSG methods are under the fully- supervised setting (Tang et al., 2021: Liu et al.. 2023b; Zhang et al., 2021; Cao et al., 2020; Lei et al., 2020; Liu et al., 2020a, 2022b; Liu and Hu, 2022; Liu et al., 2022e), where all video-query pairs and precise segment boundaries are manu- ally annotated. Based the top-down approach, some methods (Anne Hendricks et al.. 2017: Chen et al., 2018: Zhang et al., 2020b) first pre-define multiple segment proposals and then align these proposals with the query for cross-modal seman- tic matching based on the similarity. Finally, the
To tackle these issues, we propose a novel Cross- Modal Knowledge Transferring (CMKT) network, which fine-tunes and transfers the extracted appear- ance and action knowledge from other cross-modal tasks to search the best-matching video activity re- lated given queries in the unsupervised TSG set- ting. As shown in Figure 1(b), our CMKT contains three modules: (i) In the appearance knowledge

---

Appeirance knowledge
Crus-rada Eaaratia
Action knowledge
Apparsicc
penar
Testrel
BodBAy
wards
feviurn
Vilue
RadBav
Corwenss
Visol
Soature
Saluro
Crwwr
"se""
Images
Neun relttd
Sualiii Tideo-in Fustrre loile Sauslly
Know Knowledge transfie
Glove
Comp-patt
Greaslla
Fadex R-CNN
Seluait
Cosine similarity
IIII Brppken
Stalarity
11-11
Singl
Video
Video stbbe
Figure 2: Overview of our CMKT framework for unsupervised TSG. We design two knowledge modules to extract both appearance and action knowledge from other simple yet cheap cross- tasks. (i) In appearance branch, we learn the region-word pair information and the appearance information given noun. (ii) The action knowledge module contains two branch: single-action branch and multi-action branch. For the single-action branch, we extract the pure action information based set of laheled action video. Specifically, based the branch, can obtain the visual represent of the verbs. As for multi-action branch, since many video often contain multiple action, we introduce the clip-level copy-paste method to construct the action-hybrid videos. The global information these videos contain the action features. (iii) Finally, introduce the appearance and action knowledge into our TSG task by knowledge searching. Best viewed in color.
be viewed as the visual semantic of
best-matched proposal is selected as the predicted segment. Recently, some works utilize the bottom- up strategy (Chen et al., 2020a; Mun et al., 2020; Zhang et al., 2020a), which directly regresses the boundary of the target segment or predicts bound- ary probabilities frame-wisely. To alleviate the re- liance to certain extent, some state-of-ther thr----rrtttt turn the superl-- setting (Tan et al., 2021; Chen al., 2019; Mithun et al., 2019; Ma al., 2020; Lin et al., 2020; Song et al., 2020; Zhang et al., 2020d), where only video-query pairs are annotated without precise segment boundaries. To further mitigate the reliance, an unsupervised work (Liu et al., 2022c) clusters the semantic informa- tion of the whole query set by complex training process for grounding. Different from the above methods, we address the challenging unsupervised TSG task from a novel perspective: collecting and transferring the general cross-modal knowledge from other simple yet cheap tasks.
and the g-th query, NV and Nº as the number of videos and queries, respectively. For each query, unsupervised TSG has no annotation information about its corresponding related video and detailed activity location. Obviously, our unsupervised TSG is more challenging than previous supervised TSG, since drop all annotated information between V and Q including their correspondence and anno- tated segment boundaries.
Overall pipeline. To tackle the challenging unsu- pervised setting, we propose a novel Cross-Modal Knowledge Transfer (CMKT) network shown in Figure 2, which first designs two task-agnostic knowledge collection modules to learn both appear- and action knowledge from other labeled but cheap cross-modal tasks, then refine and transfer their knowledge into our complicated TSG task.
For the appearance knowledge module, we first collect a set of conceptual nouns and their corre- sponding image regions from public Image-Noun datasets (e.g., Visual Genome (VG) (Krishna et al., 2017)), where noun corresponds multiple rel- evant regions. We denote these region-level im- ages as Va No and their corresponding words (object nouns) as 4 = (w]) where N and N are the number of regions and nouns, respectively. Then, we explore the region-level
3 Our Method
3.1
Overview
Problem definition. Given set of untrimmed videos V- ( and sentence queries Q Ne {Q₂] Qg1g== g=1> we denote Vn, and Q2 as the h-th video

---

appearance knowledge by the region-word pair in- formation (w), vf)) where oy is the consensus of all the regions corresponding to wn.. For noun wy. we compute of by averaging the representations of all relevant regions. With pairwise nouns and their consensus region representations, we can obtain general cross-modal appearance knowledge.
Glove (Pennington et al., 2014) to embed each noun into dense vector { {w" {wn]A" N" the j-th noun feature and i the feature dimension. Appearance-aware visual encoder. Given N region-level images, we use Faster-RCNN (Ren et al., 2015) to extract their features { where of E E is the i-th appearance feature. Appearance knowledge acquisition. For the j- th noun w" we can obtain its prototypical region representation T. by averaging all relevant region features )09,,)km k=1
j=1 where w R is
Ji=1>
In the action knowledge module, we are given a massive number of conceptual verbs and their cor- responding action-pure clips from popular Video- Verb datasets (e.g., Kinetics (Carreira and Zisser- man, 2017)), where a verb corresponds to many labeled clips. These clips and corresponding verbs are denoted as V {w])}= respectively, and N and N" are the number of clips and verbs. We tend to learn the clip-level action knowledge (un,, ")"), where 6i is the consensus of all the regions corresponding to wy. Besides obtaining the basic action feature via averaging over all relevant single-action clips, we synthesize the multi-action clips by the copy-paste approach (Dvornik al., 2018; Xu al.. 2021; Rong et al., 2021) for simulating more complicated scenes. By refining both single- and multi-action representations, we obtain the final action knowl- edge. Finally, a knowledge transfer module is de- signed to transfer appearance and action knowledge into our unsupervised TSG.
TJ AVP(U, "o.,))
{um") N and W
where by is the consensus appearance feature and AV P(-) denotes the average pooling, Kÿ is the to- tal related region number. To ensure that Tj shares the same semantics with its corresponding noun- aware features, we learn two parametric transforma- tion matrices 8" and 8" via regularization loss (Lin et al., 2017) Lapp to enforce consensus region feature vy as close as possible to corresponding noun feature wf:
Cupp B" B" P.
(2)
where ||. ||| is Frobenius norm, RdxN" and W ( RdxN"
C
j=1
3.3 Action Knowledge Collection
Similarly, we can utilize cheap Video- Verb datasets to extract cross-modal action knowledge by learn- ing the video-verb pairs. To meet the requirement of diverse actions in the TSG task, we collect the verbs and their paired trimmed videos from highly diverse video dataset Kinetics (Carreira and Zis- serman, 2017). In the TSG task, is difficult to obtain the specific feature of every single action from multi-action videos. Thus, we simulate var- ious action videos to better generalize the mixed action features. To collect different types of action knowledge, we design two branches: single- and multi-action branches. In the single-action branch, since all the clips share the same-class action infor- mation, we aim to obtain the pure average action feature of given verb. For the multi-action branch, we refine the consensus action feature by merg- ing multiple action clips into hybrid one with a pastee strategy.
3.2 Appearance Knowledge Collection
Since the cheap Image- maage-Word datasets provide suffi- cient known image-noun pairs, we can leverage them to extract the simple cross-modal appear- ance knowledge. However, our desired appear- ance knowledge has two key properties: 1) Region- aware: Rather than extract visual backgrounds, ap- pearance knowledge is required to accurately align the region-level foreground objects and correspond- ing nouns. It is more fine-grained than other meth- ods that roughly link the whole image to words. 2) One-to-one matching: Instead of aligning noun to multiple related images in one-to sae-to-mar one-to-many manner, we focus cross-modal one-to-one matching to alleviate the appearance variations of objects by aligning the noun and its corresponding consensus region. Therefore, we collect all the nouns and their related regions on the VG dataset, which con- tains diverse contents that are useful to improve knowledge generalization.
Verb-aware textual encoder. Given No verbs, we use the Glove network (Pennington et al., 2014) to embed each verb into dense vector W No 1 E Nd
Noun-aware textual encoder. Given N nouns as the label set of No region-level images, we utilize

---

ig Action-aware visual encoder. Given Nm la- beled trimmed vidcos, we first extract their clip- wise features by a pre-trained C3D network (Tran et al., 2015); and then employ a multi-head self- attention (Vaswani et al., 2017) module to cap- ture the long-range dependencies among video frames. We denote the extracted video features Vm {0," Ji=1
where Yj the label of the j-th sample that equals I for the matched action-verb pairs and 0 for the unmatched action-verb pairs; j
hee is obtained by
For the intra-action loss in each multi-action video, we take verb-related video clips as positive samples and define samples. We adopt a hinge loss for supervision:
J E RdxNN"
video clips as negative
Action knowledge acquisition. In the single- action branch, we are given a of verbs {w]) f=1 and each verb wn corresponds to multiple action-
N'
Lintra
mar(0, 3 r3 + +"""
j=1
pure videos p= where NJ is the num- ber of action videos corresponding wy. For verb wj, we directly utilize the average pooling to obtain its average action feature: (oJa) AV AVP().)......................... wm ".
where 33 is a parameter and f denotes the verb- relevance of negative clips. Thus, the final action knowledge loss
los i8:
Laction
Linter Lintro-
(5)
Since real-world videos often contains multi- ple actions, in the mulli-action branch. we intro- duce copy-paste strategy augment the multi- action videos. As shown in Figure 2, given any two trimmed action videos, we randomly copy some clips from a video, and paste them onto the other video by replacing the same number of clips. For action-pure video oy with Nyp clips (corresponding to verb w))) and another action- pure video oy (irrelevant to verb wy), we ran- domly replace jjp clips of v% with Pm clips from -. The augmented multi-action video is denoted as (o For convenience, we design a metric denote the copy-paste amount: paste rate = 3.p /N J.p E (0, 1). i.e., the percent of replaced clips in the original video "p After the copy-paste process, since the augmented video (o"))) still contains verb-related action informa- tion (related to verb wf). we obtain the correspond- ing average action feature by the average pooling: (of")" de-
Eq. (5) tries align the same-class action features and push the diff-class action features away.
3.4 Knowledge Transfer to Unsupervised TSG
In previous sections, we can obtain consensus ap- pearance knowledge and consensus action knowl- edge, which are important for comprchending the target moments in our TSG task. However, we cannot directly utilize the collected knowledge in our TSG task due to (i) the domain gap between Image-Noun/Videee Verb task and the TSG task, and (ii) more complicated scenes in the TSG task. Thus, we aim to fine-tune collected knowledge to fit the TSG domain by knowledge transfer. In the following, we will illustrate how we transfer the collected knowledge into the TSG task.
TSG query encoder. Given a set of queries Q {Q to keep the query features same as the knowledge-based one, we utilize the Glove network to embed each word into a dense vector, For Q2 with N3 words, its word-level features are E Rd We uti- lize the NLP tool spaCy (Honnibal and Montani, 2017) to parse nouns from the given query, then remove duplicate nouns. The textual features of the No 1 where füj E Rd is the j-th noun feature and No is the reserved noun number. Similarly, we parse verbs NY
(oj")" = AV P((o)))", )*). "u....("]]n ((""""""..... (o"""""))) notes the generated action feature of 1ith after the copy-paste operation. By combining single-action (uj")) and action feature is obtained by:
(x:))) )", the final consensus
denoted as Qg = 1f
7 = (1 + (D3') )' + (1 (00--)0 ,)
(3)
where is used to adaptively control the balance between the action-pure information and hybrid action information.
reserved nouns are denoted as (
To generalize the above action knowledge, we design an inter-action loss Linter and an intra- action loss Ligtre for constraint. For the inter- take
action loss, in different augmented vidcos, the matched action-verb pairs as positive samples and the unmatched action-verb pairs negative samples, and use the weighted binary cross-entropy N" (
and obtain the reserved verb feature set { where f E Idd is the j-th reserved verb feature and N is the reserved verb number.
loss to supervise the verb-relevance (1.) 1==1:
TSG video encoder. Given NV videos V
{Vi [V we denote V,, as the h-th video and Tj as its frame number. For the appearance feature,
Linter Σ )= ( yj log(r)"") - (1 y;) log(1 - ))))

---

we utilize Fasttr-RCNN (Ren et al., 2015) built on ResNet50 (He et al., 2016) backbone detect K objects on each video to obtain a set of appearance features V =
Yo E ER and Y E R to evaluate if each noun/verb and its reconstructed one are the same not as:
a k For the action feature, we first divide video V/ into Th/8 clips. Then, we utilize a pre-trained C3D network (Tran et al., 2015) and multi-head self-attention to ex- tract the clip-aware features VA [fi,)==1 Th/8
y ({²]) (W"" (W")))" A))" S =====((Q)) ((Q))) (W, M))"
(8)
With Y and ymm we optimize W and W by 9 minimizing the cross-entropy loss:
Fitting the TSG domain via knowledge trans- fer. To transfer the pre-trained knowledge into the TSG task, we fine-tune the collected appear- ance and action knowledge based on the prin- ciple of cross-modal cycle consistency. Given No the nouns features ( and verbs features
NQ Ldce
N log( (gg))
yooz)
gm1
T n log( Y 9.21)
{ (fزر)يي we search each fgj in the appearance knowledge and each fg.j in the action knowledge with the cosine similarity, then re-represent them with the collected consensus appearance features Py and action features Tn respectively. For con- Ng < B and M, = [ No E RdxNN, As for the visual appearance and action features, we also enhance them via the collected knowledge, respectively, To fine-tune consensus features (Ag and Mg). we design a cross-modal cycle-consistent loss to make collected knowledge more applicable to the
yd.2 and you are one-hot ground-truth vectors, where the z-th and x-th value are one, and rest values are zeros. By La we use W and W to transfer all consensus appearance and action knowl- edge into TSG.
venience, we denote Ag (
3.5 Knowledge-basee Grounding
By pre-training the appearance knowledge by Leyp and the action knowledge by Laction with the knowledge transfer by Lydes we can directly uti- lize the suitable transferred appearance knowledge (w) D5) and action knowledge (w"). v to local- ize the target segment boundary without any further training. Specifically, during the inference, given some videos (V, h= N and a query set {Q g=1> we first extract their appearance feature fh ac- tion feature fm noun feature fg verb feature f respectively. Then, based on the collected appearance knowledge and action knowledge mod- ules, we can obtain the corresponding consensus appearance and action features. We compute the cosine similarity matrix between consensus appear- ance features and video appearance features, and also calculate the cosine similarity matrix between consensus action features and video action features. After that, following (Liu et al., 2022c), we select the best-matched clip with the highest scores (sum of appearance similarity and action similarity) in the target video as the target segment center. We add the left/right frames into the segment if the ra- tio of their scores to the frame score of the closest segment boundary less than a threshold 7. We repeat this step until no frame can be added. In this way, we can generate the final segment boundary.
TSG task. For appearance knowledge, we first com-
pute the cosine similarity matrix S E XVV
between noun-guided consensus appearance fea- tures and video appearance features. Then, we combine all appearance features by treating sim- ilarities as weights to obtain reconstructed nouns features:
S = (W." WW"""
(6)
where C (VaINY W is learnable transfor- /==
g mation matrix, a-(-) is the softmax operation along the column dimension, and Q" is the reconstructed noun representation. Similarly, we combine all action features obtain the reconstructed verbs features:
@ "" S: (W; M,)" W.C"
(7)
where C (V NY h=1 W is learnable trans-
formation matrix, Q" is the reconstructed verb fea- tures, and 5% C Ny is the similarity matrix between transformed verb and action features. Our cycle- consistonitt loss aims to (i) align the semantics of original nouns and their reconstructed ones, (ii) pull verbs and their reconstructed ones together. Thus, we construct two indicator matrices

---

Table 1: Performance comparison with state-of-the-art methods. 'Type' refers to supervision level, FS: fully- supervised setting, WS: weakly-supervissd setting, US: unsupervised setting.
ActivityNet Catiiinnn
Charades- STA
Method
Type
IoU=0.1
IoU=0.3
IoU=0.5
1oU=0.7
mloU 1oU=0.3
10U=0.5
IoU=0.7 7 mloU
CTRL (Gao (Gao et al., 2017)
FS
49.10
28.70
14.00
20.54
21.42
7.15
2D-TAN (Zhang et al., 2020h)
FS
58.75
44.05
27.38
42.80
23.25
LGI (Mun et al., 2020)
FS
58.52
41.51
23.07
41.13
72.96
59.46
35.48
51.38
VSLNet (Zhang et al., 2020a)
FS
63.16
43.22
26.16
43.19
70.46
54.19
35.22
50.02
VCA (Wang al., 202
WS
67.96
50.45
31.00
33.15
58.58
38.13
19.57
38.49
RTBPN (Zhang et al., 2020c)
WS
73.73
49.77
29.63
60.04
32.36
13.24
CTF (Chen et al., 2020b)
WS
74.20
44.30
23.60
32.20
39.80
27.30
12.90
27.30
MARN (Song al., 2020)
WS
47.01
29.95
48.55
31.94
14.81
SCN (Lin al., 2020)
WS
74.48
47.23
29.22
42,96
23.58
9.97
BAR (Wu al., 2020)
WS
49.03
30.73
44.97
27.04
12.23
CCL (Zhang al., 2020d)
WS
50.12
31.07
33.21
15,68
LCNet (Yang al., 2021)
WS
78.58
48.49
26.33
34.29
59.60
39.19
18.87
38.94
CNM (Zheng al., 2022)
WS
78.13
55.68
33.33
60.39
35.43
15.45
WSTAN (Wang al., 2021b)
WS
79.78
52.45
30.01
43.39
29.35
12.28
DSCNet al., 2022c)
US
47.29
28.16
44.15
28.73
14.67
CMKT
US
73.35
50.69
31.28
16.42
38.79
47.80
30.96
18.87
30.42
4 Experiment
features. For the action knowledge module, we use the pre-trained C3D (Tran et al., 2015) network to extract the single-action and multi-action infor- mation on the Kineties dataset. Also, we use all the verbs on the Kinetics dataset. To make fair comparison with previous works (Gao et al., 2017; Zhang et al., 2020b), we also utilize the pre-trained C3D model to extract video features and employ the Glove model to obtain word embeddings. As some videos are too long, we set the length of video feature sequences to 128 for Charades-STi and 256 for ActivityNet Captions, respectively. We fix the query length 10 in Charades-STA, and 20 in ActivityNet Captions. Threshold 7 is set to 0.9 on Charades-STA and 0.8 on ActivityNet Captions. All the experiments are implemented by PyTorch.
4.1 Datasets and Evaluation Metrics
ActivityNet Captions. From ActivityNet v1.3 (Heilbron et al., 2015; Lan et al., 2022), Activi- tyNet Captions contains 20,000 YouTube videos and 100,000 language queries. On average, vidco is 2 minutes and query has about 13.5 words. Fol- lowing the public split (Gao et al., 2017), we utilize 17,031 video-query pairs for testing.
Charades-STA. Built upon the Charades dataset (Sigurdsson et al., 2016), Charades-S 16,128 video-sentence pairs. The average video length is 0.5 minutes. The language annotations generated by sentence decomposition and keyword matching with manual checks. Following (Gao et al., 2017), we remove 12,408 training pairs and utilize the others for testing.
contains
4.3 Comparison with State-of-the-Arts
Evaluation metrics. Following previous works (Mun et al., 2020; Zhang al., 2020a), we use "IoU=m" for evaluation, which denotes the percent- age of queries having at least one result whose Inter- section over Union (IoU) with ground truth is larger than m. In our experiments, m E {0.3, 0.5, 0.7) for Charades-ST: STA and m E {0.11, 0.3, 0.5, 0.7) for ActivityNet Captions. Also, we utilize mean loU (mloU) as the averaged temporal ToU between the predicted boundary and the ground-truth one.
We conduct performance comparison with three cat- egories of state-of-the-aa TSG methods: (i) Fully- supervised (FS) setting (Gao et al., 2017; Zhang et al., 2020b; Mun et al., 2020; Zhang et al., 2020a); (ii) Weakly-superviee (WS) setting (Chen et al.. 2020b; Lin et al., 2020; Wang al., 2021b; Wu et al., 2020; Song al., 2020; Zhang et al., 2020d; Huang al., 2021; Wang et al., 2021c; Yang et al., 2021: Zhang et al., 2020c; Zheng et al., 2022); (iii) Unsupervised (US) setting (Liu et al., 2022c). Table reports the performance comparison. Without any supervision, CMKT still obtains the competitive performance to WS and US meth- ods on the ActivityNet Captions dataset. Com- pared with the US method DSCNet, our CMKT improves the performance by 3.40% and 3.1
4.2 Implementation details
For the appearance knowledge module, we utilize all the words on the Visual Genome dataset, so the noun number of region-level semantics is 27,801. The feature dimension d is set to 512 for all the

---

Table Main ablation study, where we remove each key component to investigate its effectiveness. "AK" denotes "appearance knowledge". "MK" denotes "action knowledge", "KT" denotes "knowledge transfer".
ActivityNer Captions
Charades STA
Model
AK
MK
KT
1oU=0.3
103=0.3
IoU=0.7
mloU
IoU=0.3
ToU=0.3
IoU=0.7
mloU
CMKT(a)
X
45.81
24.79
11.52
31.36
40.81
24.72
10.88
25.24
CMKT())
X
V
47.50
26.82
12.37
32.71
42.43
26.87
11.74
26,90
CMKT())
V
X
48.02
27.91
13.96
34.05
44.35
27.19
12.62
27.86
CMKT(FulI)
50.69
31.28
16.42
38.79
47.80
30.96
18.87
30.42
Table 3: Necessity of average pooling ActivityNet Captions, where "MIP" means min-pooling, "MAP" means nax-pooling. and "AVP" means average pooling.
Table 4: Effect of action knowledge ActivityNet Captions.
Single- action
Multi- action
IoU=0.3
IoU=0.5
IoU=0.7
mIoU
Module
Changes
loU=0.3
loU=0.5
loU=0.7
mloU
MIP
47.80
29.08
13.76
35.70
X
48.21
27.70
15.55
36.03
Appearance
knowledge
MAP
48.03
30.72
14.45
36.68
X
49.36
29.82
15.73
37.54
AVP
50.69
31.28
16.42
38.79
50.69
31.28
16.42
38.79
MAP
48.36
28.72
15.45
35.53
Action
MIP
49.27
29.26
16.20
36.61
knowledge
AVP
50.69
31.28
16.42
38.79
Table 5: Performance of different knowledge transfer on ActivityNet Captions.
terms of "IoU=0.3" and "IoU=0.5", demonstrating the effectiveness of our proposed CMKT. On the Charades-STA dataset, compared with supervised methods (FS and WS), our CMKT the US set- ting reaches competitive performance. Although our CMKT eliminates the training process, it out- performs US method DSCNet by 3.65%, 2.23% and 4.20% in terms of "loU=0.3", "IoU=0.5" and "IoU==077", which shows our effectiveness.
Appearance
Action
0.3
1o0=0.5
1oU=0.7
mloU
X
X
45.81
24.79
11.52
31.36
X
V
46.07
28.30
13.15
35.08
X
47.15
29.98
12.74
34.86
50.69
31.28
16.42
38.79
Table 6: Effect of copy-paste strategy ActivityNet Captions.
Changes
IoU=0.3
IoU=0.5
loU=0.7
mloU
w/o copy-paste
45.98
26.94
11.02
34.75
video connection
49.22
30.76
14.54
37.82
4.4 Ablation study
w/ copy-paste
50.69
31.28
16.42
38.79
Main ablation studies. To examine the effective- ness of each module, we conduct the main ablation studies in Table CMKT(a) the baseline model, which only utilizes pre-trained appearance and action features for inference without fine-tuning knowledge. CMKT(b) removes appearance knowl- edge module, and directly use the action knowl- edge to locate the target segment. In CMKT(c), we only use the appearance information to obtain the segment boundary by removing the action in- formation. CMKT(Full) our full CMKT model. CMKT(Full) performs better than all the ablation models, demonstrating the effectiveness of appear- ance and action knowledge for understanding the long yet complex sentence in TSG.
(MAP) on both knowledge collection modules, showing that AVP is suitable for integrating the in- formation among single modality for representing the same semantic as the other modality.
Influence on different types of action knowledge. To investigate the contribution of single- and multi- action videos to our final consensus action features, conduct the corresponding ablations in Table 4 evaluate the significance of single-action and multi-action knowledge. Obviously, both single- action and multi-action features contribute to our action knowledge module.
Investigation on different types of knowledge transfer. To evaluate the importance of appearance and action knowledge, experiments. As shown in Table 5, each knowl- edge contributes lot to transferring the knowledge into our task. By jointing appearance and action knowledge, the performance can further boost. Effect of the copy-paste strategy. To analyze the
Investigation the consensus features. To as- sess the effectiveness of average pooling (AVP) dur- ing the generation process of consensus appearance features, we compare different pooling strategies. As shown in Table 3, our AVP significantly outper- forms both min- toollpoling (MIP) and max-pooling
conduct corresponding

---

40
40
(M (MK
(M (CMKT()
38
CM (F
38
CMKT(C OCM
36
36
mlou
mloU
34
34
32
A
A
A
32
30
30
28
283
2
9
10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Object Number (K)
Paste Rate (1)
40
80
70
38
CMKT(C) OCMKT(FulI
Performance
603
mlou
363
+
(M)T(a) (CMKT()
50
(M (FMlT(Full)
40
34
30
32
20
10
30
0.5
0.6
0.7
0.8
0.9
1.0
1.1
1.2
1.3
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Parameter (B)
IOU
Figure 3: Effect parameters on ActivityNet Captions.
performance of our copy-paste strategy, an abla- tion experiment conducted Table 6, where "w/ video connection" means that we direct connect two videos for data synthesis. Thus, the copy-paste strategy is more effective to synthesize the multi- action videos for better grounding performance. Analysis on the hyper-parameters. As shown in Figure 3, we investigate the impact of four hyper- parameters: object number (K); paste rate (μ); positive parameter (B); evaluation metries (loU). (i) With larger K, our proposed CMKT performs better. To balance the performance and the compu- tation cost object detection, we choose (ii) All the variants obtain the best performance when the paste rate μ is set to 0.8. (iii) Our CMKT obtains the best performance when 3 = 0.9. (iv) CMKT(Full) obtains the best performance on all IoUs, demonstrating the effectiveness of each mod- ule. Thus, set K 8, a 0.8 and B 0.9 in all the experiments.
Quars: The bry fillsolf and bock bell bnfore fill Pepeeeeee Poreeeeeeie CE
Goraadt Trish
5,07%
1361s
340b
11.55
WSTAN
9.723
17.48
1334%
Qurry: pears glass.
TDNAN
3877
1302
Our CNKY
1195ks
Figure 4: Qualitative results sampled from ActivityNet Captions (top) and Charades-STT (bottom).
Conclusion
In this paper, we address the challenging yet practi- cal unsupervised TSG task from brand-new per- spective of knowledge transfer. Without further grounding training, we can directly utilize the gen- eral knowledge from other cross-modal tasks to guide to match the video and query for retrieving the target segment. To our best knowledge, make the first attempt to transfer the knowledge from other multi-modal topics our TSG task. Experimental results validate the effectiveness of CMKT, outperforming the existing unsupervised method by a large margin.
4.5 Qualitative Results
As shown in Figure we visualize the localiza- tion results. Our CMKT can predict more precise segment boundaries than fully-supervised model 2D-TAN and weakly supervisd method WSTAN.

---

6 Acknowledgements
for Computational Linguistics. Association for Com- putational Linguistics.
This work is supported by National Natural Sci- ence Foundation of China (NSFC) under Grant No. 61972448.
Zhenfang Chen, Lin Ma, Wenhan Luo, Peng Tang, and Kwan-Yee K Wong. 2020b. Look closer to ground better: Weakly-supervised temporal grounding of sentence in video. arXiv preprint arXiv:2001.09300
7 Limitations
Nikita Dvornik, Julien Mairal, and Cordelia Schmid. 2018. Modeling visual context is key to augmenting object detection datasets. In Proceedings of the Euro- pean Conference on Computer Vision (ECCV), pages 364-380.
This work analyzes an interesting yet challenging problem of how to transfer the annotation knowl- edge from other cheap cross-modal annotations to the complicated and challenging unsupervised TSG task. Although we can transfer the knowl- edge from other cheap multi-modal datasets, the contexts of these datasets are not always helpful for our grounding task. However, our proposed CMKT still provides a brand-new and interesting idea, cross-modal knowledge transfer, for promot- ing the development of these areas. Therefore, a more contextual and generalizable way to transfer cross-modal knowledge between different topics is a promising future direction.
Xiang Fang and Yuchong Hu. 2020. Double self- weighted multi-view clustering via adaptive view fu- sion. arXiv preprint 1.Ni:2011.113900
Xiang Fang. Yuchong Hu, Pan Zhou, and Dapeng Wu. 2021a. Anime: A soft approach for autoweighted noisy and incomplete multiview clustering. IEEE Transactions on Artificial Intelligence, 3(2):122-206.
Xiang Fang, Yuchong Hu, Pan Zhou, and Dapeng Oliver Wu. 2020. Vhh: View variation and view heredity for incomplete multiview clustering. IEEE Transactions on Artificial Intelligence, 13))2233-277.
References
Xiang Fang, Yuchong Hu, Pan Zhou, and Dapeng Oliver Wu. 2021b. Unbalanced incomplete multi-view clus- tering via the scheme of view evolution: Weak views meat; strong views do eat. IEEE Transactions on Emerging Topics Computational Intelligence, 6(4):913-927.
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivie, Trevor Darrell, and Bryan Russell. 2017. Localizing moments in video with natural language. In Proceedings of the IEEE international conference on computer vision, pages 5803-5812.
Da Cao, Yawen Zeng, Meng Liu, Xiangnan He, Meng Wang, and Zheng Qin. 2020. Strong: Spatio- temporal reinforcement learning for cross-modal video moment localization. Proceedings of the 28th ACM International Conference on Mullimedia, pages 4162 4170.
Xiang Fang. Daizong Liu, Pan Zhou, and Yuchong Hu. 2022. Multi-moda cross-domain alignment network for video moment retrieval. IEEE Transactions on Multimedia.
Xiang Fang. Daizong Liu, Pan Zhou, and Guoshun Nan. 2023a. You ground earlier than see: An effective and efficient pipeline for temporal sentence ground- ing in compressed videos. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pat- tern Recognition, pages 2448-2460.
Joao Carreira and Andrew Zisserman. 2017. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Patlern Recognition, pages 6299-6308
Xiang Fang, Daizong Liu, Pan Zhou, Zichuan Xu, and Ruixuan Li. 2023b. Hicrarchical bocalgglobal trans- former for temporal sentence grounding. IEEE Trans- actions on Multimedia.
Jingyuan Chen, Xinpeng Chen, Lin Ma, Zequn Jie, and Tat- Tan-Seng Chua. 2018. Temporally grounding natural sentence in video. Proceedings of the 2018 con- ference on empirical methods in natural language processing, pages 162-171.
Jiyang Gao, Chen Sun, Zhenheng Yang. and Ram Neva- tia. 2017. Tall: Temporal activity localization via language query. In Proceedings of the IEEE interna- tional conference on compuler vision, pages 5267- 5275.
Long Chen, Chujie Lu, Siliang Tang, Jun Xiao, Dong Zhang, Chilie Tan, and Xiaolin Li. 2020a. Rethink- ing the bottom-up framework for query-based video localization. In I Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10551- 10558.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778.
Z Chen, L Ma, W Luo, and KKY Wong. 2019. Weakly- supervised ppatio-temporall grounding natural sen- tence in video. In Annual Meeting of the Association

---

Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. 2015. Activitynet: A large-scale video benchmark for human activity understanding. In 2015 IEEE conference on com- puler vision and pattern recognition (CVPR), pages 961-970. IEEE
Zhouhan Lin, Minwei Feng, Cicero Nogueira dos San- tos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. International Conference on Learning Representations((CLLR))
Daizong Liu, Xiang Fang, Wei Hu, and Pan Zhou. 2023a. Exploring optical-flow seoecall--ow--ee flw-guidedd motion and detection- based appearance for temporal sentence grounding. IEEE Transactions on Multimedia.
M Honnibal and I Montani. 2017. Natural language un- derstanding with bloom embeddings, convolutional neural networks and incremental parsing. Unpub- lished software application. https://spacy. io.
Daizong Liu, Xiang Fang, Wei Hu, and Pan Zhou. 2023b. Exploring optical-flow -io----uwwwddd motion and detection-based appearance for temporal sentence grounding. IEEE Transactions on Multimedia.
Jiabo Huang, Yang Liu, Shaogang Gong, and Hailin Jin. 2021. Cross-sentence temporal and semantic relations in video activity localisation. In Proceed- ings of the IEEE/CVE International Conference on Computer Vision, pages 7199-7208.
Daizong Liu, Xiang Fang, Pan Zhou, Xing Di, Weining Lu, and Yu Cheng. 2023c. Hypotheses tree build- ing for one-shot temporal sentence localization. In Proceedings of the AAAI Conference on Artificial Intelligence.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin John- son, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vi- sion using crowdsourced dense image annotations. Internationalauan of computer vision, 123(1):32 73.
Daizong Liu and Wei Hu. 2022. Skimming, locating. then perusing: A human-like framework for natural language video localization. In Proceedings of the 30th ACM International Conference on Multimedia. pages 4536-4545.
Xiaohan Lan, Yitian Yuan, Xin Wang, Long Chen, Zhi Wang. Lin Ma, and Wenwu Zhu. 2022. A closer look at debiased temporal sentence grounding in videos: Dataset, metric, and approach. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM).
Daizong Liu, Xiaoye Qu, Xing Di, Yu Cheng, Zichuan Xu, and Pan Zhou. 2022a. Memory-guided semantic learning network for temporal sentence grounding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 1665-1673.
Jie Lei, Licheng Yu, Tamara L Berg, and Mohit Bansal. 2020. Tvr: large-scale dataset for video-subtille moment retrieval. In European Conference on Com- puter Vision, pages 447 463. Springer.
Daizong Liu, Xiaoye Qu Jianfeng Dong, and Pan Zhou. 2020a. Reasoning step-by-step: Temporal sentence localization in videos via deep rectification- modulation network. In Proceedings of the 28th International Conference on Computational Linguis- tics, pages 1841-1851.
Qian Li, Shu Guo, Cheng Ji, Xutan Peng, Shiyao Cui, Jianxin Li, and Lihong Wang. 2023a. Dual-gated fusion with prefix-tuning for multi-modal relation extraction. In Findings of the Association for Com- putational Linguistics: ACI. 2023, Torotoo, Canada, July 9-14, 2023, pages 8982-8994. Association for Computational Linguistics.
Daizong Liu, Xiaoye Qu Jianfeng Dong, and Pan Zhou. 2021a. Adaptive proposal generation network for temporal sentence localization in videos. arXiv preprint arXiv:2109.06:5
Qian Li. Shu Guo, Yangyifei Luo, Cheng Ji. Li- hong Wang, Jiawei Sheng, and Jianxin Li. 2023b. Attribute-oonssste knowledge graph representation learning for multi-modal entity alignment. In Pro- ceedings of the ACM Web Conference 2023, WWW 2023, Austin, TX, USA, 30 April 2023 4 May 2023, pages 2499-2290000 2508. ACM.
Daizong Liu, Xiaoye Qu, Jianfeng Dong, Pan Zhou, Yu Cheng, Wei Wei, Zichuan Xu, and Yulai Xie. 2021b. Context-aware biaffine localizing network for temporal sentence grounding. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 1 11235-11 11244.
Qian Li, Hao Peng, Jianxin Li, Jia Wu, Yuanxing Ning. Lihong Wang, Philip S. Yu, and Zheng Wang. 2022. Reinforcement learning-based dialogue guided event extraction to exploit argument relations. IEEE ACM Trans. Audio Speech Lang. Process., 30:520-55:5 533.
Daizong Liu, Xiaoye Qu, and Wei Hu. 2022b. Reducing the vision and language bias for temporal sentence grounding. In Proceedings of the 30th ACM Interna- tional Conference on Mullimedia, pages 4092 4101.
Daizong Liu, Xiaoye Qu Xiao Yang Liu, Jianfeng Dong, Pan Zhou, and Zichuan Xu. 2020b. Jointly cross-and self-modal graph attention network for query-based moment localization. In Proceedings of the 28th ACM International Conference on Multime- dia, pages 4070-4078.
Zhijie Lin, Zhou Zhao, Zhu Zhang, Qi Wang, and Huasheng Liu. 2020. -eakly-superrvied video mo- ment retrieval via semantic completion network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11539-11546.

---

Daizong Liu, Xiaoye Qu, Yinzhen Wang, Xing Di, Kai Zou, Yu Cheng. Zichuan Xu, and Pan Zhou. 2022c. Unsupervised temporal video grounding with deep semantic clustering. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 36, pages 1683-1691.
Yijun Song, Jingwen Wang. Lin Ma. Zhou Yu, and Jun Yu. 2020. srakly--upervised multi-level attentional reconstruction network for grounding textual queries in videos. arXiv preprint 2tt:::0003000-45
Reuben Tan, Huijuan Xu, Kate Saenko, and Bryan A Plummer. 2021. Logan: Latent graph co-attention network for weakly- seeely--upervised video moment re- trieval. In Proceedings of the IEEE/CVF Winter Con- ference on Applications of Computer Vision, pages 2083-2092.
Daizong Liu, Xiaoye Qu, and Pan Zhou. 2021c. Pro- gressively guide to attend: An iterative alignment framework for temporal sentence grounding. arXiv preprint arXiv:2 109.06400.
Daizong Liu, Xiaoye Qu, Pan Zhou, and Yang Liu. 2022d. Exploring motion and appearance informa- tion for temporal sentence grounding, In Proceedings of the AAAI Conference Artificial Intelligence, vol- ume 36, pages 1674-1682.
Haoyu Tang, Jihua Zhu, Meng Liu, Zan Gao, and Zhiy- ong Cheng. 2021. Frame-wise cross-modal matching for video moment retrieval. IEEE Transactions on Multimedia, 14:1338-1343
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torre- sani, and Manohar Paluri. 2015. Learning spatiotem- poral features with 3d convolutional networks. In Proceedings of the IEÉE international conference computer vision, pages 1489-4497.
Daizong Liu, Pan Zhou, Zichuan Xu, Haozhao Wang. and Ruixuan Li. 2022e. Few-shot temporal sen- tence grounding via memory-guided semantic learn- ing. IEEE Transactions Circuits and Systems for Video Technology.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances neural information processing systems, 30.
Minuk Ma, Sunjae Yoon, Junyeong Kim, Youngjoon Lee, Sunghun Kang, and Chang D Yoo. 2020. Vlanet: Video-language alignment network for superie- video moment retrieval. In Euro- pean conference on computer vision, pages 156-171. Springer.
Yimu Wang, Shiyin Lu, and Lijun Zhang. 2020a. Searching privately by imperceptible lying: novel private hashing method with differential privacy. In Proceedings of the 28th ACM International Confer- ence on Multimedia, page 2700-2709.
Niluthpol Chowdhury Mithun, Sujoy Paul, and Amit Soy-Chhwdhhury 2019. Weakly supervised video moment retrieval from text queries. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages I 11592-11601.
Yimu Wang and Peng Shi. 2023. Video-text retrieval by supervised multi-space multi-grained aligmment.
Jonghwan Mun, Minsu Cho, and Bohyung Han. 2020. Local-global video-text interactions for temporal grounding. In Proceedings of the IEEF/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 10810-10819
Yimu Wang. Ren-Jie Song, Xiu-Shen Wei, and Lijun Zhang. 2020b. An adversarial domain adaptation network for cross-domain fine- -na-ggr recognition. In 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1217-1225.
Jeffrey Pennington, Richard Socher, and Christopher D Manning, 2014, Glove: Global vectors for word rep- resentation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Yimu Wang, Xiu-Shen Wei, Bo Xue, and Lijun Zhang. 2020c. Piecewise hashing: A deep hashing method for large-scale fine tin-igrieed search. In Puttern Recog- nition and Computer Vision -Third Chinese Confer- ence, PRCV 2020, Nanjing, China, October 16-18, 2020, Proceedings, Part 11, pages 432-444.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28.
Yimu Wang, Bo Xue, Quan Cheng, Yuhui Chen, and Lijun Zhang. 2021a. Deep unified mroastmy hashing by pairwise data alignment. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 1139-1135.
Yu Rong, Takaaki Shiratori, and Hanbyul Joo. 2021. Frankmocap: A monocular 3d shole-body pose es- timation system via regression and integration. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pages 1159.
Yuechen Wang, Jiajun Deng, Wengang Zhou, and Houqiang Li. 202lb. Weakly supervised temporal ad- jacent network for language grounding. JEEE Trans- actions on Multimedia, 24:3276-3286.
Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. 2016. Hollywood homes: Crowdsourcing data collec- tion for activity understanding. In Computer Vision- ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14. 2016, Proceedings. Part I 14, pages 510-526. Springer.
Zheng Wang, Jingjing Chen, and Yu-Gang Jiang. 2021c. Visual co-occurrence alignment learning for weakly- supervised video moment retrieval. In Proceedings of the 29th ACM International Conference on Multi- media, pages 1459-1468.

---

Jie Wu. Guanbin Li. Xiaoguang Han, and Liang Lin. 2020. Reinforcement learning for weakly supervised temporal grounding of natural language in untrimmed videos. In Proceedings of the 28th ACM Interna- lioaal Conference on Multimedia, pages 1283-1291.
Zhenbo Xu, Ajin Meng, Zhenbo Shi, Wei Yang. Zhi Chen, and Liusheng Huang. 2021. Continuous copy- paste for stage multi-object tracking and seg- mentation. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 15323- 15332.
Wenfei Yang, Tianzhu Zhang, Yongdong Zhang, and Feng Wu. 2021. Local correspondence network for weakly supervised temporal sentence grounding. IEEE Transactions on Image Processing, 30:3252- 3262.
Hao Zhang, Aixin Sun, Wei Jing, and Jocy Tianyi Zhou. 2020a. Span- baand localizing network for natural lan- guage video localization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6543-6554.
Mingxing Zhang, Yang Yang. Xinghan Chen, Yanli Ji, Xing Xu, Jingjing Li, and Heng Tao Shen. 2021. Multi-stage aggregated transformer network for tem- poral language localization in videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12669-12678.
Songyang Zhang, Houwen Peng, Jianlong Fu, and Jiebo Luo. 2020b. Learning 2d temporal adjacent networks for moment localization with natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 1870-12877..
Zhu Zhang, Zhijie Lin, Zhou Zhao, Jieming Zhu, and Xi- uqiang He. 2020c. Regularized two-branch proposal networks for weakly-supervised moment retrieval in videos. In Proceedings of the 28th ACM Interna- tional Conference on Multimedia, pages 4098-4106.
Zhu Zhang, Zhou Zhao, Zhijie Lin, Xiuqiang Hc, al. 2020d. Counterfactual contrastive learning for weakly-supervised lision-language grounding. Ad- vances in Neural Information Processing Systems, 33:18123-1888
Minghang Zheng, Yanjie Huang, Qingchao Chen, and Yang Liu. 2022. Weakly supervised video moment localization with contrastive negative sample mining. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pages 3517-3525.


Reviews:
[{"review_id": "lUVnb3CYFy", "reviewer": "Reviewer_Zmbu", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Compare acquiring transfer knowledge and the original training process in terms of computational resources and time cost.**\n\n\n**A1:** Thanks for your valuable comment. We report the time cost of each component of our model and the original training of previous methods for detailed comparison. Since previous unsupervised TSG method (DSCNet) does not provide its code publicly, we only compare our unsupervised CMKT with representative open-source fully- and weakly-supervised methods. We conduct all the experiments on 8 NVIDIA A100 GPUs with a total GPU memory of 640GB. Besides, we measure all the data with strict adherence to the source code in the same environment.\n\n(1) Firstly, we compare our knowledge transfer module with original training as follows:\n\n| Method | mIoU(Act) | Time(Act) | mIoU(Cha) | Time(Cha) |\n|:--:|:--:|:--:|:--:|:--:|\n| CTRL (FS) | 20.54 | 5.31h (original training) | 18.94 | 2.81h (original training) |\n| CNM (WS) | 37.14 | 4.88h (original training) |  **38.11** | 2.34h (original training) |\n| LCNet (WS) | 34.29 | 4.25h (original training) | 38.04 | 2.20 (original training) |\n| WSTAN (WS) | 35.72 | 3.87h (original training) | 27.43 | 1.78h (original training) |\n| **Our CMKT (US)** | **38.79** | **1.14h (knowledge transfer)** | 30.42 | **0.38h (knowledge transfer)** |\n\nHere, \"Act\" denotes \"ActivityNet Captions\", \"Cha\" denotes \"Charades-STA\", and \"Time cost\" denotes the time required to complete the training on the whole dataset. Obviously, our knowledge transfer can significantly reduce the time cost of the grounding model with competitive performance. The reason is: As a fully-supervised method, CTRL calculates the relationship between each frame and the given language query, spending much time during training. For weakly-supervised methods (CNM, LCNet and WSTAN), they still spend much time on cross-modal alignment. Different from them, our unsupervised CMKT avoids complex cross-modal alignment computation by our computation-efficient knowledge transfer module, which reduces the time cost.\n\n(2) Secondly, for our knowledge collection modules, their time costs are 10h (appearance knowledge collection) and 43h (action knowledge collection). **Please note that, our knowledge collection modules are the pre-processing modules, which are pre-trained offline. Other downstream methods/tasks can directly utilize our knowledge collection modules to further improve their generalization-ability for avoiding complex training processes.** To assess their effectiveness, we use them as a plug-and-play module for compared supervised methods. Then, we fine-tune their hyper-parameters to obtain the optimal values. We report the plug-and-play study as follows:\n\n| Method | Time(Origin) on Act | Time(Ours) on Act | mIoU(Origin) on Act | mIoU(Ours) on Act | Time(Origin) on Cha| Time(Ours) on Cha| mIoU(Origin) on Cha| mIoU(Ours) on Cha| \n|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|\n| CTRL (FS) | 5.31h | **4.13h** | 20.54 | **23.32** | 2.81h | **1.93h** | 18.94 | **20.31** |\n| CNM (WS) | 4.88h | **4.06h** | 37.14 | **37.89** | 2.34h | **1.35h** | 38.11 | **38.95** |\n| LCNet (WS) | 4.25h | **3.74h** | 34.29 | **36.01** | 2.20h | **1.28h** | 38.94 | **40.26** |\n| WSTAN (WS) | 3.87h | **3.18h** | 35.72 | **36.24** | 1.78h | **1.04h** | 27.43 | **28.57** |\n\nIn the above table, we choose the official codes as the baseline (i.e., \"Origin\") under the official settings. In \"Ours\" setting, we first utilize our pretrained two knowledge modules as the pretrained knowledge base. Then, we utilize our knowledge transfer module to enrich the grounding knowledge. Finally, we utilize their remaining modules for video grounding. Obviously, by using our model as the plug-and-play module, other TSG models can significantly reduce the training time and obtain better performance.\n\nObviously, our CMKT is more efficient than other supervised methods with competitive performance. Besides, our model can serve as a plug-and-play module to inspire some cross-modal downstream tasks for further improving their effectiveness and effectiveness.\n\n**Q2. Model framework diagram is not sufficiently clear. There is an excessive use of symbols instead of explicit explanations.**\n\n**A2.** We will divide the model framework diagram into three figures (appearance knowledge collection, action knowledge collection, and knowledge transfer to unsupervised TSG) to improve the reading experience. Moreover, we will reduce unnecessary symbols and add more explicit explanations for a clearer framework in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. There has been limited research on unsupervised temporal sentence grounding in the past. Previous works often focused on meticulously designing models and dealing with complex training processes. However, this work takes a different approach by leveraging transfer learning, which directly eliminates the need for an extensive training process. \n2. This article’s approach involves decomposing the original task, enabling the direct transfer of knowledge from other simpler tasks to the unsupervised Temporal Sentence Grounding (TSG) task. During the knowledge acquisition stage, the authors introduced several techniques to enhance the generalization of the acquired knowledge. In the knowledge inference stage, they also addressed the issue of knowledge applicability and proposed specific optimization measures to utilize the knowledge effectively. \n3. The experimental section of this work is comprehensive, and the model’s performance shows significant improvements. Moreover, various ablation experiments conducted in the study demonstrate the effectiveness of each proposed design.", "reasons_to_reject": "1. Although this work eliminates the complex training process for Temporal Sentence Grounding (TSG), there will still be some degree of training cost during the stages of acquiring transfer knowledge and improving knowledge applicability, both in terms of computational resources and time cost. So, compared to the original training process, does it show improvement? \n2. The model framework diagram presented in the paper is not sufficiently clear and aesthetically pleasing. There is an excessive use of symbols instead of explicit explanations, which significantly impacts the reading experience.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper addresses the task of unsupervised temporal sentence grounding (TSG). The main contributions of this paper are as follows:\n1. This work is the first attempt to use transfer learning knowledge to tackle the unsupervised temporal sentence grounding task. It eliminates the need for any annotations related to TSG data and simplifies the training process.\n2. The authors adopted carefully designed techniques to acquire and utilize cross-modal knowledge obtained through transfer learning, aiming to achieve better generalization performance.\n3. The model performs well on both datasets, especially in the unsupervised task setting. It also demonstrates competitive performance compared to some models that adopt weakly supervised approaches.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wpeMlizB0Q", "reviewer": "Reviewer_4roi", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Model diagram is too complicated, and the font is too small.**\n\n**A1:** Thank you for your constructive suggestions. We will divide the model diagram into three figures (appearance knowledge collection, action knowledge collection, and knowledge transfer to unsupervised TSG) to improve readability. Also, we will increase the font size and add more explicit explanations in our revision.\n\n**Q2: Highlight crucial information and move some content to appendix.**\n\n**A2:** We will follow your suggestions to highlight the crucial information and move some of the content to the appendix in our revision.\n\n**Q3: Methodology involves complex process, and model lacks simplicity.**\n\n**A3:** We will simply our model for better readability in the future version. Specifically, our model can be summarized as: \n\n(i) We design two knowledge collection modules to extract both appearance and action knowledges from other simple yet cheap cross-modal tasks. \n\n(ii) To fit the simple action knowledge into more complicated videos in TSG, we design an effective copy-paste strategy to synthesize videos with multiple actions. \n\n(iii) To transfer the knowledge into TSG, we obtain parsed words and visual features from the TSG data, and re-represent them by searching and aggregating the consensus features from the two knowledge modules. \n\n(iv) A cycle-consistent loss is also introduced to assist the knowledge transfer.\n\n**Q4: Lack of organization and analysis of some references:**\n\n**[1] Local-global video-text interactions for temporal grounding.**\n\n**[2] End-to-End Modeling via Information Tree for One-Shot Natural Language Spatial Video Grounding.**\n\n**A4:** Actually, we have cited LGI [1] in Table 1 of our paper.  In our revision, we will cite paper [2] and provide more discussion about [1,2]:\n\n[1] is a fully-supervised TSG method, which tackles the TSG task using a regression-based model that learns to extract a collection of mid-level features for semantic phrases in a text query, and to reflect bi-modal interactions between the linguistic features of the query and the visual features of the video in multiple levels.\n\n [2] is a natural language spatial video grounding method, rather than TSG method. It aims to detect the relevant objects in video frames with descriptive sentences as the query. Especially, it designs a tree-structure approach, and learns to ground natural language in all video frames with solely one frame labeled, in an end-to-end one-shot manner. \n\nPapers [1,2] severely rely on the numerous data annotations. Different from them, our method does not need any TSG annotations. Instead, we target to transfer the annotation knowledge from other cheap cross-modal tasks to the complicated and challenging TSG task. Our unsupervised method achieves very competitive performance compared to existing fully- and weakly-supervised methods.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed approach leverages simple and cheap cross-modal alignment knowledge from other tasks to model appearance and action information in videos, which significantly reduces the reliance on expensive video-query annotations and the need for large-scale training data.\n\n2. The proposed approach introduces a novel copy-paste approach to synthesize various multi-action clips and improve the generalization ability of the model, which enables it to handle complicated videos with multiple action contexts.\n\n3. The proposed approach achieves state-of-the-art performance on two challenging datasets, outperforming existing unsupervised TSG methods and even competing with supervised methods, demonstrating its effectiveness and potential for practical applications.", "reasons_to_reject": "1. The content of the model diagram is too complicated, and the font is too small, which greatly affects readability.\n\n2. It can be observed that the methodology section occupies a large portion of the paper, containing a significant amount of content with insufficient emphasis on key information. It might be worth considering highlighting the crucial information and moving some of the content to the appendix.\n\n3. The methodology involves a highly complex process, and the model lacks simplicity.\n\n4. There is a lack of organization and analysis of important references in the paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a novel approach for unsupervised temporal sentence grounding (TSG) without relying on expensive video-query annotations. The proposed Cross Modal Knowledge Transferring (CMKT) network leverages simple cross-modal alignment knowledge from other tasks to model appearance and action information in videos. The extracted knowledge is then fine-tuned and transferred to the TSG task for inference without further training. Extensive experiments on two challenging datasets demonstrate the effectiveness of the proposed approach, outperforming existing unsupervised methods and even competitively beating supervised methods.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] Mun J, Cho M, Han B. Local-global video-text interactions for temporal grounding[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020: 10810-10819.\n\n[2] Li M, Wang T, Zhang H, et al. End-to-End Modeling via Information Tree for One-Shot Natural Language Spatial Video Grounding[C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022: 8707-8717.\n"}}, {"review_id": "0Ar4x1A9XF", "reviewer": "Reviewer_MGd2", "summary": "", "questions": "see weakness", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Introduction is unnecessarily written as if it were the first time performing an unsupervised task. Cite the previous unsupervised TSG works [a,b] in Introduction.**\n\n**[a] Zero-shot natural language video localization. ICCV'21**\n\n**[b] Prompt-based zero-shot video moment retrieval ACM MM'22**\n\n**A1:** Thank you for your patient review. We will reorganize our introduction to avoid misunderstanding.  Also, we will cite and discuss zero-shot TSG methods [a,b] in our revision. Actually, **your mentioned \"zero-shot\" setting [a,b] is not the same as the \"unsupervised\" setting.** There is only one paper [c] focusing on the unsupervised TSG. We have already cited and compared it in Table 1, where we achieve better performance. Our \"unsupervised\" differs from \"zero-shot\" in two main aspects:\n\n1) **Annotations are different.** Zero-shot TSG [a,b] means that the input data only contains video set without query set. These works utilize pre-trained object detector and NLP tools to generate language sentence semantically related to the detected segment. Note that, during training, these generated segment-query pairs are semantically aligned. Different from zero-shot setting, the unsupervised TSG means that the input data contains both video and query sets. However, during training, the correspondence between videos and queries are unknown.\n\n2) **Goal of the designed model.** Since zero-shot methods [a,b] generate semantically aligned segment-query pairs, their models are designed to train with these pseudo labels for learning to ground the query-related segment. This process is the same as the general fully-supervised setting. They severely rely on the quality of the generated sentence. \nDifferent from them, the unsupervised setting [c] aims to mine the potential relationships between the unknown video and query. It does not need any additional pre-trained model and its model is designed to generalize to the unlabeled cases.\n\n**Overall, the zero-shot setting is quite different from the unsupervised setting.** \n\n[c] Unsupervised temporal video grounding with deep semantic clustering, AAAI’22\n\n**Q2: This work is more closed to weakly-supervised training.**\n\n**A2:** Firstly, we want to clarify that our main contribution is not the unsupervised setting, but is to propose to transfer knowledge from other annotation-cheap tasks into TSG task without using TSG annotations. \n\nSecondly, our work is not the weakly-supervised setting. Generally, the weakly-supervised learning is based on the matched video-query pair without frame-level annotation on the TSG datasets as supervision, while our unsupervised method does not utilize any video-query pair information on the TSG datasets. Both the appearance knowledge and action knowledge are pre-trained from two modules respectively. These two collected knowledges are fixed during our unsupervised framework fine-tuning (i.e., knowledge transfer).\n\nWe will highlight our contributions and add more discussion about different settings in our revision.\n\n**Q3: The figure is too complicated.**\n\n**A3:** We will revise our figures to further improve the readability.\n\n**Q4: Recent unsupervised TSG works are not cited.**\n\n**[a] Zero-shot natural language video localization. ICCV'21**\n\n**[b] Prompt-based zero-shot video moment retrieval ACM MM'22**\n\n**A4:** In our revised version, we will provide more analysis of these zero-shot TSG works and provide more details about the difference between zero-shot TSG and unsupervised TSG as:\n\nAs the pioneering work of zero-shot TSG, [a] introduces two modules (temporal event proposal (TEP) and pseudo-query (PQ) generation) into a supervised TSG model. At training, pseudo-supervision composed of TEPs and corresponding PQs as a simplified sentence is generated to train the NLVL model. At inference, a natural sentence query is transformed into a simplified one, and temporal segment boundaries are predicted with the trained model.\n\nAs a respectable work of zero-shot TSG, [b] design two novel modules (Proposal Prompt and Verb Prompt) into a supervised TSG model. During training, [b] uses pseudo-supervision to train the TSG model. The pseudo-supervision comprises temporal proposals and corresponding pseudo queries (i.e., simplified sentences consisting of nouns and verb prompts). During inference, [b] transforms a sentence query to a simplified one and localizes temporal boundaries with the trained TSG model.\n\n**However, the zero-shot setting is quite different from the unsupervised setting:**\n\nZero-shot TSG means that the input data only contains video set without query set. These works utilize pre-trained object detector and NLP tools to generate language sentence semantically related to the detected segment. Different from zero-shot setting, the unsupervised TSG means that the input data contains both video and query sets. Moreover, since zero-shot methods generate semantically aligned segment-query pairs, their models are designed to train with these pseudo labels for learning to ground the query-related segment under the fully-supervised setting. \n\nDifferent from Zero-shot TSG, unsupervised TSG aims to mine the potential relationships between the unknown video and query. It does not need any additional pre-trained model and its model is designed to generalize to the unlabeled cases.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "[+] Unsupervised setting for TSG\n\n[+] Sufficient experiments\n\n[+] Enhanced retrieval performances", "reasons_to_reject": "[-] Although this paper is not the first paper to perform unsupervised TSG, the introduction is unnecessarily written as if it were the first time performing an unsupervised task. There are several citations about TSG and weakly-supervised TSG, but why did the author not cite the previous works about unsupervised TSG in Introduction. See below recent works of unsupervised TSG\n- Zero-shot natural language video localization. ICCV'21\n- Prompt-based zero-shot video moment retrieval ACM MM'22\n\n[-] This work is more closed to weakly-supervised training, the model trained with a video retrieval dataset and performing retrieval on the video is the process of weakly-supervised setting. Therefore this work has a fatal misunderstanding of contributions.\n\n[-] The figure is too complicated, which ruins the readability.\n\n[-] Many recent works about unsupervised TSG are not cited.\n- Zero-shot natural language video localization. ICCV'21\n- Prompt-based zero-shot video moment retrieval ACM MM'22", "questions_to_authors": "see weakness", "paper_topic_and_main_contributions": "This paper performs video moment retrieval under the training of image-noun pairs and video-verb pairs. Transfer learning is applied to incorporate the alignment knowledge.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "see weakness", "Typos_Grammar_Style_and_Presentation_Improvements": "presentations are too complicated."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received a mixed reviews from 3 reviewers. Reviewers Zmbu and 4roi recommend acceptance given the strong perfromance of the method and the potential of elimate annotation cost. While all reviewers recommend rejection as the paper is overall not well-written and is hard to understand, e.g., complicated figures and excessive use of symbols. Reviewer MGd2 also mentioned the work unnecessarily claimed as first work for unsupervised TSG. AC values the contributions of targeting reducing the use of annotations in the TSG task, however, AC does not agree with the authors that this work should be described as "truly unsupervised" due to its use of annotated assets (Visual Genome and Kinetics). There are work that use purely unannotated data for the task, e.g., "Zero-shot Video Moment Retrieval With Off-the-Shelf Models".
Decision: Accept-Findings