Weakly Supervised Video Scene Graph Generation via Natural Language Supervision
Authors: Kibum Kim
Kanghoon Yoon
Yeonjun In
Jaehyeong Jeon
Jinyoung Moon
Donghyun Kim
Chanyoung Park
Venue: ICLR 2025 Poster (2025)
Keywords: Video Scene Understanding
Weakly Supervised Learning
Large Language Model
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose  a weakly-supervised video scene graph generation framework that aims to relieve the annotation costs by training a model using natural language supervision.

Abstract: Existing Video Scene Graph Generation (VidSGG) studies are trained in a fully supervised manner, which requires all frames in a video to be annotated, thereby incurring high annotation cost compared to Image Scene Graph Generation (ImgSGG). Although the annotation cost of VidSGG can be alleviated by adopting a weakly supervised approach commonly used for ImgSGG (WS-ImgSGG) that uses image captions, there are two key reasons that hinder such a naive adoption: 1) Temporality within video captions, i.e., unlike image captions, video captions include temporal markers (e.g., before, while, then, after) that indicate time-related details, and 2) Variability in action duration, i.e., unlike human actions in image captions, human actions in video captions unfold over varying duration. To address these issues, we propose a Natural Language-based Video Scene Graph Generation (NL-VSGG) framework that only utilizes the readily available video captions for training a VidSGG model. NL-VSGG consists of two key modules: Temporality-aware Caption Segmentation (TCS) module and Action Duration Variability-aware caption-frame alignment (ADV) module. Specifically, TCS segments the video captions into multiple sentences in a temporal order based on a Large Language Model (LLM), and ADV aligns each segmented sentence with appropriate frames considering the variability in action duration. Our approach leads to a significant enhancement in performance compared to simply applying the WS-ImgSGG pipeline to VidSGG on the Action Genome dataset. As a further benefit of utilizing the video captions as weak supervision, we show that the VidSGG model trained by NL-VSGG is able to predict a broader range of action classes that are not included in the training data, which makes our framework practical in reality.

Published as a conference paper at ICLR 2025
WEAKLY SUPERVISED VIDEO SCENE GRAPH GENERATION VIA NATURAL LANGUAGE SUPERVISION
Kibum Kim' Kanghoon Yoon' Yeonjun In'
Jachyeong Jeon'
Jinyoung Moon?
Donghyun Kim Chanyoung Park
KKAIST EERRI KKorea University
[kb. (kb.kim, ykhoon08, yeonjun.. in, wogud405, cy parkk ac.. ar. Jymooneetri. re.kr d d.kim@korea.co
ABSTRACT
Existing Video Scene Graph Generation (VidSGG) studies are trained in a fully supervised manner, which requires all frames in a video to annotated, thereby incurring high annotation cost compared to Image Scene Graph Generation (ImgSGG). Although the annotation cost of VidSGG can be alleviated by adopt- ing a weakly supervised approach commonly used for ImgSGG (WS- WVS-ImSSGG that uses image captions, there are two key reasons that hinder such naive adop- tion: 1) Temporality within video captions, i.c.. unlike image captions, video cap- tions include temporal markers (e.g., before, while, then, after) that indicate time- related details, and 2) Variability in action duration, i.e., unlike human actions in image captions, human actions in video captions unfold over varying dura- tion. To address these issues. we propose Natural Language-basee Video Scene Graph Generation (NL-VSGG) framework that only utilizes the readily available video captions for training a VidSGG model. NL-VSGG consists of two key mod- ules: Temporality-awerr tion Variability-aware caption-free -frmme alignment (ADV) module. Specifically, TCS segments the video captions into multiple sentences in a temporal order based on Large Language Model (LLM), and ADV aligns each segmented sentence with appropriate frames considering the variability in action duration. Our approach leads to significant enhancement in performance compared simply applying the WS-ImgSGG pipeline to VidSGG on the Action Genome dataset. As a fur- ther benefit of utilizing the video captions weak supervision, we show that the VidSGG model trained by NL-VSGG is able predict a broader range of action classes that are not included in the training data, which makes our framework prac- tical in reality. Our code is available https://github.com/iiigg www./////ggiII////S/CCCCCC 107/NL-VSGG
Caption Segmentation (TCS) module and Action Dura-
INTRODUCTION
Scene graph is a Assuall-grrunnde structured graph in which objects are represented as nodes and the relationships between them as directed edges. A scene graph bridges computer vision and natural language with high-level information. facilitating its usage on various downstream tasks, such as question answering (Ghosh et al. 2019), captioning (Yang et al. 2019), and retrieval (Schroeder & Tripathi 2020).
In general, studies for scene graph generation (SGG) (Kim et al. 2024a Yoon et al., 2023 Jeon et al., 2024 Yoon et al. 2024) have been conducted in the realm of images, referred to as ImgSGG. These studies primarily excel at predicting static relationships (c.g.. standing on) within single image, while struggling to predict dynamic relationships (e.g.. running) that may exist over consecutive images, since ImgSGG models are unable capture dynamic visual relations (Chen et al.) 2023) In this regard, Video scene graph generation (Cong et al.) 2021 Feng et al.) [023)] Teng et all 2021], dubbed as VidSGG, has emerged to capture temporal context across video frames and predict dynamic relationships, extending its scope beyond merely predicting static relationships within single image.
Existing VidSGG studies (Cong et al. 2021 Nag et al. 2023 Xu et al. 2022 Wang al. 2022a)
follow a fully supervised approach, indicating à heavy reliance on costly annotation involving class
"Corresponding Author

---

Published as a conference paper at ICLR 2025
Image
cutting
Groanding
Irpst lmage tmag Cestion
person
sitting
sota
[a] Fully Supervised
Pipelline Weally Supervised Img56G
Temppers Algeement
Grounding
1400
StrENN Sentenee
Video Captine
Sestera poree bulting
Sestere pares ecaroing a
Teepaoal Sepmeniation
Training
(c) Pipeline of Weakdy Supervised Vid5GG (Ours)
Figure I: (a) The fully supervised VidSGG requires costly localized scene graphs across all frames. (b) The pipeline of WS-ImgSGG. (c) The pipeline of WS-ViG needs to consider the tempo- rality within the caption addressed by temporal segmentation and the variability of action duration addressed by temporal alignment.
information (i.e.. entity and relation) alongside bounding boxes for entities across all frames in video (See Fig. [a)). This indicates that VidSGG requires greater annotation costs compared to ImgSGG, which only requires annotations for a single image. Despite recent weakly supervised ImgSGG (i.e., WS-Img
approaches (Zhong et al., 2021 Zhang al. 2023 Ye & Kovashka, 2021 Kim et al. 2024b Li et al. 2022a) that address the annotation cost in ImgSGG, weakly supervised approach for VidSGG (i.e WidSGGG where annotation costs are more demanding, remains unexplored. Although PLA (Chen al. 2023), the first WS- VidSGG study, proposes framework for training an VidSGG model based on ground-truth unlocalized scene graph of the middle frame, argue that the assumption of ground truth scene graph existing in the middle frame, among other frames, is not only unrealistic but also still requires manual human annotation, thus continuing to impose substantial annotation costs.
In this work, we are interested in training an VidSGG model without any graphs, and there can be two different strategies. The first strategy would be to apply a pre- trained ImgSGG model, preferably the with strong zero-shot predictive ability for relations (c.g.. RLIP (Yuan et al. 2022) and RLIPv2 (Yuan et al.) (023)), to every frame of the video to obtain pseudo-labeled scene graphs in each frame. However, since such model is trained based static visual relationships in images, they are not capable of predicting dynamic visual relationships The second strategy would be to leverage language supervision from video cap- tions and follow the conventional WS- WS-ImgSGG pipeline (Zhong al. 2021 Zhang al. 2023 Ye Kovashka [222]] Kim et al.) 2024b) Li et al. 2022a) as depicted in Fig. (bb.. Specifi- cally, we could first parse a video caption to extract triplets, align them with consecutive frames, and ground the aligned triplets within each frame. However, such simple approach of adopt- ing the WS-ImgSGG pipeline to VidSGG is limited? due to the following two key reasons:
scene
Temporality within Video Captions. Contrary image captions shown in Fig. [b)), video captions often contain temporal markers (e.g., before, while, then. after) representing time-related details. Without considering them, applying the above simple approach may erroneously supervise the model. For example, in Fig. (c.. (). if we overlook the temporal marker be- fore the caption, and use <person, sitting on, sofa> annotate earlier frames rather than later frames, the model would be mistakenly supervised, resulting in the performance degradation. As temporal markers account for around 65% of the Action Genome (AG) dataset used for VidSGG, while they account for only 2% in the Visual Genome (VG) (Krishna et al.) (017b) dataset used for ImgSGG (See Fig. 2). considering temporality is especially crucial in video captions.
50 40
Va
Caption
Figure 2: Ratio of
temporal markers.
we show that the performance of RLIP and RLIP=2 is subpar in the Action Genome Jji et al. 2020) dataset under the zero-shot setting.
'In Table !
In Table show that the simple approach performs poorly the Action Genome dataset.
2

---

Published as a conference paper at ICLR 2025
Variability in Action Duration. Human actions unfold over varying duration in video. For example, in Fig. [c), Sentence occurs within two frames, while Sentence 2 extends across a longer span of four frames. However, if we overlook such variability in the action duration and naively align the first segmented sentence with the first four consecutive frames and the second sentence with the latter four consecutive frames (i.e., 8 frames + 2 sentences), the 3rd and 4th frames would end up being annotated with <person, carrying, cup>. while the last two frames would end up being annotated with <person, sitting on, sofa>>, both of which are undesired.
To this end, we propose a simple yet effective weakly-supervised VidSGG framework, called Natural Language-based Video Scene Graph Generation (NL-VSGG), that only utilizes the readily available video captions for training a VidSGG model. Our proposed framework consists of two key modules: Temporality- Caption Segmentation (TCS) module, and Action Duration Variability- Cuption-Framne Alignment (ADV) module. More precisely, TCS segments the video caption into multiple sentences while also considering the temporality within the video caption based on Large Language Model (LLM). Then, ADV aligns the temporally segmented sentences with corresponding consecutive frames in the video considering the variability in action duration. The main idea is to perform K --eans clustering on the frames, and assign each segmented sentence to the frames within a cluster based on its similarity with the cluster centroids.
Through our extensive experiments, we demonstrate the superiority of NL-VSGG over the simple adoption of 1) pre-trained ImgSGG model, and 2) the WS-ImgSGG pipeline to VidSGG. It is worth noting that for the first time, we show the capability of training a VidSGG by only utilizing the readily available video captions, i.c., language supervision. As a further benefit of utilizing the video captions weak supervision, we show that the VidSGG model trained by NL-VSGG is able to predict a broader range of action classes that are not included in the training data, which makes our framework practical in reality. Our contributions are summarized follows:
We identify two key reasons for why a simple adoption of the WS-ImgSGG pipeline to VidSGG fails, i.e., temporality within video captions and variability of action duration.
We propose a simple yet effective weakly supervised VidSGG framework, called NL-VSGG, that only utilizes the readily available video captions for training. To the best of our knowledge, we are the first to enable the training of VidSGG model with language supervision without manual annotation.
Our extensive experiments on the Action Genome dataset demonstrate the superiority of NL- VSGG. Our proposed method is practical in that utilizing the video captions as weak supervision allows the VidSGG model to be able to predict action classes that are not included in the training data.
WS- WITH NATURAL LANGUAGE SUPERVISION
In this section, we describe the pipeline of NL-VSGG for training VidSGG model based on natural language supervision of video captions. We start by outlining the problem formulation of WS- VidSGG (Section 2. Subsequently, we describe Temporality aware Caption Segmentation (TCS) module that segments a video caption in a temporal order via an LLM (Section 2.2). followed by Action Duration Variability- aawre Saption-Frame Alignment (ADV) module that aligns each of the segmented sentence with appropriate frames (Section 2.3 Then, we parse the aligned segmented sentences to extract triplets. ground them within each frame, and train the VidSGG model with pseudo-localized scene graphs (Section 2.4). Finally, we describe novel lseudo-labeling strategy that leverages negative action classes using motion cues within unaligned frames (Section 2.5). The overall framework is shown in Fig. 3
2.1 PROBLEM FORMULATION
In this work, given a video V = 12,,,, 1"' and its paired video caption S, where I' is the t-th frame of the video V and T is the number of video frames in the video V. our goal is to generate a scene graph G = ( p/. of 1 for each frame I', where Ae is the number of triplets in I'. Moreover, /o denote the subject/object, whose entity classes are sf,c/o5,e E C and their bounding boxes are given by "f/8/O5,6 pa denotes the action of ,43 interacting with of. and its class is denoted by p¹,c E Ca, where Ce and Ca are predefined entily classes and action classes,
In the Action Genome dataset, the subject is always a person.
3

---

Published as a conference paper at ICLR 2025
sideo Vare Contien
Coption A'yrmnnnt (A0))
AI Sec 24 Gesnratic of Pwwdee-eenlteedeeee
Pening
aetrouzing 01a peeponncccnnnneeec..... sogardla
Jeb
www.eooriity Orytien Seprorsiinn (roy)
Waboo Traishty
e. 1,=-,,1
Gostrali chiter
10 Sec
deries strage
Vhee incadar
LUM
Ebcodar
Figure 3: The overall framework of NL-VSGG. With an input video and its caption, (a) em- ploy the TCS module to segment the inpur video caption into sentences based on temporality. (b) In the ADV module, each segmented sentence is aligned with appropriate frames considering the variability action duration. (c) The segmented sentences are then parsed and grounded generate pseudo-localized scene graphs. (d) Furthermore, we assign negative classes based on the motion cues within unaligned frames. (e) Utilizing the pseudo-localized scene graphs and pseudo-labeled negative classes, we then train a VidSGG model.
respectively. Given a scene graph G' across all frames along the time axis (i.e., t E {1, .T.),, the scene graph for video V is represented by G (G,,Ge,... (e,,,G,..,,....). G²).
Difference with existing fully/weakly supervised approaches. Note that fully supervised ap- proach (Cong et al., 2021; Nag et al. 2023) relies on localized video scene graph g for train- ing the model. On the other hand, a recently proposed weakly supervised approach (i.e., PLA (Chen et al.] (023)) relies on a ground-truth unlocalized scene graph of the middle frame, i.e., 1nn". Gº [sf P of j==1 where 1' is 2/2 and the bounding boxes //,///0/ are not provided. Both fully/weakly supervised approaches require structured scene graphs that demand costly human labor, while NL-VSGG only necessitates a readily available video caption S.
2.2 TEMPORALITY-WWAR CAPTION SEGMENTATION (TCS)
In this module, we segment the video caption S considering the temporal order of events clearly understand the sequence of actions. To this end, we employ an L.M based on a prompt that is designed to be aware of temporality within video captions. Our newly designed prompt is described as follows. First, the initial instruction to inform the LLM about the task at hand while enforcing to consider the temporality of events occuring in the vidco caption is designed as: Your job is to split the given video caption into multiple compositional sentences and arrange them in chronological order. Moreover, video captions often require coreference resolution (Peng et al.) 2019) caused by a pronoun referring to the same object over time, which may hinder accurate extraction of triplets. To this end, we provide an additional prompt: Note that you should specify the objects for the pronouns used in each of these sentences. Following the above prompts, we provide few examples related to the task at hand (i.e., in-context few-shot learning (Brown et al. (0200) further engage the LLM. Finally, we instruct the L.M to segment the video caption S in a temporal order, which in turn yields segmented sentences ( ,.... 5. S..,S,M,, J. where M is the number of segmented sentences and M < T. For example, in Fig. Sentence / and Sentence 2 correspond to Ś and 52, respectively, with M being 2. In summary, we design prompts tailored to the WSdVid task particularly focusing on capturing the temporality within the video caption and addressing the coreference issue. Please refer to Appendix A regarding the details of the prompt, and Appendix B regarding the impact of the prompt addressing the coreference resolution.
2.3 ACTION DURATION SARIABILITY-AWAR
COPTION-FRAME ALIGNMENT (ADV)
Having obtained the segmented sentences ($1,52 52., 5m S.M) from the video caption S, we need to align each segmented caption 5T with frames that visually correspond to the scene being described therein. However, this alignment process requires careful attention to ensure effective supervision of the model, taking into account the variability in action duration, as discussed in Section 1 In doing so, it is crucial to estimate how the visual semantic of each frame 11, 12 ...., 1T
4

---

Published as a conference paper at ICLR 2025
is relevant to the textual semantic of each segmented sentence Sth. Hence, we employ vision- language mode that captures the joint space of visual and textual semantics. More precisely, we feed 12,12 I' into a visual encoder fess to extract visual features (i.e., Ü2 = (f)) and Stm into a text encoder Steat to extract textual features (i.e., I'm = [reze(S"")), followed by K-Means clustering algorithm on {&1 02 .... or) to generate K proposals with which the textual features can be aligned. The most straightforward approach to align I' with corresponding video frames would be to compute the similarity scores between Ém and the centroids {c1, C2...., CK J. and select the frames near the most similar centroid. However, this approach cannot effectively capture the variability of action duration, especially in long action duration cases, e.g., Sentence 2 in Fig. 3 To this end, we propose simple yet effective method of leveraging the variation in similarity scores.
Clustering-based Caption-Frame Alignment. For a sentence with long action duration, its cor- responding frames would span across multiple clusters, exhibiting relatively high similarity scores with multiple cluster centroids, On the other hand, a sentence with short action duration would rep- resent confidently sharpened similarity score mainly focusing
single cluster. Hence, for the representation irn of each segmented sentence 5Th we arrange its similarity scores with K centroids in descending order, and determine the point where the scores show the steepest decline. Note that this steepest decline point will determine the varying durations of actions by isolating the relevant multiple clusters, regardless of how long the action lasts, Then, we choose all the clusters preceding this point, and align the frames in those clusters with fTH For example, for Sentence in Fig. 3 we sort the cluster based on the similarity scores (i.e., C4, C32 C1 C2....) and determine the point where the score shows steepest decline (i.e., C1 + c2). Then, we choose frames in clusters ca: Cg. and cı to align with Sentence 2. In this manner, we can adaptively choose clusters according to the vari- ability of action duration. Regarding the K, our aim is to adapt the number of clusters depending on the length of the video. Thus, we set K VI 3 where /3 is hyperparameter.
Removing Unrealistic Caption-Framt Alignments. It is worth noting that we remove unrealistic alignments that violates the temporal order. For example, if S is aligned with 12, and $ is aligned with 71, 13, 74, 16, 76, we remove 7 from $2 since 1 is already aligned with 51, which precedes 52 (See Fig. 3). Finally, each segmented sentence STm is aligned with a sequence of consecutive frames within V = 12.., 1"', i.e. [7|ST
2.4 GENERATION OF PSEUDO-LOCALIZE SCENE GRAPHS
Based on the {(Smm [ M for training the model, which can be described as follows:
M obtained from ADV, we construct pseudo-localized scene graphs
Scene graph parsing. We begin by converting each segmented sentence 5Th into a triplet L.L.- parser (im et al. 2024b). Note that the bounding boxes 5m,b. 0m,8 are not defined, and the classes Sm,es Om,es Pi.ce are not necessarily included in C and C". To ensure 8m,c, Om,c E C and Pn.c E Ca we map them to the classes of our interest (i.e., entity uriit//aciin /action classes in the Action Genome dataset) by either using synsets' lemmas and hypernyms in WordNet (Miller) 1995) or L.L.-based alignment (Kim et al. 2024b). In this process, we obtain graphs. Note that we can also omit the class mapping process to let the model be able to predict a broader range of action classes that are not included in the entity/ane /action classes of Action Genome, as will be shown in Section
<.8m: Pin- , Om using either rule-based parser (Schuster et al.. 2015)
-ouudoo-nlocalzeed scene
Scene graph grounding. To define the bounding boxes 8m,b and Om,b, we follow a prior study (Chen et al. 2023) that leverages the information of a pretrained object detector. Specifically, we ground S. to detected bounding box whose entity class corresponds Sm,e, while on is grounded in similar manner. After grounding Sm and Onz we assign pm between 8m and Om-
The above processes of scene graph parsing and scene graph grounding are adopted to each pair in {(Smm. [ )) where the scene graph grounding process is applied across aligned frames video scene graphs G can be obtained.
[1] snls", after which pseudo-localizn
*For the vision-language model, we used DAC (Doveh et al. 2024), a variant of CLIP Radford et al. 2202 due to its compositional reasoning ability. Moreover, the vision-language model can be replaced with a video- language model (Refer to Appendix for experiments that replaces DAC with InternVideo).
Although each segmented caption can be converted into more than triplet, we assume here that it is converted only triplet for simplicity of explanation.
5

---

Published as a conference paper at ICLR 2025
2.5 OPTIONAL: DEALING WITH NEGATIVE ACTION CLASSES IN ACTION GENOME
It is important to note that while the Action Genome dataset contains negative action classes ge.g. not look- ing at and not contacting), video captions usually do not contain negation I For this reason, a VidSGG model trained in weakly supervised manner based on video captions as in our study would fail to generate a scene graph containing negative action classes in the Action Genome dataset. To address this issue, we propose a novel Pseudo-Labeling strategy based on Motion cues (PLM) that assigns negative action classes between subject-object pairs. The core idea is to make use of the fact that person usually does not look at (or contact) an object when moving apart from it. Specifically, we adopt the generalized loU (Rezatofighi et al. 2019 Gao et al. 2023) (GloU) as a metric to measure how close two objects are from each other. Note that for a given subject and an object, a small GIoU in the end frame (i.e., G.) and a large GloU in the start frame (i.e., G.)) indicates that they are getting farther from each other over time. For example, as a person is moving farther from sofa from the 3rd frame to the last frame, the GloU gets smaller (See Figure 4.. In other words, as G. G. gets smaller, the subject and the object are getting farther from each other over time. With this motion cue, we assign negative action classes, i.e., not looking at and not contacting.
pprsoouscchiing on the sigs
Liaalgoed Promses
Figure 4: Example of motion cue.
However, as considering all the frames in a video for costly, we utilize frames that are not aligned with any of the segmented sentences for pseudo-labeling negative action classes. More precisely, for all the unaligned frames across the entire videos given in the dataset, we compute G. G. between the same subject-object pair. Then, we sort the computed GloUs in the ascending order, and assign pseudo-labels (i.e., not looking at and not contacting) to top-a% aiming at assigning labels to more confident subject-object pairs. More precisely, we restrict the assignment of not looking at to the start and end frames within unaligned framees and not contacting only to the end frames. Please refer to Appendix regarding the different selection strategies for pseudo-label assignment. Since considering all subject-object pairs included in the training data is not only time-consuming but also prone to noise, we only select objects appearing in the pseudo-localized video scene graph G obtained from the aligned frames in each video.
of negative action classes is
It is important to note that the proposed PLM is an optional module that can be only applied when the dataset contains negative classes as in the Action Genome dataset. While we show its effec- tiveness on the original Action Genome dataset with negative classes, we also provide results for datasets without negative classes, namely the AG dataset without negative classes See Table 10 in Appendix[] [ and VidHOI dataset (Chiou et al. 202T) (See Table 6 in Appendix C
Model Training. Lastly, having obtained the pseudo-localized video scene graph & and pseudo- labeled negative classes within unaligned frames, we then follow the training protocol of existing VidSGG models, i.e., STTran (Cong et al. 2021) and DSG-DETR (Feng et al.) (023).
3
EXPERIMENT
3.1 EXPERIMENTAL SETTING
Datasets. To train a VidSGG model without ground-truth localized video scene graphs" we use three video caption datasets: the Action Genome (Ji ct al., 2020) (AG) caption, the MSVD (Chen) & Dolan, 1)1 caption, and the ActivityNet caption (Krishna et al. 201 datasets. For the AG caption dataset, we use 7,454 videos consisting of 166,785 frames, following previous studies (Chen al. [023] Cong et al. |02 Ji et 2020). The duration of each video is on average 29.9 seconds. To show a more practical setting, which is to use external caption datasets, other than a benchmark VidSGG dataset containing video captions (i.e., AG dataset), weak supervision, we use the MSVD and ActivityNet caption datasets, each of which contains 1,970 videos consisting of 40,863 frames with 2 FPS and 4,640 videos consisting of 669,836 frames with 1 FPS, respectively. The duration of each video for the MSVD and ActivityNet datasets is on average 9.5 and 117.3 seconds, respectively. Note that we mainly use the AG caption dataset for analysis throughout this paper while the MSVD and ActivityNet caption datasets are only used in Section 3.4 and 3.5 To evaluate our proposed NL-VSGG framework, we use the AG dataset containing ground-truth localized video scene graphs with 36 object classes (i.e., C*) and 25 action classes (i.e., C"), whose categories are divided into
'Only 0.09% (11/11,593) among all captions in Action Genome contain negation.
'The ground-truth localized video scene graphs are utilized for model training in fully-supervised ap- proach while we do not them for weakly supervised approach.

---

Published as a conference paper at ICLR 2025
three types, i.e., 3 attention classes, 6 spatial classes, and 16 contacting classes. Following previous
studies (Chen et al. 2023] Cong et al. (02T), we use 1,747 videos with 54,429 frames. Furthermore, to validate the generalization on other datasets, we train and evaluate our proposed framework using
the VidHOI (Ciiou et al. 2021 dataset, which is detailed in Appendix C
Evaluation Metrics. We use the widely adopted Recall@ K (R@ K) with "With Constraint" and
"No Constraint" strategies, following previous studies (Cong et al. 202 Chen et al. 2023 Feng al. 2023). "With Constraint" allows only one action prediction with the highest score between subject and object, while "No Constraint" reflects the multi-label prediction capability in evaluation metric as the AG dataset follows a multi-label task. For example, if multiple high action scores are associated with a single subject-object pair, all of them are allowed in the top-K. Regarding the evaluation task, adopt the Scene Graph Generation (SGDet) task, which is commonly used for WS-ImgSGG (Kim et al., 2024b Zhong et al., [021 Zhang et al.) 2023 Ye & Kovashka] (02T). In this task, the ground-trute (GT) bounding boxes and entity class information are not provided during evaluation, and the predicted triplet is considered valid only if the predicted bounding box overlaps with the GT bounding box with an 1oU:0.55.
Baselines. To evaluate our proposed framework, we compare it with models each of which cor- responds to one of the four types of supervision: Zero-shot supervision supervision, Weak supervision of GT Unlocalized scene graph (SG), and Weaker supervision of Natural Language, i.e., utilization of readily available video captions. Specifically, zero-shot su- pervision indicates that without any training process, each frame is considered as a static image and inferred with ImgSGG models equipped with strong zero-shot predictive ability for relations. For this purpose, we employ RLIP (Yuan et al.) 2022) and RLIPv2 (Yuan et al. 2023). Full supervision, which we consider as the upper bound of our model, involves training the model with ground-truth localized video scene graphs and we leverage STTran (Cong et al.) 2021) and DSG-DETR (Feng et al. (203)) for VidSGG models. Weak supervision of GT unlocalized SG is to utilize a GT unlo- calized scene graph in the middle frame, which is proposed by PLA (Chen et al. 2023). Moreover, we also compare with PLAsimp. a simplified version of PLA trained using only unlocalized SG from the middle frame without any component of PLA. This baseline is included
@ (No supervision), Full
demonstrate the heavy reliance of PLA on the annotated GT unlocalized scene graph, which is costly. Finally, the natural language supervision is based on the readily available video captions. It includes two approaches: straightforward approach of the WS-ImgSGG method discussed in Section (i.e.. + WS-ImgSGG) and our proposed framework (i.e., + NL-VSGG). In addition to those approaches, add a new baseline adapted from PLA (i.c. + PLAcap),, ) where the GT unlocalized scene graph of the middle frame are replaced with couuo-unlocalized scene graphs obtained from video captions. It is important to note that video captions provides weaker supervision signal to the VidSGG model compared with the GT unlocalized SG (hence we name "Weaker").
Implementation Details. For the pretrained object detector, we follow PLA (Chen et al.) 202.3, which employs VInVL (Zhang et al. (02T) with backbone ResNeXt-152 C4. This object detector only leaves bounding boxes for objects with a confidence score of 0.2 or higher. In the TCS module, use gpt-3.5-turbo in ChatGPT (OpenAl 2023) for an LLM. In the ADV module. DAC (Doveh cet al. 2024) is adopted for a vision-language model. Please refer to Appendix D regarding the experiment with open-source smaller language model (Jiang et al. (023) and another vision- language model (Wang et al., 2022ch. Additionally, 33 used determine the number of clusters K
is set to 4, and a used in the pseudo-labeling strategy is set to 15%. Please refer to Appendix [ regarding the sensitivity of hyperparameter /3 and Regarding the triplet extraction in Section 2.4 we adopt an LLM-based based approach (Kim et al. 2024b). Please refer to Appendix F regarding the result of different triplet extraction processes. For the experimental environment, we implemented NL-VSGG on both an NVIDIA GeForce A6000 48GB and an Intel Gaudi-v2.
3.2
QUANTITATIVE RESULTS
Table shows the performance comparisons across four types of supervision, utilizing STTran (Cong et al.] 2021) and DSG-DETR (Feng et al.) (023) as backbones for full and weak/weaker supervision. We have the following observations: 1) Models with zero-shot predictive ability for relation, such RLIP YYuan et al. 2022) and RLIPv2 (Yuan et al. 2023). show inferior perfor- mance, especially in With Constraint setting. This suggests that they struggle to predict key re- lationships between subject-object pairs, highlighting the limitation of simply applying ImgSGG models to the VidSGG task, as these models fail to account for dynamic relationships. Please refer to Appendix G for experiments regarding the incorporation of dynamic relationships into the RLIP.
We use the term zero-shot setting interchangeably with zero-shot supervision.
7

---

Published as a conference paper at ICLR 2025
2) PLAsimp. which solely relies on a GT unlocal- ized SG of the middle frame without any com- ponents proposed in PLA, shows competitive perfor- mance with the vanilla PLA (Chen et al.) (023). while significandly surpass- ing the performance of RLIP/RLIPV2, which are trained on large num- ber of images with complex models. This suggests that incorporating even a few ground-truth unlocalized video scene graphs is essential for strong perfor- mance in the VidSGG task, while also revealing PLA's heavy reliance annotated GT unlocalized scene graphs, which are expensive to obtain. This is further verified by the significant performance drop when the GT unlocalized scene graph of the middle frame is replaced with a scene graph obtained from the video caption (PLA vs. PLAcpp). Moreover, we argue the assump- tion of a GT scene graph existing in the middle frame among other frames is not only unrealistic but also still requires manual human annotation, which makes these methods impractical in reality. 3) Given the natural language supervision from video captions, which further relieves the annotation cost of PLA, NL-VSGG exhibits notably superior performance compared with a simple adoption of the Ws-ImgSGG pipeline to VidSGG and PLAcap- both of which disregard the two key factors discussed in Section I i.e., temporality within video captions and variability in action duration.
Table 1: Results of four types of supervision on the AG dataset.
Backhone
Method
Supersision
With Constraint R
R
No Constraiat
R@20
RLIP RIJPV2
Img5GG
Zero-shot
7.95
9.16
9.70
13.80
8.37
10:05
14.60
21.42
Vanilla
Full
33.98
36,93
36.20
48.88
+PLA
Weak (GT Unlocalized SG)
20.94
25.79
22. 34
31.69
Aymp.
21.42
30.87
+WS-ImgSGG
10.01
12.83
9.02
14.05
+PLA.ca
Weaker (Natural Language)
10:10
13.26
10.61
15.13
+NL-VSGG
15.61
19.60
15.92
22.56
Vanilla
Full
34.80
36,10
40.90
48.30
DSG-DETR
+PLA
+PLAvcmp
Weak (GT Unlocalized SCG)
21.30
20.78
25.90
22.
22.70
31.90
31.69
+WS-ImgSGG
10.05
12.96
10:29
14.77
+PLA.gz
Weaker (Natural Language)
10.36
13.53
10.57
15.41
+NL-VSGG
15.75
20.40
16.11
23.21
3.3 ABLATION STUDIES
Table 2: Ablation studies on each module of NL-VSGG:
In Table 2 we conduct ablation stud-
ies on the AG dataset to verify the
Row
TCS
ADV
PLM
With Constrsint
R 20
R@50
No Constrsint
R@20
R
R6:20
Meon
R@:00
effectiveness of each module in NL-
(a) (b)
10.01
12.83
9.02
14,05
9.52
13:44
VSGG. For the ablation studies, we
11.09
14.66
11.34
16.70
11.22
15.68
11.98
15.58
11.93
17.36
11.96
16.47
use STTran (Cong et al.) (202I) as the
15.61
19.60
15.92
22.56
15.77
21,08
backbone. The variant of not using
any module (row (a)) corresponds to a simple approach of adopting the WS-ImgSGG pipeline to the
VidSGG task. We have the following observations: 1. Effeet of TCS: We observe that the adoption of the TCS module to capture the temporality within the video captions enhances the performance compared with a simple approach (row (a) vs. (b)). Note that we apply the TCS module to the sim- ple approach by considering each triplet demonstrates the effectiveness of the TCS module in capturing the temporality, leading to accurate supervision for the model. 2. Effect of ADV: We observe that the incorporation of the ADV module responsible for capturing the variability (b) vs. row (c)), which demonstrates the importance of considering the variability of human actions. Regarding a qualitative analysis of the ADV module, please refer Appendix H 3. Effeec of PLM: Our proposed pseudo-labeling strategy with motion cues significantly improves the performance (row (c) vs. (d)), demonstrating the effeectivnness of pscudo-labeling negative action classes. We recognize that the PLM module contributes the most to the the final performance, since the negative action classes, i.e.. not looking at and not contacting, belong to head predicate classes, accounting for 16.5% and 8.7% of the predicates in the test set, respectively. Recall that PLM is an optional module that can be only applied when the dataset contains negative classes. Thus, to more precisely validate the effectiveness of the TCS and ADV modules, we intentionally removed negative classes from both training and test sets in the AG dataset, and show the results in Appendix
segmented sentence. The performance enhancement
action duration further enhances the performance (row
3.4 EXPLORING THE POTENTIAL OF UTILIZING EXTERNAL VIDEO-TEXT DATASET
Note that our proposed framework is not only limited to a benchmark dataset, i.e., Action Genome (Ti et al. 2020) (AG). In this section, we explore the potential of utilizing an external video-text paired dataset, i.c., MSVD (Chen & Dolan, 2011), for training. For this experiment, we use STTran as the backbone. In Table we observe that the performance based on sole utilization of the MSVD caption dataset for training shows inferior performance compared to that of AG caption.
Table 3: Performance when an external
video-text dato-seexxdtttetttteeeeeeeeeeeee is utilized for training.
Training Datasel (Tleseed on AG)
Construint
No Constraint
20
R030
AG
15.61
22.56
MSVD
9.0
11.31
10.23
AG+MSVD
15.71
20.00
16.07
23.21
8

---

Published as a conference paper at ICLR 2025
This is expected since the video domains of these datasets are very different (i.e., indoor scenes in AG and outdoor scenes in MSVD). However, we observe that the combination of AG and MSVD caption datasets performs the best, implying that collecting large pool of video-text pair regardless of the domains is helpful. In this regard, we believe this study paves the way of enhancing the inherent limited VidSGG datasets.
3.5 ANALYSIS OF PERFORMANCE ON LONGER VIDEO DATASET
To explore the impact of video length, we conducted experiments using the ActivityNet (Krishna et al. 2017a) cap-
Table 4: Performance over various video length. We use
backbone as STTann.
Training Dutaset (Caplimn)
Methad
Langth
With Censtraint
No Constraint
Mron
tion dataset (average length: 117.3 sec- onds), which is approximately 4 times
RR:30
R8850
RP:50
Cinorr
WS-tigggGG NL-V9GG
10:01
13
12:83
19:60
15.92
9:00
22.56
14:15
11:48
18.42
longer than the Action Genome caption dataset (average length: 29.9 seconds) and 12 times longer than the MSVD dataset (average length: 9.5 seconds). As shown in Table we made the following two observations: 1) Regardless of video length, our proposed method consistently outperformed the naive approach (i.e., WS- ImggGG.) This indicates that our proposed method remains effective across videos of var- ious lengths. 2) When comparing the performance between the MSVD and ActivityNet datasets, aside from the benchmark dataset (i.e., Action Genome), we observed that both WS-ImgSGG and NL-VSGG achieved better performance on the longer ActivityNet dataset compared to the shorter MSVD dataset. We attribute it to the fact that longer videos allow the model to learn more diverse video content, thereby improving generalization, and provide more supervision as the duration of actions increases. In this context, despite the shorter video length of the Action Genome dataset compared to the ActivityNet dataset, our proposed method performs better on the Action Genome dataset. This is because the video distribution in the Action Genome dataset is more closely aligned with the test set, which is derived from Action Genome.
MSVD
W-SJmg55G NL-VSGG
9.5 seconds
6:22
9.8
11.31
8.00
18.22
7.69
16.00
12.31
11.50
8.56
Activing
WS
VLGV
seconds
10.85
1394
12.60
3.6 QUALITATIVE RESULTS FOR EXPANSION OF ACTION CLASSES
Recall that we can allow VidSGG model trained by NL-VSGG to be able to predict broader range of action classes that are not included in the train- ing data (i.e., AG dataset), as our framework utilizes the video captions weak supervision. To validate this, we perform qualitative analysis with STTran back- bone on the AG test set. Fig. shows qualitative re- sults for scene graphs gen- erated under the following classes") that are obtained through the class mapping process described in Section 2.4 and 2) ex- panded action classes (i.e., 500 classes in tota("o that are obtained without the class mapping pro- cess. In Fig. a)) we observe that the expansion of action classes enhances the temporal coher- ence of video scene graphs in that in the 3rd frame, the prediction of walking into/open under ex- panded action classes carries stronger temporal implication compared to the prediction of standing on/holding under constrained classes. Furthermore, we observe that in Fig. 5])) the expansion of action classes allows for predicting close, which presents the opposite meaning to open in Fig, (ta), while the VidSGG model trained the constrained classes still generate holding. This indicates that the expansion of action classes with temporal implication significantly benefits VidSGG mod- els, which are capable of capturing temporal context, thereby generating video scene graphs with enhanced temporal coherence. this vein, we believe that our proposed framework facilitates the training of VidSGG models with temporal coherence.
[N)
Cipandedd Actia Class saltine
Copanded Asd Cliis Nookting
Dosi sold Actice Olus
Corotre Astter Chers
Expondedd
Lat frams
2nd frame
3rd frurne
Figure 5: Qualitative results of NL-VSGG for broader range of action classes. The red-coloree texts indicate predicates with novel meanings that are not present in the AG dataset.
two conditions: an VidSGG model trained on 1) constrained action
9 "Among the three types of action classes (i.e., attention, spatial and contacting). we generate scene graphs with the contacting class type, which conveys more descriptive information than the other two types. 1oo filter out noisy classes, select the top 500 most frequent action classes from 2,325 action classes.
Refer to Appendix reegarding for more information on 500 action classes.

---

Published as a conference paper at ICLR 2025
4 RELATED WORK
4.1 VIDEO SCENE GRAPH GENERATION (VIDSGG)
Video Scene Graph Generation (VidSGG) aims to learn the spatio-temporal dependencies of a video to predict the dynamic relationships between all object pairs. Existing VidSGG models are cate- gorized into two settings based the granularity of the generated video scene graph: video-level
VidSGG (Shang et al.,
2024) and frame-level 2017 Tsai et al. 2019 Shang et al. 2021 Wei et al.
VidSGG (Cong et al., 2021 Feng et al. 2023; Teng et al. | et al. 20226). Video-level
VidSGG models generate global scene graph for video clip, while frame-level VidSGG mod-
els generate a scene graph for each frame in a video clip. Note that our work follows the frame- level VidSGG setting. Following the release of the Action Genome dataset (Ji et al., 2020), frame- level VidSGG models have been actively researched. Specifically, STTran (Cong et al. (021) em- ploys two separate transformers to capture spatio-tempores dependencies of objects and frames, and TRACE (Teng et al. 202I) proposes a hierarchical relation tree method to enhance spatial-tempont reasoning. DSG-DETR (Feng et al. 2023) mitigates the complexity issue arising from the fully- connected graph between frames, enabling the capture of long-term -erm temporal context. Recently, TEMPURA (Nag et al. 2023) and FloCoDe (Khandelwal, (024) aim to alleviate the long-tailed predicate issue through memory-guided as these methods heavily rely on costly annotations on all frames, we propose a weakly supervised approach for VidSGG, which utilizes the readily available video captions, i.e., language supervision, for the first time.
learning and label correlation loss, respectively. However,
4.2 WEAKLY SUPERVISED SCENE GRAPH GENERATION
Weakly Supervised ImgSGG (WS-ImgSGG). To relax a heavy reliance on the costly annotation
of a fully supervised approach, ImgSGG utilizes two types of weak supervision. 1) Unlocalized scene graph: It uses ground-trune
scene graphs represented in text format, which have not yet been
grounded to bounding boxes. In this regard, related studies (Ye & Kovashka 102 Shi et al. 202T Zareian et al. 2020 Li et al. 2022a) have focused on aligning these text-based scene graphs with suitable bounding boxes. LSWS (Ye & Kovashka )021) utilizes the linguistic structure within triplets for grounding, while another study (Shi et al. 2021 proposes graph-matching module based on contrastive learning to further improve the grounding performance. 2) Language super- vision: In order to further relax the annotation costs associated with unlocalized scene graphs. the natural language description (i.e., image caption) used for training the model (Zhong et af. [002T] Kim et al, 2024b] Zhang et al. 2023). SGNLS (Zhong et al. 202T) is the first to enable model tran- ing with image captions. LLM4SGG (Kim et al., 20244))))) addresses semantic over-simplification and low-density scene graph issues arising from the triplet generation process.
Weakly Supervised VidSGG (WS-V. As the first WS- VidSGG approach. PLA CCen eet al. 2023) utilizes an unlocalized scene graph of the middle frame in a video clip as weak super- vision train a VidSGG model. However, the assumption of ground-truth scene graph existing in the middle frame among other frames is not only unrealistic but also still requires manual human an- notation. On the other hand, our proposed NL-VSGG framework only relies on the readily available video captions to train a VidSGG model, which further reduces annotation costs.
CONCLUSION
In this work, we propose weakly supervised VidSGG with Natural Language Supervision (NL- VSGG) that relieve annotation costs for VidSGG, which is the first time to enable training a VidSGG model with readily available video captions. We identify two key reasons for why a simple adoption of the WS-ImgSGG pipeline to VidSGG fails, i.e., temporality within video captions and variability of action duration. Our Temporality- aware Caption Segmentation (TCS) module captures the tempo- rality within the video captions, while Action Duration Variability-aware Caption-Franc Alignment (ADV) module addresses the variability in the action duration. Furthermore, we propose a novel pseudo-labeling strategy based on motion cues (PLM) to deal with negative action classes in the Action Genome dataset. Our extensive experiments on the Action Genome dataset demonstrate the superiority of NL-VSGG over the simple adoption of WS-ImgSGG pipeline to VidSGG. As a fur- ther appeal of NL-VSGG, it allows the VidSGG models to predict broader range of action classes that are not included in the training data, which makes our proposed framework practical in reality.
However, NL-VSGG has a limitation regarding untrimmed. durrtddn videos. For detailed in- formation related to this and the future work addressing it, please refer to the Appendix.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea governmen((MSIT) (RS-2022-11220007, Reasoning, and Inference from Heterogeneous Data), as well as another IITP grant funded by the Korea govern- ment(MSIT) (No. (N0.2020-0000004)) Furthermore, this research was supported in part by the NAVER- Intel Co-Lab. The work was conducted by KAIST and reviewed by both NAVER and Intel.
REFERENCES
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances neural information processing systems, 33:1877-1901. 2020.
David Chen and William Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th annual meeting of the association for computational linguisties: human language technologies, pp. 190-200, 2011.
Siqi Chen, Jun Xiao, and Long Chen. Video scene graph generation from single-frame weak super- vision. In The Eleventh International Conference on Learning Representations, 2023.
Meng- Jiun Chiou, Chun- Yu Liao, Li-Wei Wang, Roger Zimmermann, and Jiashi Feng. St-hoi: A spatial-temporal baseline for human-object interaction detection videos. In Proceedings of the 2021 ACM Workshop on Intelligent Cross-Date Analysis and Retrieval, pp. 9-17, 2021.
Yuren Cong, Wentong Liao, Hanno Ackermann, Bodo Rosenhahn, and Michael Ying Yang. Spatial- temporal transformer for dynamic scene graph generation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 16372-16382, 2021.
Sivan Doveh, Assaf Arbelle, Sivan Harary, Roei Herzig, Donghyun Kim, Paola Boscaatec-Bonilie Amit Alfassy, Rameswar Panda, Raja Giryes, Rogerio Feris, al. Dense and aligned captions (dac) promote compositional reasoning in vl models. Advances in Neural Information Processing Systems, 36, 2024.
Shengyu Feng, Hesham Mostafa, Marcel Nassar, Somdeb Majumdar, and Subarna Tripathi. Ex- ploiting long-term dependencies for generating dynamic scene graphs In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5130-5139, 2023.
Jessica Ficler and Yoay Goldberg, Controlling linguistic style aspects in neural language generation. arXiv preprint arXiv: 100.:0770..23 2017.
Kaifeng Gao, Long Chen, Hanwang Zhang, Jun Xiao, and Qianru Sun. Compositional prompt tuning with motion cues for open-vocabulary video relation detection. arXiv preprint 230::2200000068. 2023.
Shalini Ghosh, Giedrius Burachas, Arijit Ray, and Avi Ziskind. Generating natural language ex- planations for visual question answering using scene graphs and visual attention. arXiv preprint arXiv: /902.0575. 2019.
Linjiang Huang, Liang Wang, and Hongsheng Li. Weakly supervised temporal action localization via representative snippet knowledge propagation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3272-3281. 2022.
Jaehyeong Jeon, Kibum Kim, Kanghoon Yoon, and Chanyoung Park. Semantic diversity-aware prototype-based learning for unbiased scene graph generation. In European Conference on Com- puter Vision, pp. 379-395. Springer, 2024.
Jingwei Ji, Ranjay Krishna, Fei-Fei, and Juan Carlos Niebles. Action genome: Actions com- positions of spatio-temporal scene graphs. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 10236-10247, 2020.
Albert Q Jiang, Alexandre Sablayrolles. Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.0688 2023.

---

Published as a conference paper at ICLR 2025
Anant Khandelwal. Flocode: Unbiased dynamic scene graph generation with temporal consistency and correlation debiasing. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2516-2526, 2024.
Kibum Kim, Kanghoon Yoon, Yeonjun In, Jinyoung Moon, Donghyun Kim, and Chanyoung Park. Adaptive self-training framework for fine-grained scene graph generation. In The Twelfth Inter- national Conference on Learning Representations, 2024a.
Kibum Kim, Kanghoon Yoon, Jachyeong Jeon, Yeonjun In, Jinyoung Moon, Donghyun Kim, and Chanyoung Park. Llm4sgg: Large language models for weakly supervised scene graph genera- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 28306-283 16, 2024b.
Ranjay Krishna. Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In Proceedings of the IEEE international conference on computer vision, pp- 706-715, 2017a.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting lan- guage and vision using crowdsourced dense image annotations. International joturnalionaaouuralll of computer vision, 123:32-73, 2017b.
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang. and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint #xXiv:3355006555 2023a.
Lin Li, Jun Xiao, Guikun Chen, Jian Shao, Yueting Zhuang, and Long Chen. Zero-shot visual relation detection via composite visual cues from large language models. Advances in Neural Information Processing Systems, 36, 2024.
Mengze Li, Han Wang, Wenqiao Zhang, Jiaxu Miao, Zhou Zhao, Shengyu Zhang, Wei Ji, and Fei Wu. Winner: Weakly-supervised hierarchical decomposition and alignment for spatio-temporal video grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23090-23099, 2023b.
Xingchen Li, Long Chen, Wenbo Ma, Yi Yang, and Jun Xiao. Integrating object-aware and interaction-aware knowledge for weakly supervised scene graph generation. In Proceedings of the 30th ACM International Conference Multimedia, pp. 4204 4213, 2022a.
Yiming Li, Xiaoshan Yang, and Changsheng Xu. Dynamic scene graph generation via anticipa- tory pre-training. In Proceedings of the IÉEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1387411383, 2022b.
Huaishao Luo, Lei Ji. Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing, 508: 293-304, 2022.
George A Miller. Wordnet: lexical database for english. Communications of the ACM, 38(11): 39 41, 1995.
Sayak Nag, Kyle Min, Subarna Tripathi, and Amit K Roy- noy-Chowdh Unbiased scene graph gen- eration in videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22803-22813, 2023.
Zhifan Ni. Esteve Valls Mascaró, Hyemin Ahn, and Dongheui Lee. Human-objer interaction prediction in videos through gaze following. Computer Vision and Image Understanding, 233: 103741, 2023.
OpenAl. Chatgpt.
2023.
Haoruo Peng, Daniel Khashabi, and Dan Roth. Solving hard coreference problems, arXiv preprint 1rX.v: 2019.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188):, 2019.
12

---

Published as a conference paper at ICLR 2025
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, PP- 8748-8763. PMLR, 2021.
Hamid Rezatofighi, Nathan Tsoi, Jun Young Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: A metric and a loss for bounding box regression. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 658-666, 2019.
Brigit Schroeder and Subarna Tripathi. Structured query-based image retrieval using scene graphs. CVPR Workshops, pp. 178-179, 2020.
Sebastian Schuster, Ranjay Krishna, Angel Chang, Li Fei-Fei, and Christopher D Manning. Gen- erating semantically precise scene graphs from textual descriptions for improved image retrieval. In Proceedings of the fourth workshop on vision and language, pp. 70-80, 2015.
Xindi Shang, Tongwei Ren, Jingfan Guo, Hanwang Zhang, and Tat-Seng Chua. Video visual relation detection. In Proceedings of the 25th ACM international conference on Multimedia, pp. 1300- 1308, 2017.
Xindi Shang, Yicong Li, Junbin Xiao, Wei Ji, and Tat-Seng Chua. Video visual relation detection via iterative inference. In Proceedings of the 29th ACM international conference on Multimedia, pp. 3654-3663, 2021.
Jing Shi, Yiwu Zhong, Ning Xu, Yin Li, and Chenliang Xu. A simple baseline for weakly-supervised scene graph generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16393-16402, 2021.
Yao Teng, Limin Wang. Zhifeng Li, and Gangshan Wu. Target adaptive context aggregation for video scene graph generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13688-13697, 2021.
Yao-Hung Hubert Tsai, Santosh Divvala, Louis- Chiis-Philippe Morency, Ruslan Salakhutdinov, and Ali Farhadi. Video relationship reasoning using gated spatio-temporal energy graph. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 10424-10433, 2019.
Lan Wang, Gaurav Mittal, Sandra Sajeev. Ye Yu, Matthew Hall, Vishnu Naresh Boddeti, and Mei Chen. Protege: Untrimmed pretraining for video temporal grounding by video temporal ground- ing. In Proceedings of the /EFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 6575-6585, 2023.
Shuang Wang, Lianli Gao, Xinyu Lyu, Yuyu Guo, Pengpeng Zeng, and Jingkuan Song. Dynamic scene graph generation via temporal prior inference. In Proceedingy of the 30th ACM Interna- tional Conference on Multimedia, pp. 7793-5801, 2022a.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- cry, and Denny Zhou Self-consistency improves chain of thought reasoning in language models. arXiv preprint 1tX.Xx2223331117. 2022b.
Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint
2022c
Meng Wei, Long Chen, Wei Ji, Xiaoyu Yue, and Roger Zimmermann. In defense of clip-based video relation detection. IEEE Transactions on Image Processing, 2024.
Li Xu, Haoxuan Qu, Jason Kuen, Jiuxiang Gu, and Jun Liu. Meta spatio-temporal debiasing for video scene graph generation. In European Conference on Computer Vision, pp. 374-390. Springer, 2022.
Shen Yan, Xuehan Xiong, Arsha Nagrani, Anurag Arnab, Zhonghao Wang, Weina Ge, David Ross, and Cordelia Schmid. Unloc: A unified framework for video localization tasks. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 13623-13633 2023.
13

---

Published as a conference paper at ICLR 2025
Xu Yang, Kaihua Tang. Hanwang Zhang, and Jianfei Cai. Auto-encoding scene graphs for image captioning. CVPR, pp. 10685-10694, 2019.
Keren Ye and Adriana Kovashka. Linguistic structures as weak supervision for visual scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 8289-8299, 2021.
Kanghoon Yoon, Kibum Kim, Jinyoung Moon, and Chanyoung Park. Unbiased heterogeneous scene graph generation with relation-aware message passing neural network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 3285-32944 2023.
Kanghoon Yoon, Kibum Kim, Jaehyung Jeon, Yeonjun In, Donghyun Kim, and Chanyoung Park. Ra-sgg: Betrieval-augment scene graph generation framework via multi-protorpe learning.
arXiv preprint ar 2024.
Hangjie Yuan, Jianwen Jiang, Samuel Albanie, Tao Feng, Ziyuan Huang, Dong Ni, and Mingqian Tang. Rlip: Relational language-image pre-training for human-object interaction detection. Ad- vances Neural Information Processing Systems, 35:37416-37431, 2022.
Hangjie Yuan, Shiwei Zhang, Xiang Wang, Samuel Albanie, Yining Pan, Tao Feng. Jianwen Jiang, Dong Ni, Yingya Zhang, and Deli Zhao. Rlipv2: Fast scaling of relational language-image pre- training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 21649-21661, 2023.
Alireza Zareian, Svebor Karaman, and Shih-Fu Chang. Weakly supervised visual semantic parsing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 3736-3745, 2020.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5579-5588, 2021.
Yong Zhang, Yingwei Pan, Ting Yao, Rui Huang. Tao Mei, and Chang- Wen Chen. Learning to generate ginguage-superiseed and open-vocabulary scene graph using pre-trained visual-semantic space. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2p.2915-2292 2023.
Shu Zhao and Huijuan Xu. Less more: Toward zero-shot local scene graph generation via foun- dation models. arXiv preprint 3rXiv:2310.0/3555 2023.
Minghang Zheng, Yanjie Huang, Qingchao Chen, Yuxin Peng, and Yang Liu. Weakly supervised temporal sentence grounding with caussian-based contrastive proposal learning. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15555-15564, 2022.
Yiwu Zhong, Jing Shi, Jianwei Yang, Chenliang Xu, and Yin Li. Learning to generate graph from natural language supervision. Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 1823-1834, 2021.
14

---

Published as a conference paper at ICLR 2025
Supplementary Material
Weakly Supervised Video Scene Graph Generation via Natural Language Supervision
Details of Prompt
16
Ablation Study of Prompt Designed for Coreference Resolution
16
Experiment on the VidHOI dataset
16
Replacing ChatGPT and DAC with other models
17
Experiment for Selection Strategy of Pseudo-Labeling
17
Experiment for Different Scene Graph Parsing Approach
18
Experiment for Integration of Dynamic Relationships into RLIP
18
Qualitative analysis of the ADV module
18
Ablation Studies the AG dataset without negative classes
19
J Hyperparameter Sensitivity
19
K Details of Expanded Action Classes
19
Cost for Utilization of Large Language Model
20
M Experiment for Different Clustering in the ADV module
20
Experiment Combining weakly supervised and fully supervised datasets
20
Discussion of Improvement for Parsing and Grounding
20
Discussion of the Steepest Decline in the ADV module
21
Additional Related Work
22
Large Language Model based Scene Graph Generation
22
Future Work
22
15

---

Published as a conference paper at ICLR 2025
SAST DESCRIPTION
multiple compositioal caprion, Considerrng woeds that
events leg. then, while, and after, your job given video caption specily the objects prosoune
aronng chronological
CONTEXT EXAMINES
The person turning the vtow. He then begins to sitr food and atter that picke camera and look at pick
Dutiutt person ieturing
risve. The penan din feod, the
The person laska at
Ieputt person siting in bed and bocing on phone while holding blanket, The person the phone down and pulls the blanket
Ourpun A person is drting in bed. + person texting phone white holding a baaket The person puns the phone down. + The person pulls the Wonker up. Isputs A person picka a phone and enters the buthroom through doorway while talking on the phone. Before walking out the room, the parson puts on shoss and pids up clothes while laughing and draises thems
Dutpurt peronn picks up phone + pemonn entern the bathrom thmugh doonway while talling on the phone. The person puts thons. + The person pidk up detttee while taughing. The person dresss clether. The person walk out mom.
isputt posson siting on tolet pidl up phone and buttory that are on ground, puts the battery into the phone, tales jacket, thin stands takes seffies against bathroon
person
ground.
apainst
undressing picka Outputs A person undressing persen picks towe. + person deans some glasses + person takes drink some plasses. Ieputt Person pulls phone and begits playing with then sets down and pulss the Maariet further up.
Refore taking drink,
and cleans
Dutpurt Penson pullt out phone. + Person plays with the phone. + Person unts the phone down. Person pulls the blanket further up. Inputs A parion watching telestiion and axting sandwich while laying on the floor and reading baok, aftera while the person gats up to grib box. Ourport person watches television and eats sandwlth while laying the floor + person readi book. person gets up grab lew.
Ieputt perton walksto pantrry takes out some clothes from it, tosses one on the flane and puts on another after taklng off spat.. person a person pati
Dutsurt person wilks to pantry. + person takes cloth.
dothes pantry. A person tosscs doth on
Figure 6: Prompt used in the Temporality aware Caption Segmentation (TCS) module.
A DETAILS OF PROMPT
In Fig. 6 we provide the complete prompt used in the TCS module, which is discussed in Section 2.2 of the main paper.
B ABLATION STUDY OF PROMPT DESIGNED FOR COREFERENCE RESOLUTION
In this section, we conduct experiment to validate the impact of the prompt introduced to address corefer- ence resolution, which aids in specifying the pronouns and extracting accurate triplets. That is, we remove the following sentence in task description of Fig. 6 Note that you should specify the objects for the pro- nouns used each of these sentences., and evaluate NL-VSGG without As shown in Table 5
Table 5: Impact of the prompt designed for coreference resolution.
Method
Wilhs Contrrint R
R
No Condirsint R
R8:50
Complete Prorge
15.61
19.60
15.00
22:56
ewil Corclerence Resolation
15.48
19.09
15,90
21.76
the performance with the prompt designed for coreference resolution (i.e., Complete Prompt) is su- perior compared to the performance without that prompt (i.e., w/o Coreference Resolution). It is attributed to the increase in the number of triplets from 58K to 82K (i.e., 40% increase) when using the Complete Prompt, resulting in the alleviation of lack of supervision Kim al. (2024b). This demonstrates the effectiveness of our prompt design for coreference resolution.
C EXPERIMENT ON THE VIDHOI DATASET
Dataset. The VidHOI (Chiou et al. 2021 dataset consists of real-life videos capturing daily human activities without a scripted narrative. This dataset includes manually annotated scene graphs on based on keyframes sampled at 1 FPS. Following the processing step of prior studies (Ni et al. 2023 Chiou et al. 2021). we obtain the training and test sets contain 6,366 and 756 videos along with 193,911 and 22.976 frames, respectively. There are 78 object classes and 50 predicate classes, which are divided into 8 spatial classes and 42 action classes. Note that since the VidHOI dataset does not include video captions, we employ a video captioning model (i.e., VideoChat (Li et al.) 2023ab) to generate video captions.
Table 6: Results on VidHOr dataset.
Results. Table 6 shows performance comparisons under natural language supervision between a sim- ple adoption of WS-ImgSGG pipeline (i.e., WS- ImgSGG) and our proposed framework (i.e.. NL- VSGG) on the VidHOI dataset. Beyond the com- parisons made within the Action Genome dataset in the main paper, also observe that in the
Backhone
Methed
Superridonn
Re:20
R
R3520
Vaezila
Fall
19:46
20.30
25.65
25.05
STTEu
-V80
10.41
10.96
31.48
22.16
16

---

Published as a conference paper at ICLR 2025
VidHOI dataset, NL-VSGG continues to outperform the WS-ImgSGG method, further validating the effectiveness of our proposed framework.
D REPLACING CHATGPT AND DAC WITH OTHER MODELS
In this section, we replace the LLM used in the TCS module (i.e., CharGPT) and the vision-language model used in ADV (i.e., DAC) with alternative models, and use the same prompts shown in Fig. 6 to evaluate the performance. Table 7 shows the results, where Row (a) the performance of the baseline WS-ImgSGG model, and Row (b) is the performance of the original NL-VSGG reported in the main paper.
Table 7: Performance with other models.
Rin
Medelc
Wai Csnimiia K8-20
K4:20
B0-20
30.01
12.3
Chaadir CRaRPPT
15.11
19.00
15:00
22.56
15.38
13.66
15:00
31:48
Replacing an LLM (i.e., ChatGPT) with a smaller LM (i.e., Mistral-7B) in TCS module. Row (c) shows the performance of replacing ChatGPT- 175B OpenAl (2023) with a smaller Mistral-7B Jiang al. (2023). We observe that although the performance based on the smaller LM is inferior to that of ChatGPT as expected, it still outperforms WS WS-ImgSGG baseline shown in Row (a). This demonstrates the effectiveness of our proposed NL-VSGG framework.
Replacing a vision-language model (i.e., DAC) with a video-language model (i.e., Intern Video) in ADV module. Within the ADV module, we raise the question: is it sufficient to use a vision- language model, which is trained on image-text pair datasets, for aligning segmented sentences with frames while capturing the temporal context between frames? To explore it, we conduct experi- ments by replacing the image- based languaelanguage model (i.e., DACCDovel et al. (2024)) with a video-language model (i.e.. Intern Intern Video Wang et al. (2022c)) ). Specifically, to explicitly reflect tem- poral context using Vntern we group two consecutive frames without overlap and encode each grouped frame to obtain a visual representation, followed by performing Clustering basdd Caption- Frame Alignment, as discussed in Section 2.3 of the main paper.
As shown in Table 7 we observe that the performance based on the Intern Video (Row (d)) is com- petitive with that based on the DAC (Row (b)). It demonstrates that it is sufficient use vision- language model for aligning segmented sentences with frames while capturing the temporal context by simply averaging the frames. i.c., centroid in cluster. This result is consistent with prior research, i.e., CLIP4CLIP Luo et al. (2022), which demonstrates that simply averaging frame vi- sual representations yields competitive results compared to an advanced model reflecting temporal context between frames.
E EXPERIMENT FOR SELECTION STRATEGY OF PSEUUDO-LABELING
Here, we conduct experiments over various selection strategies for assigning pseudo-labels of negative classes (i.e., not looking at and not contacting) within unaligned frames, which is mainly addressed in Section 2.2 of the main paper. Specifically, these strategies involve assigning negative classes to the start (S), end (E), or both start and end (SE) frames for each negative class. The experiment results are shown in Fig. rreported with @550 in With Constraint setting. For not looking at. we observe that the performance of assigning it on the end frame inferior compared to the performance of the start frame (2nd vs. 3rd row), while the performance of start+end frames is best (1rd row). It indicates that the start frame for assigning not looking at provides more confident supervision than assignment on the end frame, and assignment on both frames is most beneficial.
19.53
19.60
not tooking
19:09
19.25
19.05
164
19.20
19.31
19.48
contacting Figure 7: Performance over diverse selection strategies.
For not contacting, in contrast to not looking at, observe that the performance of assigning it on the start frame is inferior compared to the performance of the end frame (1Td vs 2nd column), while the performance of start+end frames is best (3td column), except when not looking at is exclusively labeled on the end frame (2th row with 3rd column). It indicates that the end frame rather provides more confident supervision than the start frame, while assignment on both frames is generally confident. We attribute the exceptional case to the noisy supervision, where not looking at is only assigned on the end frame.
17

---

Published as a conference paper at ICLR 2025
22.6
19.6
8@00 (WC)
22.4
190
19.4
222
16.3
180
19.2
22.0
17.
29.0
210
19
21.6
18.8
21.4
15.5
2015)
201%
Figure 8: Hyperparameter sensitivity of 3. WC and NC stand for With Constraint and No Con- straint setting, respectively.
Figure 9: Hyperparameter sensitivity of a. WC and NC stand for With Constraint and No Con- straint setting, respectively.
F EXPERIMENT FOR DIFFERENT SCENE GRAPH PARSING APPROACH
In this section, we conduct experiments regarding the dif- ferent approaches for scene graph parsing discussed in Section 2.4 of the main paper. Specifically, there approaches to scene graph parsing: one involves extract- ing triplets and aligning classes with those of our interest based on an LLM Kim et al. (2024b) (LLM-bbseed ap-
Table 8: Performance over diverse
scene graph parsing approaches.
graph parding
Construint
1. M--o-aod apprunch
1561
19.0
15.92
22.56
SG Panser XB appreach
11.08
14.34
11,26
16.51
") [T] while the other relies on a rule-based scene graph parser Schuster et al. (2015) with knowledge base-based Miller (1995) alignment (SG parser+KB approach). Note that for the scene graph parsing, these two approaches are widely adopted in the realm of WS-ImgSGG Kim et al. 20244b;; Zhong et al. (202 1): Zhang et al. ); Ye & Kovashka (2021). In Table 8 we observe that the performance of the SG KB approach inferior compared the LLM-based ap- proach. It indicates that the heuristic rule-based parser for triplet extraction from video captions is ineffective, and knowledge base-based based alignment struggles to accurately map the classes with com- plicated action classes (e.g.. drinking from, in front of) in VidSGG. In this regard, we demonstrate that the LLM-based approach is particularly effective at WS-VidSGG due to the complex structure of video captions and complicated action classes. To further facilitate research in the WS- VidSGG, we make the triplets extracted based on the LLM publicly available.
G EXPERIMENT FOR INTEGRATION OF DYNAMIC RELATIONSHIPS INTO RLIP
In Table of the main paper, we observe that RLIP
Table 9: Performance of RLIP Yuan al.
Yuan et al. (2022) exhibits subpar performance un-
2022) trained with dynamic rel relationships.
der the zero-shot setting due to its inability to pre- dict dynamic relationships. To explore potential performance enhancement achievable by integrat- ing dynamic relationships into RLIP, we conduct experiments by fine-tuning a pre-trained RLIP on the Action Genome dataset. Furthermore, we ap- pend the STTran Cong al. (2021) module to RLIP in order facilitate capturing temporal context. As shown Table 9 we observe that RLIP trained with dynamic relationship substantially boosts performance. Moreover, RLIP with the STTran module further enhances performance by capturing temporal context. It indicates that the incorporation of dynamic relationships crucial VidSGG. In summary, we demonstrate the importance of incorporating dynamic relationships and reflecting temporal context in VidSGG.
Modd
Supervision
With Construint
RV-00
No Cunstraint
KW720
STTTma
33.98
36.93
RLIP
RLIPISTIas
Rell (Fine-mme)
40.04
36.26
41,00
RLIP
Zero shot
7.93
9.16
13,80
H QUALITATIVE ANALYSIS OF THE ADV MODULE
To qualitatively evaluate the effectiveness of the ADV module, visualize the results of clustering video frames in the visual space using T-SNE, and show the distribution of their visual-language (VL.) scores, i.e., the similarity scores between each segmented sentence and the cluster centroids. In Figure 10 we observe that the case where cluster clearly separated (See A), the VL score distribution is concentrated within that cluster, resulting in the selection of a single cluster. On the
In the main paper, we L.M approach for scene graph parsing.
18

---

Published as a conference paper at ICLR 2025
person eating sandwich
VL Score Distribution
(A)
6C11
G.b
(C3)
(C4) 17-19
(C5) 20-21
6.0
Cluste
The peroo is taking dathes of box
Prosabiiy 6.6
Seleted
(8)
Visual Space
CRUCN
Figure 10: Qualitative analysis of the ADV module.
other hand, in the case where visual features are concentrated but distributed across multiple clusters (See B), the ADV module adaptively selects multiple clusters. This indicates that the ADV module reflects the variability in action duration, allowing it to supervise the model accurately and thereby improve performance.
I ABLATION STUDIES ON THE AG DATASET WITHOUT NEGATIVE CLASSES
In Section of the main paper, we observed rel- atively significant performance improvement in the PLM module due the dominance of the negative class in the dataset. Therefore, to further clarify the effectiveness of the TCS and ADV modules, perform ablation studies which negative classes excluded from both training and test sets in the Action Genome (AG) dataset. As shown in Table 10 we observe that even in a dataset where the PLM module cannot be applied, TCS and ADV modules still show superiority, proving their effec- tiveness.
Table 10:
Ablation studies where negative
classes are excluded from the AG dataset.
ICS ADV
With Constraint
No Constraint
Mean
RG 30
R8500
R@
R8:50
R@920
RG:50
12.98
17
13.28
13.13
15.47
15.41
15.95
20.74
16.19
23,04
16.07
21.89
J HYPERPARAMETER SENSITIVITY
In NL-VSGG, two hyperparameters are used: / for defining K (i.e., [V]) in ADV module (Section 8 2.3), and a for assigning negative classes within unaligned frames in PLM module (Section 2.5).
We analyze the sensitivity of these hyperparameters.
Hyperparameter for ADV module. In Fig. g we conduct experiments over various 3s. We ob- serve that the performance decreases with large which reduces the cluster number K. It indicates that large 3 (i.e., small K) cannot capture the fine-grained variability of action duration, resulting in deteriorating performance. On the other hand, small / (i.e., large K) helps to capture fine- fine-grained variability of action duration, leading increasing performance. However, rather small / (i.e., 3) decreases performance in No Constraint setting. We attribute it to the fact that rather small divides the frames into overly fine- fine-grained clusters, making difficult for the vision- language model dis- tinguish highly discriminative clusters terms of similarity scores. this regard, it beneficial to appropriately set B as 4 to capture the variability of action duration.
Hyperparameter for PLM module. In Fig, observe that the performance consistently increases up to 5%, followed by fluctuation beyond 5%. is worthwhile noting that the pseudo-labels with smaller would provide more confident supervision models since their associated subject and object are distinctly getting farther over time. It suggests that up to 5%, clear supervision is provided for negative classes within unaligned frames, but beyond that ratio, noisy supervision is included. Hence, setting around 5% is preferable. However, opt for value of a at 15% as marginally enhances performance under the With Constraint setting.
conduct experiment over various We
K DETAILS OF EXPANDED ACTION CLASSES
In Table. 14 enumerate all the expanded action classes sorted by frequency in descending order.
19

---

Published as a conference paper at ICLR 2025
Table 11: Summarization of cost for LLM.
Module
Num. Output/Input tokens per video Cost per video
TCS
0.045k / /0.68k
$0.00041
L COST FOR UTILIZATION OF A LARGE LANGUAGE MODEL
For the TCS module, the cost of using ChatGPT OpenAI ChennA (2023) summarized in Table I1 Given that the cost of input tokens and output tokens is $0.5 and $1.5 per 1M tokens, respectively, the cost per video is computed by (680/IM) x 0.5 + (45/IM) 1.5.
M EXPERIMENT FOR DIFFERENT CLUSTERING IN THE ADV MODULE
To investigate the impact of differ- ent clustering strategies within the
Table 12: Performance over various clustering strategies.
ADV module, we conducted exper- iments where we replaced the K- means clustering strategy with Ag- glomerative clustering and Gaussian Mixture Model (GMM) clustering strategies. As shown in Table 12 we observed that other clustering strategies exhibit competi- tive performance compared to that of the K-means clustering strategy, indicating that our proposed framework is robust to other clustering strategies. Another observation is that the performance with the GMM clustering strategy is relatively better on average. This result aligns with previous stud- ies (Huang et al.) 2022; Zheng et al.) (2022) that assume proposal distributions Gaussian in the temporal grounding task in that GMM also assumes the same thing, resulting in effective clustering within the ADV module.
Clustering Algorithm
With Constraint
No Constraint
Mean
R@ 20
R@50
R@200
R@50
K-Means Agglormerative
15.61
19.60
15.92
22.56
18.24
15.78
19.69
16.12
23.01
18.65
GMM
15.31
19.80
15.85
23.93
18.72
N EXPERIMENT COMBINING WEAKLY SUPERVISED AND FULLY SUPERVISED DATASETS
In this section, we investigate a scenario where our proposed approach meets with the fully su- pervised approach. To this end, we attempted to fine-tune model, which was initially trained on a weakly supervised dataset (i.e., pseudo-localized scene graphs in Section 2.5), to the fully super- vised dataset, i.e., ground-truth localized scene graphs. In this process, we assumed that model trained on a larger dataset would provide more effective weight initialization, leading us to leverage the model trained on both the AG caption and MSVD caption datasets, as detailed in Section 3.6 the main paper. Interestingly, as shown in Table 13 we observed that the performance of the model fine-tuned to the AG dataset significantly outperformed that of the model initially trained on the AG dataset, implying that our proposed framework can synergize with the fully supervised approach.
Table 13: Performance for a combination of weakly supervised and fully supervised data.
Training dataset
With Contraint | No Constraint Moan
R0
R.000
R
AG (all
36.93
20
48 88
39.00
AGEMSVD (Weak) - (Full)
34.84
37.64
49.55
40.11
O DISCUSSION OF IMPROVEMENT FOR PARSING AND GROUNDING
For future work, we discuss the techniques developed for scene graph parsing and grounding, de- tailed in Section 2.4 of the main paper.
Improvement for Scene Graph Parsing, We may employ the ensemble approach (Wang et al.) 20226) to extract more high-quality scene graphs. Specifically, for an LLM--bsed parser, we can utilize temperature sampling or top-k sampling extract diverse triplets for each segmented sen- tence, followed by taking a majority vote over diverse triplets to extract the most consistent triplet. With this technique, can further develop the scene graph parsing over the state-of-the thhe----tth-tttttt method.
Improvement for Scene Graph Grounding. We may improve the grounding accuracy. Basically, a triplet is grounded to a bounding box when the bounding box's 's class matches the object class of the triplet parsed from the caption. However, inherent challenges in videos such as motion blur, fast
20

---

Published as a conference paper at ICLR 2025
Gap of Steepest Decline
1.0
8
19.8
Steepest Dedine
0.8
19.6
19.4
Recal
19.2
0.4
19.0-
18.8
0.0
18.6
6
11 12 14
0.2
0.6
0.8
Number of Total Cluster (K)
Threshold
Figure 11: The gap in the similarity point the steepest decline over the number of total clusters (i.e., K), which is proportional to the frames.
at the
Performance across various fixed thresholds in the ADV module, conducted under the With Constraint setting.
Figure 12:
movement, and occlusion often hinder accurate object class classification within the bounding box, resulting in grounding failures. To address it, we can utilize adjacent frames successfully grounded. Specifically, in the case where the object class of the bounding box in the target frame is ambiguous so that grounding fails, we can ensure grounding by selecting a bounding box with high IoU and visual similarity to bounding box of an object that is grounded in adjacent frame. This technique could compensate for the failures in the target frame, ensuring more reliable grounding.
P DISCUSSION OF THE STEEPEST DECLINE THE ADV MODULE
In the ADV module, detailed in the Section 2.3 of the main paper, originally determined the relevant clusters depending on the steepest decline in the vision-language similarity score. Here, we explore the sensitivity of the steepest decline criterion to noise in the similarity score. To this end, analyze the gap in the similarity score at the point of steepest decling 12 over the total number of clusters (i.e., K), which is proportional to the video frames. As shown Figure 1T we observe that the gap in the steepest decline decreases the number of total clusters increases (from the left to the right on the x-axis). This is because the similarity score is distributed across a greater number of clusters, thereby reducing the gap. However, it is important to note that from eight clusters onward, the gap converges 0.2, demonstrating that even with noise in the similarity score, the steepest decline approach still reliably selects relevant clusters using this relatively large 0.2 gap.
In this regard, raise question: Why not determine the relevant clusters through point where the gap in the decline exceeds 0.2 instead of using the steepest decline? Specifically, we could sort the similarity scores in descending order, find the first point where the gap exceeds 0.2, starting conducted from the highest score, and select all the clusters preceding this point. Therefore, additional experiments where we set the threshold as hyperparameter, ranging from 0.2 to 0.8, to determine the relevant clusters through point where the gap in the decline exceeds this threshold. As shown in Figure [2, we had two observations: 1) Setting the threshold 0.2 shows competitive performance with the steepest decline, aligning with an aforementioned discussion that a 0.2 gap robust to determine the relevant clusters. 2) As the threshold increases, observe decrease in performance. This is because more irrelevant clusters are assigned as the threshold increases, thereby introducing noise.
However, we argue that this thresholding approach requires meticulous analysis of the gap in the steepest decline for each dataset to set the threshold, whereas the steepest decline approach has the advantage of adaptively capturing relevant clusters without needing to analysis each dataset.
For example, in Figure 10 of the Appendix, the gap in the similarity the point of the steepest decline C1-C2 for (A), and C4-C2 for (B).
21

---

Published as a conference paper at ICLR 2025
Q ADDITIONAL RELATED WORK
Q.1 LARGE LANGUAGE MODEL-BASED SCENE GRAPH GENERATION
As Large Language Models (LLMs) got a surge of interest from various domains, LLMs have also been applied to the SGG task to leverage the rich semantic knowledge. RECODE (Li et al. 2024) utilizes LLMs to generate an informative description for predicate, enabling to capture the fine- grained visual cues. For zero-shot scene graph generation task, ELEGANT (Zhao & Xu 2023) extracts the candidate relationships generated by the LLMs, which have profound reasoning and commonsense knowledge. For weakly supervised SGG, LLM4SGG (Kim et al.) 2024b) alleviates the long-tailed predicate issue and scarcity of datasets via the LLMs. In this work, we leverage the LLMs understand the temporality within video captions for the WS- VS-VidSGG task.
R FUTURE WORK
A potential limitation of our work that NL-VSGG is mainly designed for handling relatively short-length videos (~~10 seconds). As future work, we plan to generalize NL-VSGG to longer untrimmed videos. One possible direction would to perform untrimmed video temporal local- ization (Wang et al.] 2023) Yan et al. 2023 Li et al., 2023b) tasks in computer vision, wherein short-length clips are extracted from longer videos, and apply NL-VSGG to each clip for training a VidSGG model. This approach would enable the training of the VidSGG model with untrimmed longer videos.
Another direction for future work explore the possibility of using Multimodal Large Language Models (MLLMs) to extend open-set or multi-label prediction in the VidSGG task. Specifically, for open-set prediction, we can query the temporal relationship between subject and object to the MLLMs by inserting the union box and previous frames' caption for temporal context. For multi- label prediction, temperature (Ficler & Goldberg (017) or top-k sampling (Radford et al.) 2019). commonly used in the NL.P for generating diverse answers, can be used generate various temporal relationships.
22

---

Published as a conference paper at ICLR 2025
Table 14: Enumeration of all expanded action classes.
put, pick up, open, take, hold, sit on, eat, grab, close, look at, throw, take off, drink, walk into, sit in, place, on, sit at, watch, with, put on, pour, walk to, read, set, drink from, in, clean, carry, walk over to, stand in, walk through, turn on, fold, work on, take out, look out, sit, remove, play with, use, get, look in, lie sweep, talk on, from, wash, fix, sit down on, tidy up, enter, walk out of, leave, into. move, turn off, lay on, walk in, go to, out of, look into, walk down, at, vacuum, play on, sit down, stand in front of, cook, turn, look through, off, stand on, wipe, run into, have, snuggle with, drop, walk up, make, lay down on, stand at, front of, tidy, walk around, opening, check, do, pull out, toss, adjust, of, put down, pull, stand by, walk out, sit down at, shut, touch, open up, get up from, sit down in, pick, lie in, walk, hug, snuggle, lay in, onto, type on, look, walk across, drink out of, set down, sleep on, to, come into, flip through, write in, laugh at, walk up to, go through, stand, lay, lie down on, sneeze into, put away, talk to, wrap in, stir, sneeze, dry, wear, start, write, wrap, down on, reach for, stand next to, next to, hang, play, go into, stare at, retrieve, get out of, straighten, eat from, through, run through, run, grasps, walk towards, off of, smile at, under. look inside, answer, cook on, clean up, go over to, exit, fill, rum out of, laugh, wipe off, stand near, look out of, reach into, walk over, smile, lean against, keep, cover with, tie, sweep with, rearrange, write on, stand hold, back on, sit in front of, mess with, wipe down, hold onto, hang up, organize, grasp, examine, prepare, run down, go out, wave, go back to, find, shake, walk in with, cover, bring, inspect, come in, undress, arrange, down, inside, awaken on, run to, push, dust, cover in, empty, lean on, sip from, by, stack, texte on, straighten up, walk with, interact with, walk away with, on the floor, bend down, lay down, flip. kick off, cuddle with, stand up, stand up from, clean off, get dress, on top of, leave through, walk away from, lock, fiddle with, reach, take out of, hold up, dump. go up, hand, fold up. point at, run around, change, cuddle, run in, kick, fluff, walk away, stand behind, knock, pour out, over, come through, swing, rub, enter through, get up out of, walk in through, unfold, unlock, sneeze on, scrub, take picture of,
over, button up, walk past, type, consume, look for, continue, take a bite of, lie down, see, button, getting dress, near, smell, hit, get up off. lay down with, sit down to, take pieture with, move to, get out, zips up, lift, turn out, dress in, walk toward, talk, give, rinse, look up at, read from, sit with, run up get into. stop at, wrap up in, get off, sleep in, show, cover up with, twirl, wrap around, gather, lie down in, proceed to, be, around, check out, view, tidy up with, spray,
undress out of, walk in hold, come in through, eat out of. hold on to, approach, appear to be in, swallow, fall on, sit next to, kneel on, dusting, stand nearby, walk by, search for, snuggle in, get on, stop, leave with, get up, for, search through, sniff, sit down in front of, on the ground, sweep up, undress from, back in, sit down with, asleep on, walk from, lean, look around, fidget with, look over, reach over, run out, cook at, rummage through, back into, polish, dress, come out of, dig through, return to, wake up from, swinge, comb, brush, sort, return, talk with, do work on, ball up. move towards, climb up. smile into, sit up, walk in front of, awaken, pace in, seize, undress in front of, climb into, roll up, finish, wet, pull up. place on, do something on, stand up with, spread, admire, out, wrap themselves in, untie, seat at, spill, wake up on, stand with, tap on, go in, go, juggle, behind, awaken from, work at, turn to,
dance around, shine, turn away from, up to, pat, flick, zips, clean with, pack, reach over to, turn back to, walk back to, knock over, exit through, return with, walk back out of, vacuum around, run across, walk back down, full of, walk around with, follow, go out of, plug in, stand watch, away from, bite, switch, dump out, rock in, sprinkle, stick, sip out of, sit back down on, prop. underneath, seat on, cook with, do something with, recline on, flip on, lic, seat in, wake up, replace, continue up, reach in, kneel down, unbutton, text on, clothe, come to, appear to be tidy up. tie up, fill up, climb, reach up on, climb on, lay down in, down onto, on the back of, face, move from, unpack, on their lap, walk out with, go and sit on, walk back into, get to, look down at, drape, sleep at, straighten out, rifle through, move around, appear, shake out, proceed to eat, sit by. take from, pace back and forth, walk back, film, shuffle, come back in, up on, back onto. enjoy, their, exercise, eat in, tap, sit down to watch, on the shelf, dust off, walk back across
23


Reviews:
[{"review_id": "qYRtOFDtLA", "reviewer": "Reviewer_QPb2", "summary": "This paper presents VSNLS, a weakly supervised approach for VidSGG that uses only video captions for supervision. The paper addresses two key challenges: temporality within captions and variability in action duration. The solution comprises three main components: Temporality-aware Caption Segmentation (TCS), Action Duration Variability-aware Caption-Frame Alignment (ADV), and a novel Pseudo-Labeling strategy based on Motion cues (PLM).", "questions": "- Can you provide LLM selection criteria and impact on TCS performance?\n- Can you clarify the robustness of K-means clustering for action duration estimation?\n- Can you include an analysis of performance vs. video length?\n- Can you provide an ablation experiment of TCS and PLM without ADV?\n- Can you make use of a temporal consistency metric to make the analysis fair and comprehensive?\n- What is the justification for the threshold value in the PLM? Have you done any experiments?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Enabling VidSGG training using only video captions.\n- Reduction in annotation costs compared to existing approaches.\n- Well-identified key challenges in adapting image-based weak supervision to videos.\n- Modular architecture with TCS and ADV components. \n- PLM module handling negative action classes.\n- Effective use of LLMs for temporal understanding.", "weaknesses": "- Limited discussion of LLM choice impact in TCS module.\n- No detailed analysis of failure cases.\n- An effective temporal consistency metric could be added.\n- Analysis of performance on longer videos is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "etRPaSeqXc", "reviewer": "Reviewer_aNnm", "summary": "This paper introduces a framework, VSNLS, for Video Scene Graph Generation (VidSGG) that leverages natural language supervision from video captions to reduce the high cost of manual annotation. Unlike existing methods, VSNLS uses two modules tailored for video data: a Temporality-aware Caption Segmentation (TCS) module to capture time markers in captions, and an Action Duration Variability-aware Caption-Frame Alignment (ADV) module to align captions with frames based on action duration. This approach enables the model to learn from weak supervision, allowing it to predict dynamic relationships and even generalize to unseen actions. The proposed method is shown to be effective on the Action Genome dataset, demonstrating improved performance over traditional image-based weak supervision techniques.", "questions": "The results are reported on the Action Genome dataset, but it is unclear how well the framework would perform on other VidSGG datasets with different types of actions or domain-specific captions. How about VidOR dataset?\n\nHow does VSNLS handle noisy or ambiguous captions? Since the method depends on the quality of video captions, it would be helpful to understand how robust the framework is to variations in caption detail and accuracy.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "VSNLS addresses the high annotation cost of fully supervised Video Scene Graph Generation (VidSGG) by using weak supervision from video captions. This method eliminates the need for extensive manual annotation of all frames, which is both time-consuming and costly.", "weaknesses": "-Dependency on Co-occurrence Priors: VSNLS, like many video-based scene graph models, often relies on co-occurrence priors to estimate relationships. This dependency can reduce the generalizability of the model to unseen object combinations, potentially limiting its performance on diverse or novel datasets where the objects and relations deviate from the training distribution.\n\n-Handling Long-term Relations: Clip-based methods in VSNLS might struggle with capturing relations that unfold over extended sequences, which require a larger receptive field. If clip length is limited (e.g., fixed at 30 frames), the model may miss detecting relations that span longer temporal windows, thereby impacting its ability to accurately model long-term dependencies.\n\n-Occlusion and Tracking Limitations: Although clip-based approaches are often advocated for mitigating long-term tracking issues like occlusion, the use of short-term tubelets might still face challenges in fragmented tracking. Occlusions or visual artifacts can still lead to lost tracklets or misalignments, reducing the reliability of relationship detection over long video durations.\n\n-High Computational Cost for Short Clips: Analyzing shorter clip sequences may require repeated processing of overlapping frames to maintain temporal context, leading to higher computational and memory demands. This could make the model less efficient and limit scalability for very long video sequences where full contextual understanding is crucial.\n\n-Important References Missing:\n[1] Video visual relation detection via iterative inference. ACM MM 2021.\n[2] Winner: Weakly-supervised hierarchical decomposition and alignment for spatio-temporal video grounding. CVPR 2023.\n[3] In Defense of Clip-based Video Relation Detection. TIP 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SOWr5rA7Gd", "reviewer": "Reviewer_6GYf", "summary": "This paper conducts video scene graph generation in a weakly supervised manner. Several key steps are employed to generate scene graph labels from video-text annotations. The authors first design text prompts for multimodal large language models to parse the video captions into sentences along a time horizon. Then, a clustering-based caption-frame alignment framework is designed to align video frames with the segmented sentences. With images and their aligned texts, the authors use existing scene graph parsing methods and scene grounding methods to generate scene graphs. The authors also conduct experiments on the video scene graph generation benchmark , the Action Genome (AG) dataset, to validate the effectiveness the proposed method.", "questions": "I have no further questions. Please see the weakness part.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. This paper proposes a weakly supervised pipeline for video scene graph generation, an important research topic in multimodal learning. Researchers can build a larger scene graph generation dataset with this pipeline from annotated video-text pairs. Researchers can also leverage the MLLMs to generate scene graphs without any human labels in further days.\n2. The design of the temporality-aware caption segmentation module is interesting; it can convert the video's text caption into multiple sentence segments. MLLM may find it challenging to finish such tasks. It would be helpful to add more details about the segment process. \n3. Aligning the segmented sentences with the video frames is critical for generating a scene graph. The authors provide many details about conducting the clustering-based alignment.", "weaknesses": "Although the proposed VSNLS pipeline can achieve good results on VidSGG tasks, its innovation still needs to be enhanced and emphasized. The VSNLS pipeline has four critical steps: the temporality-aware caption segmentation part, the action duration variability-aware caption-frame alignment part, the generation of pseudo-localized scene graphs part, and the adverse action classes generation part. Except for the action duration variability-aware caption-frame alignment part, I am unclear about the novelty and technical insights in other sections. \n1. In section 2.2, the authors design prompts to segment the video captions into sentences. Since there are a lot of prompt engineering works, what are the new insights in this part that can motivate the following researchers to conduct their work? Are there some key points that need to be noticed in design prompts?\n2. In section 2.4, the scene graph parsing and grounding utilize state-of-the-art methods to generate graph nodes. I guess readers are more willing to see how the authors develop some techniques to make these state-of-the-art methods conduct parsing or grounding more accurately.\n3. The experimental setting is organized in a bit of chaos. How the training, validation, and testing sets are split is not introduced. From line 316 to line 325. the reader can hardly understand whether the model is trained on the AG dataset or the combination of the AG caption and the MSVD caption dataset.\n4. There is no ablation study on the effectiveness of the microdesign in the ADV part. How different clustering strategies influence the final results is important to understand the rationale behind the VSNLS framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sCPvXKFS6g", "reviewer": "Reviewer_61fs", "summary": "This manuscript tackles the problem of extracting scene graphs from video inputs by only using raw video captions during the training stage. The proposed method focuses on extracting pseudo training data from the raw captions to be consumable by supervised VidSGG models. The proposed method consists of TCS (temporally sorting the atomic events in the complex captions), ADV (handling the variable length of atomic actions), and k-means clustering with some smoothings to assign clusters of frames to each of the atomic actions.", "questions": "1- In Table 1, what about combining the full supervision and weak supervision data to see if the weak supervision can give a boost to supervised results? The combination could be combining the training data or doing pretraining on weak supervision and then fine-tuning on fully supervised data. IMO, we can always have some supervised data, and the value of this work will shine if we see that it can boost the results on top of that.\n\n2- Lines 240-244: I think this part needs more clarification. A) Does this filtering contribute to the final results? B) In the given example, will I^1 be assigned to S1, or will it not be assigned to either S1 or S2? C) Videos can be formed of different shots that can jump between two actions back and forth. How those cases will be handled? \n\n3- Do the clustering steps have any other purpose than computation efficiency? Why not compare each singleton frame and the Segmented Sentences?\n\n4- I would like to see the examples that have been passed to the LLM. A) How are they selected? B) How does the quality of output change by changing those examples? I see a reproducibility risk here.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "VidSGG is an important problem, and IMO has been overlooked. I believe this line of research is very important, and I believe that authors selected a good approach by utilizing only captions to train a vidsgg.\n\nAlso, the temporal inconsistency and variability of action lengths are two main difficulties for any video-language tasks. I believe that this paper systematically decomposed them and addressed them each separately.", "weaknesses": "The main weakness of this manuscript is that it does not contribute to any core learning methods. For example, no visual understanding or natural language understanding method has been improved. It is mostly a wise tooling over LLMs and some basic approaches like clustering. Even the final training has been delegated to off-the-shelf vidsgg methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)