LASER: A Neuro-Symbolic Framework for Learning Spatio-Temporal Scene Graphs with Weak Supervision
Authors: Jiani Huang
Ziyang Li
Mayur Naik
Ser-Nam Lim
Venue: ICLR 2025 Poster (2025)
Keywords: neuro-symbolic
video understanding
spatial-temporal scene graph
weak supervision
Primary area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
TL;DR: We propose LASER, a neuro-symbolic approach to learn semantic video representations that capture rich spatial and temporal properties in video data by leveraging high-level logic specifications.

Abstract: Supervised approaches for learning spatio-temporal scene graphs (STSG) from video are greatly hindered due to their reliance on STSG-annotated videos, which are labor-intensive to construct at scale. Is it feasible to instead use readily available video captions as weak supervision? To address this question, we propose LASER, a neuro-symbolic framework to enable training STSG generators using only video captions. LASER employs large language models to first extract logical specifications with rich spatio-temporal semantic information from video captions. LASER then trains the underlying STSG generator to align the predicted STSG with the specification. The alignment algorithm overcomes the challenges of weak supervision by leveraging a differentiable symbolic reasoner and using a combination of contrastive, temporal, and semantics losses. The overall approach efficiently trains low-level perception models to extract a fine-grained STSG that conforms to the video caption. In doing so, it enables a novel methodology for learning STSGs without tedious annotations. We evaluate our method on three video datasets: OpenPVSG, 20BN, and MUGEN. Our approach demonstrates substantial improvements over fully-supervised baselines, achieving a unary predicate prediction accuracy of 27.78% (+12.65%) and a binary recall@5 of 0.42 (+0.22) on OpenPVSG.  Additionally, LASER exceeds baselines by 7% on 20BN and 5.2% on MUGEN in terms of overall predicate prediction accuracy.

Published as a conference paper at ICLR 2025
LASER: A NEURO-SYMBOLIC FRAMEWORK FOR LEARNING SPATIO-TEMPORAL SCENE GRAPHS WITH WEAK SUPERVISION
Jiani Huang
Ziyang Li$
Mayur Naik
Ser-Nam Lim sernam@uce edu
(jianih, 19, mhna Saaiki @eses upenn.edu
University of Pennsylvania
'University of Central Florida
ABSTRACT
Supervised approaches for learning spatio-temporal scene graphs (STSG) from video are greatly hindered due to their reliance STSG-annotated videos, which are labor-intensive to construct at scale. Is it feasible to instead use readily avail- able video captions weak supervision? To address this question, we propose LASER, neuro-symbolic framework to enable training STSG generators using only video captions. LASER employs large language models first extract logical specifications with rich spatio-temporal semantic information from video captions. LASER then trains the underlying STSG generator align the predicted STSG with the specification. The alignment algorithm overcomes the challenges of weak supervision by leveraging differentiable symbolic reasoner and using a combina- tion of contrastive, temporal, and semantics losses. The overall approach efficiently trains low-level perception models to extract a fine-grained STSG that conforms to the video caption. In doing so, enables novel methodology for learning STSGs without tedious annotations. We evaluate our method on three video datasets: OpenPVSG, 20BN, and MUGEN. Our approach demonstrates substantial improve- ments over o fully-supervise baselines. On OpenPVSG, LASER achieves a unary predicate prediction accuracy of 27.78% (+12.65%) and binary recall of 0.42 (+0.22). Furthermore, LASER exceeds baselines by 7% on 20BN and 5.2% MUGEN in terms of overall predicate prediction accuracy.
INTRODUCTION
Understanding video semantics has gained prominence due to a wide range of applications such as video search, text-video retrieval, video question answering, video segmentation, and video captioning. Video semantics constitutes two crucial aspects: spatial semantics, which concern the entities the video, their individual attributes, and their semantic relationships; and temporal semanties, which capture actions and properties evolving through time. For example, the video described in Figure] ] by the phrase "pushing a box off the desk by hand" involves entities like "box" and "hand", which are connected by the spatial relation "touching", It also features two temporally consecutive states: the "box" is first "on" " the "desk". and then "not above" the "desk".
To explicitly learn combined spatial and temporal semantics, a structured representation called Spatio- Temporal Scene Graph (STSG) (Shang al. 2017 Zhu et al. 2022) has been proposed represent entity relations throughout video. Existing approaches for learning STSG from video data are typically fullyesupervised, e.g., e.g.,Nag et al.] (2023); Cong al. (202T). They can potentially learn high-fidelity STSGs from video data but are greatly hindered in practice due to the complexity of low-level annotations that are laborious to obtain (Yang et al. 2023a).
Weak supervision emerges as promising approach to address this challenge. For example, the vast availability of video captions provides a valuable source of weak supervisory signals. However, key difficulties arise in effectively learning STSGs from such weak supervision. Is it even feasible to use video captions given the sparsity and noise in the signals they provide? Captions often focus only on the primary objects, ignoring underlying details, and many temporal signals are either hidden or must be inferred. How can we provide useful fine-grained signals under such circumstances? To address these challenges, we propose transforming captions into logical specifications using large

---

Published as a conference paper at ICLR 2025
Video
Fine tune
Spatio-Temporal Scene Graph (STSG)
Back-Prop
STSG Generator
onto, D)
on
above DJ
bebove D)
touching
tnuching H)
HouchhngeB
Spotie-
Reiational Keywords
Temperal
Alignment
Checker
Video Caption
Opatio-Tempont Specification
(Scillop)
Language
V1, v2, v3: Wiar(Vi, v2, v3) (14, v2, v3) ung vB))
Pushing box off the desk
Model
VvarlV1, V2, v8)
is,90d1, "box") and is 6_a(22, "desk") and is (s,a, "hand")
by hand
[eg: GPF1-4)
MasDii, V2, v3)
on(ys, 92) and touchinglvi, V3)
Stractured Representations
utglvi, V2, v3)
not(above(vi, and notttouching(V]. V3))
Figure I: Illustration of the learning pipeline of LASER. The goal is to fine-tune rision-language model to produce STSG without direct supervision on ground truth STSG labels. LASER relies on video captions for weak- weak-supervision labels. We apply LLM to extract a spatio-temporal specification from video captions. The LLM-inferred relational keywords, along with the input video, are then passed alignment checker uses the specification to derive an alignment loss, capturing issues in the predicted STSG. The differentiable checker effectively back- back- proppaaass the loss to the vision- Saiign-a model.
rision-languaan model to generate an STSG. At the end, a spatio-tempon
language models to explicitly reveal the hidden spatial and temporal information. This transformation creates a shared foundation to systematically align captions with predicted STSGs. The alignment process should a) capture both spatial and temporal nuances to provide fine-grained supervision for underlying STSG generators: b) allow diversity, naturalness, and fuzziness in the video and caption data; and c) account for common-sens' knowledge that may be implicit or ambiguous in the captions.
We set out by designing STSL, general and expressive Spatio-Temppel Temporal Specification Language for specifying fine-grained
properties. STSL is grounded Finite Linear Temporal Logic (LTL,) (De Giacomo & Vardi 2013) which is used to describe temporal properties finite traces of action and states, STSL subsumes action sequences commonly seen in video-action alignment tasks (Chang al.) 2019) while capturing additional temporal nuances such as "until" (U) and "finally" (0). It also allows to express common- sense constraints for extra supervision. Finally, combined with relational predicates extracted from natural language, such as "is pushing off" and "lies above", it can even specify the open-domain spatial semantics of videos.
We now introduce LASER (Learning to Align for Spatio-tEmporal Representations), a novel frame- work to enable training STSG generators using only video captions. As illustrated in Figure [ LASER enhances a langun-annguage model by aligning its predicted STSGs with STSL specifications derived from video captions using large language models. This alignment process is carried out in divide-and-conquer fashion, where the caption is broken down into temporally related events, each of which must correspond with portion of the STSG. We enhance the alignment process in two important aspects. First, to ensure precise optimization, we implement a neuro- suuro-ymmboli alignment checker atop the Scallop framework (Li et al., 2023), making the alignment both probabilistic and differentiable. This enables seamless integration into an end-to-end learning pipeline. Second, to complement the weak supervision, introduce a multi-faceted loss function that includes contrastive. temporal, and semantic components, which provide additional layers of supervision.
We conduct an extensive evaluation of LASER across various dimensions, demonstrating its broad utility and effectiveness. Being model-agnostic, LASER is capable of fine-tuning a wide range of STSG generators, including open-world generators our evaluation such as CLIP (Radford et al.) 202I), VIOLET (Fu et al. (02I), and SigLIP (Zhai et al. 2023), as well as closed-wort generators like MLP classifiers for STSG generation. To evaluate versatility, we apply LASER to diverse datasets such as OpenPVSG (Yang et al. (2033)), 20BN (Goyal et al. (0177), and MUGEN (Hayes) ct al.) 2022), which encompass open-domain vocabularies, specifications with varied patterns, and both synthetic and complex real-world videos. Despite challenges such
excessive number of relevant entities, diverse momn-domain labels, and fuzzy captions, LASER consistently outperforms even fully supervised baselines in STSG generation all three datasets. To further evaluate the data efficiency of LASER, we train the STSG generator with just 10% of the training data, resulting in an average of 70.75% of the performance gains achieved with the full dataset.
2

---

Published as a conference paper at ICLR 2025
We summarize the main contributions of this work follows:
1. We introduce a novel formulation of spatio-temporal scene graph learning as a weakly supervised task driven by video captions.
2. We design STSL, a general and expressive spatio-tempon specification language, for specifying fine-grained video semantics.
3. We implement a differentiable neuro- neuro-symbolic alignment checker to relate between an STSL specification and spatio-temporal scene graph.
4. We propose LASER, a model-agnostic, end-to-end differentiable framework for learning spatio- temporal scene graphs with weak supervision from video captions.
5. We empirically evaluate LASER on three video understanding datasets, demonstrating superior performance video semantics extraction tasks.
2 RELATED WORK
Structured Representation of Image/Video Semantics. Significant advances have been made in representing structured information within vision data. widely adopted representation for capturing
spatial semantics in images is Scene Graph (Kuznetsova et al. 20188 [u et al., 2016). with various generation techniques emerging over the years (Zhu et al., [022]] Liu al. 2021 Huang al.) 2020 2021
Yang al. 2018 Li et al., 2024). More recently, research has increasingly focused on extending these representations to integrate both spatial and temporal semantics videos (Yang et al.,
2023a Li al. 2022). However, learning spatio-tenn temporal structures remains challenging open problem, which LASER addresses by introducing a neuro meuro-symbolic approach.
Video Scene Graph Learning. Learning video scene graphs has attracted significant attention from the vision community. Various tasks, including entity tracking, object identification, dynamic relation
analysis, and pathfinding, are being explored (Sun et al.) 2023 Xu et al. [022a Shang et al. 2017)
New techniques involving spatio-temporal aware networks have been developed (Nag et al. 2023
Cong et al. [022 Ji al. 2021 Sun al. 2019). A few recent works have developed point-solutions
for extracting fine-grained video semantics (Lee et al.) [023] Apriceno et al.
2022
Chang et al.
2019). Such approaches includes both dynamic time warping (Dvornik al.
2021
Chang et al.
2019 Richard et al.] 2018: Ding & Xu 2018). soft nearest neighbor (Han et al.)
2022 Dwibedi et al.
2019). and semantic loss (uu et al.] | 20226). To our knowledge, LASER is the first framework to
train STSG generation models using video captions as weak-superr labels.
Vision Language Pretraining. Vision language pretraining is erucial in video understanding and has wide range of downstream applications. Current works have succeeded in learning visual representations using large-scale paired visual-textual data through contrastive learning in both image-text (Zhai al. [0233] Radford et al. 2021 Jia et al. (021) and video-text (Li et al.) 2021] Xu ct al.) 2021 Miech et 2019) representation learning. Recent works also explore the viability of utilizing pretrained foundation models for generating image and video scene graphs (Shindo et al.)
2024 Liang et al. 2024
Zhang et al. 2023 Yao et al. 2021).
3
METHODOLOGY
We begin by presenting the high-level problem definition. We are given a dataset D of video-caption pairs (X, where X x= [=1...., _nn] is video containing n frames, and c is its video caption. We then transform the caption into spatio-temporal specification &i in LTL, using large language model. We wish to learn a neural model Ma which extracts spatio-temporal scene graph r = Ma (X) that conforms to the corresponding specification vb. During training time, given loss function L. aim to minimize the following main objective:
((0) पें EXX.SED C(Pr(F v 0),
(1)
where Pr(f F v/ 0) is the alignment score (probability of alignment) computed by our spatio- temporal alignment checker, conditioned on the model parameter 0. We illustrate the full learning pipeline in Figure] and detail the process in this section. We first describe our video semantics representation using a probabilistic relational database ($3.1) Then, we present our specification language STSL (3.2) and its alignment checker ($3.4). We further demonstrate how to automatically convert natural language captions to a specification in STSL using LLM ($3.3). In [3.5] we present multi-faceted loss function design comprising contrastive, temporal, and semantic components.
3

---

Published as a conference paper at ICLR 2025
A man carnes child and walks the fron bchinal..
Estracted Usiry Koywerds
Input Caption
LLM
diu
case
Transe Squaraas
ottio man)
Lmary
strioy, momen)
atoooy, chill)
Video
Prepracesann
Bloary Predicate
car arrioo
CLIP
Extracted Binnry Newwods
caryto, Og
bend
behindtey Sy)
Segment Patre
lef-of(Oy, Bt)
Time
Probabilisttic Spatio-Temper Tempotal Scene Graph
(STSG)
Figure 2: Pipeline illustration with CLIP as the backbone model for probabilistic STSG generation.
3.1 VIDRO TO PROBABILISTIC RELATIONAL. DATABASE
A probabilistic spatio-temporal scene graph is probabilistic relational database that contains two types of facts denoted by relations unary om and binary_ass om, _atom, for unary and binary predicates respectively, each associated with probability denoting the likelihood that the fact is true. For exam- ple, O. 05: uuaay__ttoo ("deformed", 3, e) means that "entity is unlikely to deformed al time stamp 3," while 0. 92: : :binary_atom ("push" 10, h, b) indicates that "object is highly likely to be pushing object b at time stamp 10." This flexible representation supports the seamless incorporation of unary and binary keywords into the database. The unified probabílistic database enables LASER to be model-agnostic, supporting both closed-domain STSG classification models and porld vision-language models for converting input video data into relational repre- sentations. With a unified formalization, an STSG generator, Mg. takes pixel-basee raw video data X, and generate distribution of STSGs, which is encoded
probabilistic relational database î.
We illustrate the STSG generation pipeline using an open-world vision-language model, CLIP, in Figure 2 The pipeline generates probabilistic STSG from input video and caption pair. An LLM first extracts relational keywords from the caption and passes the keywords to the STSG model. The video is then preprocessed by cropping objects in each frame and aligning them with unary keywords, while weighted masking is applied to object pairs to distinguish between subject, object, and background, aligning them with binary keywords. Consistent relationships, such as object categories, are aggregated across the video, resulting in the desired probabilistic STSG. For closed- domain STSG classification models, the pipeline remains the same terms of video preprocessing. However, instead of relying predefined set of vocabularies. For brevity, throughout the paper, we use actual unary and binary keywords as predicate names, c.g. deformed (2, and push (10, h,
LLM to extract keywords, these models predict directly over
3.2 OPATIO-TEMPORAL SPECIFICATION L LANGUAGE (STSL)
Linear Temporal Logic (LTL) (Pnueli 1977) is formal logic system extending propositional logic with concepts about time. It is commonly used for formally describing temporal events, with applications in software verification (Chaki al. 2005 Kesten al. 1998) and control (Ding et al. 2014, Sadigh et al. 2014). As we operate on prerecorded, finite-lenghh videos, our language is developed using LTL, (De Giacomo & Vardi 2013), which supports LTL reasoning over finite traces. Thus, we use ,,TL,,
Tramework for specitying events and their temporal relationships.
Our STSL (Figures 5p further extends LTTL, by introducing relational predicates and variables. It starts from the specification v which existentially quantífies variables in an STSL formula. The formula yp is inductively defined, with basic elements as relational atoms a the form a(t),..... t.). Note that the terms i { (f...... )n) can contain quantified variables to be later grounded into concrete entities based on context r, noted by (ubstr((). From here, can be constructed using basic propositional logic components A (and), (or), and (not). The system additionally includes temporal unary operators O (always), Q (finally), O (next), and a binary operator (until) (Albers) al. 2009). For example, the description "A hand continues touch the box until it drops." can be
4

---

Published as a conference paper at ICLR 2025
Caption
Pushing a off the desk by hand
Spatio Tempporal Specification
GPT4
lccation(e1)) = dumatio((1))
location( duration(22)
Structured Representation
Description:
Description:
0,(v1, v2, va) lies ontvo, v2) and touching/vi.x33)
Vy/Vi, v3)
drop toudhing(iv1xx))
touching the box.
Predicates:
A box les on the desk and hand
i,alk2, touchingfvi, only1,, s2),
Predicates:
The hox longer ahove
desic
stop_oflivi,
Precesiing
Vaarv1 v2, v3) = i_a(v), "box") A a "desk") iB_a(vS, "hand")
v3]
Duralion:
Video Location:
Video Location:
Duration:
Relational Keywords
Unary: bos, desk, hand
touching, drop_ooff
Event 1
Event 2
Figure 3: Pipeline utilizing 3-shot GPT-4 convert natural language captions into: (1) programmatic spatio-temporal specification for alignment score calculation as input to the alignment checker, and (2) unary and binary keywords for predicting the probabilistic STSG inputs to the neural model.
represented an STSL formula
= touch(n, b) U drop(b, -)
Note that an argument to the predicate drop wildeard (,, since we do not specify where does the box drops from. This formula might seem too strict since it requires the two events to be consecutive. To make the specification more natural, one can change the above formula to "O(touch (h, b) A (drop (b, -)", Here, the two events, touch and drop, need to happen in chronological order but are not required to be consecutive.
3.3 NATURAL LANGUAGE TO PROGRAMMATIC SPATIO-TEMPORAL TEMPORAL SPECIHICATION
To leverage the abundance of video captions as weak supervision signals, we employ large language model (L.LM) to automatically extract a programmatic specification v from each video caption c. Directly converting captions into formal program particularly challenging for an LLM, especially in a low-data language like STSL. We hence use a few-shot learning approach with an LLM to generate an intermediate structured representation of the caption in JSON format. For each caption c, our goal is to convert it into series of events ē = {ei, . e. Each event includes (a) a detailed natural language description of the event, which guides the generation of subsequent details, (b) a series of unary, binary, positive, and negative predicates describing the semantics of the scenario, (c) the location the event, loc(es). ). in the video where the event occurs, represented as a fraction the video length, and (d) the duration of the event, dur(e;). also expressed fraction. In Section 3.5 we explain how these structured representations are incorporated into the loss function.
To extract such structured representations from the caption, we designed a generic prompt template, which consists of the following components: (a) examples for temporal specification in fraction numbers: "0", "1/2", "2/3", "1". (b) scene graph keywords, such object names and relations. (c) few-shot examples of caption and JSON structured representations pairs. We illustrate a caption and its structured representation in Figure and the full prompt in the appendix.
The programmatic spatio-temporal specification then generated by postprocessing the events in sequential order. Consequently, we can generate the programmatic spatio-temporal specification for the caption as a sequence of events in chronological order:
Deleeth,,
Noseus Øg.
(3)
3.4 SPATIO-TEMPORAL ALIGNMENT CHECKING
Given a probabilistic database that encodes a distribution of STSGs (3.1). and specification e in STSL, we aim to measure the alignment score Pr(r - x)) in an end-to-end and differentiable manner. Conceptually, each probabilistic fact f in the database can toggled on off, resulting in 2|rl distinct worlds. Denoting each world (i.e. discrete STSG) as E P(r), where P. represents power-set, we can check whether the world w satisfies the specification vù or not (Figure 5). From here, the alignment score can computed as the sum of the probabilities of worlds satisfying th:
Pr(r -= 1)) Ewer() Pr(10) 1 w), Pr(w)= [ [ Pr(f')
(4)
5

---

Published as a conference paper at ICLR 2025
(Formula) : a(I) A p2
V pp
Time Siteps
Oip 4P1 U 62 4 Op
(Specification) 1) -...... Uk, s.t.
DB
0.7:touch
0.6::touch
0.2:touch
0.3:cropp
0.8::drop
Figure 4: The formal syntax of STSL. Here, A, V, and represents logical "and", "or", and "not". Formula may also contain temporal operators O (next), U (until), D (global), and (finally).
Iterativv Evaluntion Procedure
0.58: al58:aligned touch U drop)
0.3:ddrop
0.7:touch
0.4:ddrop
0,42:touch
0.336::aligned(uucch drop)
(w, 8) = 1b
iff ar, (T, 8) =
0.42:touch
0.8::drop
(T, m,
(f))
iff a()) E w
substr (f)
0. 0.064:ttucch
(T, 8)
PI A
iff (T, w, P
0.0756:alignee(tooo drop):
(T,w,s) P2
0.084
0.9:xdrop
(T', w. 8)
iff (T, p
(T.
Oup
iff [
Target Specification
touch
drop
(T, $
Up2
iff S Vk.s i,
Figure 6: The evaluation process aligning spatio-temporal scene graph (DB) with spec- ification t touch U drop. This figure omits the predicate arguments, concentrating solely on matching sequential events.
Figure 5: Formal semantics of STSL. (w,, s) F means the STSL specification vi is aligned with the ST-SG w starting from time s. We + as an abbreviation for (w, 1) + 3/.
Enumerating all possible worlds is intractable due to its exponential complexity. Existing general- purpose neuro-symbolic systems like Scallop (Li et al. 2023) employ scalable algorithms to approx- imate this probability and greatly reduce the probabiliste listic reasoning time. We also note that some of the STSG sampled from P(r) might be infeasible due involving conflicting facts (e.g.. box is above and below desk at the same time). To further enhance the logic deduction efficiency, extend Scallop's "top-k proofs" provenance support general disjunctive constraints and early removal of infeasible STSGs that do not satisfy the specification.
LASER implements the aligmment checker using our extended version of Scallop. It inductively computes the aligument between temporal slice of r and an STSL formula. The whole specification & is aligned if the full r satisfies jb with a concrete variable grounding Т, which maps variables to concrete entities. We also extend Scallop with variable assignment interface enumerating T for constraint solving. We illustrate one simplified evaluation process in Figure [6] The checker iteratively aligns the predicted probabilistic events (simplified to just t touch and crop) with the specification. At the 4th iteration, 4 different satisfying alignments are derived, yielding a final aggregated alignment score of 0.9916.
3.5
LOss FUNCTION
Contrastive Learning, Unavoidable dataset biases exist in the specification. Contrastive learning can effectively reduce the bias and generate explanations of better quality. Let (X,, th:) and (X,, wby) be two datapoints in mini-batch B, where v/; and t5 are the specifications for video X; and X; correspondingly. X; == 3 then is an extra positive sample to the video X;; otherwise, it is negative sample to X;. We can thus define our per-batch contrastive loss (.(B::
(B BB Exx. C(P - v³)) 1lvs v)
(5)
Time-Span Supervision. A video caption expanded into sequence of events using LLM, with each event assigned a specific temporal target, detailing its location and duration within the video, as illustrated Section 3.3 By aligning the spatio-temporal specification y with the video, we can identify when its sub-specifications. v4,, vi...... th. met. This alignment facilitates weak supervision across the entire time span. We define (s, ) E [0. 1]. the time span alignment score, a function actual time stamp s, expected time stamp I, and expected event duration d. In particular, ar (s,, I, d) should peak at when the event happens exactly at the expected locations (s = l). In practice, we embed into the computation of probabilistic alignment between STSG and an

---

Published as a conference paper at ICLR 2025
Unary
Binary
Method (with LASER)
R@l
R@5
RΦ 10
R@1
R@5
RΦ 10
Base
0.0660
0.1855
0.2983
0.0460
0.1307
0.2636
VIOLET
Fine-tuned
0.0878
0.2574
0.3463
0.0501
0.2028
0.3451
Incr.
t 0.0218
1 0.0719
1 0.0480
0.0041
0.0722
0.0815
Base
0.0000
0.0179
0,0483
0.0000
0.0362
0.1667
SigLIP
Fine-tuned
0.1467
0.2627
0.3152
0.0347
0.1624
0.3012
Iner.
t 0.1467
0.2444
1 0.2669
0.0347
0.1226
0.13345
Base
0.1633
0,3381
0.4404
0.0197
0.0673
0.0988
CLIP
Fine-tuned
0.2778
0,5231
0.6402
0.1482
0,4214
0,5398
Iner.
1 0.1145
1 0.1850
1 0.1998
0.1284
0.3540
0.4410
Table 1:
We show the performance improvements of base backbone models and their fine-tuned version, on the R@ @@ metrics of unary and binary predicate prediction. As shown by the increments, LASER's 's weak supervisory learning framework significantly enhances all three models' performance on the STSG extraction tasks.
atomic specification a()), where we utilize the expected location loc(a) and the duration dur(a):
(s. 1, d) = max 1- 21s-!1) Pr((T' w. = at)) = Pr(a(e) a()) E w(s) A ē stt=ssuubtrr)) a dor(a)))u((a))
(7)
Semantic Loss. To provide further supervision, resort human knowledge encoded in the form of integrity constraints. We introduce semantic loss reflecting the probability of violating the integrity constraints, For example, an entity video cannot be open and closed at the same time; an entity that is not bendable cannot be de formed. These integrity constraints may interweave so heavily that it is hard to use simple disjoint multi-class classifier to enforce. We encode all integrity constraints in the form of first-order logic rules, and our reasoning engine generates the probability that these constraints are violated. We thus have the per-sample semantic loss L, (X,) as an extra- extra-weighted weighted term after calculating the other loss components. Let be the number of integrity constraints and let IC; be the i-th integrity constraint, we have
(x(X;) = Σ ((Pr(Mo(XX)) O).
4 EVALUATION
Our evaluation attempts to answer several key questions about LASER. How effective is LASER in learning STSG generators? How does its performance compare to fully supervised basclines and existing methods? Is STSL versatile and expressive enough when applied to diverse specification patterns? Finally, how essential is the multifaceted design of LASER ' loss function?
To address these issues, we evaluate LASER three datasets: OpenPVSG (Yang al. 2023a realistic dataset with diverse and fine ane-grained STSG annotations. 20BN (Goyal et 2017 a video dataset focusing on daily actions, and MUGEN (Hayes et al.) 2022), synthetic dataset containing gameplay footages. These datasets vary significantly in their temporal patterns, showcasing the versatility of STSL. Specifically, OpenPVSG captions focus on natural and complex events, 20BN captions are in the form of action pre-conditions and post-conditions, while MUGEN captions describe consecutive action sequences performed by the main protagonist in the game.
Our main result shows significant improvements of LASER over fully supervised baselines. On OpenPVSG, LASER achieves unary predicate prediction accuracy 27.78% and a binary r of 0.42, surpassing the best fully supervised baseline by 12.65% and 0.22, respectively. Furthermore, LASER outperforms baselines by 7% on 20BN and 5.2% on MUGEN in overall predicate prediction accuracy. We now delve into the experiments conducted on each of these datasets.
4.1 OPENPVSG DATASET
Dataset. The OpenPVSG Yang et al. 2023a) dataset comprises 400 videos sourced from Ego4D
(Gramman al. 2021), VidOr (Shang al. 2019 Thomee al. 2016). and EpicKitchen (Damen)
7

---

Published as a conference paper at ICLR 2025
Unary Rocallats
Unary Recallses
Unary 1o
Sintry Recallit
Rissry Recillés
Binary Resaliginn
Method
Unary
Binary
Acc. (%)
R@
R@s
RØ 10
0.75
VS-Vanilla
15.13
0.0741
0.1081
0.1109
0.50
IPS-Filter
13.14
0.0777
0.1040
0.1133
IPS-Conv
15.13
0.0861
0.1143
0.1218
0.45
Recall
IPS-Trans
14.67
0.1419
0,2032
0.2207
0.30
0
2233
12410
VPS-VanillA Vanilla
5.49
0.0374
0.0517
0.0531
VPS-Filter
5.46
0.0405
0.0480
0.0488
0.16
VPS-Com
7.46
0.1616
0.1781
0.2343
0.00
VPS- Trans
5.46
0,1019
0. 0.1499
0. 0.1562
0%
tofs
gots
100%
LASER-CLA
27.78
0.1482
0.4214
0.5398
Training Duta Usage
Figure 7: Data-efficient fine-tuning on OpenPVSG dataset with LASER: Provid- ing only 10%, 50%, and 100% of the train- ing dataset significantly enhances the per- formance of CLIP model.
Table 2:
Comparison between weakly supervised LASER-CLIC and fully supervised IPS and VPS meth- ods on various backbones trained on the full OpenPVSG. LASER-CLID significantly outperforms all baselines, ex- cept on Binary R@ 1, despite using weak supervision.
et al. 2022 2018). This dataset offers fine- fineggriined ground truth annotations of STSGs for 150K frames, encompassing 126 object classes and 57 relation classes. We train on 1, 832 video-caption pairs, and evaluate on 438 video-STSG pairs.
Experimental Setup. As illustrated in Figure 2 our objective is train an STSG generator capable of taking a video clip with object bounding boxes as input and predicting the properties, attributes, and relationships between objects. We leverage LASER to fine-tune three vision-language mooelss-VOOEE" (Fu et al. (2021). SigLIP (Zhai et al. 2023), and CLIP (Radford et al. (002)) using weak supervision from captions. These models predict both similarity scores between cropped objects and unary predicate keywords, as well as between object pairs and binary predicate keywords, resulting a probabilistic STSG. All backbone models support open- world vocabularies and are thus robust the fuzziness present in GPT-4-generated structured representations.
Evaluation Metric. We evaluate model performance using Recall (R@k) which estimates whether the ground truth label is within the top-k prediction of given model. During evaluation, the model processes (a) the full vocabulary of object and relation classes and (b) preprocessed cropped objects and object pairs, predicting the probabilistic STSG. In particular, unary R@ k assesses object category prediction capability, while binary R R@ k evaluates pair-wise prediction of binary relations.
Backbone models significantly improve after been weakly supervised by LASER. We validate LASER' effectiveness in learning STSGs with weak supervision by comparing the performance improvements of the backbone models after fine-tuning. As shown in Table I LASER significantly enhances backbone performance using only captions for weak supervision on the OpenPVSG dataset.
Data efficiency of LASER. To further assess LASER's 's data efficiency, we train the model 10% and 50% of the training dataset. As illustrated in Figure 7 even with just 10% of the training data (183 video-caption pairs), LASER significantly enhances the unary R@ I from 0.1633 to 0.2305 and the binary R@I I from 0.0197 to 0.1261. On average, using just 10% of the data achieves 70.75% of the performance obtained with the full dataset, highlighting LASER's 's data efficiency.
Weak-supervision may outperform full-supervision. To better understand the efficacy of weak supervision, we also compare them against fully supervised methods. We study 8 fully supervised baselines, which employ two different video panoptic segmentation strategies: Image Panoptic Segmentation with Tracker (IPS) and Video Panoptic Segmentation (VPS). For relation extraction, the basclines employ 4 model architectures: (1) Vanilla: fully-connected layers, (2) Filter: handcrafted filters, (3) Conv: ID-convolutional layers, and (4) Trans: transformer encoders. As shown in Table 2 LASER significantly outperforms the best fully supervised methods in all metrics except for binary R @11 1, where it ranks just after the top- perforrfnnm VPS-Conv.
4.2 20BN DATASET
Dataset. The 20BN dataset consists of (a) video and action pairs of humans performing everyday actions with ordinary objects (Goyal et al. 2017). (b) expert designed pre-conditions and post-
8

---

Published as a conference paper at ICLR 2025
0.8
Score
0.8
0.4
0.2
GPA LASER LASER
Is Is-bendable (s-fluid Ishoddable
is-rigid Is-tearbble
ahove
attached
behind
broken
close
closed
deformed
empty
far
lite tolded has
hole
high
in-front
lefi
low next-o
on on-surface
on
open
right
stacked
stretched
twisted under upright visible
Figure Per-predicate Fl score performance comparison LASER, LASER-P, and GPA, all trained on the full 20BN dataset. LASER-P outperforms GPA 71 1 of predicates, and LASER outperforms GPA on 59% of the predicates.
Birow-suriin surffce
0:989:
0.506: (tonching
0.614: touching
0.420 touching (k.
0.70 touching
0.362: bowching
0.136:
0.188::def ::B:::eddmedd
0.522:
deforoed
D.866::aformee
0-877::dafcrmm deferrddd
:broken broker (A)
971:brokenn.o
605
bboker
9.75::broken (A)
860 bbeddabbeeee
0.000:
bendable :
(A)
0. 06000 bendable
060:
0.000: bendable (A)
060: bendable
Figure 9: Qualitative study of the model trained with LASER on the full 20BN dataset. Each row displays sequence of frames from video, with bounding boxes labeled by object IDs. The left side of each row shows the action label, while the bottom of each row lists the attributes and relationships associated with the objects, along with the corresponding likelihoods of these facts holding true.
conditions for the actions the PDDL language (Migimatsu & Bohg 2022). and (c) frame-based object bounding boxes (Materzynska et al.) 2020). There are 172 actions with 37 underlying predicates in this dataset, capturing object attributes, object states, and relationships between two objects. Specifically, there are 6 static predicates, 21 unary predicates, and 10 binary predicates. We train on 10, 000 training datapoints and test on 14, 816 data points.
Experimental Setup. In the 20BN dataset, each video is assigned action label from set of 172 possible actions. Each action annotated with a natural language description and a logical specification, represented
pair of pre-condition and post-condition in PDDL format. These PDDL specifications naturally map to the (Wpre A (1ppa) structure in STSL. This setup enables us to assess the performance difference between caption-generatee specifications and ground truth programmatic specifications. LASER received the generated specifications from natural language captions using GPT-4 as weak supervision label. For ablation study, refer variant of our method that uses ground truth specifications as weak supervision labels as LASER-P.
Evaluation Metrics. We consider exact accuracy and Fi-score as the metrics to evaluate STSG generators for 20BN dataset. Specifically, we compute F1-score and accuracy for each predicate in the vocabulary, and evaluate the weighted average of F1-score and accuracy for overall evaluation.
Model Architectures. Our backbone STSG generator model comprises an S3D (Xie et al. 2018 video encoding model pretrained on Kinetics 400 (Kay et al.) 2017). followed by ROlpooler for extract object embeddings, then passed to MLP layers for relation classification.
Quantitative and Qualitative Study. We compare our approach against GPA (Migimatsu & Bohg 2022), weakly supervised baseline which uses the ground truth specification as learning signals. Our evaluation shows that LASER-P -P, with the same programmatic supervision. achieves higher average Fl Fi-score of 0.77 and accuracy of 91%, compared to the baseline's 's 0.74 Fi-score and 76% accuracy. LASER, using only natural language descriptions, achieves a comparable Fl-score of 0.73 and better accuracy of 83%. Furthermore, LASER-P outperforms the baseline in 71% of the

---

Published as a conference paper at ICLR 2025
Setup
Contr.
Sem.
Tmp.
FI
#Data
LASER
VI-TWINS
TempCLR
Supervised
0.31
100
46.0
37.0
16.4
52.0
0.31
1000
48.5
22.8
17.10
52.0
0.58
5000
59.6
55.6
31.91
54.4
<'
0.77
Table 3: Ablation study loss function components (contrastive, semantic, and temporal losses), trained and evaluated on the full 20BN dataset. By ablating on loss function components, we find all are essen- tial to the performance of our method.
Table 4: Data efficiency study of the LASER model trained on the MUGEN dataset, compared against VT- twins, TEMP-CLR, and fully supervised baselines. The models were trained on 100, 1,000, and 5,000 data points from the MUGEN dataset, evaluated using action prediction accuracy (%).
fine-grained predicate recognition tasks, and LASER outperforms 59%, shown in Figure For the qualitative study, we present the STSG generator's 's frame-wise predictions on several test data points in Figure 9 In addition this, conduct an ablation study, detailed in Table 3 evaluate the impact of different components of our loss function design.
4.3 MUGEN DATASET
Dataset. MUGEN Haaes 2022 is synthetic dataset that is based on an open-sourced platform game CoinRun (Cobbe et al. 2019). Each datapoint contains a 3.2s video snippet of the game-play and a corresponding automatically generated text describing the video. The agent in the game can perform 6 actions: walk, jump, kill, collect, die, and c 1imb.. A video may contain any sequence of actions, which may be simple or complex. train on 5, 000 training datapoints and test on 12, 851 datapoints.
Experimental Setup. In the MUGEN dataset, each video paired with natural language description that can be represented as ordered list of actions [a1, d22. ...,,........................... a.). The corresponding STSL specifications are naturally expressed in the logical form aj Uaz... Ua,. LASER uses these generated specifications, derived from natural language captions via a crafted semantic parser, as weak supervision labels. We evaluate the model's performance based on action prediction accuracy.
Model Architectures. Since the MUGEN videos differ significantly from natural videos, backbone model is trained from scratch. Our backbone STSG generator model comprises an S3D video encoding model pretrained Kinetics 400, followed by ROIpooler for extract object emheddings, then passed to MIPP layers for relation classification.
Quantitative Study. We evaluate action prediction performance MUGEN and compare it to caption supervised VT-TWINS (Ko et al. (2022). TempCLR (Yang et al. (023b), and a directly super- vised baseline using the same model architecture as ours but with ground truth labels (Supervised). We provide all baselines with extra annotations on the start- and end- end-frame of each dip-based textual description. As shown in Table LASER has better action prediction accuracy despite receiving less supervision than the baselines. Moreover, our weakly supervised model even achieves better accuracy than the Supervised method on when trained on 5000 data points.
CONCLUSION, LIMITATION, AND FUTURE OUTLOOK
We propose LASER, model-agnostic, end-to-end differentiable framework for learning spatio- temporal scene graphs with weak supervision from video captions. We evaluate our work on three datasets OpenPVSG, 20BN, and MUGEN which exhibit diverse temporal properties. LASER significantly outperforms even the fully supervised baselines on STSG generation tasks.
Limitations. LASER still faces scalability challenges regarding both video duration and the number of objects present in the video. Learning STSGs over long time horizons with weak signals remains an open problem. Additionally, LASER's performance is limited by the quality of the video captions. unclear whether more specifications can be elicited from LLMs to reduce reliance solely on captions. We aim to explore these directions in future work.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We sincerely thank the anonymous reviewers for their valuable feedback and constructive suggestions. We also gratefully acknowledge Mayank Keoliya, Matthew Kuo, Amish Sethi, and Neelay Velingker for their help on the project. This research was supported by ARPA-H program Safe and Explainable Al under the award D24AC00253-00 and NSF award CCF 2313010.
REFERENCES
Susanne Albers, Alberto Marchetti-Spaccanee Yossi Matias, Sotiris Nikoletseas, and Wolfgang Thomas. Automata, Languages and Programmming: 36th International Colloquium, ICALP 2009, Rhodes, Greece, July 5-12, 2009, Proceedings, Part I, volume 5555. Springer Science & Business Media, 2009.
Gianluca Apriceno, Andrea Passerini, and Luciano Serafini. A neuro-symbolic approach for real-world event recognition from weak supervision. In 29th International Symposium on Temporal Representation and für Informatik, 2022.
Reasoning. Schloss Dagstubl-Lecibniz-Zeme
Sagar Chaki, Edmund Clarke, Johl Ouaknine, Natasha Sharygina, and Nishant Sinha. Concurrent software verification with states, events, and deadlocks. Formal Aspects of Computing, 17(4):461-483, 2005.
Chien-Ye Yi Chang. De-An Huang, Yanan Sui, Fei-Fei, Juan Carlos Niebles. D3tw: Discriminative differentiable dynamic time warping for weakly supervised action alignment and segmentation. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 3546-3555, 2019.
Karl Cobbe, Oleg Klimov, Chris Hesse, Tachoon Kim, John Schulman. Quantifying generalization in reinforcement learning. In International Conference on Machine Learning, pp. 1282-1289. PMLR, 2019.
Yuren Cong, Wentong Liao, Hanno Ackermann, Michael Ying Yang. and Bodo Rosenhahn. Spatial-temporte transformer for dynamic scene graph generation. CoRR, abs/2107.12300 2021.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scaling egocentric vision: The epic-kitchens dataset. In European Conference on Computer Vision, 2018.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision: Collection, pipeline and challenges epic-kitchens- 100. International Journal of Computer Vision, 130: 33-55, 2022.
Giuseppe De Giacomo and Moshe Y Vardi. Linear temporal logic and linear dynamic logic on finite traces. In IJCAI'13 13 Proceedings of the Twenty- international joint conference on Artificial Intelligence, pp- 854-860. Association for Computing Machinery, 2013.
Ding and Chealiang
beakly-supervised action segmentation with iterative soft boundary assignment. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6508-655 6516, 2018.
Xuchu Ding, Stephen L Smith, Calin Belta, and Daniela Rus. Optimal control of markov decision processes with linear temporal logic constraints. IEEE Trunsactions on Automatic Contrul, 5945):1244-1233 2014.
Nikita Dvornik, Isma Hadji, Konstantinos G. Derpanis, Animesh Garg, and Allan D. Jepson. Drop-dtw: Aligning common signal between sequences while dropping outliers. CoRR, abs/2108.1 1999, 2021.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. Temporal cycle-consistency learning. CoRR, abss/194.07775 2019.
Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu, Violet: End- to-end rideo-language transformers with masked visual-token modeling. arXiv preprint arXiv:2111.12681, 2021.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski. Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fründ, Peter Yianilos, Moritz Mueller-Freitae Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The "something something" video database for learning and evaluating visual common sense. CoRR, 7tlps//706.426.11 2017.
11

---

Published as a conference paper at ICLR 2025
Kristen Grauman, Andrew Westbury, Eugene Byme, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, San- thosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Cartillier, Scan Crane, Tien Do, Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer, Adriano Fragomeni, Qichen Fu, Christian Fuegen, Abrham Ge- breselasie, Cristina González, James Hillis, Xuhua Huang. Yifei Huang, Wenqi Jía, Weslie Khoo, Jáchym Kolár, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li, Zhenqiang Li, Karttikeya Mangalam. Raghava Modhugu, Jonathan Munro, Tullie Murrell, Takumi Nishiyasu. Will Price. Paola Ruiz Puentes. Merey Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo. Yuchen Wang. Xindi Wu. Takuma Yagi. Yunyi Zhu, Pablo Arbeláez, David Crandall, Dima Damen, Giovanni Maria Farinella, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard A. Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg, Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani, Mingfei Yan, and Jitendra Malik. Ego4d: Around the world in 3, 000 hours of egocentric video. CoRR, abs/21 10.07058, 2021.
Tengda Han, Weidi Xie, and Andrew Zisserman. Temporal alignment networks for long-term video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2216, 2022.
Thomas Hayes, Songyang Zhang, Xi Yin, Guan Pang, Sasha Sheng, Harry Yang, Songwei Ge, Qiyuan Hu, and Devi Parikh. Mugen; A playground for vidco-audio-text multimodal understanding and generation. 10.48855/ARX/1.2224444 08058, 2022.
Jiani Huang. Calvin Smith, Osbert Bastani, Rishabh Singh, Aws Albarghouthi, and Mayur Naik. Generating pro- grammatic referring expressions via program synthesis. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 1 119 of Proceedings of Machine Learning Research, pp. 4495 4506. PMLR, 13 18 Jul 2020.
Jiani Huang. Ziyang Li, Binghong Chen, Karan Samel, Mayur Naik, I.c Song, and Xujic Si. Scallop: From probabilistic deductive databases to scalable differentiable reasoning. Advances in Neural Information Processing Systems, 34:25134444 134-25145, 2021.
Jingwei Ji, Rishi Desai, and Juan Carlos Niebles. Detecting human-object relationships in videos. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 8 8106-8116, 2021.
Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le. Yun-Hsuan Sung, Zhen Li. and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. CoRR, ahs/2102.05918, 2021.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloc Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, al. The kinetics human action video dataset. arXiv preprint 30.iv:1705.0699553 2017.
Yonit Kesten, Amir Pnueli, and Li-on Raviv. Algorithmic verification of linear temporal logic specifications. In Automata, Languages and Programming: 25th International Colloquium, ICALP 1998, Aalborg, Denmark, July 13 17, 1998 Proceedings 25, pp. 1-16. Springer. 1998.
Dohwan Ko, Joonmyung Choi, Juyeon Ko, Shinyeong Noh, Kyoung-Wao Woon On, Fun-Sol Kim, and Hyunwoo J Kim. Video-text representation learning via differentiable weak temporal alignment. In Proceedings of the JEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 5016-5025, 2022.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper R. R. Uijlings, Ivan Krasin, Jordi Pont-Tuser Shahab Kamali, Stefan Popov, Matteo Malloci, Tom Duerig, and Vittorio Ferrari. The open images dataset V4: unified image classification, object detection, and visual relationship detection at scale. CoRR, 0bs//811,00998 2018.
Jac Hee Lee, Michael Sioutis, Kyra Ahrens, Marjan Alirezaie, Matthias Kerzel, and Stefan Wermter. Neuro- symbolic spatio-temporal reasoning. In Compendium of Neurosymbolic Artificial Intelligence, pp. 410-429. IOS Press, 2023.
Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq R. Joty, Caiming Xiong, and Steven C. H. Hoi. Align before fuse: Vision and language representation learning with momentum distillation. CoRR, abs/2107.07651, 2021.
Xinghang Li, Di Guo, Huaping Liu, and Fuchun Sun. Embodied semantic scene graph generation. In Aleksandra Faust, David Hsu, and Gerhard Neumann (eds.), Proceedings of the 5th Conference Robot Learning, volume 164 of Proceedings of Machine Learning Research, pp. 1585-1594 PMLR, 08-11 1 Nov 2022.
12

---

Published as a conference paper at ICLR 2025
Ziyang Li, Jiani Huang, and Mayur Naik. Scallop: A language for neurosymbolic programming. Proc. ACM Program. Lang. 7(PLDI, jun 2023, doi: 10.1 145/3591288
Ziyang Li. Jiani Huang. Jason Liu. Felix Zhu, Eric Zhao, William Dodds. Neelay Velingker, Rajeev Alur, and Mayur Naik. Relational programming with foundation models. AAAI 241AA 24. AAAI Press, 2024. ISBN 978-1-57735- 887-99 doi: 10. 10.1609/a.i, v3819.28934.
Nanhao Liang, Yong Liu, Wenfang Sun, Yingwei Xia, and Fan Wang. Ckt-rem: Clip-hased knowledge transfer and relational context mining for unbiased panoptic scene graph generation. In ICASSP 2024-2024 IEFE International Conference on Acoustics, Speech and Signal Processing, pp. 3570-3574. IEEE, 2024.
Hengyue Liu, Ning Yan, Masood Mortazavi, and Bir Bhanu, Fully convolutional scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11546- 11556, June 2021.
Cowu Lu, Ranjay Krishna, Michael S. Bernstein, and Li Fei-Fei. Visual relationship detection with language priors. CoRR, abs/1608.00111 2016.
Joanna Materzynska. Tete Xiao, Roei Herzig, Huijuan Xu, Xiaolong Wang, and Trevor Darrell. Something-elsee Compositional action recognition with spatial-temporal interaction networks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 1049-1059, 2020.
Antoine Miech, Jean- Saan-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to- end learning of visual representations from uncurated instructional videos. CoRR, abs/1912.06430, 2019.
Toki Migimatsu and Jeannette Bohg. Grounding predicates through actions. In 2022 International Conference on Robotics and Automation, pp. 3498-3504. IEEE, 2022.
Sayak Nag, Kyle Min, Subamn Tripathi, and Amit K. Roy Chowdhury. Unbiased scene graph generation in videos. arXiv preprint arXiv:2304.0072 2023.
Amir Pnueli. The temporal logic of programs. 18th Annual Symposium on Foundations of Computer Science, pp. 46-57, 1977.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Alexander Richard, Hilde Kuehne, Ahsan Iqbal, and Juergen Gall. Neuralnetwork-viterhii framework for weakly supervised video learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 7386-73660 7395, 2018.
Dorsa Sadigh, Eric S Kim, Samuel Coogan, S Shankar Sastry, and Sanjit A Seshia. learning based approach to control synthesis of markov decision processes for linear temporal logic specifications. 53rd IEEE Conference on Decision and Control, pp. 1091-1096. IEEE, 2014.
Xindi Shang, Tongwei Ren, Jingfan Guo, Hanwang Zhang, and Tat-Seng Chua. Video visual relation detection. In Proceedings of the 25th ACM international conference on Multimedia, pp. 1300-1308, 2017.
Xindi Shang, Donglin Di. Junbin Xiao, Yu Cao. Xun Yang, and Tat-Seng Chua. Annotating objects and relations in user-generated videos. In Proceedings of the 2019 on International Conference on Multimedia Retrieval, pp. 279-287. ACM, 2019.
Hikaru Shindo, Manuel Brack, Gopika Sudhakaran, Devendra Singh Dhami, Patrick Schramowski, and Kristian Kersting. Deisam: Segment anything with deictic prompting. arXiv preprint arXiv:2402.14122 2024.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 7464 7473, 2019.
Yu Sun, Qian Bao, Wu Liu, Tao Mei, and Michael J. Black. Trace: 5d temporal regression of avatars with dynamic cameras in 3d environments. arXiv preprint arXiv:2306.02850, 2023.
Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. 100m The new data in multimedia research. Communications of the ACM, 59(2):64-73 2016.
Saining Xie. Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification. In Proceedings of the European conference on computer vision, pp. 305-321, 2018.
13

---

Published as a conference paper at ICLR 2025
Hu Xu, Gargi Ghosh, Po-Yao Huang. Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoeer and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. arXiv preprint arXiv:2109. 14084, 2021.
Liu. Meta spatio-temporal debiasing for video scene graph generation. arxiv preprint arXiv:2207. /444,, 2022a.
Xu, Haoxuan Qu, Jason Kuen, Jiuxiang
Ziwei Xu, Yogesh S Rawat, Yongkang Wong, Mohan Kankanhalli, and Mubarak Shah. Don't pour cereal into coffee: Differentiable temporal logic for temporal action segmentation. Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho Advances in Neural Information Processing Systems, 2022b.
Jianwci Yang, Jiasen Lu, Stefan Lec, Dhruv Batra, and Devi Parikh. Graph r-cnn for seene graph generation. In Proceedings of the European conference on computer vision, 670-685, 2018.
Jingkang Yang. Wenxuan Peng, Xiangtai Li. Zujin Guo, Liangyu Chen, Bo Li, Zheng Ma, Kaiyang Zhou, Wayne Zhang. Chen Change Loy, and Ziwei Liu. Panoptic video scene graph generation. arXiv preprint arXiv:2311. 2023a.
Yuncong Yang, Jiawei Ma, Shiyuan Huang, Long Chen, Xudong Lin, Guangxing Han, and Shih-Fu Chang. Tempelr: Temporal alignment representation with contrastive learning, arXiv preprint arXiv:2212.13738, 2023b.
Yuan Yao, Ao Zhang, Xu Han, Mengdi Li, Cornelius Weber, Zhiyuan Liu, Stefan Wermter, and Maosong Sun. Visual distant supervision for scene graph generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 35816-15828 October 2021.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. arXiv preprint ArXiv:2303. 15343, 2023.
Yong Zhang, Yingwei Pan, Ting Yao, Rui Huang. Tao Mei, and Chang Wen Chen. Learning to generate language- supervised and open -pcabuocab scene graph using pre-trained semantic space. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2915-2924, 2023.
Guangming Zhu, Liang Zhang, Youliang Jiang, Yixuan Dang, Haoran Hou, Peiyi Shen, Mingtao Feng, Xia Zhao, Qiguang Miao, Syed Afaq Ali Shah, et al. Scene graph generation: A comprehensive survey, arXiv preprint
2022.
14

---

Published as a conference paper at ICLR 2025
A NATURAL LANGUAGE TO STSL SPECIHICATION
We designed generic prompt template to convert caption into its structured representation. The prompt template consists of the following components:
Set the LLM to be super user in logic programming".
2. Define video location and duration as fractions of the whole video length.
3. Include desired scene graph keywords, such as object names and relations.
4. Provide n-examples of captions and their corresponding JSON structured representations.
We include the full prompts for OpenPVSG dataset and Something-Somettiiee datasct here.
A.1 CHARACTER SETUP
We setup the agent character for all the datasets with this prompt.
You are super user in logic programming.
A.2 CONSTANT DEFINITION
We sctup the examples for temporal specification for all datasets.
Rxamples of precise video locati cns: 1/4, 1/2, 2/3, 1. Examples of event durations: 1/4, 2/3, 1.
We give the concept of "umary predicate" and "binary predicate" for all datasets.
static predicate represents the property of the target object. For example, is_tearable (A) means the can teared.
A unary predicate takes in one argument.
For example, close(A) means A is close to the camera. A binary predicate takes in two argurents. above B.
For example, above (A, B) means
A.3 SCENE GRAPH VOCABULARIES
To bias the GPT-4 model to generate more related words align with the desired scene graph vocabulary, we added the OpenPVSG vocabularies into the prompt as well.
We show the scene graph vocabulary part of OpenPVSG prompt below.
The entites in the video tree, wall, water
be:
acult, baby, bag, The relations in the video can be:
besice, biting, blowing, brushing, .... watering, wearing
We show the scene graph vocabulary part of Sar--Sommmniinmmeettit prompt below.
Static predicates:
is be_deabael,,, is fluid, is heldable, is is_rigid, is_tearable, neq
Unary predicates:
broken, close, closed, deformed, empty, far, twisted, upright
Binary predicates:
above, attached, behind, fits, in, ... touching, under, visible Constants:
hand
A.4 FEW-SHOT EXAMPLE
In both cases, we provide the prompt with examples converting the caption into a structured representation with json format. The target json file includes detailed version of caption. the programmatic version in logical predicates, video location of the event and the duration of the event.
We show example mapping the caption into json format below for OpenPVSG dataset below.
15

---

Published as a conference paper at ICLR 2025
Caption:
"The young receives another gift and sits the floor." "
Action Jaon:
"caption";
"The young boy seceives another gift and sits on the floor.", , "seueential desoriptions"
"boy A receives gift B", "boy A sits on the flcor e*
"t me stamps":
"1";
"decription":
"bo receives gift B",
"progrenmatic": [
holddnn D)",
"nne (8, boy)
"aae (3, gALL)",
b "ducation"; "1/2",
"video location": "0"
"2"
"decription"; "boy sits
floor
1,
"progremmatic"; [
sitting (A, e)", "noee (A, boy) " (C, floor)",
b "duration": "vidao location": "1"
We show one example mapping the caption into json format below for 20BN dataset below.
Action: dig
Act lon predicates dig (A,
Action description: Digging (A] out [8).
Actilon json:
*action": "dig",
"action_pred"; "dig I4, s)",
*explanation"; "Digging [A] out [R)",
"static properties"; [
"A is holdsble", rigid",
"entity A aad 3 aze net equvalent*
1,
proggaamctic veraion": [
"is ho__holdalee
(is_ggiig))
"Aeq(A, R)*
time stamps":
"ing
"deor Montiption":
Not and are far away from the camera",
"A D*,
*A and B are not touched by hand", visible,
visible, and there is hand that is visible"
}.
"programmatte":
not (far (A1), not (B)r
* (n, ")
"not (Bouching hand) not (touching (B, hand)) (vissbiel 18),, visible (B1, visible (hand:"
b
"duration"; *1/5",
"video location": "g"
#2n;
"decription":
become visible",
*A in in a hand", not touching R*
16

---

Published as a conference paper at ICLR 2025
1,
"programmatic";
vissible (A1",
(An hand)",
*not (touching (A, B))
"duration": *1/5",
"vidao locationn: "1"
We have implemented corresponding post-prrcessoos to parse the json structures into STSI. programs.
B EXPERIMENTAL DETAILS
B. HARDWARE
All experiments are conducted on two machines:
two 20-core Intel Xeon CPUs, four GeForce RTX 2080 Ti GPUs, and 768 GB RAM, 2. 3.00GHz Intel Xcon machine with 48 CPUs, 8 A A100s, and 1.5T RAM.
B.2 OPENPVSG
Dataset. The OpenPVSG Yang et al. (2023a) dataset comprises 400 videos sourced from Ego41 Grauman et al. (202]). VidO/Shang et al. (2019); Thomee al. (2016). and EpiceiicheDammnn et al. (2022 2018). This dataset offers fine-grained ground truth annotations of STSGs for 150K frames. These scene graphs encompass 126 object classes and 57 relation classes, including pixel-level object masks. Each video is accompanied by multiple captions with their corresponding vidco snippets. The training dataset contains 1, 538 caption-vidos pairs, while the testing dataset includes 295 caption-vidco pairs. We train and test on these caption-vidco snippet pairs.
Experimental Setup. As illustrated in Figure our objective is to train STSG extraction model capable of taking video clip along with object bounding boxes input and predicting the properties. attributes of objects, and relationships between objects. the ground truth programmatic specification not provided in the OpenPVSG dataset, we only adopt the LASER-C architecture. The 3-shot GPT-4 model processes the captions and converts them into structured representations of events with logical predicates th. By postprocessing, we extract all the unary and binary keywords from the GPT-4 generated sb and use them input to the STSG extraction model. We finetune the SigLIP model to generate the probabilistic spatio-temporal scene graph. This model predicts both similarity scores between the cropped objects and the unary predicate keywords, and the similarity scores between object pains and the binary predicate keywords, yielding a probabilistic STSG. Due to SigLIP's characteristic as an embedding baaeed model, it supports open-world vocabulary and can handle the fuzziness that may be present in the GPT-4 generated structured representations. Due to the hardware limitation, the hatch size to I, and contrastive loss is obtained by randomly sampling another negative specification other than the current specification.
Evaluation Metrics. We use @ecall@k k for evaluating the unary and binary STSG generating performance. The recall metric checks given the ground truth bounding boxes and object pairs, whether the predicted unary predicate, in this case, object names, ranks within the top-k among the ground truth unary predicates. Then compare the accuracy against the ground truth binary labels.
To obtain the evaluation results for the OpenPVSG baselines, the object identification results need to isolated from the entire prediction pipeline. The IPS-based OpenPVSG models first predict a class label for each pixel, then aggregate similar predictions into single object using Unitrack. In contrast, the VPS-based models directly predict object tubes with corresponding labels. To separate object identification from the segmentation task, we identify predicted objects that sufficiently overlap with ground truth objects (with an IOU greater than 0.5), We then calculate the average label for each predicted object across the entire video. The label probability is determined by the proportion of frames in which the object is assigned particular label, relative to the total frames in which the object appears.
Learning Setup For STSG generator, finetune the "google/siglii-bas--paaee 1atassepaacch6-224444 6--244" "openai/elpp-viilare"" patch14", and the pretrained violet model checkpoint with learning rate 0.000001 for 10 epoches. To ensure the inference efficiency, we adopt sampling technique, uniformly sample the whole video frames, and ensure the final generated video has at most 12 frames. Further. we preserve only 10 largest objects on the image, and for each frame, we sample 100 relation pairs at training time, and 300 relation pairs test time. We set the learning batch size to 1.
17

---

Published as a conference paper at ICLR 2025
B.3 LLAVA-VIDEE VIDEO ZERO-SHOP TRANSFERABILITY
We trained CLIP model with LASER method on 10K LLaVA-Vides dataset, and test OpenPVSG evaluation datasct, showing generalizability, robustness, and scalability.
B.3.1 DATASET PREPROCESSING WITH LLM + COMPILER PIPELINE
OpenPVSG Caption to STSL Program The OpenPVSG captions average 12.69 words, containing average of 1.69 events in GPT-extracted representations. The failure rate for converting captions into valid executable STSL formulas 0.78%. A total 453 unique unary keywords 679 unique binary keywords were identified, covering: 92.86% ground-truth unary and 94.74% ground-truth binary keywords in the OpenPVSG dataset. 55.56% ground-truth unary and 61.54% ground-truth binary keywords in the Action Genome dataset. 20% ground-truth unary and 12.88% ground-truth binary keywords the VidVRD dataset. Notably, these closed- world datasets include keywords like "unsure," "other relation," and "not contacting," which rarely occur in natural scenes.
LLa VAooideo 10K Dataset Caption to STSL Program To demonstrate the capability of our LLM + compiler pipeline in handling long and complex caption descriptions, evaluated it on LLaVA- dataset with detailed captions released October 4, 2024. We study the quality of the extracted captions sampled subset of 10,000 video clips (each under 30 seconds). The LLaVA. VA-Video 10K captions average 233.05 words, and the pipeline extracted 4.03 events per video on average. The failure rate for converting captions into valid executable STSL formulas was 0%. A total of 18,458 unique unary keywords and 4,492 unique binary keywords identified, covering: 91.27% ground-trute suuud- unary and 89.47% ground-trun binary keywords in the OpenPVSG dataset. 80.56% ground truth unary and 69.23% ground- truth binary keywords in the Action Genome dataset. 82.86 86% ground-truth unary and 31.82% ground-truth binary keywords in the VidVRD dataset. Notably, 91.99% of LLLaVA- Video samples originate from YouTube, with the remaining samples sourced from ActivityNet, Charades, Ego4D, NexIQA, and YouCook2. This highlights significant domain shift between the LLaVA- Video dataset and all evaluation datasets.
randomly
B.3.2 KEYWORD ANALYSIS
We present the top 10 most frequent unary and binary keywords extracted from captions in both datasets to further illustrate the diversity and robustness of method.
LLa Video Unary
LLaVA-Video Binary
OpenPVSG Unary
OpenPVSG Binary
Category
Count
Category
Count
Category
Count
Category
Count
women
1020
hold
2531
adult
823
on
264
hand
886
wear
2275
child
569
holding
211
man
863
on
1103
man
322
picking
163
text
499
in
811
ball
254
placing
148
child
421
with
725
I
206
using
135
camera
419
adjust
386
dog
187
toward
113
character
414
color
362
toy
127
in
90
hands
256
stand
346
woman
111
throwing
83
room
243
near
286
Baby
106
playing
66
object
239
sit
275
camera
104
sitting on
65
Table 5: Comparison of LLa and OpenPVSG categories and their counts.
B.3.3 COST ANALYSIS
Cost of Generating the STSL from Intermediate Representations Generating all intermediate representations with GPT for the 10K dataset costs approximately $50 and takes around 15 minutes using parallelization techniques. The compilation process from the 10K GPT-generated structured representations executable STSL, programs requires about minute.
18

---

Published as a conference paper at ICLR 2025
B.4 ZERO-SHOT TRANFERABILITY
Training with Noisy Object Trajectories and Long Captions on LLdeeoVideo As the ALLVA-Vides dataset does not provide ground truth object mask-level trajectories. To address this, we preprocess the videos using SAM2.1 to extract object trajectories and train our model with noisy object trajectories and weak supervision labels. A uniform
of hyperparameters for the SAM2.1 mask generator and mask quality was manually verified. On average, we extract 19.87 object trajectories each video, and on average 10.61 objects occur on single frame. Compared to the OpenPVSGi dataset, there are 20.47 ohject trajectories for each video, and on average 10.36 objects occur on single frame. For the Action Genome and VidVRD datasets, only coarse- grersegg bounding boxes available. During evaluation, these hounding boxes converted into masks by setting all parts within the bounding boxes True and areas outside to False.
determined through grid search,
The chart below illustrates the learning performance of our method, leveraging CLIP as the backbone. The results demonstrate that our approach is robust handling complex caption descriptions, noisy object trajectories and out-of-domain transfer scenarios.
Eval Dataset & LASER FT Strategy
Unary R@K
Binary R@K K
Dataset
Strategy
R@I
R@5
R@ 10
R@1 1
R @@5
R 10
Base
0.1633
0.3381
0.4404
0.0197
0.0673
0.0988
OpenPVSG
LLaVA-FT
0.2368
0.5000
0.5789
0.1191
0.3534
0.5346
OpenPVSG-FT
0.2778
0.5231
0.6402
0.1482
0.4214
0.5398
Base
0.1487
0.4166
0.5911
0.0509
0.2464
0,5478
Action Gnome
LLaVA-FT
0.1768
0.4795
0.6471
0.1255
0.5250
0.6772
OpenPVSG-FT
0.1422
0.4367
0.6277
0.2407
0.3085
0.6927
Base
0.6267
0.8408
0.9332
0.0499
0.1696
0.2445
VidVRD
LLaVA-FT
0.6444
0.8899
0.9585
0.0678
0.1979
0.2966
OpenPVSG-FT 0.5925
0.8322
0.9161
0.0211
0.1179
0.1715
Table 6: Performance comparison of LASER finetuning strategies across different evaluation datasets. Note that the LLaVA-FT is out of distribution and OpenPVSG-FT is in domain.
B.4.1 USING STSL AS STSG
We conducted an experiment on the OpenPVSG eval split, comparing the performance of model directly using the generated STSI. data (referred to as STSL-aS-STSG) against our LASER fine-tuned CL.IP-based STSG model. Here is the detailed experimental sctup, partially addressing the aforementioned challenges:
the spaCy package project caption- caption-extracted STSL. keywords into the dptaset-speeific vocabulary. When direct match was found, we took the top- 10 predictions based on similarity scores and assigned the similarity score as the predicted likelihood. Assuming Perfect Grounding: To address challenge (b) and (c), we assumed a hypothctical, idcal algorithm that grounds entities and their relations into specific frames of the video perfectly, bypassing emrors related to spatial-temporal grounding. Evaluation Metric An object is counted likely predictions of the projected STSL within predicted event duration. Similarly, a pair objects counted as a match for binary if their binary relation is among the top-k most likely predictions of the projected STSL within the predicted event duration. Even under these over-optimistic assumptions, our experiment revealed that model directly using STSL data without fine-tuning performs significantly worse compared the fine-tuned CLIP-based STSG model. Through our investigation, we found that the performance degradation is primarily due to (d) mismatched granularity, which cannot he easily addressed without fine-tuning. As a side note, our fine-tuned STSG model produces results much faster than STSL-as-STSS during inference time. On average, calling the full STSL -asSSTG pipeline takes about 34.5 seconds per video for extracting the STSI. from the video, while directly calling the fine-tuned CLIP model takes about 2.6 seconds per video on the OpenPVSG dataset.
Vocabulary Alignment: To address challenge (a),
match for unary if its name appears the top-k most
B.4.2 STATE-OF-TIE--AT FULLY SUPERVISED BASELINES
While STTran and TRACE demonstrate strong performance in video scene graph generation, they are closed- domain models with fixed classification layers, limiting their ability transfer knowledge across datasets.
19

---

Published as a conference paper at ICLR 2025
Eval Dataset & LASER FT Strategy
Unary R@K
Binary @@K
Dataset
Strategy
R@1 1
R@5
R@ 10
R@1
R@5
R@ 10
Base
0.1633
0.3381
0.4404
0.0197
0.0673
0.0988
LLaVA-FT
0.2368
0.5000
0.5789
0.1191
0.3534
0.5346
OpenPVSG
OpenPVSG-FT
0.2778
0.5231
0.6402
0.1482
0,4214
0.5398
(NEW) aTSL-as-STSG
0.0514
0.1132
0.1288
0.0362
0.0583
0.0716
Table 7: Performance comparison of LASER FT CLIP strategies on the OpenPVSG dataset.
Additionally, their architectures are not designed to handle noisy labols directly extracted from captions, making them less robust in such scenarios.
It is important note that the primary contributions of STTran and TRACE lie in their carefully designed model architectures for capturing visual features. In contrast, our work focuses on an orthogonal problem: developing a modcl-agnoste weak supervision training pipeline. Our approach is not in competition with STTran or TRACE but rather complements them. In fact, STTran and TRACE can be adopted as backbone models within our LASER framework, demonstrating that our method enhances their capabilities rather than being mutually exclusive.
Nevertheless, we report unary and binary predicate recall STTran using author- provided checkpoint. Note that STTran is trained in tomaiaain on the Action Genome dataset a fully supervised manner, while our models are trained on out-of-domain datasets using a weakly supervised approach.
Eval Dataset & Strategy
Unary R@K K
Binary R@K K
Dataset
Strategy
R@1
R@5
R@10
R@I I
R@5
R@ @
10
Base CLIP
0.1487
0,4166
0.5911
0.0509
0.2464
0.5478
LLaVA-FT
0.1768
0.4795
0.6471
0.1255
0.5250
0.6772
Action Gnome
OpenPVSG-FT
0.1422
0.4367
0.6277
0.2407
0.3085
0.6927
STTran
0.0671
0.2879
0.4632
0.4097
0.7815
0.8886
Table Performance comparison of different strategies on the Action Gnome dataset. We note that LLaVA-FT and OpenPVSG-FT is out of distribution, and weakly supervised, while STTran is in domain and fully supervised.
B.5 MUGEN
Task
SDSC
LASER-P
Video-to-Tex Retrieval Text-to- Video Retrieval
87.00%
93.80 %
86,80%
90.00% %
Table 9: Comparison to baselines downstream retrieval tasks with the MUGEN dataset.
Model Architecture We present the model architecture overview in Figure 10b The video is first segmented into clips through sliding window of length 2. Then, each clip is encoded through S3D model and yields clip-wise embedding, The embedding further passed into BiLSTM model obtain the context for each clip. An action classifier takes the concatenation the clip- based embedding and its context embedding, and classifies each clip into 6 actions: walk, jump, kill, collect, dic, and cl imb. Further, a modifier classifier predicts the 1 possible direction of the actions: left, right, up, down. The matching process of action and its modifier is also performed in the reasoner. For example, combination of col lect, left will be invalidated during reasoning.
Learning Setup We train the model on 5000 training datapoints, and 12, 851 test datapoints. The contrastive loss is obtained over a batch size 3: the violation loss is constructed with regard to the axioms in GPA MMgimmtsuu & Bohg 2022). We train the model with learning rate of 0.0001, violation weight 0.01, number of epochs 100, and batch size 3. The Scallop reasoning engine setup using difitopkproofs provenance with l set 5.
20

---

Published as a conference paper at ICLR 2025
Object
Viepi (hodde / CXX RESTTD
Itonder
00100 30
Actio ClasEsr
Clasifiss
Attrilate Clasific
Reluiio Clasiliar
Oiped
(a) 20BN-Sone Homeehing-Somethiien classifier model architecture used in LASER-P.
(b) MUGEN model classifier architecture used LASER-P.
Suap1
jump ()
collect (coin)
walk teight)
collect (coin)
walk (eight)
kill (alimw)
walk (eight)
collect (coin)
Time
Figure 11: Examples of the predicted MUGEN actions predicted. predicate for brievity.
We elide showing M in each
We construct the temporal specification from the given text description using simple heuristic-basee semantic parser. We first extract ordered action list (01, Ouy..., 6..] from the text with heuristics, then construct the temporal specification as 0₃ Uaz Ua.. This specification means actions are performed one by one with no intermediate gaps. The generated specifications are used as ground truth programmatic STSL labels. As the MUGEN video data is very different from the natural videos, we train a classification model from scratch. Our neural perception model consists of an S3D video encoding model pretrained Kinetics 400 Kay et al. (2017). and MLP layers classify the actions. The video is first passed through the S3D Xie et al. (2018) model for frame-basee based embeddings. Then the embeddings input to a 2-layer MLP classifier. We set the number of frames per clip to 2, and the batch size to due to the hardware limitation. The classifier thus produces clip-wise video representations. We employ contrastive loss and semantic loss for training. More details are shown in Appendix B.5
occur a video clip are concatenated as
Quantitative Study. We evaluate action prediction performance MUGEN and compare it caption supervised VT-TWINS Ko al. (2022). TempCLR Yang et al.) (2023b). and directly supervised baseline (Supervised). We provide all baselines with extra annotations on the start- and end-frame of each clip-based textual description. As the ground truth actions for the video are diffcrent from the text description, we use heuristics obtain the start and end frames. For a fair comparison, using Kinetics 400 (the same as LASER) along with the bascline models. This label used for training both VT-TWINS and fully supervised models. As shown in Table 4 LASER has better action prediction accuracy despite receiving less supervision than the baselines. Moreaver, our weakly supervised model even achieves better accuracy than the Supervised method on #Data = 5000. We further evaluate our approach on downstream video-specification retrieval task. Given videos and specifications, correctly aligned pairs, retrieve specification given video and vice versa. denote two tasks as spec-retrieval and video-retrieval respectively. Specifically, we evaluate the accuracy with which we infer whether the correct specification has the highest alignment score, and vice versa. LASER outperforms an embedding based baseline SDSC Hayes et al. (2022) both tasks (Table D). Our approach can even identify actions that persist for a very short period of time, such as k an enemy.
fine-tune S3D backbone pre-trained
want match the
21

---

Published as a conference paper at ICLR 2025
B.6 -OBETHNMT SOOMEHHIG-SSOETTING
Frame
Video:
pull right
temporal
1.000: (Aft(A)
1.000: lAft(A)
1.000: left(A)
1.000: left(A)
1,000: right (A)
1.000: right(A)
1.000: right
1.000: right (A)
temporal + contrastive
0. 950: left(A)
0. 732: left(A)
O. 0.991: left(A)
O. 912: left(A)
0.318: right(
0.484: right
0.827: right
0.657: right
temporal + contrastive
0.302: (Af
(Af 66:: right (A)
(eft(A) 0.819: right
left(A) 0.994: right (A)
0.587: right (A)
violation
Figure 12: Comparing model performance trained with different loss functions.
Dataset Details trained on 10, 000 datapoints and tested on 14, 816 datapoints. There are 172 actions with 37 underlying predicates in this dataset. Specifically, there are static predicates, 21 unary predicates, and 10 binary predicates. The static predicates represent object attributes that persist over time; the unary predicates reflect single object's state; the binary predicates reflect the relationship between two objects.
Model Architecture We use a 4-layer convolutional model to extract the fcatures for cach frame and ROIpooler obtain the embedding for each object on the frame. To obtain the static predicates, we pass frame-wise wise object embeddings through an LSTM encoder to obtain the video-wise object embedding, and a 2-layer MLP classifier yields the output distribution. For the unary predicates, 2-layer MLP classifier takes in the concatenation of the frame embedding, object embedding, and the object bounding box, and generates the output distribution. For the binary predicates, a 3-layer MLP classifier takes in concatenation the frame embedding, two object embeddings, and two object bounding boxes, and generates the output distribution. The overview of the architecture is shown in Figure
We use convolutional model for the frame features, an ROT Pooler for object features, an LSTM model for video-wis wise object embeddings, and MLP layers for predicate output distributions. We adopt temporal supervision loss with the prior that the precondition and postcondition should be far away from each other: contrastive learning loss with batch size 3: and semantic loss with a weight of 0.05, where the integrity constraints, such as "a rigid object is not fluid", are obtained from the original PDDL file. More training details are included in the appendix.
Instead of using just querying for the alignment score Pr(r + v). we instead query for the conditional alignment score Pr(r }= vild) which conditioned on distance d between Whrr and alyvar are satisfied, In the experiment, we set the threshold dain to be 0.9dmax-
Learning Setup. The contrastive loss is obtained over a batch size 3; the violation loss is constructed with regard the axioms in GPA Migimatsu & Bohg (022). We train the model with lcarning rate of 0.0001, violation weight 0.05, number of epochs 50, and batch size 3. The Scallop reasoning engine setup using difftopkproofs provenance with k=3.
Ablation Studies. We study how different loss impact the qualitative evaluation result, as shown Figure 12 In this task, the spatiotemporal specifications are manually crafted, which also means a lot of biases are introduced. As can sec, using only temporal loss yields us counter-intuitive result, 1.00 probability for both the object the left of the camera and the right of the camera. This is mainly due the imbalanced low-level supervision that introduced by human bias. By adding contrastive loss, we can see an improvement in that the model predicts the ground truth position with higher probability compared to its counterpart. Incorporating a violation loss can further improve the performance in that the sum of the probabilities is closer to 1.0.
More Qualitative Studies We include qualitative examples predicted by the LASER-P model in Figure
22

---

Published as a conference paper at ICLR 2025
0.969:
0.986: :eocchiniiigggggo (A.3)
614:: torcching (A.
0.420::: touching
0.507: touchiing (A.3)
0.362: toruching (A.B)
0e479::elosed
(Al
0.565:
0.856: copen
0. 2.691: open
0.743:
0.379:
407407:000 :07::open000
break
deformed
660:::d deforeed
06866:
326::broeench (A)
543- broken
brrokenn
605: bookkn
754: broken
705
broken
0.860: ::0D:bbndable
0. 0.860: bendabla l (A)
0.860: bendabblll
0.860:beedae 460::bendable (A)
860: bendabba La
360:
:bendable (A)
0.634: 6.634:ccese:
818::close
855:
push-off
961: :attchh 93293322t0 :attach
0.910:
11:920::aa :Attcchh
0.806: rattach
650:: (A,
Time
Figure 13: The STSGs predicted by LASER on the 20BN dataset. The actions are labeled on the left.
23


Reviews:
[{"review_id": "xeqjjFnawP", "reviewer": "Reviewer_gWyP", "summary": "This paper introduces a neural-symbolic framework to learn spatio-temporal scene graph generation from the weak supervisory signals provided by video captions. It is achieved by utilizing an LLM to convert a video caption into a programmatic specification and then compute the alignment scores by a pretrained vision-language embedding model. Experiments are conducted to show the effectiveness of the proposed weakly-supervised method in performance.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.This paper is well-written and the organization is reasonable and easy to follow.\n\n2.Experimental results show that the proposed method achieves state-of-the-art performance that can even outperform fully-supervised methods.\n\n3.The introduction of neural-symbolic representations and reasoning sounds interesting and could be a promising direction to explore.", "weaknesses": "1.Although the authors claimed that they proposes the first framework to learn weakly-supervised scene graph generation from video caption data, the utilization of image caption data in weakly-supervised scene graph generation has been widely explored [1, 2], and I think this work falls into a straightforward adaptation to the video domain, so this contribution is not that appealing.\n\n2.The idea to incorporate LLMs into scene graph generation has also been investigated by several previous works [3, 4]. This makes the novelty of this framework less significant.\n\n3.For using neural-symbolic techniques, as have been mentioned in the manuscript, it could be hard to accurately transform a video caption into a structured representation like formal program. However, the proposed method heavily relies on the extracted programmatic specification for making predictions and this would possibly become a weakness that reduces the robustness of the method. Moreover, if the video captions become even more complex and longer, I am wondering whether the method will simply fail since the programmatic representations can be very inaccurate in such scenario. It is suggested to add experimental results on the model's sensitivity to the LLM's capability.\n\n4.In terms of performance comparison, the authors claimed that their weakly-supervised framework outperforms fully-supervised frameworks, but I think this is kind of unfair since the complexity and external models like LLMs adopted in this work are much more expensive and stronger than many fully-supervised method. So what would be the comparative result if no LLMs are used in this work? If LLMs are necessary for this method to work, then how much complexity and resources will it cost to use the LLMs?\n\nReferences\n\n[1] Linguistic Structures As Weak Supervision for Visual Scene Graph Generation, Ye et al.\n\n[2] Learning to Generate Scene Graph from Natural Language Supervision, Zhong et al.\n\n[3] GPT4SGG: Synthesizing Scene Graphs from Holistic and Region-specific Narratives, Chen et al.\n\n[4] LLM4SGG: Large Language Models for Weakly Supervised Scene Graph Generation, Kim et al.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bfGf5r2AC1", "reviewer": "Reviewer_p4rQ", "summary": "This paper introduces LASER, a framework that leverages video captions as weak supervision to train spatio-temporal scene graph (STSG) generators, addressing the challenge of costly manual STSG annotations. LASER utilizes large language models to derive spatio-temporal semantic specifications from video captions, then aligns STSG predictions with these specifications using a differentiable symbolic reasoner and specialized loss functions. This approach enables efficient training of STSG models without annotated data, achieving notable improvements over fully-supervised baselines on datasets such as OpenPVSG, 20BN, and MUGEN.", "questions": "Lines 428-430 describe the baseline; however, it seems that no state-of-the-art (SOTA) methods for relation extraction are mentioned. Does this paper surpass the SOTA methods for image or video relation extraction?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper proposes a new framework the uses specification language, for specifying fine-grained video semantics.\n2. The experimental results show good performance compared with fully-supervised one.", "weaknesses": "1.  The design of this approach seems puzzling. If the STSG can be generated based on a series of advanced models such as CLIP and LLM, why need to train an additional model to make predictions based on the generated data? What are the problems with directly using the generated data as the results, and what is its performance?\n2. The description of the experimental section is also quite unclear. For instance, it is not specified which state-of-the-art (SOTA) methods are used for comparison or whether they represent the most advanced approaches. Additionally, there are no specific references provided, making it difficult to assess the relative effectiveness of the proposed method.\n3. The motivation behind STSL is not very clear. Its relationship with STSG is also unclear, as well as why this step is necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cnNdu1rKI1", "reviewer": "Reviewer_sYch", "summary": "The paper proposes LASER, a framework for training spatio-temporal scene graph (STSG) generators using video captions as weak supervision. LASER uses large language models to extract logical specifications from captions and aligns these with predicted STSGs through a differentiable symbolic reasoner and a mix of contrastive, temporal, and semantic losses. This approach enables training STSG generators without manually annotated STSG data. LASER is evaluated on OpenPVSG, 20BN, and MUGEN datasets, showing notable improvements over supervised baselines.", "questions": "Could you clarify the specific criteria or methods used by the language model to extract logical specifications? Additional details on handling ambiguous or underspecified captions would be helpful.\n\nHave you considered using additional forms of weak supervision, such as visual cues or scene descriptions, to compensate for the limitations of captions alone?\n\nHow well does the approach generalize to more complex, real-world videos beyond the tested datasets? Testing on diverse, large-scale datasets could provide valuable insights.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The paper introduces a novel approach for training spatio-temporal scene graph (STSG) models using video captions as weak supervision, which removes the need for labor-intensive STSG annotations. It leverages large language models to extract logical specifications, which is innovative in combining neuro-symbolic learning with weakly supervised STSG generation.\n\nThis work has high significance as it opens a new path for STSG learning, making it accessible without costly annotation processes. Its performance gains on OpenPVSG, 20BN, and MUGEN suggest practical implications for scalable STSG generation in video understanding.", "weaknesses": "The reliance on large language models to extract logical specifications may introduce noise and inaccuracies, especially if the captions lack detailed descriptions. Enhancing this extraction process or adding error-handling mechanisms could improve overall model performance.\n\nThe evaluation covers three datasets but could be strengthened by testing on additional complex datasets for scene graph generation to validate generalizability, such as Action Genome, VidVRD, and VIdOR.\n\nThe paper does not include strong works on scene-graph generation, such as STTran, TRACE. It is hard to evaluate the robustness of the proposed framework without the comparison with such methods.\n\nSTTran: https://openaccess.thecvf.com/content/ICCV2021/papers/Cong_Spatial-Temporal_Transformer_for_Dynamic_Scene_Graph_Generation_ICCV_2021_paper.pdf\n\nTRACE: https://openaccess.thecvf.com/content/ICCV2021/papers/Teng_Target_Adaptive_Context_Aggregation_for_Video_Scene_Graph_Generation_ICCV_2021_paper.pdf", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SoyLEX3mKC", "reviewer": "Reviewer_nqrt", "summary": "This paper uses large language models to generate pseudo-labels for spatiotemporal scene graph generation models without intensive human annotations. It generally contains a probabilistic relational database construction section, a spatiotemporal specification language description section, a natural language parsing section, and a spatiotemporal alignment checking section. A loss function that combines contrastive, temporal, and semantic components is exploited. The authors conduct extensive experiments on three popular datasets: OpenPVSG, 20BN, and MUGEN.", "questions": "I have no further questions. See the weakness part please.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "1. Exploring large language models to generate pseudo labels for vision-language learning tasks is intuitive nowadays. The proposed method designs an elegant pipeline to generate pseudo labels for the spatio-temporal scene graph generation task.", "weaknesses": "1. The significance of the target task spatiotemporal scene graphs needs to be discussed further. Since the large language models generate the pseudo label, they should be able to create scene graphs with appropriate prompts. So, the problem formulation and significance do not seem solid enough. It would be better to compare the proposed approach more explicitly to directly using LLMs to generate scene graphs. If the experiments are too expensive, the authors can discuss the potential advantages of the proposed method over state-of-the-art MLLMs. The authors can also explain why training a separate STSG model is beneficial.\n2. Some technical details are not clear enough. The symbols in equation (1) are not explained well. \n\n $~~~~~~~~~~~~~~~~~\\mathcal{L}\\left ( Pr\\left ( M_{\\theta} \\left ( X \\right ) |= \\psi  \\right ) ,1 \\right )  $\n\nI am unclear about what the $\\psi$ stands for and why $ M_{\\theta} \\left ( X \\right )$ needs to condition on 1. What does the \"1\" stand for?\n \n3. There are many key concepts and methodologies that are not explained in the method section. For example, in line 181, what does \"0.05::unary_atom(\"deformed\", 3, e)\" stand for, especially the number \"0.05\"? The same problem appears in line 182. \n\n   In lines 213-215, the logic component set is not given. It's an essential point for readers unfamiliar with the spatiotemporal scene graph to understand the paper.\n\n4. The spatio-temporal scene graph part in the right-up corner of Figure 1 is unclear. The same problem appears in Figure 6.\n\nIn Figure 1, in the third row, there are \"$\\neg $touching (B, H)\" both on the left and the right side. What this means seems to be ambiguous.\n\n5. The paper does not present state-of-the-art MLLM results. To demonstrate the advantage of the proposed method, authors should consider some multimodal large language models, such as BLIP-2, InstructBLIP, MiniGPT-4, and LLaVA-1.5.\n\n6. The title uses the keyword \"neuro-symbolic,\" but it seems that the proposed method doesn't emphasize this point. It deserves further discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QwetqrmAXd", "reviewer": "Reviewer_ym8p", "summary": "The paper presents LASER, a model-agnostic neuro-symbolic framework designed to learn Spatio-Temporal Scene Graphs from videos using weak supervision derived from video captions. Traditional supervised methods for training STSGs are hindered due to reliance on labor-intensive annotated datasets, while LASER addresses this challenge by employing LLMs to extract logical specifications and then training the underlying generator to align the STSG with the specification.", "questions": "1. In Table 3, setup 1 & 2 have the same F1 score, while the full setting of losses is higher than setup 3. Can you explain this phenomenon? \n2. See weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The overall presentation is clear and the figures are well-illustrated.\n2. The introduction of the STSL provides a flexible and expressive way to capture the nuances of video semantics, and the alignment checking ensures the relations between STSL and STSG.\n3. Extensive experiments and ablation studies demonstrate the effectiveness of LASER in learning STSG.", "weaknesses": "1. The performance of LASER heavily relies on the quality of video captions. If captions are sparse or inaccurate, the resulting generated STSGs may also suffer in quality.\n2. The model seems to have a problem building scene graphs with longer videos and more instances. \n3.  While the framework is promising, the scalability of the approach to very large datasets or real-time applications may pose challenges to the model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)