LASER: A Neuro-Symbolic Framework for Learning Spatio-Temporal Scene Graphs with Weak Supervision

Authors: Jiani Huang, Ziyang Li, Mayur Naik, Ser-Nam Lim
Primary area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords: neuro-symbolic, video understanding, spatial-temporal scene graph, weak supervision

Abstract:
Supervised approaches for learning spatio-temporal scene graphs (STSG) from video are greatly hindered due to their reliance on STSG-annotated videos, which are labor-intensive to construct at scale. Is it feasible to instead use readily available video captions as weak supervision? To address this question, we propose LASER, a neuro-symbolic framework to enable training STSG generators using only video captions. LASER employs large language models to first extract logical specifications with rich spatio-temporal semantic information from video captions. LASER then trains the underlying STSG generator to align the predicted STSG with the specification. The alignment algorithm overcomes the challenges of weak supervision by leveraging a differentiable symbolic reasoner and using a combination of contrastive, temporal, and semantics losses. The overall approach efficiently trains low-level perception models to extract a fine-grained STSG that conforms to the video caption. In doing so, it enables a novel methodology for learning STSGs without tedious annotations. We evaluate our method on three video datasets: OpenPVSG, 20BN, and MUGEN. Our approach demonstrates substantial improvements over fully-supervised baselines, achieving a unary predicate prediction accuracy of 27.78% (+12.65%) and a binary recall@5 of 0.42 (+0.22) on OpenPVSG.  Additionally, LASER exceeds baselines by 7% on 20BN and 5.2% on MUGEN in terms of overall predicate prediction accuracy.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730036674169

thread_type:
official_review

<thread_object>
creator: reviewer_ym8p

<content>
summary:
The paper presents LASER, a model-agnostic neuro-symbolic framework designed to learn Spatio-Temporal Scene Graphs from videos using weak supervision derived from video captions. Traditional supervised methods for training STSGs are hindered due to reliance on labor-intensive annotated datasets, while LASER addresses this challenge by employing LLMs to extract logical specifications and then training the underlying generator to align the STSG with the specification.

strengths:
1. The overall presentation is clear and the figures are well-illustrated.
2. The introduction of the STSL provides a flexible and expressive way to capture the nuances of video semantics, and the alignment checking ensures the relations between STSL and STSG.
3. Extensive experiments and ablation studies demonstrate the effectiveness of LASER in learning STSG.

weaknesses:
1. The performance of LASER heavily relies on the quality of video captions. If captions are sparse or inaccurate, the resulting generated STSGs may also suffer in quality.
2. The model seems to have a problem building scene graphs with longer videos and more instances. 
3.  While the framework is promising, the scalability of the approach to very large datasets or real-time applications may pose challenges to the model.

questions:
1. In Table 3, setup 1 & 2 have the same F1 score, while the full setting of losses is higher than setup 3. Can you explain this phenomenon? 
2. See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730549495560

thread_type:
official_review

<thread_object>
creator: reviewer_nqrt

<content>
summary:
This paper uses large language models to generate pseudo-labels for spatiotemporal scene graph generation models without intensive human annotations. It generally contains a probabilistic relational database construction section, a spatiotemporal specification language description section, a natural language parsing section, and a spatiotemporal alignment checking section. A loss function that combines contrastive, temporal, and semantic components is exploited. The authors conduct extensive experiments on three popular datasets: OpenPVSG, 20BN, and MUGEN.

strengths:
1. Exploring large language models to generate pseudo labels for vision-language learning tasks is intuitive nowadays. The proposed method designs an elegant pipeline to generate pseudo labels for the spatio-temporal scene graph generation task.

weaknesses:
1. The significance of the target task spatiotemporal scene graphs needs to be discussed further. Since the large language models generate the pseudo label, they should be able to create scene graphs with appropriate prompts. So, the problem formulation and significance do not seem solid enough. It would be better to compare the proposed approach more explicitly to directly using LLMs to generate scene graphs. If the experiments are too expensive, the authors can discuss the potential advantages of the proposed method over state-of-the-art MLLMs. The authors can also explain why training a separate STSG model is beneficial.
2. Some technical details are not clear enough. The symbols in equation (1) are not explained well. 

 $~~~~~~~~~~~~~~~~~\mathcal{L}\left ( Pr\left ( M_{\theta} \left ( X \right ) |= \psi  \right ) ,1 \right )  $

I am unclear about what the $\psi$ stands for and why $ M_{\theta} \left ( X \right )$ needs to condition on 1. What does the "1" stand for?
 
3. There are many key concepts and methodologies that are not explained in the method section. For example, in line 181, what does "0.05::unary_atom("deformed", 3, e)" stand for, especially the number "0.05"? The same problem appears in line 182. 

   In lines 213-215, the logic component set is not given. It's an essential point for readers unfamiliar with the spatiotemporal scene graph to understand the paper.

4. The spatio-temporal scene graph part in the right-up corner of Figure 1 is unclear. The same problem appears in Figure 6.

In Figure 1, in the third row, there are "$\neg $touching (B, H)" both on the left and the right side. What this means seems to be ambiguous.

5. The paper does not present state-of-the-art MLLM results. To demonstrate the advantage of the proposed method, authors should consider some multimodal large language models, such as BLIP-2, InstructBLIP, MiniGPT-4, and LLaVA-1.5.

6. The title uses the keyword "neuro-symbolic," but it seems that the proposed method doesn't emphasize this point. It deserves further discussion.

questions:
I have no further questions. See the weakness part please.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730682342337

thread_type:
official_review

<thread_object>
creator: reviewer_sych

<content>
summary:
The paper proposes LASER, a framework for training spatio-temporal scene graph (STSG) generators using video captions as weak supervision. LASER uses large language models to extract logical specifications from captions and aligns these with predicted STSGs through a differentiable symbolic reasoner and a mix of contrastive, temporal, and semantic losses. This approach enables training STSG generators without manually annotated STSG data. LASER is evaluated on OpenPVSG, 20BN, and MUGEN datasets, showing notable improvements over supervised baselines.

strengths:
The paper introduces a novel approach for training spatio-temporal scene graph (STSG) models using video captions as weak supervision, which removes the need for labor-intensive STSG annotations. It leverages large language models to extract logical specifications, which is innovative in combining neuro-symbolic learning with weakly supervised STSG generation.

This work has high significance as it opens a new path for STSG learning, making it accessible without costly annotation processes. Its performance gains on OpenPVSG, 20BN, and MUGEN suggest practical implications for scalable STSG generation in video understanding.

weaknesses:
The reliance on large language models to extract logical specifications may introduce noise and inaccuracies, especially if the captions lack detailed descriptions. Enhancing this extraction process or adding error-handling mechanisms could improve overall model performance.

The evaluation covers three datasets but could be strengthened by testing on additional complex datasets for scene graph generation to validate generalizability, such as Action Genome, VidVRD, and VIdOR.

The paper does not include strong works on scene-graph generation, such as STTran, TRACE. It is hard to evaluate the robustness of the proposed framework without the comparison with such methods.

STTran: https://openaccess.thecvf.com/content/ICCV2021/papers/Cong_Spatial-Temporal_Transformer_for_Dynamic_Scene_Graph_Generation_ICCV_2021_paper.pdf

TRACE: https://openaccess.thecvf.com/content/ICCV2021/papers/Teng_Target_Adaptive_Context_Aggregation_for_Video_Scene_Graph_Generation_ICCV_2021_paper.pdf

questions:
Could you clarify the specific criteria or methods used by the language model to extract logical specifications? Additional details on handling ambiguous or underspecified captions would be helpful.

Have you considered using additional forms of weak supervision, such as visual cues or scene descriptions, to compensate for the limitations of captions alone?

How well does the approach generalize to more complex, real-world videos beyond the tested datasets? Testing on diverse, large-scale datasets could provide valuable insights.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731059285203

thread_type:
official_review

<thread_object>
creator: reviewer_p4rq

<content>
summary:
This paper introduces LASER, a framework that leverages video captions as weak supervision to train spatio-temporal scene graph (STSG) generators, addressing the challenge of costly manual STSG annotations. LASER utilizes large language models to derive spatio-temporal semantic specifications from video captions, then aligns STSG predictions with these specifications using a differentiable symbolic reasoner and specialized loss functions. This approach enables efficient training of STSG models without annotated data, achieving notable improvements over fully-supervised baselines on datasets such as OpenPVSG, 20BN, and MUGEN.

strengths:
1. This paper proposes a new framework the uses specification language, for specifying fine-grained video semantics.
2. The experimental results show good performance compared with fully-supervised one.

weaknesses:
1.  The design of this approach seems puzzling. If the STSG can be generated based on a series of advanced models such as CLIP and LLM, why need to train an additional model to make predictions based on the generated data? What are the problems with directly using the generated data as the results, and what is its performance?
2. The description of the experimental section is also quite unclear. For instance, it is not specified which state-of-the-art (SOTA) methods are used for comparison or whether they represent the most advanced approaches. Additionally, there are no specific references provided, making it difficult to assess the relative effectiveness of the proposed method.
3. The motivation behind STSL is not very clear. Its relationship with STSG is also unclear, as well as why this step is necessary.

questions:
Lines 428-430 describe the baseline; however, it seems that no state-of-the-art (SOTA) methods for relation extraction are mentioned. Does this paper surpass the SOTA methods for image or video relation extraction?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731093294342

thread_type:
official_review

<thread_object>
creator: reviewer_gwyp

<content>
summary:
This paper introduces a neural-symbolic framework to learn spatio-temporal scene graph generation from the weak supervisory signals provided by video captions. It is achieved by utilizing an LLM to convert a video caption into a programmatic specification and then compute the alignment scores by a pretrained vision-language embedding model. Experiments are conducted to show the effectiveness of the proposed weakly-supervised method in performance.

strengths:
1.This paper is well-written and the organization is reasonable and easy to follow.

2.Experimental results show that the proposed method achieves state-of-the-art performance that can even outperform fully-supervised methods.

3.The introduction of neural-symbolic representations and reasoning sounds interesting and could be a promising direction to explore.

weaknesses:
1.Although the authors claimed that they proposes the first framework to learn weakly-supervised scene graph generation from video caption data, the utilization of image caption data in weakly-supervised scene graph generation has been widely explored [1, 2], and I think this work falls into a straightforward adaptation to the video domain, so this contribution is not that appealing.

2.The idea to incorporate LLMs into scene graph generation has also been investigated by several previous works [3, 4]. This makes the novelty of this framework less significant.

3.For using neural-symbolic techniques, as have been mentioned in the manuscript, it could be hard to accurately transform a video caption into a structured representation like formal program. However, the proposed method heavily relies on the extracted programmatic specification for making predictions and this would possibly become a weakness that reduces the robustness of the method. Moreover, if the video captions become even more complex and longer, I am wondering whether the method will simply fail since the programmatic representations can be very inaccurate in such scenario. It is suggested to add experimental results on the model's sensitivity to the LLM's capability.

4.In terms of performance comparison, the authors claimed that their weakly-supervised framework outperforms fully-supervised frameworks, but I think this is kind of unfair since the complexity and external models like LLMs adopted in this work are much more expensive and stronger than many fully-supervised method. So what would be the comparative result if no LLMs are used in this work? If LLMs are necessary for this method to work, then how much complexity and resources will it cost to use the LLMs?

References

[1] Linguistic Structures As Weak Supervision for Visual Scene Graph Generation, Ye et al.

[2] Learning to Generate Scene Graph from Natural Language Supervision, Zhong et al.

[3] GPT4SGG: Synthesizing Scene Graphs from Holistic and Region-specific Narratives, Chen et al.

[4] LLM4SGG: Large Language Models for Weakly Supervised Scene Graph Generation, Kim et al.

questions:
Please refer to the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732370105874

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Overall Rebuttal

comment:
We sincerely thank all reviewers for their thoughtful feedback and constructive comments on our work. We deeply appreciate the recognition of our innovation in using large language models (LLMs) to weakly supervised STSG learning (sYch, nqrt), the novel neural-symbolic paradigm (gWyP, sYch, nqrt), LASER's strong performance (gWyP, p4rQ), the expressiveness and flexibility of our STSL design (p4rQ, ym8p), the elegance of our pipeline design (nqrt), and the clarity of our writing and comprehensiveness of our experiments (gWyP, p4rQ, ym8p). In the following, we address common concerns shared across multiple reviewers and provide detailed responses to individual feedback.

## **1. Clarifying Symbolic Components and Their Role in Neural-Symbolic Programming (gWyP, p4rQ, nqrt)**

To improve the clarity of our approach, we will include a dedicated background section on neuro-symbolic programming. We further elaborate on our neuro-symbolic component here using the example shown in Figure 6 and described in lines 305–309.

Our neuro-symbolic programming methodology is rooted in the concept of a **probabilistic relational database**, where relations are expressed in the format `<prob>::<predicate_name>(<n-tuple>)`. For example, the relation `0.8::name("o1", "hand")` indicates that the entity `"o1"` has a 0.8 probability of being identified as a "hand." In our framework, each entity corresponds to an object trajectory within the video.

The **STSL program** is represented deterministically in our database, such as `1.0::name(v1, "hand")`, which specifies that the variable `v1` must be grounded to an object identified as a `"hand"`. An alignment between this program and the relationship in the STSG above suggests that `v1` could be `"o1"` with a likelihood of $0.8$.

### **(a) Construction of the Probabilistic Database with Video STSG and STSL Program**

In our framework, probabilities are derived from the outputs of our STSG generation model to construct the video’s probabilistic database, while the extracted STSL program remains deterministic. Using this representation, object trajectory recognition results can be expressed as follows:

- `0.8::name("o1", "hand"); 0.2::name("o1", "sponge")`: `"o1"` is recognized as a "hand" with 0.8 probability and as a "sponge" with 0.2 probability.
- `0.1::name("o2", "hand"); 0.9::name("o2", "sponge")`: `"o2"` is identified as a "hand" with 0.1 probability and as a "sponge" with 0.9 probability.

Similarly, relations are represented probabilistically, e.g., `0.7::touch("o1", "o2")` indicates that `"o1"` is touching `"o2"` with a 0.7 probability. For simplicity, we use shorthand notations like `0.7::touch` in Figure 6, which extends these representations across multiple frames:

- **Frame 1**: `0.7::touch`, `0.3::drop`
- **Frame 2**: `0.6::touch`, `0.4::drop`
- **Frame 3**: `0.2::touch`, `0.8::drop`
- **Frame 4**: `0.1::touch`, `0.9::drop`

This probabilistic relational framework bridges neural outputs with symbolic reasoning, forming the basis of our neuro-symbolic approach.

### **(b) Example of STSL Program Grounding**

The STSL specification we aim to ground into this video STSG database is `"touch U drop"`, meaning the agent performs "touch" followed by "drop." The full STSL program is:

`Until(Finally(touch(v1,v2)), Finally(drop(v1, v2)))`.

To evaluate this, our checker uses a dynamic programming algorithm to determine the top-`k` most likely groundings that satisfy the specification. For example, the top-3 proofs are as follows:

1. The agent performs "touch" at frames 1 and 2, and "drop" at frames 3 and 4, with a probability of: $0.7 * 0.6 * 0.8 * 0.9 = 0.3024$
2. The agent performs "touch" at frame 1, and "drop" at frames 2, 3, and 4, with a probability of: $0.7 * 0.4 * 0.8 * 0.9 = 0.2016$
3. The agent performs "touch" at frames 1, 2, and 3, and "drop" at frame 4, with a probability of: $0.7 * 0.6 * 0.2 * 0.9 = 0.0756$

The aggregate probability of these three proofs is: $0.3024 + 0.2016 + 0.0756 = 0.5796$

We refer to this as the **alignment score** between the video STSG and the STSL program.

In real-world scenarios, logical expressions are often far more complex. To handle such cases, we designed a full **domain-specific language** (shown in Figures 4 and 5) and implemented a **neuro-symbolic alignment checker** (detailed in Appendix C) capable of computing alignment scores for any program expressed in this language.

During training, our goals are to:

1. Maximize the alignment score between a video and its corresponding STSL specification, pushing it towards $1$.
2. Minimize the alignment score between a video and an unmatched STSL specification, pushing it towards $0$.

This approach enables contrastive learning and provides fine-grained feedback for our STSG model.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents LASER, a model-agnostic neuro-symbolic framework designed to learn Spatio-Temporal Scene Graphs from videos using weak supervision derived from video captions. Traditional supervised methods for training STSGs are hindered due to reliance on labor-intensive annotated datasets, while LASER addresses this challenge by employing LLMs to extract logical specifications and then training the underlying generator to align the STSG with the specification. A loss function that combines contrastive, temporal, and semantic components is exploited. LASER is evaluated on OpenPVSG, 20BN, and MUGEN datasets, showing notable improvements over supervised baselines.

Strengths:
+ This paper proposes a new framework that uses specification language, for specifying fine-grained video semantics.
+ The experimental results show good performance compared with a fully-supervised one.
+ This paper is well-written and the organization is reasonable and easy to follow.

Initial Weaknesses:
+ The idea to incorporate LLMs into scene graph generation has also been investigated by several previous works. The novelty of the proposed method is limited.
+ The motivation is not clear, and the design of the approach seems puzzling.
+ The paper does not include strong works on scene-graph generation, such as STTran, or TRACE. It is hard to evaluate the robustness of the proposed framework without the comparison with such methods.
+ More results on state-of-the-art MLLMs are needed.