Kati8/ICLR_2023_2025 huggingface dataset:
authors: Jiani Huang
authors: Ziyang Li
authors: Mayur Naik
authors: Ser-Nam Lim
reviews: 
Comments: The paper presents LASER, a model-agnostic neuro-symbolic framework designed to learn Spatio-Temporal Scene Graphs from videos using weak supervision derived from video captions. Traditional supervised methods for training STSGs are hindered due to reliance on labor-intensive annotated datasets, while LASER addresses this challenge by employing LLMs to extract logical specifications and then training the underlying generator to align the STSG with the specification.
Strengths: 1. The overall presentation is clear and the figures are well-illustrated.
2. The introduction of the STSL provides a flexible and expressive way to capture the nuances of video semantics, and the alignment checking ensures the relations between STSL and STSG.
3. Extensive experiments and ablation studies demonstrate the effectiveness of LASER in learning STSG.
Weaknesses: 1. The performance of LASER heavily relies on the quality of video captions. If captions are sparse or inaccurate, the resulting generated STSGs may also suffer in quality.
2. The model seems to have a problem building scene graphs with longer videos and more instances. 
3.  While the framework is promising, the scalability of the approach to very large datasets or real-time applications may pose challenges to the model.
Recommendation: 6
Reviewer confidence: 2
Soundness/correctness: 3
Clarity: 2
reviews: 
Comments: This paper uses large language models to generate pseudo-labels for spatiotemporal scene graph generation models without intensive human annotations. It generally contains a probabilistic relational database construction section, a spatiotemporal specification language description section, a natural language parsing section, and a spatiotemporal alignment checking section. A loss function that combines contrastive, temporal, and semantic components is exploited. The authors conduct extensive experiments on three popular datasets: OpenPVSG, 20BN, and MUGEN.
Strengths: 1. Exploring large language models to generate pseudo labels for vision-language learning tasks is intuitive nowadays. The proposed method designs an elegant pipeline to generate pseudo labels for the spatio-temporal scene graph generation task.
Weaknesses: 1. The significance of the target task spatiotemporal scene graphs needs to be discussed further. Since the large language models generate the pseudo label, they should be able to create scene graphs with appropriate prompts. So, the problem formulation and significance do not seem solid enough. It would be better to compare the proposed approach more explicitly to directly using LLMs to generate scene graphs. If the experiments are too expensive, the authors can discuss the potential advantages of the proposed method over state-of-the-art MLLMs. The authors can also explain why training a separate STSG model is beneficial.
2. Some technical details are not clear enough. The symbols in equation (1) are not explained well. 

 $~~~~~~~~~~~~~~~~~\mathcal{L}\left ( Pr\left ( M_{\theta} \left ( X \right ) |= \psi  \right ) ,1 \right )  $

I am unclear about what the $\psi$ stands for and why $ M_{\theta} \left ( X \right )$ needs to condition on 1. What does the "1" stand for?
 
3. There are many key concepts and methodologies that are not explained in the method section. For example, in line 181, what does "0.05::unary_atom("deformed", 3, e)" stand for, especially the number "0.05"? The same problem appears in line 182. 

   In lines 213-215, the logic component set is not given. It's an essential point for readers unfamiliar with the spatiotemporal scene graph to understand the paper.

4. The spatio-temporal scene graph part in the right-up corner of Figure 1 is unclear. The same problem appears in Figure 6.

In Figure 1, in the third row, there are "$\neg $touching (B, H)" both on the left and the right side. What this means seems to be ambiguous.

5. The paper does not present state-of-the-art MLLM results. To demonstrate the advantage of the proposed method, authors should consider some multimodal large language models, such as BLIP-2, InstructBLIP, MiniGPT-4, and LLaVA-1.5.

6. The title uses the keyword "neuro-symbolic," but it seems that the proposed method doesn't emphasize this point. It deserves further discussion.
Recommendation: 6
Reviewer confidence: 3
Soundness/correctness: 4
Clarity: 3
reviews: 
Comments: The paper proposes LASER, a framework for training spatio-temporal scene graph (STSG) generators using video captions as weak supervision. LASER uses large language models to extract logical specifications from captions and aligns these with predicted STSGs through a differentiable symbolic reasoner and a mix of contrastive, temporal, and semantic losses. This approach enables training STSG generators without manually annotated STSG data. LASER is evaluated on OpenPVSG, 20BN, and MUGEN datasets, showing notable improvements over supervised baselines.
Strengths: The paper introduces a novel approach for training spatio-temporal scene graph (STSG) models using video captions as weak supervision, which removes the need for labor-intensive STSG annotations. It leverages large language models to extract logical specifications, which is innovative in combining neuro-symbolic learning with weakly supervised STSG generation.

This work has high significance as it opens a new path for STSG learning, making it accessible without costly annotation processes. Its performance gains on OpenPVSG, 20BN, and MUGEN suggest practical implications for scalable STSG generation in video understanding.
Weaknesses: The reliance on large language models to extract logical specifications may introduce noise and inaccuracies, especially if the captions lack detailed descriptions. Enhancing this extraction process or adding error-handling mechanisms could improve overall model performance.

The evaluation covers three datasets but could be strengthened by testing on additional complex datasets for scene graph generation to validate generalizability, such as Action Genome, VidVRD, and VIdOR.

The paper does not include strong works on scene-graph generation, such as STTran, TRACE. It is hard to evaluate the robustness of the proposed framework without the comparison with such methods.

STTran: https://openaccess.thecvf.com/content/ICCV2021/papers/Cong_Spatial-Temporal_Transformer_for_Dynamic_Scene_Graph_Generation_ICCV_2021_paper.pdf

TRACE: https://openaccess.thecvf.com/content/ICCV2021/papers/Teng_Target_Adaptive_Context_Aggregation_for_Video_Scene_Graph_Generation_ICCV_2021_paper.pdf
Recommendation: 6
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 2