Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper extended weakly supervised image scene graph generation to videos, and presented a new task of weakly supervised video scene graph generation, where a model must learn from unlocalized scene graphs on a sparse set of video frames. In tandem with the new task, the paper also proposed an interesting solution. The key to the solution is to generate pseudo scene graph labels by leveraging existing object detectors, distilling from image scene graph models, and modeling temporal continuity in videos. The proposed method is evaluated on the Action Genome dataset with some promising results.  

strength_and_weaknesses: **Strength**

In spite of an extension from images to videos, the proposed task of weakly supervised video scene graph generation is new.

The proposed method harnesses multiple cues (object detector, image scene graph model, transition probability between predicates) to generate pseudo scene graph labels for training. The method is interesting and could provide a baseline for future research. 

**Weakness**

While the task and the method is interesting, the technical components of the proposed method are less exciting. Many of the ideas can be found in prior works, including (1) using object detector to ground unlocalized scene graphs (e.g., Zhong et al., 2021), (2) distilling from another model (see works on knowledge distillation), (3) using visual tracking to augment training samples (described as model-free teacher), and (4) modeling of relation transitions (Mi et al., 2021).  The proposed method seems like a reasonable combination of multiple existing ideas from video understanding and image scene graph generation. It is thus difficult to gauge the key innovation here. 

The gap between a strawman baseline (learning a model from the same set of sparse frames with unlocalized scene graphs) and the proposed method is rather small (1% on SGDet). In fact, a stronger and perhaps more realistic baseline is listed as model C in Table 1 (within a gap of 0.7% on SGDet), where a model is trained using sparse scene graphs plus augmented annotations from simple tracking. It is not totally clear if this gap is sufficient to justify the proposed method, which is arguably more sophisticated in its training scheme.

**Other Minor Comments**

A key argument in the introduction to motivate this work is that relations that involve motion (e.g., walking vs. running) can not be readily distinguished in static images. This unfortunately remains an issue for the proposed method, as it relies on image features from an object detector (L285). I wonder if the authors have considered using video features (e.g., from 3D convolutional networks or video transformers) for the proposed task. 

I thought the baseline listed Table 2 is referring to model A in Table 1, yet the numbers do not quite match. It will be great if the authors can provide some clarification here. 

L322, I think the text is referring to Table 2 instead of Table 1.

clarity,_quality,_novelty_and_reproducibility: The paper is globally well written with some minor rough edges in the text. The proposed task is interesting yet the solution is less innovative. For reproducibility, some implementation details are provided in the paper and the authors promised to release the code in the future. 
summary_of_the_review: The paper presented an interesting new task of weakly supervised video scene graph generation, and proposed a solution to the task. While the solution is quite reasonable and combines several existing ideas, it lacks technical innovation and the results are less satisfactory. Overall, I am not enthusiastic about this paper. 

The author response has addressed most of concerns, and I have raised my rating accordingly.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper propose a new task that only leverage a single frame with unlocalized annoation for video-level scene graph generation. To do this, the author propose an approach called PLA to create pseudo labels for unannotated frames, and train the fully-supervised VidSGG model with these pseudo labels.  The results on Action Genome benchmark demonstrates the effectiveness of PLA. 
strength_and_weaknesses: Strength:
1. The paper propose to do vidSGG with weak supervision, which is a good contribution. 
2. The proposed approach of pseudo label is reasonable and easy to follow. 
3. The ablation study shows the effectiveness of proposed modules. 

Weakness:
1. Approach

a. The approach seems only supports short video without drastic changes. If the scene changes, the proposed approach may not work since the pseudo label assigiment will not be correct. 

b. The approach is not very novel. The pipeline is a simple combination of three parts, and all of them are not very new, but just a simple pseudo label->training pipeline. 

2. Experiments

a. How to choose the annotation frame? Is it chosen randomly, or mid-one? How about choose the first frame? An experiment of different annotation frame choosing strategy should be added. 

b. Should add experiments to compare with fully-supervised approach to show how much is behind this upper-bound. 

c. Should compare with existing VidSGG methods metioned in Related Work. 

d. Please show more qualitative illustrations. Maybe in supplementary file. 

clarity,_quality,_novelty_and_reproducibility: Quality: I think the approach is not novel and experiments are not sufficient. 
Orginality, Clarity: Good. Easy to follow. 
Originality: The main contribution is to use weak supervision for VidSGG task, but the proposed method is not very novel. 
summary_of_the_review: Due to the lack of experiments and the lack of novelty, I think the paper is below the acceptance bar.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposed a weakly supervised setting where single-frame ground truth annotation is provided among many video frames; and the single ground truth annotation only provides an unlocalized scene graph. To tackle this problem, a new Psuedo Label Assignment (PLA) method is proposed. 
strength_and_weaknesses: [advantages]
1. The paper is well-written and easy to follow.
2. The proposed setting is important in practice.
3. The method of using model-based and model-free methods to generate pseudo labels and fusion of the updated weight is interesting. Experiments demonstrate the effectiveness of the proposed PLA.

[weaknesses]
The experiments are somewhat insufficient to justify the claims made by the authors. See suggestions below. 

The paper can be improved in the following aspects:

1. The problem setting is claimed to be weakly supervised. However, the proposed method requires a pre-trained object detector. If my understanding is correct, this object detector is trained in a fully supervised setting on ground truth bounding boxes and class labels of a given image. To some extent, the proposed method still requires bounding boxes. It does not fit the weakly supervised setting as described.

2. Follow-up on 1, what if the authors are NOT using pre-trained object detectors or using an imperfect object detector with noisy outputs, would PLA perform efficiently? Can the authors quantify the performance of PLA when there are e.g. 10%, 20%, 40%, and 80% errors made by the pre-trained object detectors?

3. Can the authors show how would the size of the weakly supervised labels influences PLA performance? e.g. the performance of PLA against 10%, 20%, 40%, and 80% annotated video frames out of the entire number of video frames? How would all the baselines perform in these titrated cases?

4. So far, the authors only demonstrate the effectiveness of the method on Action Genome. There is around 1% improvement over various testing scenarios (Table 1). To make the claim convincing, please include the same table but test on another video graph generation dataset to benchmark with SOTA.

5. Data availability (see below)

[minor] Explain what is R (recall) in R@10, R@20 and so on. 
clarity,_quality,_novelty_and_reproducibility: 5. This is a nice engineering work consisting of many concepts and components explored by the community already. Given the complicated designs of various components and hyperparameters, would the source code become available for reproducibility purposes? Please include such statements in the paper.
summary_of_the_review: So far, my ranting is weakly rejected due to insufficient experiments; however, I am happy to change the rating depending on the feedback provided by the authors in the rebuttal.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper deals with the video scene graph generation problem. 1) this paper proposes a new task named SF-VidSGG, which requires to train the VidSGG models with only weak supervision. 2) this paper proposes a novel pseudo-label assignment (PLA) method for the SF-VidSGG task. In particular, PLA decouples the problem into two steps: 1) assigning a pseudo-localized scene graph to each frame in the video, and 2) training a VidSGG model in a fully supervised manner using the pseudo-localized scene graph. Extensive experiments show the effectiveness.
strength_and_weaknesses: Strengths
1)The proposed SF-VidSGG task is interesting and meaningful. The weakly supervised scene that this task focuses on are more in line with real-world cases and thus help bridge the gap between video scene graph generation research and practical applications.
2)The proposed PLA method is novel and model-agnostic. In particular, it generates different “pseudo labels” using multiple-teachers, and dynamically assigns weights for these teachers by considering the temporal dependencies in videos.

Weaknesses
The paper does not give an exact reason for choosing the “single-frame weak supervision”. It is required to provide a comprehensive discussion on different choices of weak supervision and explain why not using other types of weak supervision.
clarity,_quality,_novelty_and_reproducibility: This paper is well-written and is easy to follow
This paper proposes a new task for VidSGG. It is meaningful for real-world application. The proposed method seems work and interesting.
summary_of_the_review: This paper proposes a weakly-supervised VidSGG task. It utilizes the unlocalized scene graph of intermediate frames and proposes a model-agnostic pseudo label assignment method PLA. The experiments showthe effectiveness.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper investigates a new task, i.e., weakly supervised video scene graph generation (VidSGG). The authors then proposed a novel approach to address this challenging task, and the key idea is to generate pseudo labels for unannotated frames. Experimental results on the Action Genome dataset are reported and discussed. 

Overall, this paper is well written and clearly organized. A major contribution of this work is to present a new task, i.e., weakly supervised VidSGG. The proposed method is well motivated and easy to follow, and the experiments can demonstrate the effectiveness of the method.

Reviewers raised some concerns regarding novelty, technical details and experimental settings. Reviewers provided detailed responses to all questions as well as additional results, which addressed most of the concerns from reviewers. Consequently, reviewers raised the scores. The authors are strongly encouraged to incorporate the suggestions from reviewers in the final version.
justification_for_why_not_higher_score: This paper presents a new task and also designs a new solution, however, there are still limitations in paper writing and experiments. 
justification_for_why_not_lower_score: This paper presents a new task and also designs a new solution, and reviewers have positive ratings.
Decision: Accept: poster