Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: - This paper introduces a Transformer-based architecture for video situation recognition that a) augments prior work on Semantic Role Labeling in videos with spatiotemporal grounding, and b) removes the requirement of ground-truth role-verb mapping, enabling end-to-end Grounded Video Situation Recognition (GVSR).
- The proposed model -- VideoWhisperer -- has 3 main components -- a) a Transformer encoder to learn object embeddings per frame, and event embeddings, both contextually enriched via self-attention; b) a Transformer decoder to predict the entities for each role, conditioned on the object and event embeddings; c) a Transformer decoder to autoregressively predict the caption for each role conditioned on the output features of (b).
- To evaluate the model on grounding, the authors collected annotations by asking users to draw bounding boxes around roles in each event clip, for the validation set. These annotations will be released and/or incorporated in the official evaluation toolkit. These annotations are not used for training (and so the proposed model is only weakly-supervised).
- Experiments are thorough, both the ablations to demonstrate the importance of each design choice, as well as comparison to prior work on the SRL benchmark. On the SRL benchmark, the proposed approach significantly outperforms prior work, and in addition, supports spatiotemporal localization, unlike prior works.
strengths_and_weaknesses: The proposed model makes sense, is well-validated by experiments, and importantly enables joint prediction of actions, roles, spatiotemporal localization of roles, and captions (unlike prior approaches that need ground-truth role-verb annotations at test time). Great work!
questions: - Which pretrained object detector backbone does VideoWhisperer use?
- It seems like role prediction accuracy (~45%) is limiting downstream performance to quite an extent. Some discussion on next steps to further improve this would be useful to include in the paper.
limitations: - The authors have included a brief overview of limitations in the appendix.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studied the problem of video situation recognition, which predicts structured information like multiple events, relationships, actions, and verb-robe pairs.  The authors propose a three-stage Transformer model, VideoWhisperer to handle the task. The model first learns contextualized embeddings for video features in parallel with key objects that appear in the video clips. The model then attends and pools information from object embeddings to localize answers to questions. The final stage of the model generates answers as captions to describe each verb-role pair in the video. Experiments are conducted on the VidSitu dataset, achieving better performance than the baseline models.
strengths_and_weaknesses: **Strengths**
1. The paper overall is well written and easy to follow.
2. The idea to decompose the task into different stages seems to be technically sound and the ablation study in the experimental section shows its effectiveness.
3. As the conclusion of the introduction, the authors also provide new data annotation to an existing dataset.

**Weaknesses**
1. The first concern is about the novelty of the main idea of the paper. The proposed mode seems to be complicated to decompose the task into three stages. Whether it is possible to compare the computation complexity between the proposed method and the baseline.
2. Second, as a method paper, this paper only conducts experiments on a recent proposed dataset. And the dataset contains only a simple baseline. This makes the paper less convincing. Whether it is possible to extend the proposed method to other video situation understanding datasets like (1) to further prove the effectiveness of the proposed method? 
3. It will also be great to implement more baselines on the existing benchmark.

 (1). Wu, Bo, et al. "STAR: A benchmark for situated reasoning in real-world videos." Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track 2021.
questions: Please see the **weaknesses** in the previous section.
limitations: Yes, the authors adequately addressed the limitations and potential negative social impact of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: VidSitu involves predicting the actions, entities involved in the actions, and simple relations between the actions, given a short video clip. This paper proposes spatio-temporal grounding (GVSR — Grounded Video Situation Recognition) as an addition to the VidSitu task. Spatio-temporal grounding involves localizing the entities which helps disambiguation of entities and co-referencing entities across different actions. 

The paper also proposes a novel Transformer model that (unlike previous works) jointly predicts all task outputs — verbs, verb-role pairs, nouns, grounding. The VideoWhisperer model has 3 stages:

1. Learning contextualized features for the video and key objects simultaneously. Predicts the action label (verb roles) for each event. 
2. Semantic role labeling, i.e., generating a caption (noun) for each verb role in each event. 
3. Localization is achieved by finding the largest attention score for a bounding box in the (ROTD) Transformer, for each verb-role in each event.

The proposed model has a few advantages — (1) results are predicted jointly, in a single forward pass, (2) improves entity captioning accuracy and (3) localizes verbs accurately despite not having access to grounding annotations during training. 

strengths_and_weaknesses: Grounding semantic predictions in the given video, is an important problem that is well-motivated. Accurate grounding can improve the reliability, robustness, trust in the model, and possibly also prediction accuracy. 

The GVSR task is an obvious extension of GSR [13] (which focused on grounding in imSitu [7]) to VidSitu [6]. Although the originality of the task is low, it is indeed useful, and improves the experimental setup of the existing VidSitu task. 

The main contribution of the paper is (a) localization annotations (for testing) on VidSitu, and (b) the VideoWhisperer model. Unlike previous works on VidSitu, the proposed model is trained end-to-end, and outputs the entire set of structured outputs in a single forward-pass. This architecture is a significantly simpler implementation. Such an architecture allows learning (common) representations, and modeling the context across sub-tasks of VidSitu. 

By leveraging cross-attention across object bounding box regions in the transformer model, localization can be achieved by weak supervision (via the caption loss). There is no need for ground-truth localization annotations. This allows one to potentially scale up the training as long as only language annotations are available. 

VidSitu requires the prediction of the action label (verb), the entities (nouns) associated with the verb-role and the relationship between events. The ground-truth language descriptions of the entities may be ambiguous, and have some variance — e.g., “Woman in blue shirt”, “woman in glasses”, etc. could all refer to the same person. Thus, evaluating captions is challenging, and imprecise. The entity localization task (measured by IoU) is less ambiguous. Further, it provides an additional dimension of understanding and interpretability of the model predictions. Thus GVSR alleviates some of the evaluation issues in VidSitu (to an extent). 

There are significant improvements in performance in entity-captioning accuracy + localization of verb-roles. 

questions: 
-  The main challenge with verb prediction seems to be the long-tailed distribution of verbs in VidSitu. Have the authors considered simple approaches to handle this, e.g., class-balanced loss [a] and/or focal loss [b] ? While these are unlikely to completely solve the verb imbalance problem, they may result in some improvements. 
- The object features are only computed from subsampled frames from the video. However, the event embeddings are computed from video features from the Slow-Fast network. I wonder if the authors have tried to compute event embeddings from subsampled frame features instead of full video. This would reduce the computational overhead (video vs. keyframes) if true. 

References
[a] Cui, Yin, et al. "Class-balanced loss based on effective number of samples." CVPR. 2019.
[b] Lin, Tsung-Yi, et al. "Focal loss for dense object detection." CVPR. 2017.

limitations: This work contributes to the general direction of understanding semantics in videos. However, current learning methods require ground-truth labels and large amounts of data. A huge challenge in this area is the difficulty in scaling this up to a large number of action labels (verbs) and roles.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces a new task of performing structured situation recognition over time in videos, called Grounded Video Situation Recognition (GVSR). The idea is that for a new video, a model must identify which verbs are happening, identify their arguments (specifying who did what to whom), and then determine where in the video that event happened. 

The authors create a dataset for this task built around VidSitu [6], also devising a set of metrics to handle multiple events happening in the same video (5 events per video, each video having 10 seconds). The authors collect extra annotations of the validation set to benchmark human performance.

The paper introduces a new model for this task called VideoWhisperer. Frames are sampled from a video, and a pretrained object detector is used to extract object embeddings from the resulting frames. A separate pretrained video backbone extracts features from the whole video. A transformer encoder fuses these features together, and a separate transformer decoder is used to perform structured prediction (for the semantic role labeling over videos). Results and ablations show that this model performs well at both localized video situation recognition, as well as the non-localized case (VidSitu).
strengths_and_weaknesses: To this reviewer, this paper is a strong overall contribution. It tackles an important and unsolved problem (fine-grained video understanding) and introduces a new dataset (Grounded VidSitu) on top of VidSitu that might help evaluate this. The proposed model seems like a first step in this direction (pipelining the problem of structured video understanding), and helpful ablations are provided to show where the model might be improved in future work.

Weaknesses:
* To this reviewer, the model feels a bit hacky and overly complex. I have a suspicion that a simpler model would perform better, but I'm also OK with this being a problem that future work ought to address.
* Adding a limitations/social impact section would improve the paper considerably.
* Grounded seems like an overloaded term, do you mean something like "Localized Video Situation Recognition"?
* The modeling section (Sec 3) was confusing to this reviewer. It might be helpful to discuss the high-level strategy first before jumping into the details, as well as the function of the ROTD/VOTE networks. Those acronyms like they are referred to that often, so maybe dropping them would boost understanding.
* Some qualitative examples and discussion would help.
questions: * Qualitatively, what examples does this approach work well on and where does it struggle?
* How important are the choice of video representations (\phi_{vid} vs. \phi_{obj}) ?
limitations: There isn't a limitations section or a discussion on the societal impacts, adding this would improve the paper.
review_title: null

Meta-review: metareview: All four reviewers agreed that this paper tackles an important problem, and the proposed approach is novel and well supported by sufficient empirical evidence. They also acknowledged the new data annotation contribution and that that the paper is generally written well. There were several nice suggestions to improve the paper. The authors are strongly encouraged to incorporate them in the final version. 
confidence: Certain
Decision: Accept