Grounded Video Situation Recognition
Authors: Zeeshan Khan
C.V. Jawahar
Makarand Tapaswi
Venue: NeurIPS 2022 Accept (2022)
Keywords: Video Understanding
Vision and Language
Visual Semantic Role Labelling
Spatio-temporal Grounding

Abstract: Dense video understanding requires answering several questions such as who is doing what to whom, with what, how, why, and where. Recently, Video Situation Recognition (VidSitu) is framed as a task for structured prediction of multiple events, their relationships, and actions and various verb-role pairs attached to descriptive entities. This task poses several challenges in identifying, disambiguating, and co-referencing entities across multiple verb-role pairs, but also faces some challenges of evaluation. In this work, we propose the addition of spatio-temporal grounding as an essential component of the structured prediction task in a weakly supervised setting, and present a novel three stage Transformer model, VideoWhisperer, that is empowered to make joint predictions. In stage one, we learn contextualised embeddings for video features in parallel with key objects that appear in the video clips to enable fine-grained spatio-temporal reasoning. The second stage sees verb-role queries attend and pool information from object embeddings, localising answers to questions posed about the action. The final stage generates these answers as captions to describe each verb-role pair present in the video. Our model operates on a group of events (clips) simultaneously and predicts verbs, verb-role pairs, their nouns, and their grounding on-the-fly. When evaluated on a grounding-augmented version of the VidSitu dataset, we observe a large improvement in entity captioning accuracy, as well as the ability to localize verb-roles without grounding annotations at training time.

Grounded Video Situation Recognition
Zeeshan Khan
C. V. Jawahar
Makarand Tapaswi
CVTT, MIT Hyderabad
httpp:://eeeshank05 hww.:shhannngg...hbbbbb. http::i...gthubboooooogoo io/grvidsitt
Abstract
Dense video understanding requires answering several questions such as who is doing what to whom, with what, how, why, and where. Recently, Video Situation Recognition (VidSitu) is framed as task for structured prediction of multiple events, their relationships, and actions and various verb-role pairs attached to descriptive entities. This task poses several challenges in identifying, disambiguat- ing, and co-referencing entities across multiple verb-role pairs, but also faces some challenges of evaluation. In this work, we propose the addition of spatio- temporal grounding as an essential component of the structured prediction task weakly supervised setting, and present novel three stage Transformer model, VideoWhispere that is empowered to make joint predictions. In stage one, we learn contextualised embeddings for video features in parallel with key objects that appear in the video clips to enable fine-grained spatio-temporal reasoning. The second stage sees verb-role queries attend and pool information from object embeddings, localising answers to questions posed about the action. The final stage generates these answers as captions to describe each verb-role pair present in the video. Our model operates on a group of events (clips) simultancously and predicts verbs, verb-role pairs, their nouns, and their grounding ol- When evaluated on large improvement entity captioning accuracy, as well as the ability localize verb-roles without grounding annotations at training time.
version of the VidSitu dataset, we observe a
Introduction
At the end of The Dark Knight, we see a short intense sequencethat involves Harvey Dent toss coin while holding gun followed by sudden action. Holistic understanding of such video sequence, especially one that involves multiple people, requires predicting more than the action label (whar verb). For example, why they are doing it (purpose goal), how are they doing it (manner). where are they doing (location), and even what happens after (multi-event understanding). While humans perceive the situation and are good at answering such questions, many works often focus on building tools for doing single tasks, e.g. predicting actions [8] detecting objects [2, 4] or image/video captioning [18, 28]. We are interested in assessing how some of these advances can be combined for holistic understanding of video clips.
may wish to answer questions such who performed the action (agent),
able to
recent and audacious step towards this goal is the work by Sadhu el al. [27]. They propose Video Situation Recognition (VidSitu), structured prediction task over five short clips consisting of three sub-problems: (i) recognizing the salient actions in the short clips; (ii) predicting roles and their entities that are part of this action; and (iii) modelling simple event relations such enable or cause. Similar to the predecessor image situation recognition (imSitu [39]), VidSitu annotated using Semantic Role Labelling (SRL) [21]. A video (say 10s) is divided into multiple small events (~22) and each event associated with a salient action verb (e.g. hit). Each verb has a fixed set of roles arguments, e.g. argnt prtient-Argl tool-Arg2, LocationAArgM(LLcatton manner-
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

ArgM(manner), etc., and each role is annotated with free form text caption, egg. agent: Blonde Woman, as illustrated in Fig. 1.
Grounded VidSitu. VidSitu poses various challenges: long-tailed distri- bution of both verbs and text phrases, disambiguating the roles, overcoming semantic role-noun pair sparsity, and co-referencing of entities in the en-
Video
Video
Video 3
Event NV
tire video. Moreover, there is ambi- guity in text phrases that refer to the
same unique entity (e.g. "man in white shirt" or "man with brown hair"). A model may fail to understand which attributes are important and may bias towards a specific caption (or pattern like shirt color), given the long-tailed distribution. This is exacerbated when multiple entities (e.g. agent and pa- tient) have similar attributes and the model predicts the same caption for them (see Fig. 1). To remove biases of the captioning module and gauge the model's 's ability to identify the role, we propose Grounded Video Situa- tion Recognition (GVSR) - an exten- sion of the VidSitu task to include spatio-temporal grounding. In addi- tion to predicting the captions for the role-entity pairs, we now expect the structured output to contain spatio- temporal localization, currently posed as a weakly-supervised task.
Evant 3, Werbs ROLL
Foent Verbo
Vorbu
poller
Aeylvirriedd
Aeg pidter)
Men in armee
Bondd woman
Arg: (Tingg rolled
Himsal
Axgo (ontunee)))
Spear
Aught (Dewesooo
Up
Ary Scene
Sackwird
Evont h. Vorts RLB
Exent M. bwrt WINC
Ang Soors
An open hold
[luker))
Pensent shat
Lwent N. Verbs WALK
fry: [hing (ibbe)) Dogg
Kig Sorne
Nigt (lecmr)
Bosde wonin
Agg Scane
Badovard
Angi
Ang Sonne
An open Sid
Figure I: Overview of GVSR: Given a video consisting of multiple events, GVSR requires recognising the action verbs, their corresponding roles, and localising them in the spatio- temporal domain. This is a challenging task as it requires to disambiguate between several roles that the same entity may take in different events, e.g. in Video 2 the buld man is a patient in event 1, but an agent in event N. Moreover, the entities present in multiple events are co-referenced in all such events. Colored arguments are grounded in the image with bounding boxes (figure best seen in colour).
Joint structured prediction. Previous works [27,37 37] modeled the VidSitu tasks separately, e.g. the ground-truth verb is fed to the SRL task. This setup does not allow for situation recognition new video clip without manual intervention. Instead, in this work, we focus on solving three tasks jointly: (i) verb classification; (ii) SRL; and (iii) Grounding for SRL. We ignore the original event relation prediction task in this work, as this can be performed later in a decoupled manner similar to [27].
We propose Video Whisperer, new three- transformer architecture that enables video under- standing at a global level through self-attention across all video clips, and generates predictions for the above three tasks at event level through localised event-role representations. In the first stage, we use a Transformer encoder to align and contextualise 2D object features in addition to event-level video features. These rich features are essential for grounded situation recognition, and used to predict both the verb-role pairs and entities. In the second stage, a Transformer decoder models the role as query, and applies cross-attention to find the best elements from the contextualised object features, also enabling visual grounding. Finally, in stage three, we generate the captions for each role entity. The three-stage network disentangles the three tasks and allows for end-to-end training.
Contributions summary. (i) We present a new framework that combines grounding with SRL for end-to-end Grounded Video Situation Recognition (GVSR). We will release the grounding annotations and also include them in the evaluation benchmark. (ii) We design a new three-stage transformer architecture for joint verb prediction, semantic-role labelling through caption generation, and
grounding of visual entities. (iii) We propose role prediction and use role queries contextualised by video embeddings for SRL, cireumventing the requirement of ground-truth verbs or roles, enabling end-to-end GVSR. (iv) At the encoder, we combine object features with video features and highlight multiple advantages enabling weakly-supervised grounding and improving the quality of SRL captions leading to a 22 points jump in CIDEr score in comparison to a video-only baseline [27]. (v) Finally, we present extensive ablation experiments to analyze our model. Our model achieves the state-of-the-ee results on the VidSitu benchmark.
2

---

2 Related Work
Image Situation Recognition, Situation Recognition in images was first proposed by [10] where they created datasets to understand actions along with localisation of objects and people. Another line of work, imSitu [39] proposed situation recognition via semantic role labelling by leveraging linguistic frameworks, FrameNet [3] and WordNet [19] to formalize situations in the form of verb-role-noun triplets. Recently, grounding has been incorporated with image situation recognition [23] to add a level of understanding for the predicted SRL. Situation recognition requires global understanding of the entire scene, where the verbs, roles and nouns interact with each other to predict a coherent output. Therefore several approaches used CRF [39]. LSTMs [23] and Graph neural networks [14] to model the global dependencies among verb and roles. Recently various Transformer [32] based methods have been proposed that claim large performance improvements [6, 7, 35].
Video Situation Recognition. Recently, imSitu was extended to videos as VidSitu [27], large scale video dataset based on short movie clips spanning multiple events. Compared to image situation recognition, VidSRL not only requires understanding the action and the entities involved in a single frame, but also needs to coherently understand the entire video while predicting event-level verb-SRLs and co-referencing the entities participating across events. Sadhu el al. [27] propose to use standard video backbones for feature extraction followed by multiple but separate Transformers to model all the tasks individually, using ground-truth of previous the task to model the next. A concurrent work to this submission, [37], proposes to improve upon the video features by pretraining the low-level video backbone using contrastive learning objectives, and pretrain the high-level video contextualiser using event mask prediction tasks resulting in large performance improvements
SRL. Our goals are different from the above two works, we propose to learn and predict all three tasks simultaneously. To achieve this, we predict verb-role pairs on the fly and design a new role query contextualised by video embeddings to model SRL. This eliminates the need for ground-truth verbs and enables end-to-end situation recognition in videos. We also propose to learn contextualised object and video features enabling weakly-supervised grounding for SRI, which was not supported by previous works.
Video Understanding. Video understanding is a broad area of research, dominantly involving tasks like action recognition [5, 8, 9, 29, 34, 36). localisation [16, 17], object grounding [26, 38], question answering [31, 40], video captioning [25], and spatio-temporal detection [9, 30j. These tasks involve visual temporal understanding in a sparse uni-dimensional way. In contrast, GVSR involves hierarchy of tasks, coming together to provide fixed structure, enabling dense situation recognition. The proposed task requires global video understanding through event level predictions and fine-grained details recognise all the entities involved, the roles they play, and simultaneously ground them. Note that our work on grounding is different from classical spatio-temporal video grounding [41, 38] or referring expressions based segmentation [11] as they require text query as input. In our case, both the text and the bounding box (grounding) are predicted jointly by the model.
3 Video Whisperer for Grounded Video Situation Recognition
We now present the details of our three stage Transformer model, Video WideoWhisperer. visual overview is presented in Fig. 2. For brevity, we request the reader to refer to [32] for now popular details of self- and cross-attention layers used in Transformer encoders and decoders.
Preliminaries. Given a video V consisting of several short events & {e,), the complete situation in V. is characterised by 3 tasks. (i) Verb classification, requires predicting the action label V associated with each event e;; (ii) Semantic role labelling (SRL), involves guessing the nouns (captions) C, = {Cas} for various roles R, = (r/r P(u;)Vr E R} associated with the verb v;. P is a mapping function from verbs to set of roles based on VidSitu (extended PropBank [21]) and R. the set of all roles); and (iii) Spatio-temporal Grounding of each visual role-noun prediction C.2 is formulated as selecting one among several bounding box proposals B obtained from sub-sampled keyframes of the video. We evaluate this against ground-truth annotations done at a keyframe level.
3.1 Contextualised Video and Object Features (Stage 1)
GVSR is challenging task, that requires to coherently model spatio-temporal information to understand the salient action, determine the semantic role-noun pairs involved with the action, and simultancously localise them. Different from previous works that operate only on event level
3

---

Onirraad
Wonse io berater
Chorcal
C
Role Object Decoder (RO)
Rele Careries
Wert deciitiattonn ewent
2
Pan
Event oware Cross-attention
Video Object Encoder (VO)
Crject
Viecc
Captioning Decodor
Entodldnng
Vilio
C
Cross Attention
Role Object Decoder
frore
Eactore
B
RO
VO
Video Object Encoder
FraneT
Event
Figure 2: Video Whisperer: We present new 3-stage Transformer for GVSR. Stage-I I learns the contextualised object and event embeddings through a video-object Transformer encoder (VO), that used predict the verb-role pairs for each event. Stage-2 models all the predicted roles by creating role queries contextualised by event embeddings, and attends to all the object proposals through role-object Transformer decoder (RO) to find the best entity that represents role. The output embeddings are fed to captioning Transformer decoder (C) to generate captions for each role. Transformer RO's cross
cross-attention attention ranks all the object proposals enabling localization for each role.
video features, propose to model both the event and object level features simultaneously. We use pretrained video backbone @vid to extract event level video embeddings x = Pva For representing objects, we subsample frames J {f+) from the entire video V. We use a pretrained object detector dabj and extract top M object proposals from every frame. The box locations (along with timestamp) and corresponding features are
= (bme) J m [I,..., MI, [l...., 7], and {x J = (fr(((t) respectively. (1)
The subset frames associated with an event e; are computed based on the event's timestamps,
= (file
StSc.
end
(2)
Specifically, at a sampling rate of 1fps, video V of 10s, and events e; of 2s each, we associate frames with each event such that the border frames shared. We can extend this association to all object proposals based on the frame in which they appear and denote this as B,
Video-Object Transformer Encoder (VO). Since the object and video embeddings come from different spaces, we align and contextualise them with a Transformer encoder [32]. Event-level position embeddings PE, added to both representations, event x, and object x (t E F.). In addition, 2D object position embeddings PEms are added to object embeddings x Together, they help capture natiootemppoal information. The object and video tokens passed through multiple self-attention layers to produce contextualised event and object embeddings:
[
e Transformervo (I.., x + PE, +
X .E.
Verb and role classification. Each contextualised event embedding ef is not only empowered to combine information across neighboring events but also focus on key objects that may be relevant. We predict the action label for each event by passing them through 1-hidden layer MLP,
0
MLP.(e))
(4)

---

Each verb is associated with fixed set of roles based on the mapping P(.). This prior information required model the SRL task. Previous works [27, 37] use ground-truth verbs to model SRL and predict both the roles and their corresponding entities. While this setup allows for task specific modelling, it not practical in the context of end-to-end video situation recognition. To enable GVSR, we predict the relevant roles for each event circumventing the need for ground-truth verbs and mapped roles, Again, we exploit the contextualised event embeddings and pass them through a role-prediction MLP and perform multi-label role classification. Essentially, we estimate the roles associated with an event as
R = (rl, > G
(5)
where (-) is the sigmoid function and Prote a common threshold across all roles (typically set to 0.5). Armed with verb and role predictions, 0; and R,, we now look at localising the role-noun pairs and generating the SRL captions.
3.2 Semantic Role Labelling with Grounding (Stage 2, 3)
major challenge in SRL disambiguate roles, as the same object (person) may take on different roles in the longer video V. For example, if two people are conversing, the agent and patient roles will switch between speaker and listener over the course of the video. Another challenge to generate descriptive and distinctive captions for each role such that they refer to specific entity. We
|R.] which are capable of learning distinctive role propose to use learnable role embeddings {ra F=
representations. As mentioned earlier, roles such as agent, patient, tool, location, manner, etc. ask further questions about the salient action.
Creating role queries. Each role gets updated by the verb. For example, for an action jump, the agent would be referred contextualised event embeddings to each role, instead of encoding ground-truth verbs. This eliminates the dependency the ground-truth verb-role pairs, and enables end-to-end GVSR. Similar to the first stage (VO), we also add event-level temporal positional embeddings to obtain role query vectors
the jumper. We strengthen the role embeddings by adding the
qik
r2 + e + PR.
R, Role-Object Transformer Decoder (RO). It is hard to achieve rich captions while using features learned for action recognition. Different from prior works [27, 37], use fine-grained object level representations instead of relying on event-based video features. We now describe the stage two of Video Whisperer model, the Transformer decoder for SRI.
Depending on the setting, l can span all roles R. ground- truth roles R, or predicted roles
Our Transformer decoder uses semantic roles as queries and object proposal representations as keys and values. Through the cross-attention layer, the event-aware role query attends to contextualised object embeddings and finds the best objects that represent each role. We incorporate an event-based attention mask, that limits the roles corresponding to event to search for objects localised in the same event, while masking out objects from other events. Cross-attention captures local event-level role-object interactions while the self-attention captures the global video level understanding allowing event roles to share information with each other.
We formulate event-aware cross-attention as follows. We first define the query, key, and value tokens fed to the cross-attention layer as
L Wo k/ = WKo and Vnne = Wyo
Here, Wio KIV] are learnable linear layers. Next, we apply a mask while computing cross-attention to obtain contextualised role embeddings
r'A
Σ vid
Omm Vents
where
amt = softmaxme (/glu. kme) 1(fe E F.).
(8)
where (:-)) is inner product and 1(.) is indicator function with value 1 when true and -oo otherwise to ensure that the cross-attentinn is applied only to the boxes B,, whose frames f. appear within the same event ex.
After multiple layers of cross- and self-attention, the role query extracts objects that best represent the entities for each role.
Zike TransformerRo qiks ..... ..... O'm .))
(9)
5

---

Captioning Transformer Decoder (C). The final stage of our model is caption generation mod- ule. Specifically, we use another Transformer decoder [32] whose input context is the output role embedding zaz from the previous stage and unroll predictions in an autoregressive manner.
Ci Transformerc(zi))
(10)
The role-object decoder in stage 2 shares all the necessary information through self-attention, and allows us to generate the captions for all the roles in parallel; while [27, 37] generate captions sequentially i.e. for a given event, the caption for role is decoded only after the caption for role 5-1, This makes Video SiieeWWhispere efficient with a wall-clock runtime of 0.4s for inference on 10s video, while the baseline [27] requires 0.94 seconds.
The entire model is designed in a way to naturally provide SRL with grounding in weakly-supervisee way, without the need for ground-truth bounding boxes during training. Cross-attention through the Transformer decoder RO scores and ranks all the objects based the role-object relevance at every layer. We extract the aross-attnttion scores ant for each role k and event e, from the final layer of Transformergo. and identify the highest scoring box and the corresponding timestep
Grounded Semantic Role Labelling.
b 0j arg max amt m2,
(11)
3.3 Training and Inference
Training. Video Whispeerrr be trained in an end-to-end fashion, with three losses. The first two losses, CrossEntropy and BinaryCrossEntropy, are tapped from the contextualis ed event embeddings and primarily impact the Video-Object Transformer encoder
L CE 0z) and Loole
BCE(r Ri E R.). PER.
Σ
(12)
The final component is derived from the ground-trute captions and helps produce meaningful SRL outputs. This is also the source of weak supervision for the grounding task,
L
caption
Σ
CE(C der Cik)
(13)
where the loss is applied in an autoregressive manner to each predicted word w. The combined loss for any training video V is given by
L
Σ L. Lyeeb
Σ Leole
caption
(14)
Inference. At test time, we split the video into similar events e; and predict verbs é; and roles R. for the same. Here, we have two options: (i) we can use the predicted verb and obtain the corresponding roles using ground-truth mapping between verbs and roles P(o;), (ii) only predict captions for the predicted roles R, We show the impact of these design choices through experiments.
4 Experiments
We evaluate our model in two main settings. (i) This setup mimics VidSitu [27], where tasks are evaluated separately. We primarily focus on (a) Verb prediction, (b) SRL and (c) Grounded SRI. This setting uses ground-truth verb-role pairs for modelling (b) and (c). (ii) End-to-end GVSR, where all the three tasks are modelled together without using ground truth verb-roles.
Dataset. We evaluate our model on the VidSitu [27] dataset that consists of 29k videos (23.6k train, 1.3k val, and others in task-specific test sets) collected from diverse set of 3k movies. All videos are truncated to 10 seconds, have 5 events spanning 2 seconds each and are tagged with verb and SRL annotations. There roles among 11 possible options, however not all are used for evaluation due to noisy annotations (we follow the protocol by [27]). For each role the corresponding value free-form caption.
total of 1560 verb classes and each verb is associated with a fixed set of
Metries. For verb prediction, we report Ace @ K, i.e. event accuracy considering 10 ground-truth verbs and top-K model predictions and Mlacro-Average Verb Recall @ K. For SRL we report CIDEr [33],

---

CIDEr-Vb: Vb: Macro-averaged across verbs, CIDEr-Arg: Macro-averager across roles, LEA [20], and ROUGE-L [15]. For more details on the metrics pleas refer to [27].
Implementation details. We implement our model in Pytorch [22]. We extract event (video)
features from a pretrained SlowFast model [8] for video representation (provided by [27]). For object features, we use FasterRCNN model [24] provided by BUTD [2] pretrained on the Visual Genome dataset [13]. We sample frames at I fps from 10 second video, resulting in T 11 frames. We extract top M = 15 boxes from each frame, resulting in 165 objects per video.
All the three Transformers have the same configurations they have layers with 8 attention heads, and hidden dimension 1024. We use the tokenizer and vocabulary provided by VidSitu [27] which uses byte pair encoding. We have 3 types of learnable embeddings: (i) event position embeddings PE; with 5 positions corresponding to each event in time; (ii) object localization 2D spatial embedding; and (iii) role embeddings, for each of the 11 roles. The verb classification MLP has single hidden layer of 2048 d and produces an output across all 1560 verbs. The role classification MLP also has a single hidden layer of 1024 d and produces output in a multi-label setup for all the 11 roles mentioned above. We threshold role prediction scores with O = 0.5.
We use the Adam optimizer [12] with learning rate of 10-4 to train the whole model end-to-end. As we use pretrained features, we train our model on a single RTX-2080 GPU, batch size of 16.
4.1 Grounding SRL: Annotation and Evaluation
As free form captions and their evaluation can be ambiguous, we propose to simultaneously ground each correct role in the spatio-temporal domain. To evaluate grounding performance, we obtain annotations on the validation set. We select the same T = 11 frames that are fed to our model sampled ffss. For each frame, we ask annotators to see if the visual roles (agent, patient, instrument), can be identified by drawing bounding box around them using the CVAT tool [i] (see Appendix ?? for thorough discussion). For each event and role k. we consider all valid boxes and create a dictionary of annotations Gi with keys as frame number and value as bounding box. During prediction, for each role r E R2, we extract the highest scoring bounding box as in Eq. 11. The Intersection cerooonnov----nnne Union (IoU) metric for an event consists of two terms. The first checks the selected frame appears the ground-truth dictionary, while the second compares if the predicted box has an overlap greater than 0 with the ground annotation,
RR. e Gal
loU@0-
1
Σ
1 16;
((o Gu(tt) > 0].
(15)
[R
4.2 Grounded SRL Ablations
We analyze the impact of architecture choices, role query embeddings, and ap- plying a mask in the cross- attention of the role-object decoder. All ablations in this section assume access to the ground-truth verb or roles as this allows us to analyze the effect of vari- ous design choices. Simi- lar to [37] we observe large
Table 1: Architecture ablations. All the models use event-aware cross- attention. indicates stages of the model. V: Video encoder, VO: Video- Object encoder, VOR: Mideo-Object-Rondd decoder, RO: Role-Object decoder, and C: Captioning Transformer.
encoder, RV: Kole-Vider
# Architecture
Query Emb.
CIDEr IoU@0.3 1.5@0.5
1 RV +C
Role GT-verb 47.91 0.53
2 RO +
Role + GT-verb 70.48 1: 1.09
0.14
1: 0.01
0.06 1 0.003
3 VOR +
Role Event
67.4 0.81
0.22 0.00
0.09
0.002
4 V RO+C
Role Event
69.15 0.62
0.23
0.03
0.09
1 0.01
5 VO RO Role Event
68.54 i: 0.48 0.29 1: 0.013
0.12 :: 0.01
variance across runs, therefore we report the average accuracy and the standard deviation over 3 runs for all the ablation experiments and 10 runs for the proposed model (VO+RO+C).
Architecture design. We present SRL and grounding results in Table 1. Rows 1 and 2 use a two-stage Transformer decoder (ignoring the bottom video-object encoder). As there is no event embedding e/,, role queries are augmented with ground-truth verb embedding. Using role-object pairs (RO) critical for good performance on captioning as compared to role-video (RV), CIDEr 70.48 vs. 47.91. Moreover, using objects enables seakly-supervised grounding. Row 3 represents a simple
7

---

Transformer encoder that uses self-attention to model all the video events, objects, and roles (VOR) jointly. As before, role-object attention scores are used to predict grounding. Incorporating videos and objects together improves the grounding performance.
We switch from two-stage to three-stage model between rows 1, 2, 3 vs. 4 and 5. Rows 2 vs. 5 illustrates the impact of including the vidco-object encoder. We see significant improvement in grounding performance 0.14 0.29 for I.U@0.3 and 0.06 to 0.12 for IoU@0.5 without significantly affecting captioning performance. Similarly, rows 4 vs. 5 demonstrate the impact of contextualizing object embeddings by events. In particular, using contextualised object representations o seems to help as compared against base features X
Role query embeddings design. Prior
Table 2: Comparing role query embeddings.
works in situation recognition 17, 27, 35] use verb embeddings identify en- tities from both images or videos. In this ablation, we show that instead of learning verb embeddings that only cap- ture the uni-dimensional meaning of verb and ignore the entities involved, event (or video) embeddings remember details and are suitable for SRL. In fact, Table (architecture: VO RO C) row 2 vs. 3 show that event embeddings are comparable and slightly better than GT-verb embeddings when evaluated on SRL and Grounding respectively. eliminating the need for GT verbs. Somewhat surprisingly, we see that the role em- beddings alone perform quite well. We believe this may be due to role embeddings (i) capture the generic meaning like agent and patient and can generate the correct entities irrespective of the action information; and (ii) the role query attends to object features which are contextualised by video information, so the objects may carry some action information with them.
# Query Emb.
CIDEr
IoU@0.3
I.5
1 Role only
68.61 + 0.61
0.27 =
0.11 + 0.009
2 Role GT-verb
68.71 + 1.06
0.25 = 0.02
0.10 + 0.01
3 Role + Event
68.54 0.48 0.29 0.013 0.12 1 0.01
Masked cross-Attention in RO decoder. We use masking in event-aware cross-attention to ensure that the roles of an event attend to objects coming from the same event. As seen in Table 3 (model: VO + RO C, query is role + event embedding), this reduces the object pool to search from and improves both the SRI.
Table 3: Impact of masking in RO decoder.
Mask
CIDEr
loU@0.3
IoU@0.5
No Yes
67.02 + 0.51
0.25 0.02
0.10 +:: 0.012
68.54 +: 0.48
0.29 + 0.013
0.12 + 0.01
and Grounding performance.
4.3 SRL So comparison
In Table 4, we compare our results against VidSitu [27] and a concurrent work that uses far better
features [37]. We reproduce results for VidSitu [27] by teacher-forcing the ground-truth role pairs to make a fair comparison while results for work [37] are as reported in their paper. Nevertheless, we achieve state-of-the-an art performance with 22 points gain in CIDEr score over [27] and 8 point gain over [37], while using features from [27]. Moreover, our model allows grounding, something not afforded by the previous approaches.
Table 4: SoTA comparison, results for SRL and grounding with GT verb and role pairs.
Method
CIDEr
C-Vb
C-Arg
R-L
Lea
IoU@0.3 1oU@0.5
Slow/FattTTEE+TDDD [27]
46.01
56.37
43.58
43.04
50.89
Slow-D+TxE-TTD [37]
60.34
0.75
69.12 + 1.43 53.87 + u3r 43.77
= 038
46.77 +061
VidcoWhisperer (Ours)
68.54 + 0.48 77.48
+ 1.52 61.55 + 0.7 45.70
+ 0300
47.54
+05 0.29 2013 0.12 = 0.00
Human Level
84.85
91.7
80.15
39.77
70.33
4.4 GVSR: Joint Prediction of Video Situations
The primary goal of our work enable joint prediction of the verb, roles, entities, and grounding.
Verb prediction is an extremely challenging problem due to the long-tail nature of the dataset. In Table 5, we evaluate verb prediction performance when training the model for verb prediction only (rows 1-3) or training it jointly for GVSR (rows 4, 5). Using a simple video-only transformer encoder boosts performance over independent predictions for the five event clips (46.8% to 48.8% 8888, rows vs.
2). Including objects through the video-object encoder (row 3) provides an additional boost resulting in the highest performance at 49.7 73% on Accuracy@) @
8

---

A similar improvement is observed in rows 4 to 5 (V vs.
Table 5: Verb prediction performance.
VO stage 1 encoder). Interestingly, the reduced perfor- mance of rows 4 and 5 as compared against rows 1-3 is primarily because the best epoch corresponding to the highest verb accuracy does not coincide with highest SRL performance. Hence, while the verb Accuracy@* @1 of the GVSR model does reach 49% during training it degrades subsequently due to overfitting. Nevertheless,
Rows 1-3 train only for verb prediction.
Rows 4, 5 are trained for GVSR.
# Architecture Acc@ @ 1 Acc@5 Rec@5
1 Baseline [27]
46.79
75.90
23.38
2 V
48.82
78.01
23.32
3 VO
49.73
78.72
24.72
observe that the mmaco-averaged Recall @5 is high-
4 V RV +
40.83
70.73
24.37
est for our model, indicating that our model focuses on all verbs rather than just the dominant classes. In Ap- pendix ??, we show the challenges of the large imbalance and perform experiments that indicate that classic re-weighting or re-sampling methods are unable to improve performance in meaningful mannner. Addressing this aspect is left for future work.
5 VO RO+
45.06
75.59
25.25
Understanding role prediction. The verb-role prediction accuracy is crucial for GVSR, since the SRL task is modelled on role-queries. In Table 6 we analyse role prediction in various settings to understand its effect on SRL. Previous work [27] used ground-truth verbs for SRL, while roles and their entities or values are predicted sequentially. This setting is termed "GT, Pred" (row 2) as it uses the ground-truth verb but predicts the roles. We argue that as the verb-role mapping P is a deterministic lookup table, this setting is less interesting. We enforce a "GT, GT setting with ground-truth verbs and roles in [27] by teacher-forcing the GT roles while unrolling role-noun predictions (row 1). Another setting is where the verb is predicted and roles are obtained via lookup, "Pred, GT map" (row 3). Note that this enables end-to-end SRL, albeit in two steps. The last setting, "Pred, Pred" predicts both verb and role on-the-fly (row 4).
Comparing within variants of [27], surprisingly, row does not perform much better than row 2 CIDEr.
Table 6: Role prediction in various settings. Role Fl is the Fl score averaged over all role classes.
# Architecture Verb
Role
V. Acc@1 I Role Fl CIDEr
This may be because the model is trained on GT verbs and is able to pre- dict most of the roles correctly (row 2, Role F1 0.88). Subsequently, both rows and 4 show a large perfor- mance reduction indicating the over- reliance on ground-truth verb. We see similar trends for our models. Rows
1
GT
GT
46.01
2
GT
Pred
0.88
45.52
VidSitu [27]
3
Pred
GT map
46.79
29.93
4
Pred
Pred
46.79
0.47
30.33
5
RO+C
GT
GT
70.48
VO+RO+C
Pred
GT
45.06
68.54
7 VO+RO+C Pred
GT map
45.06
51.24
7 and 8 with predicted verb-role pairs
8 VO+RO+C
Pred
Pred
44.05
0.44
52.30
lead to reduced SRL performance as compared against rows 5 and 6. Nev-
ertheless, our "Pred, Pred" CIDEr score of 52.3 is still higher than the baseline "GT, GT" at 46.0.
Appendix ?? discusses further challenges of multi-label and imbalance in predicting roles.
GVSR. We evaluate our end-to-end model for grounded video situation recognition. In order enable end- to-end GVSR in [27], we use it in the "Pred, Pred" setting discussed above, that allows verb, role, and SRL predictions. Table 7 shows that our model improves SRL perfor- mance over Vidsitu [27] by margin of 22% on CIDEr score. In addition to that, our model also enables Grounded SRL, not achievable in VidSitu [27].
Table 7: GVSR: Results for end-to-end situation recognition.
Our model architecture is VO + RO + C.
Prediction
Verb Acc@I 1
loU
Model
CIDEr
Verb Role SRL
0.3 0.5
VidSitu [27]
/
46.79
30.33
/ 44.06
52.30 0.13
0.05
Video VideoWhisperes
GT
<
45.06
68.54
0.29 0.12
4.5 Qualitative Results and Limitations
We visualize the predictions of and see that it performs reasonably well given the complexity of the task. VideoWhisperer correctly
(Pred-GT) in Fig. 3 for one video of 10 seconds
Mooe examples on our project page, https:://etesssee github. io/grvidsitu//VSS html.

---

Video ID: Se___OOS_NNAA, Se__NN_M_e_____________ 175_188
Prace
Web
Agga
Aggt
Aaga
AM
WILLIT
Mavlyy
fowends
sterly
VILLOO
Ev
PRUDUCO
016
We show the results for a 1Os clip that can be viewed here: https: //yutuu/uuuuuuu b NVS__NM?t=175. The video is broken down to 5 events indicated by the row labels Ev1 to Ev5. At 1fps sampling rate, we obtain boxes from 3 frames for each event (with Frame3 of event i - being the same as Framel of event i). On the right side of the table, we show the predictions for the verb and various roles in the "Pred GT" mode, discussed in Table 6 (row 6). Predictions are depicted in blue, while the ground-truth is in green. Each role is assigned specific color (see table header), and boxes for many of them can be found overlaid on the video frames (with the same edge color).
Figure 3:
predicts verbs for actions like "open" and "walk". Given the large action space and complex scenes, there can be multiple correct actions, e.g. in Ev2 see reasonable "walk" instead of "turn".
For SRL, the model generates diverse captions with good accuracy, like "woman white dress". Even though the ground-truth is syntactically different, "woman wearing white", they both mean the same. In fact, this our primary motivation to introduce grounding. Ev3, the model incorrectly predicts "walk" as the verb instead of "reach". While "walk" does not have the role Arg2, we are able to predict a valid caption "to get to the door" while grounding the woman's 's arm in Frame3. We see that model correctly understands the meaning of Arg2 combined with event features for SRL. This shows the importance of event embeddings, as they may recall fine-gen ane-ggained details about the original action even when there are errors in verb prediction.
ground truth role embeddings
For grounding SRL, we see that the model able to localize the roles decently, without any bounding box supervision during training. While we evaluate grounding only for ArgO, Arg1, and Arg2 (when available), we show the predictions for other roles as well. In Fig. 3, the model is able to ground the visual roles ArgO and Argl correctly. For non-visual roles like "Manner", the model focuses its attention to the face, often relevant for most expressions and mannerisms.
Limitations for our current model are with verb and role prediction and disambiguation, improving the quality and diversity of captions to go beyond frequent words, and the division of attention towards multiple instances of the same object that appears throughout a video (details in Appendix ??). Nevertheless, we hope that this work inspires the community to couple videos and their descriptions.
5 Conclusion
We proposed GVSR as means for holistic video understanding combining situation recognition recognizing salient actions, and their semantic role-noun pairs with grounding. We approached this challenging problem by proposing Video WhispeW,s,rrer, that combines a video-object encoder for contextualised embeddings, video contextualised role query for better representing the roles without the need for ground-truth verbs and an event-aware cross-attention that helps identify the relevant nouns and ranks them to provide grounding. We achieved sfate-of-the art performance on the VidSitu benchmark with large gains, and also enabled grounding for roles in supkri-supervieed manner.
10

---

References
[1] CVAT Annotation Tool, httpp::/ggiee com//ppen/innttookktt/cccaa
[2] Peter Anderson, Xiaodong He, Chris Buchler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 1,7
[3] Collin F. Baker, Charles J. Fillmore, and John B. Lowe. The Berkeley FrameNet project. In Association of Computational Linguistics (ACL), 1998. 3
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko, ko. End-to-End Object Detection with Transformers. In European Conference on Computer Vision
(ECCV), 2020. 1
[5] Joao Carreira and Andrew Zisserman. Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 3
[6] Junhyeong Cho, Youngseok Yoon, and Suha Kwak. Collaborative Transformers for Grounded Situation Recognition. In Conference on Computer Vision and Puttern Recognition (CVPR), 2022. 3
[7] Junhycong Cho, Youngseok Yoon, Hyeonjun Lee, and Suha Kwak. Grounded Situation Recognition with Transformers. 2021. 3. 8
[8] Christoph Feichtenhofer. Haoqi Fan, Jitendra Malik, and Kaiming He. FlowFast Networks for Video
Recogmition. In International Conference on Computer Vision (ICCV), 2019. 3,
[9] Rohit Girdhar, Joao Carreira. Carl Doersch, and Andrew Zisserman. Video Action Transformer Network. In Conference Computer Vision and Pattern Recognition (CVPR). 2019. 3
[10] Saurabh Gupta and Jitendra Malik. Visual Semantic Role Labeling. arXiv preprint arXiv:
2015. 3
[11] Anna Khoreva, Anna Rohrbach, and Bernt Schiele. Video Object Segmentation with Referring Expressions.
In European Conference on Computer Vision (ECCV), 2018. 3
[12] Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint
arXiv:1412.6980 2014. 7
[13] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations. International Journal of Computer Vision (IJCV),
123(1):32-73, 2017. 7
[14] Ruiyu Li, Makarand Tapaswi, Renjie Liao, Jiaya Jia, Raquel Urtasun, and Sanja Fidler. Situation Recognition with Graph Neural Networks. In International Conference on Computer Vision (ICCV), 2017.
[15] Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, 2004.7 7
[16] Tianwei Lin, Xiao Liu, Xin Li, Errui Ding. and Shilei Wen. BMN: Boundary-matching Network for Temporal Action Proposal Generation. In International Conference on Computer Vision (ICCV), 2019. 3
[17] Tianwei Lin, Xu Zhao, Haisheng Su, Chongjin Wang, and Ming Yang. BSN: Boundary Sensitive Network for Temporal Action Proposal Generation. In European Conference on Computer Vision (ECCV), 2018. 3 [18] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for ssiion-and-Lnguurr Tasks. In Advances in Neural Information Prucessing Systems (NeurIPS), 2019.
[19] George Miller. Wordnet: A lexical database for english. Commun. ACM, 38(1 1):39-41,, 41, nov 1995. 3 [20] Natise Sadat Moosavi and Michael Strube. Which Coreference Evaluation Metric Do You Trust? A Proposal for Link- Entity Aware Metric. In Association of Computational Linguistics (ACL). 2016.
[21] Martha Palmer. Daniel Gildea, and Paul Kingsbury. The proposition bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1):71-106, 2005. 1.3
[22] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep lcarning library. In Advances Neural Information Processing Systems (NeurIPS), 2019.
[23] Sarah Pratt, Mark Yatskar. Luca Weihs. Ali Farhadi, and Aniruddha Kembhavi. Grounded Situation Recognition. In European Conference Computer Vision (ECCV), 2020. 3
[24] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards Real-time Object Detection with Region Proposal Networks. In Advances in Neural Information Processing Systems
(NeurlPS), 2015. 7
[25] Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Christopher Pal, Hugo Larochlll, Aaron Courville, and Bernt Schiele. Movie Description. International Journal of Computer Vision (IJV),,
123:94-120, 2017. [26] Arka Sadhu, Kan Chen, and Ram Nevatia, Video Object Grounding using Semantic Roles in Language Description. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 3
11

---

[27] Arka Sadhu, Tanmay Gupta, Mark Yatskar, Ram Nevatia, and Aniruddha Kembhavi. Visual Semantic Role Labeling for Video Understanding. In Conference on Computer Vision and Pattern Recognition (CVPR).
2021. 1, 2, 3, 5, 6. 7. 9
[28] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. VideoBERT: A Joint
Model for Video and Language Representation Learning. In International Conference on Computer Vision
(ICCV), 2019. 1
[29] Chen Sun, Abhinav Shrivastava, Carl Vondrick. Kevin Murphy, Rahul Sukthankar, and Cordelia Schmid. Actor-Centric Relation Network. In European Conference Computer Vision (ECCV), 2018. 3
[30] Makarand Tapaswi, Vijay Kumar. and Ivan Laptev. Long term spatio-temporal modeling for action
detection. Computer Vision and Image Understanding (CVIU), 210, 2021. 3
[31] Makarand Tapaswi. Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun. and Sanja Fidler. MovieQA: Understanding Stories in Movies through Question-Ans Ansseiin.... In Conference on Computer
Vision and Pattern Recognition (CVPR), 2016.
[32] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need, In Advances in Neural Information Processing
Systems (NeurIPS), 2017. 3, 4, 6
[33] Ramakrishna Vedantam. C. Lawrence Zitnick, Devi Parikh. CIDEr: Consensus-basse image description
evaluation. In Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
[34] Limin Wang. Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang. and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In European Conference on Computer Vision (ECCV), 2016. 3
[35] Meng Wei, Long Chen, Wei Ji, Xiaoyu Yuc, and Tat-Seng Chua. Rethinking the Two-Stage Framework for Grounded Situation Recognition. AAAI Conference on Artificial Intelligence (AAAI), 2022. 3, 8
[36] Chao- Yuan Wu, Christoph Feichtenhofer. Haoqi Fan, Kaiming He, Philipp Krahenbuhl, and Ross Girshick. Long-term Feature Banks for Detailed Video Understanding. In Conference Computer Vision and Pattern Recognition (CVPR), 2019.
[37] Fanyi Xiao, Kaustav Kundu, Joseph Tighe, and Davide Modolo. Hicrarchical Self-supervised Represen- tation Learning for Movic Understanding. In Conference Computer Vision and Pattern Recognition (CVPR), 2022. 2, 3, 5,
[38] Antoine Yang, Antoine Micch, Josef Sivic, Ivan Laptev, and Cordelia Schmid. TubcDETR: Spatio- Temporal Vidco Grounding with Transformers. In Conference on Computer Vision and Pattern Recognition
(CVPR), 2022.
[39] Mark Yatskar, Luke Zettlemoyer, and Ali Farhadi. Situation Recognition: Visual Semantic Role Labeling for Image Understanding. In Conference Computer Vision and Pattern Recognition (CVPR), 2016. 1,
[40] Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao, ActivityNct-QA:
A Dataset for Understanding Complex Web Videos via Question Answering. In AAAI Conference on Artificial Intelligence (AAAI). 2019. 3
[41] Zhu Zhang, Zhou Zhao, Yang Zhao, Qi Wang, Huasheng Liu, and Lianli Gao. Where Does It Exist: Spatio-Temporal Video Grounding for Multi-Form Sentences. In Conference on Computer Vision and Puttern Recognition (CVPR), 2020.
12


Reviews:
[{"review_id": "zafWwKug73Q", "reviewer": "Reviewer_4WkR", "summary": "This paper studied the problem of video situation recognition, which predicts structured information like multiple events, relationships, actions, and verb-robe pairs.  The authors propose a three-stage Transformer model, VideoWhisperer to handle the task. The model first learns contextualized embeddings for video features in parallel with key objects that appear in the video clips. The model then attends and pools information from object embeddings to localize answers to questions. The final stage of the model generates answers as captions to describe each verb-role pair in the video. Experiments are conducted on the VidSitu dataset, achieving better performance than the baseline models.", "questions": "Please see the **weaknesses** in the previous section.", "limitations": "Yes, the authors adequately addressed the limitations and potential negative social impact of their work.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. The paper overall is well written and easy to follow.\n2. The idea to decompose the task into different stages seems to be technically sound and the ablation study in the experimental section shows its effectiveness.\n3. As the conclusion of the introduction, the authors also provide new data annotation to an existing dataset.\n\n**Weaknesses**\n1. The first concern is about the novelty of the main idea of the paper. The proposed mode seems to be complicated to decompose the task into three stages. Whether it is possible to compare the computation complexity between the proposed method and the baseline.\n2. Second, as a method paper, this paper only conducts experiments on a recent proposed dataset. And the dataset contains only a simple baseline. This makes the paper less convincing. Whether it is possible to extend the proposed method to other video situation understanding datasets like (1) to further prove the effectiveness of the proposed method? \n3. It will also be great to implement more baselines on the existing benchmark.\n\n (1). Wu, Bo, et al. \"STAR: A benchmark for situated reasoning in real-world videos.\" Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "n_9Fxu45Kd", "reviewer": "Reviewer_tM2B", "summary": "VidSitu involves predicting the actions, entities involved in the actions, and simple relations between the actions, given a short video clip. This paper proposes spatio-temporal grounding (GVSR — Grounded Video Situation Recognition) as an addition to the VidSitu task. Spatio-temporal grounding involves localizing the entities which helps disambiguation of entities and co-referencing entities across different actions. \n\nThe paper also proposes a novel Transformer model that (unlike previous works) jointly predicts all task outputs — verbs, verb-role pairs, nouns, grounding. The VideoWhisperer model has 3 stages:\n\n1. Learning contextualized features for the video and key objects simultaneously. Predicts the action label (verb roles) for each event. \n2. Semantic role labeling, i.e., generating a caption (noun) for each verb role in each event. \n3. Localization is achieved by finding the largest attention score for a bounding box in the (ROTD) Transformer, for each verb-role in each event.\n\nThe proposed model has a few advantages — (1) results are predicted jointly, in a single forward pass, (2) improves entity captioning accuracy and (3) localizes verbs accurately despite not having access to grounding annotations during training. \n", "questions": "\n-  The main challenge with verb prediction seems to be the long-tailed distribution of verbs in VidSitu. Have the authors considered simple approaches to handle this, e.g., class-balanced loss [a] and/or focal loss [b] ? While these are unlikely to completely solve the verb imbalance problem, they may result in some improvements. \n- The object features are only computed from subsampled frames from the video. However, the event embeddings are computed from video features from the Slow-Fast network. I wonder if the authors have tried to compute event embeddings from subsampled frame features instead of full video. This would reduce the computational overhead (video vs. keyframes) if true. \n\nReferences\n[a] Cui, Yin, et al. \"Class-balanced loss based on effective number of samples.\" CVPR. 2019.\n[b] Lin, Tsung-Yi, et al. \"Focal loss for dense object detection.\" CVPR. 2017.\n", "limitations": "This work contributes to the general direction of understanding semantics in videos. However, current learning methods require ground-truth labels and large amounts of data. A huge challenge in this area is the difficulty in scaling this up to a large number of action labels (verbs) and roles.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Grounding semantic predictions in the given video, is an important problem that is well-motivated. Accurate grounding can improve the reliability, robustness, trust in the model, and possibly also prediction accuracy. \n\nThe GVSR task is an obvious extension of GSR [13] (which focused on grounding in imSitu [7]) to VidSitu [6]. Although the originality of the task is low, it is indeed useful, and improves the experimental setup of the existing VidSitu task. \n\nThe main contribution of the paper is (a) localization annotations (for testing) on VidSitu, and (b) the VideoWhisperer model. Unlike previous works on VidSitu, the proposed model is trained end-to-end, and outputs the entire set of structured outputs in a single forward-pass. This architecture is a significantly simpler implementation. Such an architecture allows learning (common) representations, and modeling the context across sub-tasks of VidSitu. \n\nBy leveraging cross-attention across object bounding box regions in the transformer model, localization can be achieved by weak supervision (via the caption loss). There is no need for ground-truth localization annotations. This allows one to potentially scale up the training as long as only language annotations are available. \n\nVidSitu requires the prediction of the action label (verb), the entities (nouns) associated with the verb-role and the relationship between events. The ground-truth language descriptions of the entities may be ambiguous, and have some variance — e.g., “Woman in blue shirt”, “woman in glasses”, etc. could all refer to the same person. Thus, evaluating captions is challenging, and imprecise. The entity localization task (measured by IoU) is less ambiguous. Further, it provides an additional dimension of understanding and interpretability of the model predictions. Thus GVSR alleviates some of the evaluation issues in VidSitu (to an extent). \n\nThere are significant improvements in performance in entity-captioning accuracy + localization of verb-roles. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dEwSZTAiE3H", "reviewer": "Reviewer_1XR4", "summary": "- This paper introduces a Transformer-based architecture for video situation recognition that a) augments prior work on Semantic Role Labeling in videos with spatiotemporal grounding, and b) removes the requirement of ground-truth role-verb mapping, enabling end-to-end Grounded Video Situation Recognition (GVSR).\n- The proposed model -- VideoWhisperer -- has 3 main components -- a) a Transformer encoder to learn object embeddings per frame, and event embeddings, both contextually enriched via self-attention; b) a Transformer decoder to predict the entities for each role, conditioned on the object and event embeddings; c) a Transformer decoder to autoregressively predict the caption for each role conditioned on the output features of (b).\n- To evaluate the model on grounding, the authors collected annotations by asking users to draw bounding boxes around roles in each event clip, for the validation set. These annotations will be released and/or incorporated in the official evaluation toolkit. These annotations are not used for training (and so the proposed model is only weakly-supervised).\n- Experiments are thorough, both the ablations to demonstrate the importance of each design choice, as well as comparison to prior work on the SRL benchmark. On the SRL benchmark, the proposed approach significantly outperforms prior work, and in addition, supports spatiotemporal localization, unlike prior works.", "questions": "- Which pretrained object detector backbone does VideoWhisperer use?\n- It seems like role prediction accuracy (~45%) is limiting downstream performance to quite an extent. Some discussion on next steps to further improve this would be useful to include in the paper.", "limitations": "- The authors have included a brief overview of limitations in the appendix.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed model makes sense, is well-validated by experiments, and importantly enables joint prediction of actions, roles, spatiotemporal localization of roles, and captions (unlike prior approaches that need ground-truth role-verb annotations at test time). Great work!", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "CElP8wrhwCH", "reviewer": "Reviewer_iCYa", "summary": "This paper introduces a new task of performing structured situation recognition over time in videos, called Grounded Video Situation Recognition (GVSR). The idea is that for a new video, a model must identify which verbs are happening, identify their arguments (specifying who did what to whom), and then determine where in the video that event happened. \n\nThe authors create a dataset for this task built around VidSitu [6], also devising a set of metrics to handle multiple events happening in the same video (5 events per video, each video having 10 seconds). The authors collect extra annotations of the validation set to benchmark human performance.\n\nThe paper introduces a new model for this task called VideoWhisperer. Frames are sampled from a video, and a pretrained object detector is used to extract object embeddings from the resulting frames. A separate pretrained video backbone extracts features from the whole video. A transformer encoder fuses these features together, and a separate transformer decoder is used to perform structured prediction (for the semantic role labeling over videos). Results and ablations show that this model performs well at both localized video situation recognition, as well as the non-localized case (VidSitu).", "questions": "* Qualitatively, what examples does this approach work well on and where does it struggle?\n* How important are the choice of video representations (\\phi_{vid} vs. \\phi_{obj}) ?", "limitations": "There isn't a limitations section or a discussion on the societal impacts, adding this would improve the paper.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "To this reviewer, this paper is a strong overall contribution. It tackles an important and unsolved problem (fine-grained video understanding) and introduces a new dataset (Grounded VidSitu) on top of VidSitu that might help evaluate this. The proposed model seems like a first step in this direction (pipelining the problem of structured video understanding), and helpful ablations are provided to show where the model might be improved in future work.\n\nWeaknesses:\n* To this reviewer, the model feels a bit hacky and overly complex. I have a suspicion that a simpler model would perform better, but I'm also OK with this being a problem that future work ought to address.\n* Adding a limitations/social impact section would improve the paper considerably.\n* Grounded seems like an overloaded term, do you mean something like \"Localized Video Situation Recognition\"?\n* The modeling section (Sec 3) was confusing to this reviewer. It might be helpful to discuss the high-level strategy first before jumping into the details, as well as the function of the ROTD/VOTE networks. Those acronyms like they are referred to that often, so maybe dropping them would boost understanding.\n* Some qualitative examples and discussion would help.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept