Bringing Image Scene Structure to Video via Frame-Clip Consistency of Object Tokens
Authors: Elad Ben Avraham
Roei Herzig
Karttikeya Mangalam
Amir Bar
Anna Rohrbach
Leonid Karlinsky
Trevor Darrell
Amir Globerson
Venue: NeurIPS 2022 Accept (2022)
Keywords: video models
object centric models
image-video

Abstract: Recent action recognition models have achieved impressive results by integrating objects, their locations and interactions. However, obtaining dense structured annotations for each frame is tedious and time-consuming, making these methods expensive to train and less scalable. At the same time, if a small set of annotated images is available, either within or outside the domain of interest, how could we leverage these for a video downstream task? We propose a learning framework StructureViT (SViT for short), which demonstrates how utilizing the structure of a small number of images only available during training can improve a video model. SViT relies on two key insights. First, as both images and videos contain structured information, we enrich a transformer model with a set of object tokens that can be used across images and videos. Second, the scene representations of individual frames in video should ``align'' with those of still images. This is achieved via a Frame-Clip Consistency loss, which ensures the flow of structured information between images and videos. We explore a particular instantiation of scene structure, namely a Hand-Object Graph, consisting of hands and objects with their locations as nodes, and physical relations of contact/no-contact as edges. SViT shows strong performance improvements on multiple video understanding tasks and datasets, including the first place in the Ego4D CVPR'22 Point of No Return Temporal Localization Challenge. For code and pretrained models, visit the project page at https://eladb3.github.io/SViT/.

Bringing Image Structure to Video via Frame-Clip Consistency of Object Tokens
Elad Ben-Avraham Tel Aviv University eladba40gmail. com
Roei Herzig
Karttikeya Mangalam UC Berkeley mangalam@cs. berkeley. wtw.aaaem......eeeeeeeeeae.aeeee
Tel Aviv University, IBM Research roeiherz@gmail. com
edu
Amir Bar
Anna Rohrbach UC Berkeley
Leonid Karlinsky MIT-IBM Lab leonidkašibm com
Tel Aviv University
amirb4r@gmail. com
anna. amma.rolnbecchtberkkeeee... edu
Trevor Darrell
Amir Globerson
UC Berkeley
Tel Aviv University, Google Research gamir&tauex tau. ac.il
Streordarreelleeerelllg edu
Abstract
Recent action recognition models have achieved impressive results by integrating objects, their locations and interactions. However, obtaining dense structured annotations for each frame is tedious and time-consuming, making these methods expensive train and less scalable. On the other hand, one does often have access to small set of annotated images, either within or outside the domain interest. Here we ask how such images be leveraged for downstream video understanding tasks. We propose learning framework Structure ViT (SVIT for short), which demonstrates how utilizing the structure of a small number of images only available during training can improve a video model. SViT relies on two key insights. First, both images and videos contain structured information, we enrich a transformer model with a set of object lokens that can be used across images and videos. Second, the scene representations of individual frames in video should "align" with those of still images. This is achieved via a Frame- Clip Consistency loss, which ensures the flow of structured information between images and videos. We explore a particular instantiation of scene structure, namely Hand- Obeect Graph, consisting of hands and objects with their locations as nodes, and physical relations of contact/no-contaee performance improvements on multiple video understanding tasks and datasets, including the first place in the Ego4D CVPR' 22 Point of No Return Temporal Localization Challenge. For code and pretrained models, visit the project page htttps://eladee github. io/SViT/
as edges. SVIT shows strong
Introduction
Semantic understanding of videos is key challenge for machine vision and artificial intelligence. It intuitive that video models should benefit from incorporating scene structure, e.g., the objects that appear in video, their attributes, and the way they interact. Indeed, several works [3] 25 26
H4 B5
42
have demonstrated that incorporating structured representations into models improves both performance and sample efficiency. In particular, 34
81
33 89
81 [ showed how to incorporate
structured representations by utilizing static image object annotations in videos.
Recently, vision transformers (ViT) 20 have emerged as the leading model for many vision appli- cations 4 21l [2]. This raises the question: how can structured representations be leveraged for video tasks in video transformer? Past works 4 2 have proposed video transformer models
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

"Washing" Patch Tokens
Hand-Object Graph
enjject
Object Tokens
Object
Canta
Shared Video & Image Transformer
Band
Ima ges
Object Prompts
Video Input/ Image Input
Only Available Training Our approach SViT brings structured scene representations from still images into video. We use the HAnd-Object Graph (HAOG) annotations of still images that are only available during training, and videos, which may come from different domain, with their downstream task annota- tions. We design a shared video & image transformer that can handle both video and image inputs. During training, given an image, the patch and object tokens are processed logether, and the object prompts learn to predict the HAOG, whereas given a video input, the transformer predicts the video label (e.g. Washing) based on the patch tokens. During inference, the learned object prompts, which have captured the structured scene information from images, are used to predict the video label.
Figure I:
but these approaches were not focused on learning structured representations. A recent line of works [34 81 [ utilized objects for structured approaches. [ developed an object-centric model based on region crop features from SlowFast [22] A proposed new block that incorporates
object representations, and [89] proposed structured model for action detection. However, these approaches are limited because they use external pretrained detectors for the frames, which require additional processing time (also during inference). Moreover, they ignore the potential benefits of the connection between the detection and the other tasks, and it is unclear how to use them with more detailed structured annotations. Our approach, described next, builds on these prior works and proposes an effective way to leverage scene structure in video transformers, using a relatively small set of annotated static images.
Most structured mannformer-basee models for videos require structured annotations during training. For example, frame-level object bounding boxes are considered available in [5] When it comes to structured annotations of static images, there are numerous datasets with annotations such as boxes, visual relationships, attributes, and segmentations [C 49 [59] Since we cannot always expect to have annotated images that perfectly align with our downstream video task, how can these resources be used to build better transformer-based video models? this work, we propose such an approach that is specifically designed for transformer architectures and offers an effective mechanism for improving video transformers by leveraging images with structured annotations that are not necessarily associated with the video domain and can be within or outside the domain of interest.
59 K1].
natural first step towards iidgo-video knowledge sharing is to use the same transformer model to process both: namely, to view images as "single frame vidcos", and feed them to video transformer model. However, this still leaves two key questions: how to model structured information, and how to account for domain mismatch between images and videas. Towards this end, we introduce two key concepts: (i) A set of transformer "object-tokens" additional tokens initialized from learned embeddings (we refer to these embeddings as "object prompts"" that are meant to capture object- centric information in both still images and video. These tokens can be supervised based on the scene structure of images, and that information will also be propagated to videos. To formalize the image structure, we propose the HAnd-Object Graph (HAOG), a simple representation of the interactions between hands and objects in the scene. (ii) A novel "Frame-Clij Consistency loss" that ensures consistency between the "object-tokens" when they are part of a video vs. still image. Moreover, as we show in our experiments, our approach does not require any alignment between object categories appearing in still images and the ones appearing in the videos the target downstream task. We name proposed approach Structure (Bringing Scene Structure from Images to Video via Frame- Consistency of Object Tokens, or SVIT for short). See Figure D for an overview.
Prior work has explored several other video & image models. The I3D model KK was one of the first successful attempts to leverage ImageNet architectures in video. They proposed to simply convert image 2D classification models into 3D ConvNets by inflating all the filters and pooling kernels.

---

Here, our focus is on learning the shared structured representations between the two domains in order transfer this knowledge into videos. Several other methods have explored simultaneous image and video training in the context of multi-task [6] and multi-modal 2 learning. However, these works did not utilize structural information, nor did they promote consistency between image and video predictions which, as we show, leads to significant gains in downstream video task performance.
To summarize, our main contributions are follows: (i) we propose a new method for exploiting structured information present in images in order to improve performance of video understanding tasks; (ii) we propose a novel concept of "object tokens" to capture object-centric structure in transformer models of images and videos a form of learned prompts to video & image transformer, which during training are associated to the available structure representation of the still images. (iii) we introduce new Frame-Clip Consistency loss, which promotes consistency between image-level and video-level predictions made by a shared transformer backbone. We show how this loss helps to drive performance improvements in downstream video tasks even in cases when image labels are unrelated to the video task; (iv) we demonstrate improved performance on several video understanding benchmarks, highlighting the effectiiveness of the proposed approach.
2
The SVIT Approach
Our SVIT approach learns a structured representation that is exhibited in both static images and video. We consider the setting where the main goal is to learn video understanding downstream tasks while leveraging structured image data. In this paper, we focus on the following video tasks: action recognition, ppatio-tempora action detection and object state classification & localization. In training. we assume that we have access to task task-specffic video labels as well as structured scene annotations for images. Based on the structured representations obtained by using the object tokens and regularized by the Frame-Clip Consistency loss, the inference is performed only for the downstream video task without explicitly using the structured image annotations.
We begin by describing an image annotation structure we refer to as Hand-Object Graphs (HAOG) (Section We then introduce our SVIT model (Section 2.2). and the Frame-Clip Consistency loss function (Section) 2.3)
Our method is schematically illustrated in Figure 2
2.1 The Hand-Object Graph
As mentioned above, our motivation is to bring scene structure from still images into video. In order achieve this, one component of this work is the construction of semantic representation of the interactions between the hands and objects in the scene. Specifically, we propose to use graph-structure we call Hand-Objer Graph (HAOG), see Figure D The nodes of the HAOG represent two hands and two objects with their locations, whereas the edges correspond to physical properties such as contact/no contact. Formally, an HAOG a tuple (O, E)) defined as follows:
Nodes O: A set O of n objects. Human-object interaction scenes, which we focus on in this work, usually consist of up two hands and objects. Thus node categories and they may also contain attributes (e.g., spatial coordinates). We assume that each image comes with the bounding boxes describing the locations of the two hands, and the objects they interact with. The bounding boxes for the left and right hand are denoted as b,, b2 E R4, respectively, while for the two corresponding objects interacting with the hands are denoted by, b4 We also assume access to four binary variables e e4 E {0, }} that indicate whether the corresponding hands or objects in fact exist in the image (c.g., if e 0 the left hand is not in the image, and b should be ignored).
"hand" and "object",
Edges E: The edges are represented as labeled edges between hand and object nodes. The edge labels describe the physical properties of the scene. In our current modeling, each edge is characterized by a physical property of "contact" or "no-contact". We denote two binary variables C1, C2 E {0. 1} that specify if each of the hands is in direct contact with the corresponding object (the left hand b can only interact with the object by. similar for the right hand). In principle, we can also include properties such as distances between objects, 3D hand poses, etc., where available.
We note that images annotated consistent with our defined HAOG format can be found in several datasets 31 59 [2] and we leverage these in experiments.
3

---

loss
Frame- Clip Consistency
Video
Image
HAAG
loss
Clip Object Tokens
Frame Object Tokens
Object
Token
Shared Video & Image Transformer
Object
Prompt
Patch
Token
Figure 2:
Our shared video & image transformer model processes two different types of tokens: standard patch tokens from the images and videos (blue) and the object prompts (purple), that are transformed into object tokens (purple) in the output. During training, the object tokens (purple) are trained to predict the HAOG for still images. For video frames that have no HAOG annotation, we use our "Frame- Clip" loss to ensure consistency between the "frame object tokens" (resulting from processing the frames separately) and the "clip object tokens" (resulting from processing the frame as part of the video). Last, the final video downstream task prediction results from applying a video downstream task head on the average of the patch tokens in the transformer output (after they have interacted with the clip object tokens (purple)).
2.2 The SVIT Transformer Model
We next present the SVIT model: video transformer that employs object tokens as a means to capture structure both images and videos. We begin by reviewing the video transformer architecture, which our model extends. Next, we explain how we process both images and videos. Finally, we describe object tokens and how they are used in videos and still images.
Shared Video & Image Transformer. Video transformers
extend the Vision Trans- former model the temporal domain. Similar vision transformers, the input is first "patchified" but with temporally extended 3-D convolution (instead of 2-D) producing a down-sampled tensor X of size T H x W d. We refer this as "patch tokens". Then, spatio-temporal position embeddings are added for providing location and time information. Next, multiple stacked self-attention blocks are applied repeatedly on the down-sampled tensor X produce the final vector representation using mean pooling over the patch tokens.
want to be able process batches of images videos. key desideratum in this context is to be able to input both videos and still-images into the same model. Towards this end, our first change is rely only 2-D convolutions in the first transformer step instead of 3-D. This allows single images to be used as inputs without padding, and does not hurt performance in practice.
In approach,
Object tokens. Our goal is to learn shared structured representation between the video and the image domains. We do this by adding transformer tokens to represent objects. These tokens are functionally similar to the patch tokens, with two exceptions. First, their initial embedding is learned. We refer to these embeddings as "object prompts". Second, they are used to predict the structured representations when those are available (here, use the Hand-Object Graphs structure for still images). An object token is used to predict properties of the corresponding node in the HAOG, and the concatenation of two object tokens is used to predict edge properties.
Next, we describe the model more formally. Let n the maximum number of modeled objects per frame (or still image). We have n tokens for each frame, and thus a total of T x tokens. The token for object i at frame is initialized with the vector 0, + т, where o, E RRd a learned object prompt and r. Rd is learned temporal positional embedding (the same one used for initializing the patch tokens). With these new tokens, we have T x (Hl x W + n.) tokens (i.e, vectors in IRd), and these together will go through the standard self-attention layers.
We denote the operator that outputs the final representation of the object tokens by Fo, where for a video V we have (V(V) Txxd and for an image I we have FI E Raxd The Fo representation used to predict structured representations for single images. We also include a loss that makes single frame representations align with those of clip representations (see Section 2.3 The operator that outputs the final representation vector used for the video downstream task denoted by FCLS, where for video V we have FoLs (VLssVV= gd

---

Finally, our method can be used on top of the most common video transformers (MVIT 21 TimeSformer [T. Mformer [66). Video Swin (56]). For our experiments, we use the MViTv2 model because it performs well empirically.
2.3 Losses and Training
During training have set of images annotated with HAOGs and videos with downstream task labels. We use these as inputs to our model (Section 2.2) and optimize the losses described below.
Video loss. Each training video V has corresponding ground-truth category Y E {1..... K).. As noted above, we have a vector representation for the entire video denoted by (V). Thus, we simply use a neural network predict logits for Y. Specifically, RK where FC is trainable fully connected network. We then consider standard cross-entropy between the predicted logits Y and true labels Y.
predicted logits are:
y= = FC(FcLs(VV)
Lvia = CE(Softmax(Y, P)),
HAOG loss. The transformer outputs set of n. object tokens for each image. Since we have supervision for these objects, we predict structured representations from the tokens, and introduce loss to optimize these predictions. Recall our notation from Section 2.T We let the transformer use n = object tokens which correspond to the two hands (bounding boxes b b2) and two manipulated objects (bounding boxes by, b4). The jth token is used to predict corresponding bounding box bj. as well as the existence variable E,. Formally, we let:
b, FCss (F))))) èg (F)
where FCM,, FC, are fully connected networks with four and one output respectively, F )) the jth object token, and / is the input image. Formally, this defines the loss of the nodes follows:
Nooes
BCE(Sigmoid(()).e ex) + e; (G, b.) + L, (6,6,,,)) b:))
(3)
where LI is the standard LI distance, and GloU is used as in 669 Furthermore, we predict the contact variable from the concatenation of the corresponding hand and object tokens. Namely for j E (1,2):
C Softmax Fg (D))
(4)
where FC, is fully connected with one output. Thus, we define the loss of the edges as follows:
Eddee
Σ CE(C,.))
(5)
The final prediction loss the sum of the node and edge losses:
LHAOG = -Nodds + Edgess
(6)
Frame-Clip Consistency Loss. Since have different losses for images and video, the model could find a way to minimize the image loss in a way that only applies to the images and does not transfer video To avoid this, need make sure that the video representation contains the same type of information as the still images, and in particular the structured information carried by the object tokens. To achieve this, we explicitly enforce the object tokens to be consistent across still images (frames) and videos (clips) using frame-clip consistency loss. To calculate the loss, we process each video twice: once as a clip, and once consisting of T frames II.... Ir, will be processed once as (V(V) and once as list of images (Fo ..., Fo (Ir))) This results in two groups of T object tokens, the first are the clip object tokens, denoted as, Octip
batch of T frames. Namely, a clip V
the frame object tokens, denoted O frames [Fo (v) ).... Fo (U(T)) ) Rxxxx Since both groups originate from the same input, each clip object token has a single corresponding frame object token. Namely, there is an alignment
(V) TR'xxxx the second
For example, it could "overfit" to relying on the positional embedding of the first frame only when minimizing image-related losses.
5

---

between the clip object tokens and the frame object tokens. Following the intuition that each clip object token should contain the information from its corresponding frame object token, we minimize the 7.1 distance between each clip object token and frame object token. Namely:
LCon = L1 (Oelip: O Otrames)
Overall loss. The total loss consists of Loo.. AH and Lvid, where Cvia is the loss of the main task. Each of the three terms in the loss is multiplied by a hyper-parameterr (ACon. AH Avia), and the total loss is the weighted combination the three terms:
Lrtal == ACo + AHAOGGHAOG AviLvia
(8)
3 Experiments and Results
We begin by describing the datasets, implementation details, and baselines and variants. We then evaluate our approach on several benchmarks. Specifically, we consider the following tasks: Com-
positional Action Recognition (Section 3.4. Object State Change Classification & Localization (Section 3.5
Action Recognition (Section and Spatio-Temporal Action Detection (Section
3.1 Datasets
We first describe the datasets used for the downstream video tasks, and those used as "auxiliary" datasets with annotated images. We use the following video datasets: (1) Soneehhing-Somethnng v2 (SSv2) 30 is a dataset containing 174 action categories of common human-object interactions. (2) SomethingElse [59] which exploits the compositional structure of SSv2, where combination of a verb and a noun defines an action. We follow the official compositional split from [59], which assumes the set of noun-verb pairs available for training is disjoint from the set given at test time. (3) Ego4D [3I] is new large-scale dataset of more than 3,670 hours of video data, capturing the daily-life scenarios of more than 900 unique individuals from nine different countries around the world. (4) Diving48 [. contains 48 fine-grained categories of diving activities. (5) A Atomic Visual
Actions (AVA) B21 is benchmark for human action detection, we report Mean Average Precision
(mAP) on AVA-V2.3 For "auxiliary"" image datasets we use the Ego4D [3I, and the 100 Days of Hands (L0ODOH) hp datasets, We collected 79,921 annotated images from 100DOH, and 57,213 annotated images from Ego4D P The AVA or SSv2 datasets are also used as "auxiliary" images in some experiments. For more details, see Section D.I in the supplementary. For the image datasets, the image annotations are based on frames selected from videos, but these are often sparsely selected from the videos, so it natural to treat them as annotated still images
3.2 Implementation Details
SViT is implemented in PyTorch, and the code is available on our project page. Our training recipes and code are based on the MViTv2 model, and were taken from /ttps:///gtth:i com/ We pretrain the SVTT model the K400 451 video dataset with auxiliary image datasets. Then, we finetune on the target video understanding task (detailed in Section B.I. together with the auxiliary image datasets and the SVIT loss. Each training batch contains 64 images and 64 videos in order to minimize the overall loss in Equation g For more details about datasets and evaluation see Section C of Supplementary.
Facebookresearch/mve
3.3 Baselines and SViT variants
Baselines. In the experiments, we compare SViT to several models from prior work reported for the corresponding datasets. These include the following approaches: I3D [3] SlowFast [22]. and the state-of-the -ftte-of-the-art transformers Mformer [66] and MVITv2 [53]. In addition, we provide the MVTTv2 multi-task (MViTv2 MT) baseline for comparing a naive multi- multi-tasking approach to our model. This is a version of MViTv2 which performs video classification and HAOG prediction from images. The
PComparatively, SomthingElse contains 2,550,700 frames, showing that image datasets are relatively small. 'The SomethingElse/SSv2 datasets has annotations for all clip frames, but we show in the experiments that SVIT only needs small percentage of these.

---

Boxes Annotations
Compositional
Base
Few-Shot
Model
Top-1
Top-5
Top-1
Top-5
5-Shot
10-Shot
SlowFast
PR
X
45.2
73.4
76.1
93,4
22.4
29.2
TimeSformer
X
44.2
76.8
79.5
95.6
24.6
33.8
Mformer
66
X
60.2
85.8
82.8
96.2
28.9
33.8
MViTv2
53
X
63.3
87.5
83.7
96.8
32.7
40.2
MViTv2 MT
V
63.8
87.3
85,2
97.1
33.7
41.4
SViT-SFT
64.2
87.9
85.0
97.1
33.8
41.7
SViT-DD
65.1
88.0
85.8
97.4
34.6
42.5
SVIT x2%
65.6
88.1
85.9
97.4
34.4
43.1
SViT-ID
65.8
88.3
85.6
97.3
34.4
42.6
Table 1: Compositional and Few-Shot Action Recognition on the "SomethingElse" dataset.
MV model enhanced with two heads: the first head is composed of an MLP that predicts the video labels given final representation vector, the second head consists of MLPs that predict the HAOG information (i.e., hands and object boxes, and contact information).
SVTT Variants. As mentioned above, we use several sources of auxiliary image datasets. In some cases, we have auxiliary images that are frames from video understanding datasets. For example, in Ego4D, we have both videos and frames annotated from some of these videos. In this sense, the auxiliary data is "in-domain" for some of the video tasks. In order to evaluate the importance of this effect, we explore "Different Domain" (SVI--DD) training, which avoids using images from a given video task auxiliary during finetuning and pretraining, Similarly, "In Domain" (SViT-ID) refers to the case where we use only images from the video task during finetuning, but not necessarily during pretraining We consider another variant that does not use the auxiliary images during finetuning at all (but does use them during pre-training), and refer to it as SVI7-SF7. Finally, to explore the effect of data-size the "In Domain" setting, we evaluate version SVTT xX% that only uses X% of the in-domain finetuning data. For MViTv2 MT, the auxiliary images correspond to the "In Domain" setting (i.e., same setting as for SViT-ID).
Auxiliary Images datasets. For SViT-DD, we generally use 100DOH and Ego4D for pretraining and finetuning. while for the Ego4D tasks, we only use 100DOH. SViT-ID uses 100DOH and Ego4D for pretraining and in-domain downstream frames for finetuning. The SViT-SFT pretraining setting similar to that of SViT-DD. SVIT xX% setting is similar that of SViT-ID.. For more details, see Section D.I in the supplementary.
3.4 Compositional & Few-Shot Action Recognition
Several video datasets define an action combination of a verb and a noun. One challenge then is to recognize combinations that were not seen during training. This "compositional"" setting was explored in the "SomethingEllse" dataset (59) where verb-noun combinations in the test data do not occur in the training data. The split contains 174 classes with 54,919/54,877 8,76 videos for training/validationn We also evaluate on the few-shot compositional action recognition task in 559 (See Section in supplementary).
Table D reports the results for these two tasks. All of our variants outperform all prior works and the MV MT on the Compositional and Few-shot task. From the table, we can see that using 2% of the in-domain data as auxiliary images (SViTx2% is almost comparable to using all the available in-domain images (SViT-ID). Furthermore, the results are comparable SViT-DD (0.7 difference) when trained on auxiliary images from different domains. This suggests that our approach optimizes the use of structural information across domains. Last, SVIT-DD outperforms SVIT-SFT when using the same data, demonstrating the value of finetuning video and image together.
3.5 Object State Change Classification and Localization
Hands and objects are key elements in human activity. Two tasks related to hand-object interaction have recently been introduced in the Ego4D 3I dataset. The first is temporal localization, which is defined as finding key frames in video clip that indicate change in object state. The second is object state change classification, which indicates whether an object state has changed or not. SVIT won the first place at the Ego4D CVPR 22 Point of No Return Temporal Localization Challenge.
7

---

(a) Object State Change in
(b) Action Recognition in SSv2.
(c)
Action Recognition in Diving48.
Ego4D.
Model
Temporal loc. error
Cls.
Model
Top-1
Model
Top-1
top-1
Bi-LSTM
0.790
65.3
SlowFast, R101
63.1
SlowFast, R101
77.6
BMN
0.780
ViViT-L
65.4
TimeSformer"
74.9
I3D ResNct-50"
0.739
68.7
MViTv1
64.7
TimeSformer-B
78.0
MVVT-v2'
0.702
71.6
MViTv2
68.2
MViTv2
73.1
MVVTv2 MT
0.678
71.1
MViTv2 MT
68.3
MViTv2 MT
75.5
SVIT-SFT
0.654
(-12)
SViT-SFT
68.9
SVIT-SFT
77.9 (4-4.8
SVIT-DD
0.646 056)
73.6
SViT-DD
69.2 (+1)
SViT-DD
79.8 (+-6.7)
SVIT x2%
0.649 063)
(-1.8)
SViT x2%
69.4(-12)
SVIT-ID
0.64 (-0)2)
73.8 (+2.2)
SVIT-ID
69.7 (-1.5)
Table Results on Ego4D, SSv2, and Diving48 datasets. Evaluation metries for the Ego4D object
state change task are absolute error in seconds (temporal loc. error. Lower is better) for temporal localization, and top-1 accuracy (Cls. top-1) for state change classification. We denote methods that do not use any additional structured annotation with""
Table Za reports results the temporal localization and object state change classification tasks the Ego4D dataset. We observe that SViT-ID, SVIT 2% and SViT-DD perform better than SViT-SFT and are very close. These results consistent with those presented in Section 3.4 which indicates that our method successfully leverages structural information across domains, and that a joint image and video finetuning is beneficial.
3.6 Action Recognition
Table 25 and Table report the results for standard action recognition task on the SSv2 and Diving48 datasets respectively. It can be that in Diving48 dataset, method improves the MVîT-v2 baseline by 6.7%, outperforming all the methods. This demonstrates that our method has some ability generalize the learned structural information of hand-object interactions even to fine- fine-grained "human-motion" tasks such as Diving48. We also note that Diving48 has no structured annotations available, thus SVIT-ID (and SViT 2%) cannot be provided. On SSv2, SViT-ID improves the baseline by 1.5%, SVTTx2% improves by 1.2%, and SViT-DD improves by 1.0%. As we observed significant improvements the more challenging SomethingElse setup, hypothesize that Something is more likely to benefit from the ability to generalize to unseen combinations of objects and hands exhibited by SVIT. We provide visualizations of the HAOGs predicted by SViT in Section B of the supplementary.
3.7 Spatio-temporal Action Detection
Table reportt the results for spatio-temporal action detection on the AVA dataset. In the literature, the action detection task on AVA formulated as a two stage prediction procedure. The first step is the detection of bounding boxes, which are obtained through an off-the-shelf pretrained person detector. The second step involves predicting the action being performed at each de- tected bounding box. The performance is benchmarked on the end result of these steps and is measured with the Mean Aver- age Precision (MAP) metric. Typically, for fair comparison, the detected person boxes are kept identical across approaches and hence the final performance depends directly on the ability of the approach utilize the video and box information. We observe that all SViT variants improve over the MVīT-v2 baseline. Specifically, SViT-ID improves the MViT-v2 baseline by 1.7, SViT-DD improves by 1.5, and SViT-SFT improves by 1.0.
Model
mAP
SlowFast MViTv2
23.8
26.8
MViTv2 MT
27.2
SViT-SIT SViT-DD SViT-ID
27.8 (+1.0)
28.3 (41.5)
28.5 (41.7)
Table 3: Spatio-temporal Ac- tion Detection on AVA-V2.2.
3.8 Ablations
We perform a comprehensive ablation study on the compositional action recognition task 59 on the SomethingElse dataset to measure the contribution of the different SVIT components (Table

---

(a) Method components
(b) Images Amount
(c) Absence of hands/objects
Model
Top-1
% of data
Top-1
Model
Objects
Hands
Top-1
MVITv2
63.3
2
65.6
MViTv2
64.6
MViTv2 MT
63.8
25
65.6
SViT-ID
/
x
66.5
-Vjev2+Objeet Tokens (OT)
65.0
50
65.7
MViTv2
X
64.9
M +OT+FC Loss (SViT)
65.8
100
65.8
SViT-ID
x
67.0
Table 4: Ablations. We show (a) Contributions of SViT components. (b) Amount of annotated images used in training. (c) Examine the robustness of SViT to clips that does not contain hands/contains only hands. The experiments are performed on the SomethingElse split. For more ablations, see see in supplementary.
Components of the SViT model. In Table 4a we ablate our model to show the significance of the individual components. First, train a standard MVîT 2 in a multi-task setting (MVITv2 MT), where we use videos for video action recognition, and images for HAOG prediction. There a small improvement (+0.5) over the baseline. We then take the MVITv2 and add the object tokens and corresponding SViT loss; this leads to a more significant improvement (+1.7). Last, when adding both the consistency loss and the object tokens, we observe the largest improvement (+2.5).
The amount of in-domain data. In Table 4b we demonstrate the effect of the amount of in no-oomain image data used during training. Specifically, since "SomethingEllse" includes HAOG annotations for each frame, we train our method with different amount of images (2%, 25%, 50%, and 100%) from the "SoothingFlsse" dataset. It can be seen that our approach takes advantage of the structured information within a small amount of data and, as a result, has enhanced sample-efficiency
Robustness to absence of hands/objects. In table Table 4c we verify whether our approach works in the cases where no hands or objects are visible, we conducted experiments on SomethingElse (ground- -ruth annotations available): (i) After filtering the videos containing (annotated) hands, we tested our model and the MViTv2 model on the filtered split. MViTv2 achieved an accuracy of 64.6 while our model achieved 66.5. This implies an improvement of +1.9. (ii) After filtering the videos containing (annotated) objects in more than 40% of the frames, we tested our model and the MViTv2 model on the filtered split. MVîTv2 achieved an accuracy of 64.9 while our model achieved 67.0. This implies an improvement of +2.1. 1. We can observe that our model outperforms the baseline even when there are no objects or hands in the videos, demonstrating the robustness of our approach. The total improvement (+1.9 and +2.1) is also a little bit lower than before the filtering (+2.5), which indicates that there has been slight degradation. Nevertheless, hand-object graphs remain valuable.
How important are the HAOGs. To examine how important the information provided by the HAOG, we suggest learning HAOGs without any useful information. Thus, we run an experiment in which the HAOGs are completely random. This means that, for each image, random HAOG is generated by sampling the boxes and their relationships uniformly. We refer to this experiment as IInd-I. On the SomethingElse dataset, the SViT-Random-ID result is 50.6 while the SVIT-ID is 65.8. This shows that predicting actual HAOG attributes provides important information that can be helpful for video downstream tasks.
4 Related Work
Joint Training with Images and Videos. Vision Transformers 20 are versatile architectures designed to he flexible in terms of input sequence length. Aside from the data ""patchification" layer, the rest of the transformer is agnostic to the domain of the input. Consequently, an exciting direction of joint training image-video has emerged in computer vision, a direction previously less natural with convolutional architectures. Specifically, this allows for training networks simultaneously on image and video, with most of the network parameters shared across both domains. Recently, multi-moda models became widely popular, from shared backbones [[5]
to separate encoders for each In our case, we focus on video and image input, similar to 27 87). In contrast to these works, our training approach utilizes two domains, while one of them (image) is used only during training and is supervised by a different task. Additionally, as we demonstrate here, we are not greatly impacted by domain mismatches between images and videos, in previous works. In related area of research, multi-task learning [14]
57
61
input modality [74, 777, and even non-vision modalities 69 40

---

involves developing models that predict output for multiple tasks using a common input. Our work is different since it uses two inputs from different domains, images and videos. Similarly, our work can be viewed as a form of auxiliary task, as already shown in many works 22 38 73 However,
it differs in that the shared object tokens are utilized to learn the shared structured representation between video and image domains, which enhances the self-attention layers with the structured representation for the main video task.
Learning from video using other modalities. Researchers have proposed different approaches
to learn from videos using audio and natural language [1 B 19
Vision tasks
require understanding of both visual and natural language. Another line of vision and language works explored object labels, supervision of textual descriptions, or the use of region crops [50 52). in contrast to SVIT, which does not require any of these components. Additionally, since high quality captions are expensive, researchers have proposed self-supervised ways to learn from both modalities without In these previous works, the two modalities must be consistent and aligned (e.g., text, audio, and RGB should be consistent for same sample). However, in our work, we do not require that the annotated images be precisely aligned with our downstream video task. Last, other new works Io 70 explored different structured representations for videos, different from what we do, which is to leverage structure from images to videos.
like Video Captioning [9 24
55 82
and Visual Quesion Answering
labeled data 62 6 68
Structured Models. Structured models have recently been successfully applied in many computer vision applications: video relation detection [55]
76. vision and language [I7
[5] 52
78
relational reasoning | 9
3T.
48
41
77
K
86 human-object interactions 21
and even
image & video generation 7
A recent line of works 25 26 34
35
60
6% 7
8C
8T 83
focuses on capturing spatio-tempora interactions for video understanding. Some of these used static image object annotations to build the representations. However, these works differ from ours.
89
works
83 [9
[3 proposed the Object Transformers (OT) for long form video
understanding, which model a video as a set of objects by extracting region crops from SlowFast (2 based on bounding boxes from an external detector. Contrary to this work, our transformer contains both patch tokens and object tokens, whereas the OT contains only object tokens. Additionally, our model learns the object-centric representation without the need for an external detector. Another work (9). proposed a structured model for action detection that explicitly utilizes image annotations. They use GCN 4 reason about actors and objects interaction, on top of an 13D network. Unlike this work, we examine how image annotations can be exploited as part of a video transformer model.
Video Understanding Models. We work on several well-benchmarked down-stream video tasks such as compositional action recognition on Something-Elk dataset (59) and action recognition with focus on temporal cues de-emphasizing appearance such as in Something-Somethine
V2 dataset 33 and Diving48 [54]. Several recent video transfromer works performed well on SSv2, such as ViVIT MVIT 21). MFormer 1661. TimeSformer [II MViTv2 [33, and Video Swin [56). We choose to work with MViTv2, although our method can be used on top of any of these. Additionally, we also adapt our proposed method to interesting new tasks on the Ego4D dataset [T). We believe such fine-nrained video understanding tasks require richer scene and object semantic reasoning, a requirement well afforded by our proposed Hand-Object Graph.
Conclusion
Video understanding is a key element of human visual perception, but modeling remains a challenge for machine vision. In this work, we demonstrated the importance of learning from the scene structure of small set of images to facilitate video learning within or outside the domain of interest. According to our empirical study, our SViT approach improves performance on four video understanding tasks. Note, that we did not prioritize making the HAOG annotations richer, i.e., it may be possible to add other physical properties to improve structure modeling. In addition, we noticed that leveraging the structure is effective even with 2% of the in-domain frames, well as when applying it from outside the domain. We leave for future work the challenge of utilizing different physical properties and more complex structures. Regarding potential impact of the method, We do not anticipate a specific negative impact, but, as with any Machine Learning method, we recommend to exercise caution.
Acknowledgements
This project has received funding from the European Research Council (ERC) under the European Unions Horizon 2020 research and innovation programme (grant ERC HOLI 81 819080). Prof. Darrell's group was supported in part by DoD including DARPA's Semafor, PTG and/or L.I.I. programs, as well as BAIR's 's industrial alliance programs.
10

---

References
[1] Triantafyllos Afouras, Andrew Owens, Joon Son Chung, and Andrew Zisserman. Self- supervised learning of audio-visure visual objects from video. In European Conference Computer Vision, pages 208-224. Springer, 2020.
[2] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, Lawrence Zitmick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference computer vision, pages
2015.
[3] Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In Proceedings of the IEEE International Conference Computer Vision, pages 609-617, 2017.
[4] Anurag Amab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: A video vision transformer, 2021.
[5] Anurag Arnab, Chen Sun, and Cordelia Schmid. Unified graph structured models for video understanding. In ICCV, 2021.
[6] Max Bain. Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In IEEE International Conference on Computer Vision 2021.
[7] Amir Bar, Roei Herzig, Xiaolong Wang, Anna Rohrbach, Gal Chechik, Trevor Darrell, and A. Globerson. Compositional video synthesis with action graphs. In ICML, 2021.
[8] Fabien Baradel, Natalia Neverova. Christian Wolf, Julien Mille, and Greg Mori. Object level visual reasoning in videos. In ECCV, pages 105-121, 2018
[9] Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi. Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01260 2018.
[10] Rodrigo Benenson, Stefan Popov, and Vittorio Ferrari. Large-scale interactive object segmenta- tion with human annotators. In CVPR, 2019.
[11] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? Proceedings of the International Conference on Machine Learning (ICML), July 2021.
[12] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and San-Michael Frahm, editors, Computer Vision ECCV 2020, pages 213-229, 2020.
[13] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 0299-6308, 2017.
[14] Rich Caruana. Multitask learning. In Encyclopedia of Machine Learning and Data Mining,
1998.
[15] Lluís Castrejón, Yusuf Aytar, Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Learning aligned cross-modal representations from weakly aligned data. 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 2940-2949, 2016
[16] Brian Chen, Xudong Lin, Christopher Thomas, Manling Li, Shoya Yoshida, Lovish Chum, Heng Ji, and Shih-Fu Chang. Joint multimedia event extraction from video and article. In EMNLP. 2021.
[17] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In ECCV, 2020.

---

[18] Ekin Dogus Cubuk, Barret Zoph, Jon Shlens, and Quoe Le. Randaugment: Practical automated data augmentation with reduced search space. In H. Larochelle. M. Ranzato. R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 18613-18624. Curran Associates, Inc., 2020.
[19] Chaorui Deng, Shizhe Chen, Da Chen, Yuan He, and Qi Wu. Sketch, ground, and refine: Top-down dense video captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 234-243, 2021.
[20] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
[21] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In ICCV, 2021.
[22] C. Feichttenhofer, H. Fan. J. Malik, and K. He. Slowfast networks for video recognition. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 6201-6210, 2019.
[23] Chen Gao, Jiarui Xu, Yuliang Zou, and Jia-Bin Huang. Drg: Dual relation graph for human- object interaction detection. ArXiv, abs/2008. 11...111144 11714, 2020.
[24] Lianli Gao, Zhao Guo, Hanwang Zhang, Xing Xu, and Heng Tao Shen. Video captioning with attention-based Istm and semantic consistency. IEEE Transactions on Multimedia, 19(9):2045- 2055, 2017.
[25] Rohit Girdhar, Joao Carreira, Carl Doersch, and Andrew Zisserman. Video action transformer network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 244-253, 2019.
[26] Rohit Girdhar, Deva Ramanan, Abhinav Kumar Gupta, Josef Sivic, and Bryan C. Russell. Ac- tionvlad: Learning spatio-temporal aggregation for action classification. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3165-31177 2017.
[27] Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten, Armand Joulin, and Ishan Misra. Omnivore: A Single Model for Many Visual Modalities. In CVPR, 2022.
[28] Chengyue Gong, Dilin Wang, Meng Li, Vikas Chandra, and Qiang Liu. Vision transformers with patch diversification. 2021.
[29] Priya Goyal, Dhruv Kumar Mahajan, Abhinav Kumar Gupta, and Ishan Misra. Scaling and benchmarking self-supervised visual representation learning. 2019 IEEE/CVE International Conference on Computer Vision (ICCV), pages 6390-6399, 2019.
[30] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freittee et al. The" something something" video database for learning and evaluating visual common sense. In ICCV, page 5, 2017.
[31] Kristen Grauman, Andrew Westbury, Eugene Byree Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagara- jan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong Xu, Chen Zhao. Siddhant Bansal, Dhruv Batra, Vincent Cartillier, Sean Crane, Tien Do. Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer, Adriano Fragomeni, Qichen Fu, Christian Fuegen, Abrham Gebreselasie, Cristina Gonzalez, James Hillis, Xuhua Huang. Yifei Huang. Wenqi Jia Weslie Khoo, Jachym Kolar, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li, Zhenqiang Li, Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro, Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz Puentes, Merey Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo, Yuchen Wang, Xindi Wu, Takuma Yagi, Yunyi Zhu, Pablo Arbe- laez, David Crandall, Dima Damen, Giovanni Maria Farinella, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard Newcombe, Aude Oliva,
12

---

Hyun Soo Park, James M. Rebg, Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresami, Mingfei Yan, and Jitendra Malik. Ego4d: Around the World in 3,000 Hours of Egocentric Video. CoRR, abs/21 10.07058, 2021.
[32] Chunhui Gu, Chen Sun, David A. Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheen- dra Vjjaanaraaimman, George Toderici, Susanna Ricco, Rahul Sukthankar, Cordelia Schmid, and Jitendra Malik. AVA: video dataset of spatio-temporally localized atomic visual actions. In 2018 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 22, 2018, pages 6047-6056. IEEE Computer Society, 2018.
[33] Roei Herzig, Amir Bar, Huijuan Xu, Gal Chechik, Trevor Darrell, and Amir Globerson. Learning canonical representations for scene graph to image generation. In European Conference on Computer Vision, 2020.
[34] Roei Herzig, Elad Ben-Avraham, Karttikeya Mangalam, Amir Bar, Gal Chechik, Anna Rohrbach, Trevor Darrell, and Amir Globerson. Object-region video transformers. In Confer- ence on Computer Vision and Pattern Recognition (CVPR), 2022.
[35] Roei Herzig, Elad Levi, Huijuan Xu, Hang Gao, Eli Brosh, Xiaolong Wang, Amir Globerson, and Trevor Darrell. Spatio-temporal action graph networks. In Proceedings of the IEEE International Conference Computer Vision Workshops, pages 0-0, 2019.
[36] Roei Herzig, Moshiko Raboh, Gal Chechik, Jonathan Berant, and Amir Globerson. Mapping images to scene graphs with permutation-invariann structured prediction. In Advances in Neural Information Processing Systems (NIPS), 2018.
[37] Geoffrey E. Hinton, Nitish Srivastava, A. Krizhevsky, Ilya Sutskever, and R. Salakhutdinov. Im- proving neural networks by preventing co-adaptation of feature detectors. ArXiv, http//207.05566 2012.
[38] Max Jaderberg, Volodymyr Mnih, Wojciech M. Czarnecki, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. ArXiv, abs/1611 1.05397, 2017.
[39] Andrew Jaegle. Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107.14795
2021.
[40] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International Conference on Machine Learning, pages 4651-4664. PMLR, 2021.
[41] Achiya Jerbi, Roei Herzig, Jonathan Berant, Gal Chechik, and Amir Globerson. Learning object detection from captions via textual scene attributes. ArXiv, abs/2009. 14558, 2020.
[42] Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos Niebles. Action genome: Actions as composition of spatio-temporal scene graphs. arXiv preprint arXiv:1912.06992, 2019.
[43] Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image generation from scene graphs. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1219- 1228, 2018
[44] Keizo Kato, Yin Li, and Abhinav Gupta. Compositional learning for human object interaction. In ECCV, 2018.
[45] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint 100.::1705.065500 2017.
[46] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv.1412.6980, 2014.
13

---

[47] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint
2016.
[48] Ranjay Krishna, Ines Chami, Michael S. Bernstein, and Li Fei-Fei. Referring relationships. ECCV, 2018.
[49] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32-73) 2017.
[50] Dongxu Li, Junnan Li, Hongdong Li, Juan Carlos Niebles, and Steven C. H. Hoi. Align and prompt: Video-and-language pre-training with entity prompts. ArXiv, abs/21 12. 12.09583, 2021.
[51] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kei- Chang. Visualbert: A simple and performant bascline for vision and language. ArXiv,
2019.
[52] Xiujun Li, Xi Yin, Chunyuan Li, Xiaowei Hu, Pengchuan Zhang, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. Oscar: Object-semantics aligned pre-traine treitraining for vision- lasion-language tasks. ECCV 2020, 2020.
[53] Yanghao Li, Chao- Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vision transformers for classification and detection. In CVPR, 2022.
[54] Yingwei Li, Yi Li, and Nuno Vasconcelos. Resound: Towards action recognition without representation bias. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
[55] Junwei Liang, Lu Jiang, Juan Carlos Niebles, Alexander G. Hauptmann, and Li Fei-Fei. Peeking into the future: Predicting future person activities and locations in videos. 2019 JEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 5718-55 118--577,220 2019.
[56] 7e Liu, Jia Ning, Yue Cao,, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. arXiv preprint arXiv:2106. 13230, 2021.
[57] Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 12-in-1: Multi- task vision and language representation learning. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10434-10443, 2020.
[58] Chih- Yao Ma, Asim Kadav, lain Melvin, Zsolt Kira, Ghassan Al-Regib, and Hans Peter Graf. Attend and interact: Higher-order object interactions for video understanding. 2018 IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 6790-6800, 2018.
[59] Joanna Materzynska, Tete Xiao, Roei Herzig, Huijuan Xu, Xiaolong Wang, and Trevor Darrell. Something-else: Compositional action recognition with spatial-temport interaction networks. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[60] E. Mavroudi, Benjamín Béjar Haro, and René Vidal. Representation learning on visual-symbolic graphs for video understanding. In ECCV, 2020.
[61] Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9879-9889, 2020.
[62] Antoine Miech. Dimitri Zhukov, Jean- Baptiste Alayrac. Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howtol00m: Learning text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 2630-2640, 2019.
[63] Tushar Nagarajan, Yanghao Li, Christoph Feichtenhofer. and Kristen Grauman. Ego-topo: Environment affordances from egocentric video. 2020 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 160-169, 2020.
14

---

[64] Andrew Owens, Jiajun Wu, Josh H McDermott, William T Freeman, and Antonio Torralba. Learning sight from sound: Ambient sound provides supervision for visual learning. Interna- lional Journal of Computer Vision, 126(10): 120-1137, 2018.
[65] Yingwei Pan, Ting Yao, Houqiang Li, and Tao Mei. Video captioning with transferred semantic attributes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6504-6512, 2017.
[66] Mandela Patrick, Dylan Campbell, Yuki M. Asano, Ishan Misra Florian Metze, Christoph Feichtenhofer, Andrea Vedaldi, and Joao F. Henriques. Keeping your eye on the ball: Trajectory attention in video transformers, 2021.
[67] Moshiko Raboh, Roei Herzig, Gal Chechik, Jonathan Berant, and Amir Globerson. Differen- tiable scene graphs. In WACV, 2020.
[68] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 9748-8763 PMLR, 2021.
[69] Hamid Rezatofighi, Nathan Tsoi, Jun Young Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[70] Arka Sadhu, Tanmay Gupta, Mark Yatskar, Ramakant Nevatia, and Aniruddha Kembhavi. Visual semantic role labeling for video understanding. 2021 IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pages 5585-5596, 2021.
[71] Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski. Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[72] Dandan Shan, Jiaqi Geng, Michelle Shu, and David Fouhey. Understanding human hands in contact at internet scale. In CVPR, 2020.
[73] Trevor Scott Standley, Amir Roshan Zamir, Dawn Chen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together multi-task learning? In ICML, 2020.
[74] Weijie Su, Xizhou Zhu Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre- training of generic visual-linguisic representations. In International Conference on Learning Representations, 2020.
[75] Chen Sun, Abhinav Shrivastava, Carl Vondrick, Kevin Murphy, Rahul Sukthankar, and Cordelia Schmid. Actor-centric relation network. In Proceedings of the European Conference on Computer Vision (ECCV), pages 318-334, 2018
[76] Xu Sun, Tongwei Ren, Yuan Zi, and Gangshan Wu. Video visual relation detection via multi- modal feature fusion. Proceedings of the 27th ACM International Conference Multimedia, 2019.
[77] Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. pages 5099-5110. Association for Computational Linguistics, 2019.
[78] Hao Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. In EMNLP, 2019.
[79] Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. Movieqa: Understanding stories in movies through mnestion-answering. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 4631- 4640, 2016.
15

---

[80] Bugra Tekin. Federica Bogo, and Mare Pollefeys. H+o: Unified egocentric recognition of 3d hand-object poses and interactions. 2019 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 2506-4515, 2019.
[81] Xiaolong Wang and Abhinav Gupta. Videos space-time region graphs. In ECCV, 2018.
[82] Xin Wang, Wenhu Chen, Jiawei Wu, Yuan-Fang Wang, and William Yang Wang. Video captioning via hierarchical reinforcement learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
[83] Chao-Yuan Wu and Philipp Krähenbühl. Towards Long-Form Video Understanding. In CVPR. 2021.
[84] Bingjie Xu, Yongkang Wong, Junnan Li, Qi Zhao, and M. Kankanhalli. Learning to detect human-object interactions with knowledge. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2019-2028, 2019.
[85] Keyulu Xu, Jingling Li, Mozhi Zhang, Simon S. Du, Ken ichi Kawarabayashi, and Stefanie Jegelka. What can neural networks reason about? In International Conference on Learning Representations, 2020.
[86] Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, et al. Relational deep reinforcement learning. arXiv preprini arXiv:/806.018. 01888000.083.. 2018.
[87] Bowen Zhang, Jiahui Yu, Christopher Fifty, Wei Han, Andrew M. Dai, Ruoming Pang, and Fei Sha. Co-training transformer with videos and images improves action recognition. ArXiv, abs/21 12.07175, 2021.
[88] Chuhan Zhang, Ankush Gputa, and Andrew Zisserman. Temporal query networks for fine- grained video understanding. In Conference Computer Vision and Pattern Recognition (CVPR), 2021.
[89] Yubo Zhang, Pavel Tokmakov, Martial Hebert, and Cordelia Schmid. A structured model for action detection. In 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 9967-9976, 2019.
16

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See line 341-342.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Sec Section 3.1 (the Dataset Section).
(b) Did you mention the license of the assets? [Yes] See the Dataset Section in supplemen-
tary.
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent obtained from people whose data you'l 're using/curating? [Yes] See the Dataset Section in supplementary.
(c) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [Yes] See the Dataset Section in supplementary.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
17


Reviews:
[{"review_id": "yhZ04PMJsLp", "reviewer": "Reviewer_bPRy", "summary": "Brief Summary: The paper tackles a few-shot / semi-supervised setup where a small number of training samples about hand-object interaction are provided to drive downstream video performance. The two key ideas are to use hand object information (bounding boxes, and contact edges), and using a frame-consistency loss so that the supervision is propagated to other frames which don't have annotations.\n\nExperiments are carried out on Something-Something, Something-Else, Ego4D, Diving48, and show improvements over compared baselines.", "questions": "Q1. (Minor) In section 3.1, the dataset numbering is incorrect. (3) is skipped.\n\nQ2. Why does diving-48 show mvit performing so much worse than slow-fast and time-sformer? \n\n", "limitations": "I feel that authors should emphasize the limitation of using hand-object graph which limits the proposed methods to a subset of videos where hands are prominent and not occluded. Moreover, it is unclear how the proposed method can be extended to more general action videos such as Kinetics data.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\n1. Interesting finding that using 2% of the in-domain data is comparable to using the entire in-domain data. \n\n2. Multiple datasets are considered for evaluation, \n\n\nCons:\n\n1. On naming conventions: I don't think it is fair to say annotated video frames count as image-annotations. They are still video annotations, just that they are sparse. \n\n2. On relations to previous work:\n\n(i) The authors should distinguish their work better compared to previous work. For instance, the idea of using object tags has been previous used in [44], so improvements just based on object tags are not exactly unexpected. Similarly, [Ref1] shows using entity-prompt can improve video-text pretraining. \n\n(ii) Structured representations for videos using semantic roles has been previously investigated, see [Ref2] and [Ref3], but discussion on them is missing.\n\n3. On the task setup: \n\n(i) L71 suggests that in training we have access to video-labels and structured scene annotations. It is not clear how scalable this is, how much preprocessing time and annotation it requires.\n\n(ii) Use of Hand-object graphs seems to be relevant for very specific domains. What if hands are not visible, or they are not interacting with a specific object?\n\n4. On data used: According to L196, video frames are annotated, but no details of how they are obtained are provided. Is it completely automatic / semi-automatic / purely human annotated?\n\n5. On model and experiment:\n\n(i) In L163, the motivation for frame-consistency loss is that image loss may not transfer to video loss. It is unclear if the proposed solution is the best way to approach this. For instance, one could simply perturb the image temporal position embedding to any random index. Alternatively, a object tracking system could interpolate / extrapolate the bounding boxes. Some experiments comparing such additional ways is needed.\n\n(ii) The relative performance improvement in Table 1. are very small, and it is unclear if the differences are actually significant.\n\n(iii) Improvements are not always consistent, for instance, in Table 1, MViT v2 MT outperforms SViT-SFT on Top-1 base. Similarly, and perhaps more surprisingly, SViT-2% outperforms SViT-ID on 10-shot case. \n\n(iv) The authors should provide SViT-2% results for Table 2a,b,c as well. \n\n(v) (Minor) Given that the model works for both images and videos, the authors could show results on human object interaction based datasets such as [Ref4]\n\n\n[Ref1]: Li, Dongxu, Junnan Li, Hongdong Li, Juan Carlos Niebles, and Steven CH Hoi. \"Align and Prompt: Video-and-Language Pre-training with Entity Prompts.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4953-4963. 2022.\n\n[Ref2]: Sadhu, Arka, Tanmay Gupta, Mark Yatskar, Ram Nevatia, and Aniruddha Kembhavi. \"Visual semantic role labeling for video understanding.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5589-5600. 2021.\n\n[Ref3]: Chen, Brian, Xudong Lin, Christopher Thomas, Manling Li, Shoya Yoshida, Lovish Chum, Heng Ji, and Shih-Fu Chang. \"Joint Multimedia Event Extraction from Video and Article.\" arXiv preprint arXiv:2109.12776 (2021).\n\n[Ref4]: Chao, Yu-Wei, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng. \"Learning to detect human-object interactions.\" In 2018 ieee winter conference on applications of computer vision (wacv), pp. 381-389. IEEE, 2018.\n\n========================\nGiven the extensive response by the authors, I raise my score by 1 point. In particular, the experiments on Kinetics-data, and additional ablative studies are helpful.\n\nHowever, I am still confused why, according to the new expts, the proposed method shows improvements in the cases where no hands are visible? This might suggest the real improvements are due to some other hyper-parameter, and not HAOG.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MdEFKx14pec", "reviewer": "Reviewer_DF7L", "summary": "This paper proposes the StructureViT (SViT) architecture designed to incorporate structured information about hand-object interaction labels in images to aid in video-related classification tasks. The authors propose a modified video-transformer architecture that, in addition to predicting video-level labels, also predicts the hand-object graph structure for annotated images via additional learned 'object tokens'. To distill information of the hand-object graphs of images to the video domain the authors further propose a Frame-Clip Consistency loss, where the hand-object graphs predicted for a video and its individual frames are forced to be consistent. The authors assess their technique on several different datasets and video-related tasks and show improvements in accuracy over the state-of-the-art (SOTA) across-the-board.", "questions": "Q. What procedure/losses did the authors employ for the pre-training task?\n\nQ. One of the main weaknesses of the proposed technique is the requirement of a large number of labeled images (~100K) to distill from. While in Table 1 and Table 3, the authors show the effect of reducing the number of annotated images for the Compositional and Few-Shot Action Recognition tasks, I am curious to know how these effects play out for the experiments presented in Table 2 as well. Does the authors' claim that the number of images can be reduced to 2% (or some similar small number) of the annotated image training images hold in general or is it only true for this task?\n\nQ. As an additional ablation, I would be curious to see the results of an experiment where the authors first learn a vision transformer to perform the HAOG task alone and then fine-tune its backbone network for the video-related task.\n\n------------------\nPost-rebuttal:\n\nI thank the reviewers for addressing all my concerns. However, after having considered all the other reviews, I share the concerns of the other reviewers in terms of limited novelty of the current work given the existence of prior non-transformer-based works that incorporate structured information into video-based tasks. Hence, I have lowered my original rating by 1 point.", "limitations": "Is the requirement for a large number of annotated images a strict one for the success of the proposed method? If it is, I would like or see the authors acknowledge this limitation more clearly.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: The proposed work is novel in many aspects. It is novel in attempting to exploit the transformer architecture's seamless ability to process multiple domains (video and image in this case) using the same set of shared weights. It is also novel in proposing a clever solution to distill information from the image domain to the video domain via the frame-clip consistency loss. Lastly, it is novel in attempting to injecting information of structured hand-object interaction labels to improve several downstream video-related tasks.\n\nQuality: All methodology and experiments in the paper are technically sound and correct.\n\nClarity: The material is presented in a clear and organized fashion. Some editorial errors are noted below. \n- ln 30: \"perfect\" should be \"perfectly\"\n- ln 129: $r_i$ should be $r_t$\n- ln 130: suggested to write this as $T \\times ( H \\times W + n)$ for improved clarity\n- ln 199: should be  \"to treat them\"\n- ln 305: \"share\" should be \"shared\"\n\nSignificance: This work presents a significant result towards advancing multi-modality processing and information sharing with transformer architectures. Transformers are quickly becoming the dominant architecture for visual information processing surpassing CNNs. They present the additional advantage of being able to align multi-modality information much more seamlessly versus CNNs. Hence, this work is an interesting exploration that advanced our understanding of the use of transformers for multimodal (image and video) information sharing via transformers. The proposed approach is fairly general and may be applicable with minor modifications to other video and image related tasks as well besides HAOG and the video related tasks considered by the authors.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "D5FGdoyTwz", "reviewer": "Reviewer_oj8q", "summary": "This paper proposes an approach to use annotated images from different domain to help with the training of video-based models. The transformer can take both inputs and there are various objectives involved during training for images and videos, respectively. During inference, the only input is video, and the object prompts are used for label prediction.", "questions": "1) I think the motivation of this approach is clear, but in terms of the technical contributions, my understanding is that the joint training of video and image inputs is enabled by ViT to tokenize, and there are other works jointly training images and videos [6, 25, 73, 26, 33, 61]. The authors mentioned that this work \"involves explicit interactions between the main video task and the auxiliary image task\", so can you further elaborate what is the explicit interaction and how is it not applicable to other approaches like [26, 33, 61]?\n\n2) For experimental results, it appears in Table 3 (a) and (b) that the amount of annotated images is not critical, and neither is the type of HAOG attributes. Does it indicate that the auxiliary task is in fact not functioning as providing additional information, but more as a regularization factor? In other words, it might be worth to systematically demonstrate what are the minimum requirements to make the model work, maybe not the best, but reasonably well (e.g., from ~63 to ~65 in Table 3 examples). This will provide guidance when there's no annotation available but possible annotations can be collected with minimum efforts.\n\n3) Is it possible to use the model in a zero-shot setting and compare with CLIP on action recognition tasks like Kinetics? Also, when comparing with MViTv2 results, is it a fair comparison because additional information/computation has been used in the proposed approach?", "limitations": "I think they properly addressed the limitations as mentioned in \"We do not anticipate a specific negative impact, but, as with any Machine Learning method, we recommend to exercise caution.\"", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is generally well-written and easy to follow. It is well-motivated to use annotated images due to the relatively low-cost annotations. The experiments demonstrate consistent performance gain over existing method like MViTv2, and the ablations are providing a reasonable break-down of improvements coming from different components.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3CRrZRToGo5", "reviewer": "Reviewer_LCAU", "summary": "The authors extend prior work on utilizing scene structure information to improve action recognition performance to transformers. In particular, their model is jointly trained to detect hands and objects in images and to classify activities in videos. The images can come from the same or from a different dataset. Differently to a naive multi-task approach they introduce dedicated tokens that are used for object detection, which improves the perfromance somewhat. Additional loss encourages consistency between object tokens (which are only supervised in images) in frames encoded separately and as a part of a clip to make sure that they are not ignored during video inference (a form of domain adaptation). This consistency loss brings further improvements. Overall, minor to moderate improvements are demonstrated on several ego-centric actions recognition datasets (and, for some reason, Diving48).", "questions": "Please improve overview of the prior work to included methods that use out-of-domain object detection labels to improve action recognition via object graph reasoning and tone down the novelty claims accordingly. \n\nPlease provided an analysis of the method's performance on Diving48 to explain why the proposed approach demonstrated the largest improvements on this dataset despite it not containing any objects. \n\nPlease report results on a 3rd person action recognition dataset of your choosing which does require object reasoning to demonstrate to what degree the learned object representations transfer to the 3rd person scenario. ", "limitations": "It's not clear whether the benefits of the proposed approach are limited to the ego-centric domain.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper is well written and is easy to follow.\n\nThe idea of utilizing scene structure information for improving action recognition perfromance is sound, though not novel.\n\nThe proposed video transformer architecture with object tokens is novel to the best of my knowledge.\n\nNon-negligible improvements over baselines are reported on SomethingElse, Ego4d and Diving48.\n\nA thorough ablation study is provided.\n\n\nWeaknesses:\n\nThe authors ignore prior work which utilizes inference on object graphs to improve action recognition without requiring in-domain object detection label (there is at least A Structured Model For Action Detection by Zhang et al., CVPR'19). There is no conceptual novelty in the proposed approach, only technical novelty of adapting those ideas to the transformer architecture (which is still a valuable contribution).\n\nThe authors refer to their approach as multi-modal, but images and videos are not different modalities. These claims should be removed. \n\nThe biggest improvement is observed on the Diving48 dataset which does not require any object reasoning. This is counterintuitive and not explained.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept