Compositional Video Synthesis with Action Graphs
Authors: Amir Bar
Roei Herzig
Xiaolong Wang
Gal Chechik
Trevor Darrell
Amir Globerson
Venue:  (2021)
Keywords: Video Synthesis
Vision and Language
Representation Learning

Abstract: Videos of actions are complex signals, containing rich compositional structure. Current video generation models are limited in their ability to generate such videos. To address this challenge, we introduce a generative model (AG2Vid) that can be conditioned on an Action Graph, a structure that naturally represents the dynamics of actions and interactions between objects. Our AG2Vid model disentangles appearance and position features, allowing for more accurate generation. AG2Vid is evaluated on the CATER and Something-Something datasets and outperforms other baselines. Finally, we show how Action Graphs can be used for generating novel compositions of actions. 

Under review as a conference paper at ICLR 2021
COMPOSITIONAL VIDEO SYNTHESIS WITH ACTION GRAPHS
Anonymous authors Paper under double-blind review
Action Graph
Tw1, image and layout
Golden
place
Contain, 1-4
Input
Small. Rubber Coea
Contalo, 3-5
Green, Large. Metal Cone
Grey. Langs Metal, Cylinder
Output
T=2
T=3
T=4
We focus video synthesis from actions and propose new task called Action Graph Video. To represent imput actions, we use graph structure called Action Graph, and together with the first frame and first scene layout, our goal is to synthesize video that matches the input actions. For illustration, we include above (partial) example. Our model outperforms various baselines and can generalize to previously unseen compositions of actions.
Figure 1:
ABSTRACT
Videos of actions are complex signals, containing rich compositional structure. Current video generation models are limited in their ability to generate such videos. To address this challenge, we introduce generative model (AG2Vid) that can be conditioned on an Action Graph, structure that naturally represents the dynamics of actions and interactions between objects. Our AG2Vid model disentangles appearance and position features. allowing for more accurate genera- tion. AG2Vid evaluated on the CATER and Something-Somettiing outperforms other baselines. Finally, show how Action Graphs can be used for generating novel compositions of actions.
datasets and
INTRODUCTION
Learning to generate visual content is fundamental task in computer vision, with numerous appli- cations from sim-to-real training of autonomous agents, creating visuals for games and movies. While the quality of generating still images has leaped forward recently (Karras et al., 2020; Brock al., 2019), generating videos is much harder. Generating actions and interactions perhaps the most challenging aspect of conditional video generation. Actions create long-range spatio-temporal dependencies between people and the objects they interact with. For example, when player passes ball, the entire movement sequence of all entities (thrower, ball, receiver) must be coordinated and carefully timed. The current paper focuses on this difficult obstacle, the task of generating coordinated and timed actions,
important step towards generating videos of complex scenes.
Current approaches for conditional video generation are not well suited to condition the generation actions. First, future video prediction (Ye al., 2019; Watters al., 2017), generates future frames based on an initial input frame, but a first frame cannot be used to predict coordinated actions. Second, in video-to-video. the goal is translate sequence of semantic masks into output video.

---

Under review as a conference paper at ICLR 2021
However, segmentation maps contain only class information, and thus do not explicitly capture the action information. As Wang et al. (2018a) notes, this is a limitation that leads to systematic mistakes, such as in the case of car turns. Finally, text-to-video (Li et al., 2018; Gupta et al., 2018) is potentially useful for generating videos of actions because language can describe complex actions. However, in applications that require precise description of the scene, language is not ideal due to ambiguities (MacDonald et al., 1994) or subjectivity of the user (Wiebe et al., 2004), Hence, we address this problem with a more structural approach.
To provide a better way to condition on actions, we introduce a formalism we call an "Action Graph" (AG), propose new task of 'AAtion Graph to Video" (AG2Vid), and present a model for this task. An AG is a graph structure aimed at representing coordinated and timed actions. Its nodes represent objects, and edges represent actions annotated with their start and end time (Fig. 1). We argue that AGs are an intuitive representation for describing timed actions and would be natural way to provide precise inputs to generative models. key advantage of AGs is their ability to describe the dynamics of object actions precisely in a scene.
In our AG2Vid task, the input is the initial frame of the video and an AG. Instead of generating the pixels directly, our AG2Vid model uses three levels of abstraction. First, we propose ac- tion scheduling mechanism we call "Clocked edges" that tracks the progress of actions in different timesteps. Second, based on this, a graph neural network (Kipf & Welling, 2016) operates on the AGS and predicts a sequence of scene layouts, and finally, pixels are generated conditioned on the predicted layouts. We apply this AG2Vid model to the CATER (Girdhar & Ramanan, 2020) and Something Sometiinng (Goyal et al., 2017) datasets and show that this approach results in realistic videos that are semantically compliant with the input AG. To further demonstrate the expressiveness of AG representation and the effectiveness of the AG2Vid model, we test how it generalizes to pre- viously unseen compositions of the learned actions. Human raters then confirm the correctness of the generated actions.'
Our contributions are as follows: 1) Introducing the formalism of Action Graphs (AG) and proposing a new video synthesis task. 2) Presenting a novel action- siriion-rrp--t--Wee -viidoo (AG2Vid) model for this task. 3) Using the AG and AG2Vid model, we show this approach generalizes to the generation of novel compositions of the learned actions.
2 RELATED WORK
Video generation is challenging because videos contain long range dependencies. Recent generation approaches (Vondrick et al., 2016; Kumar et al., 2020; Denton Fergus, 2018; Lee et al., 2018; Babaeizadeh et al., 2018; Villegas et al., 2019) extended the framework of unconditional image generation to video, based on latent representation. For example, MoCoGAN (Tulyakov et al., 2018) disentangles the latent space representations of motion and content to generate sequence of frames using RNNs; TGAN (Saito et al., 2017) generates each frame in a video separately while also having a temporal generator to model dynamies across the frames. Here, we tackle different problem by aiming to generate vidcos that comply with AGs.
Conditional video generation has attracted considerable interest recently, with focus two main tasks: video prediction (Mathieu et al., 2015; Battaglia et al., 2016; Walker et al., 2016; Watters et al., 2017; Kipf et al., 2018; Ye et al., 2019) and video-to-video translation (Wang et al., 2019; Chan et al., 2019; Siarohin et al., 2019; Kim et al., 2019; Mallya et al., 2020). In prediction, the goal to generate future video frames conditioned on few initial frames. For example, it was proposed to train predictors with GANs (Goodfellow et al., 2014) to predict future pixels (Mathieu et al., 2015). However, directly predicting pixels is challenging (Walker et al., 2016). Instead of pixels, researchers explored object-centric graphs and perform prediction on these (Battaglia et al., 2016; Lue et al., 2018; Ye et al., 2019). While inspired by object-centric representations, our method is different from these works as our generation is goal-ori,nted, guided by an AG. The video-to-video translation task was proposed by Wang et al. (2018a), where a natural video was generated from frame-wise semantic segmentation annotations. However, densely labeling pixels for each frame is expensive, and might not even be necessary. Motivated by this, researchers have sought to perform generation conditioned more accessible signals including audio or text (Song al., 2018; Fried
Our code and models will be released upon acceptance.
2

---

Under review as a conference paper at ICLR 2021
Action Graph
Clocked Graph
Carkärs Saich
Gallen Statch
Goldon Snicch
Comais
mokplacedss
Conaiis
placcats
Conuin 75
place 1005.
Bed.
Conusin,35
Red. Small. Ruiderr
Contain
Red. Rubbet
Cuntuiss 100%
Green, Large
Small, Ruiber
Small Rubleer
Lunge Cone
Metil
Mital
Crnw
Cree
Came
Cone
Rotate 54
Lange, Celinder
Crey, Lenge.
Roteso 0%
Large Cylinder
Gney, Largo
A
A
As
A₃
Figure 2: Example of a partial Action Graph execution schedule in different time-steps.
et al., 2019; Ginosar et al., 2019). Here, we propose to synthesize videos conditioned on a novel AG, which is easy to obtain compared to semantic segmentation and is more structured representation compared to natural audio and text.
A recent method, HOI-GAN (HG) (Nawhal et al., 2020b), proposed for the generation task of a single action and object. Specifically, this work addresses the zero-shot setting, and the model is tested action and object compositions which are first presented at test time. Our focus is generation of multiple simultaneous actions over time, performed by multiple objects. Our approach directly addresses this challenge via the AG representation and the notion of clocked edges.
Various methods have been proposed to generate videos based on input text (Marwah et al., 2017; Pan al., 2017; Li al., 2018). Most recent methods typically used very short captions which do not contain complex descriptions of actions. For example, (Li et al., 2018) used video-caption pairs from YouTube, where typical captions are "playing hockey" or "flying kite". Gupta et al. (2018) proposed the Flinstones animated dataset and introduced the CRAFT model for text-to-video generation. While the CRAFT model relies on text-to-vides retrieval, our approach works in an end-to-end manner and aims to accurately synthesize the given input actions.
Scene Graphs (SG) (Johnson et al., 2015; 2018) are a structured representation that models scenes, where objects are nodes and relations are edges. SGs have been widely used in various tasks in- cluding image retrieval (Johnson al., 2015; Schuster al., 2015), relationship modeling (Krishna et al., 2018; Schroeder et al., 2019; Raboh et al., 2020), SG prediction (Xu et al., 2017: Newell & Deng, 2017: Zellers et al., 2018; Herzig et al., 2018), and image captioning (Xu et al., 2019). Re- cently, SGs have been applied to image generation (Johnson al., 2018; Deng et al., 2018; Herzig et al., 2020), where the goal is to generate a natural image corresponding to the input SG. More generally, spatio-temporal graphs have been explored in the field of action recognition (Jain et al., 2016; Sun et al., 2018; Wang & Gupta, 2018; Yan et al., 2018; Girdhar et al., 2019; Herzig et al., 2019; Materzynska al., 2020). For example, space-time region graph is proposed by (Wang & Gupta, 2018) where object regions are taken as nodes and a GCN (Kipf & Welling, 2016) is applied to perform reasoning across objects for classifying actions. Recently, it was also shown by (Ji al., 2019; Yi et al., 2019; Girdhar & Ramanan, 2020) that key obstacle in action recognition is the ability to capture the long-range dependencies and compositionality of actions. While inspired by these approaches, we focus on generating videos which is very different challenge.
Recently, Ji et al. (2019) presented Action Genome, new video dataset annotated by SGs. This dataset includes spatio-temporal SG annotations, where for each video, few individual frames were chosen and spatially annotated by SGs. Here, we use the Something-Somethe 2017) dataset that is larger (200K 10K videos) and more diverse since it includes basic hu- man activities created by large number of crowd workers. Finally, we propose the Action Graph representation, which we view temporal extension of SGs.
V2 (Goyal et al.,
ACTION GRAPHS
Our goal in this work is to build a model for synthesizing videos that contain a specified set of actions. A key component in this effort is developing semantic representation to deseribe the actions performed by different objects in the scene. Towards this end, we introduce a formalism we call Action Graph (AG). In an AG, nodes correspond to objects, and edges correspond to actions that
3

---

Under review as a conference paper at ICLR 2021
Concateration
+ Addittion
Retat; O%
Crey, Lamps, Metal. Cylinder
Colden
Bel Gelle: Siiic
Creen
Sequertial Generator
snisch
Contain, * 30%
GCN
Red. Small,
Contaia 0%
Large- Metal
Vtytiner
Rubber Cone
Pick place,
Conz
4
CCuse
Flow Warping
dig
Smitch
4-1
P/-1
Figure 3: Our AG2Vid Model. The AG Ar describes the execution stage each action at time t. Together with the previous layout 6-1, it is used to generate the next layout & which has object representations that are enriched with A, actions information. Then, &,, 4-11, ft-1 are used to generate the next frame.
these objects participate in. Objects and actions are annotated by semantic categories, and actions also annotated by their start and end times.
More formally, an AG is tuple (C, A,,,,,, O, E") described as follows:
An alphabet of object categories C. Categories can be compounded and include attributes. For example "Blue Cylinder" or "Large Box".
An alphabet of action categories A. For Example "Slide" and "Rotate". Similarly, actions can contain attributes (e.g., rotation speed).
Object nodes O: A set E C objects.
Action edges E: Actions are represented as labeled directed edges between object nodes. Each edge is annotated with an action category and with the time period during which the action is performed. Formally, each edge is of the form (i, a. lg. (e)) where i, E (1,-.., n) are object instances, a E AA is action and t, te N are action start and end time. Thus, this edge implies that object i (which has category o;) performs an action over object j, and that this action takes place between times f, and te. We note that an AG edge can directly model actions over single object and a pair of objects. For example, "Swapp the positions of objects and j between time O and 9t' an action over two objects corresponding to edge (i, swap, j. O, 9). Some actions, such as "Rotate", involve only one object and will therefore be specified as self-loops.
ACTION GRAPH TO VIDEO VIA CLOCKED EDGES
We now turn the key challenge of this paper: transforming an AG into video. Naturally, this transformation will be learned from data. The generation problem is defined as follows: we wish to build generator that takes as input an AG and outputs video. We will also allow conditioning on the first video frame and layout, so we can preserve the visual attributes of the given objects.²
There are multiple unique challenges generating a video from an AG that cannot be addressed using current generation methods. First, each action in the graph unfolds over time, so the model needs to "keep track" the progress of actions rather than just condition on previous frames as commonly done. Second, AGs may contain multiple concurrent actions and the generation process needs to combine them realistic way. Third, one has to design a training loss that captures the spatio-temporal video structure ensure that the semantics of the AG accurately captured.
Clocked Edges. As discussed above, we need a mechanism for monitoring the progress of action execution during the video. A natural approach is to keep "clock" for each action, for keeping track of action progress the video progresses. See Fig. 2 for an illustration. Formally, we keep clocked version of the graph A where each edge augmented with temporal state. Let
Using the first frame and layout can he avoided by using SG2Image model (Johnson et al., 2018; Ashual Wolf, 2019; Herzig al., 2020) for generating the first frame.
4

---

Under review as a conference paper at ICLR 2021
(a,t)) E E be an edge in the AG A. We define the progress of at time to be r r, and clip [0, 1]. Thus, if r O the action has not started yet, if (0,1) it is currently being executed, and if = 1 it has completed. We then create an augmented version of the edge e at time t given by e, = J, (a: F). We define A = {erle A] be the AG at time t. To summarize the above, we take the original graph A and turn it into a sequence of AGs Au...., Ar, where T the last time-step. Each action edge in the graph now has its unique clock for its execution. This facilitates both a timely execution of actions and coordination between actions.
4.1 THE AG2VID MODEL.
Next, we describe our proposed AG-to-video model (AG2Vid). Fig. 3 provides high-level illustra- tion of our model architecture. The rationale of our generation process is that first the AG used to produce intermediate layouts. and then these layouts are used to produce frame pixels We let le = (Ia, Mt. We, h4, 21) denote the set of predicted layouts for all the n objects in the video at time t. The values Tzz, Yt, WA, h4 E [0, 1]" are the bounding box coordinates for all objects, and 2 is descriptor vector for the object (later used for frame generation). Let is denote the generated frame time i, and p( U2,-... ., UT, f2.... fr A, 01, f1) denote the generating distribution of the frames and layouts given the AG and the first frame U1 and scene layout 4,
We assume that the generation of the frame and layout directly depends only on recent generated frames and layouts. Specifically, this corresponds to the following form for p p(U2,... ..... DT, l2,..., er A, 4) = Ii=2 p(fx/ (t (t-1)p(va (,,,1,,,,-11 bution p(l,l-) as The Layout Generating Function (LGF) and to p(uxl-) as The Frame Generating Function (FGF). Next, we describe how we model these distributions as functions.
We refer to the distri-
The Layout Generating Function (LGFF). At time 1 we want to use the previous layout 4-1 1 and current AG A, to predict the current layout &. The rationale is that As captures the current state of the actions and can thus "propagate" " Êς-1 to the next layout. This prediction requires integrating information from different objects as well as the progress of the actions given by the edges of A,- Thus, a natural architecture for this task is Graph Convolutional Network (GCN) that operates on the graph A, whose nodes are "enriched" with the layouts &. Formally, the same structure as Ar, with new features on nodes and edges. At the graph node corresponding to object i the features are comprised of the previous object location defined in f and object class embedding. The features on the edges are the embedding of action and the progress of the action r. taken from (i, a. j.r) from A,. Then, node and edge features are repeatedly re-estimated for K steps using GCN. The resulting activations of the ith object at timestep f. are 2f E IR." which we use as the new object descriptor. An MIPP is then applied to it to produce the new box coordinates, which together form the predicted layout f,- For more details refer to Sec. in the Suppl.
construct new graph of
The Frame Generating Function (FGF). After obtaining the layout & which contains updated objects representations 2t, we wish to use it along with Vt-1 and f, to predict the next frame. The idea is that e,, f--1 characterize how objects should move, žt, žt-1 should capture the object- actions dynamics, and Ut-1 shows their last physical appearance. Combining these information sources we should be able to generate the next frame accurately. As first step, we construct a mask m4-1, m, TR x D using the embedding and layout pairs. Then, we estimate the optical flow at time f, denoted by ft. We let ft = F(v m,-1, m,). The idea is that given the previous frame and two consecutive layouts, we should be able to predict in which direction pixels in the image will move, namely predict the flow. The flow prediction network F' is similar to (Ilg et al., 2017), and it trained using an auxiliary loss and does not require additional supervision (see section 4.2). Given the flow fe and previous frame 2t-1 natural estimate of the next frame is to use warping function (Zhou et al., 2016) w, W(fe. 1t-1). Finally we fine-tune we via a network S(m,, we) that provides an additive correction resulting in the final frame prediction: V.- = wy + 5(m,, wx). where the S network a SPADE generator (Park et al., 2019).
Note that layout contain descriptors that can encode all generation history.
5

---

Under review as a conference paper at ICLR 2021
Actions in CATER
Actions in Something Something
Multiple Simultaneous Actions
Push Left
Move Down
Uncover
Push
Slide
Contain
Pick Place
Rotate
Push Right
Move Up
Cover
Take
Figure 4: Qualitative examples of generation on CATER Something Something. AG2Vid generated videos of four and eight standard actions on CATER and Something Something. respectively. For CATER we also used AGs with multiple simultaneous actions, and the generated actions indeed correspond to those (verified manually). For more examples please refer to Figure and in the Supp. Click the image play the video clip browser.
4.2 LOSSES AND TRAINING
We ground truth frames oF and layouts GCT for training. and use the following losses: Layout Prediction Loss Lr. Defined as L, || oedT|1, the Lı loss between ground-truth bounding boxes ęCT and predicted boxes &. Here ignore the object descriptor part of & Pixel Action Discriminator Loss LA. For the generated pixels we employ GAN loss that uses a discriminator between generated frames tt and GT frames Die conditioned on A, and 4. Formally, let DA be discriminator with output in (0, 1). First, a GCN is applied over A, to ob- tain objects representations, which are then using together with the GT layout boxes to construct scene layout. The layout and frames are then concatenated and fed into a multi-scale PatchGAN discriminator (Wang et al., 2018b). The loss is then the GAN loss (e.g., see Isola et al. (2017)):
LA = max Eer [log Da(Ai, oT ((TT)) Ep llog(1 Da(Ai, 0t, (GT
where GT corresponds to sampling frames from the ground truth videos, and p corresponds to sam- pling from the generated videos. Optimization of this loss done the standard way of alternating gradient ascent D parameters and descent generator parameters.
Flow Loss Ly. The flow loss measures the error between the warps of the previous frame and the ground truth of the next frame ofT; L3 - ET= T-1 ju: valla. where w,, W - as defined in Section 4.1. This loss was proposed previously by (Zhou et al., 2016; Wang et al., 2018a). Perceptual and Feature Matching Loss Lp. We use these losses as proposed pix2pixHD (Wang et al., 2018b; Larsen et al., 2016) and other previous works.
The overall optimization problem to minimize the weighted sum of the losses.
EXPERIMENTS AND RESULTS
We evaluate our AG2Vid model on datasets: CATER and Something Something V2 (Smth). For each dataset, we learn an AG2Vid model with given set of actions. We then evaluate the visual quality of the generated videos and measure how they semantically comply with the input actions. Last, estimate the generalization of the AG2Vid model to novel composed actions.
Datasets. We use two datasets: (1) CATER (Girdhar & Ramanan, 2020) synthetic video dataset originally created for action recognition and reasoning. Each video contains multiple objects performing actions. The dataset contains bounding-box annotations for objects, as well as labels of the actions. These include: "Rotate". "Cover", "Pick Place" and "Slide". See Figure for
*GT layouts be obtained automatically from videos using object tracking pre-processing step.

---

Under review as a conference paper at ICLR 2021
Hand
Hand
R E.oo
Came
Move Right.
Move Left
Cartalls 1-10
Consain
1:10
3:B
1:10
Move Up
Dows,
Centain IJD
Conten
Slide
I-10
Spheeee
Cone
G Cine
Lec(S-CCone)
GT
CVP
V2V
Ours
Right Up
Left Down
Huddle
Swap
Figure 5: Comparison of baselines methods. The top row are based on CATER videos, while the bottom row are based Something Something. Click the images play the video clips in browser.
Figure Composing Something and CATER. For example, "swap" ac- tion is composed by combining the "Pick-Place" and "Slide" actions on frames 10 and their locations.
actions in Something-
Table 1:
Human evaluation of action generation with respect to Semantic Accuracy and Visual Quality, For each metric, raters selected the better of two generation methods. In the results XX/YY means that AG2 was selected as better for X% of the presented pairs. Image resolution is 256 x 256.
Semantic Accuracy
Visual Quality
Methods AG2Vid Baseline
CATER
Smth
CATER
Smth
AG2Vid CVP et 2019)
85.7/14.3 -/
90.6/9.4
76.2//3.8
93.8/6.2
AG2Vid HG (Nawhal et al., 2020a)
84,6 15.4
88.5/
AG2Vid V2V (Wang et al., 2018a)
68.8 /1.2
84,4/15.8
68,8/313
96.9/3.1
AG2Vid RNN
56.0 56.0/44.0
80.6/19.4
52.0/48.0
77.8 /2.22
AG2Vid AG2Vid, GTL
48.6/51.4
46.2/53.8
42.9/57.1
50.0/50.0
qualitative examples. For "Pick Place" and "Slide" we include the action destination coordinates. We use these actions to create action graphs for training and evaluation.
We employ the standard CATER training partition (3849 videos) and split the validation into 30% val (495 videos) and use the rest for testing (1156 videos). (2) Something Something V2 (Goyal et al., 2017) contains real world videos of humans performing basic actions. Here we use the eight most frequent actions (e.g., "Putting [something] with [something]"". All videos contain up to three different objects, including the hand which is performing the action. We use the box annotations of the objects from Materzynska et al. (2020). See Sec. 3 in Suppl for the full list of actions.
a surface" and "Covering [something]
Implementation details. The GCN model uses K 3 hidden layers and embedding layer of 128 units for each object and action. For optimization we use ADAM Kingma & Ba (2014) with Ir = le and (81, 82) (0.5, 0.99). Models weights (see section 4.2) we use λB Af = Xp = 10 and A = 1. For training we use a batch size 2. We use videos of 8 FPS and FPS for CATER and Smth and evaluate on videos consisting of 16 frames which correspond to spans of 2.7 and 2 seconds accordingly.
trained on an NVIDIA V100 GPU. For loss
Performance metries. The AG2Vid outputs can be quantitatively evaluated as follows. a) Visual Quality: is common in video generation to evaluate the visual quality of videos, regardless of the semantic content. To evaluate visual quality, we use the Learned Perceptual Image Patch Sim- ilarity (LPIPS) (Zhang et al., 2018) (lower is better) over predicted and GT videos. For the Smth dataset, since videos contain single actions (meaning the AG contains a single triplet), we can report the Inception Score (IS) Salimans et al. (2016) (higher is better) and Fréchet Inception Distance (FID) (Heusel et al., 2017) (lower better) using a TSM (Lin et al., 2019) model, pretrained on Smth. We note that cannot report FID and IS CATER since it provides multiple activities simultaneously. and hence does not support a pretrained video classifier. Finally, we also evaluate relative visual quality of two models by asking human annotators select the video with higher quality. b) Semantic Accuracy: The key goal of AG2Vid to generate videos which contain spec- ified actions. To evaluate this, ask human annotators select which of two video generation

---

Under review as a conference paper at ICLR 2021
Table 2: Visual quality metrics of conditional iideo-generation methods in CATER and Smth. All methods resolution 256 x 256 except for HG, which only supports 64 x 64.
Inception f
FID
LPIPS L
Resolution
Methods
Smith
Smth
CATER
Smth
Real Videos
3.9 = 0.12
0.0
0.0 = 0.0
0.0 0.0
64x64
HG (Nawhal et al., 2020a)
1.66 = 0.03
35.18 + 3.6
0.33 0.08
AG2Vid (Ours)
2.51 + 0.08
26.05 + 0.73
0.04 0.01
0.13 0.01
Real Videos
7.58 0.2
0.0
0.0 1 0.0
0.0 0.0
(Yc al., 2019)
1.92 0.03
67.77
0.24 0.04
0.55
RNN
1.99 0.05
74.17 + 1.54
0.14 0.05
0.26 - 0.08
256x256
V2V (Wang ct al., 2018a)
2.22 0.07
67,51 + 1.42
0.11
0.02
0.29 = 0.09
AG2Vid (Ours)
3.02 0.11
66.7 + 1.29
0.07
0.02
0.25 0.08
AG2Vid, GTL (Ours)
3.52 + 0.14
65.04
1.25
0.06 0.02
0.22 0.09
Table 3: Ablation experiment for components of the frame generation. Losses are added one by one.
Inception f Smth
FID 1
LPIPS |
Loss
Smth
CATER
Smth
Flow
1.59 T 0.02
107.26 1 1.46
.14 1 .01
.70 .06
Perceptual
2.21 1 0.07
71.70 + 1.46
.08 1 .03
.29 1 .07
Action Disc.
3.02 1 0.11
66.7
1.29
.07 .02
.25 i .08
models provides a better depiction of actions the real video. The protocol is similar to the visual quality evaluation above. We also evaluated action timing, see below.
Compared methods. Generating videos based on action-graphs is a new task. There are no off-the- shelf models that can be used for direct evaluation with our approach, since no existing models take as input an action graph and output a video. To provide fair evaluation, we compare with two types of baselines. (1) First, existing baseline models that share some functionality with AG2Vid. (2) Second, variants of the AG2Vid model that shed light on its design choices. Each baseline serves to evaluate specific aspects of the model, as described next. Basclines: (1) HOI-GAN (HG) (Nawhal et al.,, 2020b) generates videos given a single action-object pair, an initial frame and a layout. It can be viewed as operating on two-node action graph without timing information. we compare HG to AG2Vid on the Smth dataset because it contains exactly such action graphs. HG is not applicable to CATER data, (2) CVP (Ye et al., 2019) uses as input the first image and layout for future frame prediction without access to action information. CVP allows us to asses the visual quality of AG2 videos. However, it is not expected that CVP captures the semantics of the action- graph, unless the first frame and action are highly correlated (e.g.. hand at the top-left corner always moves downwards). (3) V2V (Wang et al., 2018a): This baseline uses a state-of-the-an Vid2Vid model based (Wang et al., 2018a) to generate videos from ground- truth layout. Since it uses ground- truth layout it provides an upper bound on Vid2Vid performance for this task. We note that Vid2Vid cannot use the action graph, and thus it is not provided input. AG2Vid variants: (4) RNN: This AG2Vid variant replaces the layout generation GCN with an RNN that processes the action graphs. The frame generation part the same as AG2Vid. More details are provided in the Supp Sec.4. 1. (5) AG2Vid, GTL: An AG2Vid model that uses ground truth layout at inference time. It allows us to test if using the GT layout for all frames improves overall AG2Vid video quality and semantics.
Layout Generation Ablations. We experiment with an RNN architecture as an alternative to the GCN implementation of the LGF. The motivation behind the RNN experiment to compare the design choice of the GNN to a model that processes edges sequentially (RNN). This RNN has access to the same input and supervision to the GCN, namely, 4-1 and A., and the results from Table 4 confirm the advantage of GCN processing. For more details, see Sec. 4.1 in the Supplementary.
Loss Ablations. Table 3 reports ablations over the losses of FGF, confirming that the perceptual loss and actions discriminator losses improve the overall visual quality on CATER and Smth.
Semantic and Visual Quality. Fig. 4 shows sample videos generated by AG2Vid, and Fig. 5 shows comparison to generation by baselines. Table shows the results of human evaluations for semantic and visual quality. It can be seen that AG2Vid is more semantically accurate and has better visual
8

---

Under review as a conference paper at ICLR 2021
quality than the baselines, and it is comparable AG2Vid,GTL. See Sec 4.4 in Suppl for additional evaluations of AG2Vid correctness of the generated actions. Table evaluates visual quality using several metrics, with similar takeaways as in Table 1.
Action Timings. To evaluate the extent to which the AG2Vid model can control the timing of actions, generated AGs of actions at different times and asked annotators choose in which video the action is executed first. In 89.45% of the cases, the annotators confirmed the intended result. For more information see Sec. 4.3 in Suppl.
Composing New Actions. To evaluate the extent to which the AG2Vid model can generalize at test time to unseen actions, we manually defined four compositions of learned actions. As seen in Fig. 6, we are using learned atomic actions to generate new action combinations that did not appear in the training data (either by having the same object perform multiple objects at the same time, or multiple objects performing coordinated actions). For example, in CATER, we created the actions "swap" based on "pick-place" and "slide" and "huddle" based on "contain". For Smth we composed the "push-left" and "move-down" to form the "left-down" action. For each generated video, raters were asked to choose the correct action class from a list of possible actions. The avg. class re- call for CATER and Smth is 96.65 and 87.5 respectively, see the Supplementary for results by action.
6 DISCUSSION
We present a video-synthesis approach with a new Action Graph formalism, that describes how mul- tiple objects interact in scene over time. By using this formalism, we can synthesize complicated compositional videos and construct novel actions and action combinations. Although our approach outperforms previous methods, our model still fails in several situations. First, our model depends on the initial frame and layout. This could be potentially addressed by using an off-the-shell im- age generation model. The formal AG representation designed for describing complex semantic information in dasy-to-grasp way. The formalism could be further extended to handle other ac- tions or their properties that were not present in today's datasets. For instance, may be desired to capture features of actions described by adverbs. This can be achieved by adding attributes over actions, which we leave for future work. Finally, in this work we present an hierarchical and mod- ular pipeline of video synthesis: first actions are scheduled for specific timestep, then the scene layout is predicted, and finally the future flow is predicted and refined. While this pipeline fairly general, we believe these representations can be further adopted to different datasets. For example, pose representation can be added for videos of people.
7 BROADER IMPACT
The paper proposes new framework for video generation, which focuses on coordinated multiple simple actions, operating on simple daily objects. We believe it has potential for improving the qual- ity and versatility of video generation. Video synthesis technology has many practical implications, such generating simulated data for training robots and improving content search in video. These clearly have positive societal impact. The current work does not focus on generating faces or human movement, and as a result, we estimate that the potential for negative societal and ethic aspects is low.
REFERENCES
Oron Ashual and Lior Wolf. Specifying object attributes and relations in interactive scene genera- tion. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4561-4569, 2019.
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H. Campbell, and Sergey Levine. Stochastic variational video prediction. International Conference on Learning Representations, 2018.
Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. In Advances in neural information processing systems, pp. 4502-4510, 2016.

---

Under review as a conference paper at ICLR 2021
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.
Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A Efros. Everybody dance now. In Pro- ceedings of the IEEE International Conference on Computer Vision, pp. 5933- 5942, 2019.
Zhiwei Deng, Jiacheng Chen, Yifang Fu, and Greg Mori. Probabilistic neural programmed networks for scene generation. In Advances in Neural Information Processing Systems, pp. 4028-4038, 2018.
Emily Denton and Rob Fergus. Stochastic video generation with a learned prior. In ICML, pp. 1174-1183, 2018.
Ohad Fried, Ayush Tewari, Michael Zollhöfer, Adam Finkelstein, Eli Shechtman, Dan Gold- man, Kyle Genova, Zeyu Jin, Christian Theobalt, and Maneesh Agrawala, Text-based editing of talking-head video. ACM Transactions on Graphics (TOG), 38(4):1-14, 2019.
Shiry Ginosar, Amir Bar, Gefen Kobavi, Caroline Chan, Andrew Owens, and Jitendra Malik. Learn- ing individual styles of conversational gesture. Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pp. 3497-3506, 2019.
Rohit Girdhar and Deva Ramanan. CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning. ICLR, 2020.
Rohit Girdhar, Joao Carreira, Carl Doersch, and Andrew Zisserman. Video action transformer net- work. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp-
244-253, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-266660 26800, 2014.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Aheller-Freeitgg et al. The" something something" video database for learning and evaluating visual common sense. In ICCV, pp. 5, 2017.
Tanmay Gupta, Dustin Schwenk, Ali Farhadi, Derek Hoiem, and Aniruddha Kembhavi. Imagine this! scripts to compositions to videos. In Proceedings of the European Conference on Computer Vision (ECCV), pp.
2018.
Roei Herzig, Moshiko Raboh, Gal Chechik, Jonathan Berant, and Amir Globerson. Mapping im- ages to scene graphs with permutation-invarian structured prediction. In Advances in Neural Information Processing Systems (NIPS), 2018.
Roei Herzig, Elad Levi, Huijuan Xu, Hang Gao, Eli Brosh, Xiaolong Wang, Amir Globerson, and Trevor Darrell. Spatio-temporal action graph networks. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 0-0, 2019.
Roei Herzig, Amir Bar, Huijuan Xu, Gal Chechik, Trevor Darrell, and Amir Globerson. Learn- ing canonical representations for scene graph to image generation. European Conference on Computer Vision, 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Informaiion Processing Systems 30, pp. 6626-6631 Curran Associates, Inc., 2017.
E. 1lg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jul 2017. URL http: / 1mb. fnfrmttikk uni - fre Freiburg, de/ crublicatioss/// I/MKDB117
10

---

Under review as a conference paper at ICLR 2021
Phillip Isola, Jan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-inet translation with conditional adversarial networks. CVPR, 2017.
Ashesh Jain, Amir R Zamir, Silvio Savarese, and Ashutosh Saxena. Structural- Deep learning on spatio-temporal graphs. Proceedings of the ieee conference on computer vision and pattern recognition, pp. 5308-5317, 2016.
Jingwei Ji, Ranjay Krishna, Fei-Fei, and Juan Carlos Niebles. Action genome: Actions com-
position of spatio-temporal scene graphs. arXiv preprint arXiv:1912.06992, 2019.
Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma, Michael Bernstein, and Li Fei-Fei. Image retrieval using scene graphs. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 3668-3677 2015.
Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image generation from scene graphs. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 1219-1228, 2018.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
Dahun Kim, Sanghyun Woo, Joon-Young Young Lee, and In So Kweon. Deep video inpainting. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5792-5801, 2019.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6998 2014.
Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. arXiv preprint 1rX1v:7802.04687, 2018.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. arXiv preprint 1 2016.
Ranjay Krishna, Ines Chami, Michael S. Bernstein, and Li Fei-Fei. Referring relationships. ECCV, 2018.
Manoj Kumar, Mohammad Babaeizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine, Laurent Dinh, and Durk Kingma. Videoflow: A conditional flow-based model for stochastic video gener- ation. In International Conference on Learning Representations, 2020.
Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther, Autoen- coding beyond pixels using learned similarity metric. In Proceedings of The 33rd International Conference on Machine Learning, pp. 1558-1566, 2016.
Alex X. Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint arXiv:1804. 01523, 2018.
Yitong Li, Martin Renqiang Min, Dinghan Shen, David E Carlson, and Lawrence Carin. Video generation from text. In AAAI, pp. 5, 2018.
Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
Pauline Luc, Camille Couprie, Yann Lecun, and Jakob Verbeek. Predicting Future Instance Seg- mentation by Forecasting Convolutional Features. ECCV, pp. 593-608, 2018.
Maryellen C MacDonald, Neal J Pearlmutter, and Mark Seidenberg. The lexical nature of syntactic ambiguity resolution. Psychological review, 101(4)),76, 1994.
Arun Mallya, Ting-Chun Wang, Karan Sapra, and Ming- Yu Liu. World-consistent video-to-video synthesis. In European Conference Computer Vision (ECCV), 2020.

---

Under review as a conference paper at ICLR 2021
Tanya Marwah, Gaurav Mittal, and Vineeth N Balasubramanian. Attentive semantic video genera- tion using captions. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1426-1434, 2017.
Joanna Materzynska, Tete Xiao, Roci Herzig, Huijuan Xu, Xiaolong Wang, and Trevor Darrell. Something-else: Compositional action recognition with spatial-temporal interaction networks. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond
mean square error, arXiv preprint wrX.v: 2015.
Megha Nawhal, Mengyao Zhai, Andreas Lehrmann, Leonid Sigal, and Greg Mori. Generating videos of zero-shot compositions of actions and objects. In Proceedings of the European Confer- ence on Computer Vision (ECCV), 2020a.
Megha Nawhal, Mengyao Zhai, Andreas Lehrmann, Leonid Sigal, and Greg Mori. Generating videos of zero-shot compositions of actions and objects. In European Conference on Computer Vision (ECCV), 2020b.
Alejandro Newell and Jia Deng. Pixels to graphs by associative embedding. In Advances in Neural Information Processing Systems 30 (to appear), pp. 1172- 1772-1180 180. Curran Associates, Inc., 2017.
Yingwei Pan, Zhaofan Qiu, Ting Yao, Houqiang Li, and Tao Mei. To create what you tell: Generat- ing videos from captions. In Proceedings of the 25th ACM international conference on Multime- dia, pp. 1789-1798, 2017.
Taesung Park, Ming-Yu Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2337-2346, 2019.
Moshiko Raboh, Roei Herzig, Gal Chechik, Jonathan Berant, and Amir Globerson. Differentiable scene graphs. In Winter Conf. App. of Comput. Vision, 2020.
Masaki Saito. Eicchi Matsumoto, and Shunta Saito. Temporal generative adversarial nets with sin- gular value clipping. In ICCV, 2017.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved techniques for training gans. In D. D. Lee, M. Sugiyama, U. V. Luxburg, Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp- 2234-2244.. Curran Associates, Inc., 2016.
Brigit Schroeder, Subarna Tripathi, and Hanlin Tang. Triplet-aware scene graph embeddings. In The IEEE International Conference on Computer Vision (ICCV) Workshops, Oct 2019.
Sebastian Schuster, Ranjay Krishna, Angel Chang, Li Fei-Fei, and Christopher Manning. Gen- erating semantically precise scene graphs from textual descriptions for improved image retrieval. In Proceedings of the fourth workshop on vision and language, pp. 70-80, 2015.
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. Animating arbitrary objects via deep motion transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2377-2386, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 14X9:::409.1556 2014.
Yang Song, Jingwen Zhu, Dawei Li, Xiaolong Wang, and Hairong Qi. Talking face generation by conditional recurrent adversarial network. arXiv preprint arXiv:1804 NKKv:1800.00786, 2018
Chen Sun, Abhinav Shrivastava, Carl Vondrick, Kevin Murphy, Rahul Sukthankar, and Cordelia Schmid. Actor-centric relation network. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 318-334, 2018.
Sergey Tulyakov. Ming- Yu Liu, Xiaodong Yang, and Jan Kautz. MoCoGAN: Decomposing motion and content for video generation. In IEEE Conference on Computer Vision and Pattern Recogni-
fion (CVPR). pp. 1526-1535. 2018.
12

---

Under review as a conference paper at ICLR 2021
Ruben Villegas, Arkanath Pathak, Harini Kannan, Dumitru Erhan, Quoe V Le, and Honglak Lee. High fidelity video prediction with large stochastic recurrent neural networks. In Advances in Neural Information Processing Systems 32, pp. 81-91. Curran Associates, Inc., 2019.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In Advances in neural information processing systems (NIPS), 2016.
Jacob Walker, Carl Doersch, Abhinav Gupta, and Martial Hebert. An uncertain future: Forecasting from static images using variational autoencoders. In European Conference on Computer Vision, pp. 835-851. Springer, 2016.
Ting- Hing-Chun Wang, Ming-Ya Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Video-to-video synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2018a.
Ting-Chum Wang, Ming-Yi Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8798 8807, 2018b.
Ting- Ling-Chun Wang, Ming- Yu Liu, Andrew Tao, Guilin Liu, Jan Kautz, and Bryan Catanzaro. Few- shot video-to-video synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Xiaolong Wang and Abhinav Gupta. Videos as space-time region graphs. In ECCV, 2018.
Nicholas Watters, Daniel Zoran, Theophane Weber, Peter Battaglia, Razvan Pascanu, and Andrea Tacchetti. Visual interaction networks: Learning a physics simulator from video. Advances in neural information processing systems, pp. 4539-4547, 2017.
Janyce Wiebe, Theresa Wilson, Rebecca Bruce, Matthew Bell, and Melanie Martin. Learning sub- jective language. Computational linguistics, 30(3):277-308, 2004.
Danfei Xu, Yuke Zhu, Christopher B. Choy, and Li Fei-Fei. Scene Graph Generation by Iterative Message Passing. In Proc. Conf. Comput. Vision Pattern Recognition, pp. 3097-3106, 2017.
Ning Xu, An-An Liu, Jing Liu, Weizhi Nie, and Yuting Su. Scene graph captioner: Image cap- tioning based on structural visual representation. Journal of Visual Communication and Image Representation, pp. 477-485, 2019.
Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for skeleton-based action recognition. In Thirty-second AAAI conference artificial intelligence, 2018.
Yufei Ye, Maneesh Singh, Abhinav Gupta, and Shubham Tulsiani. Compositional video prediction. In Proceedings of the IEFE International Conference Computer Vision, pp. 10353-10362, 2019.
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprint arXiv: /rXiv:/910.0144 191.01444, 2019.
Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with global context. In Conference on Computer Vision and Pattern Recognition, 2018.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018
Tinghui Zhou, Shubham Tulsiani, Weilun Sun, Jitendra Malik, and Alexei A. Efros. View synthesis by appearance flow. In ECCV, 2016.
13

---

Under review as a conference paper at ICLR 2021
In this supplementary file provide additional information about model, training losses, perimental results, and qualitative examples.
GRAPH CONVOLUTION NETWORK
As explained in the main paper, we used a Graph Convolution Network (GCN) (Kipf & Welling, 2016) to predict the layout at time step t. The GCN uses the structure of the action graph, and propagates information along this graph (in K iterations) to obtain set of layout coordinates per object.
Each object category E C is assigned learned embedding Φ. E -RD and each action a E R is assigned a learned embedding 4h R.. We next explain how obtain the layouts & using GCN. Consider the action graph A, at time t with the corresponding clocked edges (i, a. j. Denote the layout for node i time 1 by 1,- The GCN iteratively calculates a representation for each object and each action in the graph. Let 2,,k Rd be the representation of the 8th object in the k layer of the GCN. Similarly, for each edge in As given by (i. a. j.j.)) r) let ue R be the representation of this edge in the kAA layer. These representations are calculated follows. At the GCN input, we set the representation for node i to be: Z = [ f.-1.). And, for each edge
= (i, a; j, r") set Ue.0 = [ r, 1, 1-1.) All representations at time 0 are transformed to D dimensional vectors using an MLP. Next, we use three functions (MLPs) Fs, Fa, For each from RD x IR x R 2 to IR". These can be thought of as processing three vectors on an edge (the subject, action and object representations) and returning three new representations. Given these functions, the updated object representation is the average of all edges incident on i:3
2
Σ
Fa(zi,k Me,k, Zj,e) +
Σ (.a,ir)))))
FolEj,k Ue Zi,k)
(=(ia.j)))
Similarly, the representation for edge e updated via: ue Fa Ue 2j,++)).
Finally, we transform the GCN representations above at each time-step to layout & as follows. Let K denote the number GCN updates. The layout coordinates of l,, are the output of an MLP applied to z which simply the set of the predicted normalized bounding box coordinates. The object descriptor =
LOSSES AND TRAINING
We elaborate on the Flow and Perceptual losses from Section 4.2.
Optical flow loss Lr. The flow loss Lr is the warping loss which measures the error between the warps of the previous frame and the ground truth of the next frame o.
C, --
we - 0t GT l|2
(3)
T-11-
t=1
where w, = W(f,, 11-1) as defined in Section 4.1. This flow loss proposed previously in Wang al. (2018a); Zhou et al. (2016).
Perceptual loss Lp. This is the standard perceptual loss as in HiX2pixHD (Wang et al., 2018b). In particular, we use the VGG network (Simonyan & Zisserman, 2014) as a feature extractor and minimize the error between the extracted features from the generated and ground truth images from layers.
L. Cp Σ - (1660 (10
Note that box can appear both "subject" and "object" thus two different in the denominator.
14

---

Under review as a conference paper at ICLR 2021
Slide
Contain
Pick Place
Rotate
Swap
Huddle
Figure 7: Qualitative examples generation actions CATER dataset. use AG2Vid model to generate videos of four standard actions and two composed unseen actions ("Swap" and "Huddle"), The objects involved in actions are highlighted. Click the image to play the video clip in browser.
where (00) denotes the (-th layer with Pi elements of the VGG network. We sum the above all frames in the videos.
The overall optimization problem minimize the weighted sum of the losses:
min max LA (DA) + XcC Az XpCp. DA
(5)
where 9 all the trainable parameters of the generative model, Li the Layout loss, and C the pixel action discriminator loss from Section 4.2. In addition to the loss terms in Equation 5, we use a feature matching loss (Larsen al., 2016; Wang et al., 201 (018b) to match the statistics of features extracted by the GAN discriminators.
ACTIONS
For the Something Something dataset (Goyal et al., 2017), the eight most frequent actions. These include: "Putting (something] on a surface", "Moving [something] up". "Pushing [something] from left right", "Moving [something] down". "Pushing [something] from right to left", "Covering [something] with [something]"" "Uncovering [something]", "Taking [one of many similar things on the table]" See Figure 8 for qualitative examples. The box annotations of the objects from the videos are taken from Materzynska et al. (2020).
mIOu 1
R@0.3 1
R@0.5 1
Model
CATER
Smth
CATER
Smth
CATER
Smth
Random
5.05
13.55
5.94
16.50
01.86
4.81
RNN
75.71
41.28
80.67
61.70
78.91
39.23
AG2Vid
93.09
51.32
99.55
74.50
98.04
53.85
Table Layout generation evaluation.
15

---

Under review as a conference paper at ICLR 2021
pooc c
Uncover
Cover
Put
Right Up
Down Left
Push Right Push Left
Move Up
Move Down
Take
Figure 8: Qualitative examples for the generation of actions on Something Something dataset. We use our AG2Vid model to generate videos of eight standard actions and two composed unseen actions (" ("Right Up" and "Down Left"). Click the image to play the video clip in browser.
4 EXPERIMENTS AND RESULTS
4.1 RNN BASELINE
We experiment with an RNN architecture as an alternative to the GCN implementation of the LGF. This RNN has access to the same input and supervision to the GCN, namely, to 4-1 and A, We next explain how to obtain the layouts l, using the RNN.
Each object category e E assigned a learned embedding de E RR and each action a E R is assigned learned embedding vha R + 1. Consider the action graph A, at time / with the corresponding clocked edges (i, j.
let U, E R|E|x4D+1 denote the matrix, where every row that corresponds to edge is comprised of the object, action, subject embeddings, the embedding of the ith object, and the target the progress feature. We apply the RNN over U, and denote 2 as the last hidden state of the result. The new object descriptor of 4, is then 2,,,, and to obtain new bounding box location, MLP is applied over ži,t- 4 is the new updated bboxes and object descriptors. The RNN model has 3 layers and 512 hidden layer size.
16

---

Under review as a conference paper at ICLR 2021
Composed Actions
Standard Actions
Composed Actions
Swap Huddle
DI.
Slide
Contain
Pick Place
Rotate
Swap
Huddle
92.1
98.6 75.0 100.
96.7
100.0
90.0
56.7
93.3
100.0
Table Human evaluation of the semantic accuracy of the ac- tions in the generated videos.
Table 6: Human evaluation of timing generated videos (see
tion 4.3). The table reports accuracy human annotator answer with respect to the true answer.
4.2 LAYOUT ACCURACY
AG2Vid produces bounding boxes of objects function of time. Since our datasets contain ground-truth boxes, we can compare our predictions these. We evaluate the intersection over union (IOU) over the predicted and ground truth boxes. We report the mean intersection over union (mIOU) which is the mean over the entire set of boxes. Additionally, we measure the recall over the by considering higher than 0.3 (R.0.3) or 0.5 (R@0.5) Results are reported in Table It can be seen that the RNN underperforms, supporting the choice of GCN for layout generation in the AG2Vid model. The RNN is likely to under-perform as assumes order over the AG list of edges, which is not natural way process graph.
object be correct detected if the IOU between the GT and predicted box
4.3 HUMAN EVALUATION OF ACTION TIMING IN GENERATED VIDEOS
As described in Section 5.1, we evaluated to which extent the action graphs (AGs) control the timing execution of actions on the CATER dataset. Thus, we generated 90 pairs action graphs where the only difference between the graphs is the timing of action. We then asked the notators to select the video where the action is executed first. The full results are depicted in Table 6, and visual examples are shown in Figure 9. The results for all actions but "Rotate" are consistent with the expected behavior, indicating that the model correctly executes actions in timely fashion. The "Rotate" action is especially challenging to generate since it occurs within the intermediate layout. It is also easier to miss as it involves relatively subtle change in the video.
Slide
Contain
Pick Place
Rotate
Swap
Huddle
Figure 9: Timing experiment examples CATER. We show the clock edges can manipulate the timing of a vidco by controlling when the action is performed to achieve goal-on -rientedd vidco synthesis. The objects involved in "rotate" are highlighted. Click the image to play the video clip in browser.
4.4 HUMAN EVALUATION OF SEMANTIC QUALITY IN GENERATED VIDEOS
To test the degree to which the generated videos match their corresponding actions, generated twenty videos per action for the Something-Sone Something dataset and asked three different human annotators evaluate each video. Each annotator was asked to pick the action that best describes
17

---

more easily Under review as a conference paper at ICLR 2021
the video out of the list of possible actions. We provide the results in Fig. 7. Each cell in the table corresponds to the class recall of a specific action. To determine a video correctly matches its corresponding action, we used the majority voting over the answers of all annotators.
turns out that humans do not perform perfectly in the above task. We quantified this effect in the following experiments the Something-SSmethinn dataset. We used the above annotation process for ground-truth videos (see "Real" row in Table 7). Interestingly, it can be seen from the reported accuracy in Table 7 that our generated action videos of "Move Down" and "Take" recognizable by humans than the ground truth videos. For the CATER dataset, we did not perform such human evaluation of predicted actions, since CATER videos contain multiple activities.
To evaluate the extent to which the AG2Vid model can generalize at test time to unseen actions, we manually defined four compositions of learned actions. In Table we show the semantic accuracy of the human evaluation we did for the new unseen actions: "huddle", "Swap", "Right Up" and "Down Left".
Standard Actions
Video Source
Right
Up
Down
Left
Put
Take
Uncover
Cover
Generated Real
100.
50.
100.
75.
95.
80.
25.
55.
100.
100.
90.
100.
100.
65.
100.
85.
Table 7: The semantic quality evaluated by humans of the generated and real action videos. We asked raters to select the action described in the video for each synthesized video with a given action. The table reports the accuracy of the human annotators with respect the true action underlying the video. Actions above correspond "Pushing [something] from left to right', 'Moving [something] up', 'Moving [something] down', Pushing (somcthing] from right to left', 'Putting /something] on a surface', "Taking (one of many similar things on the table]', "Uncovering [something]', 'Covering [something] with [something]'
4.5 COMPARING AG2VID TO SCENE-GRAPE BASED GENERATION
Scene graphs expressive formalism for describing image content. Both datasets have frame-level scene graph annotation. Thus, we wanted to compare generation from these scene graphs with generation from action graphs. Towards this end, we used scene- scene-graph-te soaphtt--imaee model (John- son et al., 2018) trained to generate the images in the videos from their corresponding scene graphs. This model does not condition the action or initial frame and serves only for comparison in terms of realistic generation. It can be seen in Figure 10 that the temporal coherency of AG2Vid is more consistent and coherent than the sequence scene graphs.
18

---

Under review as a conference paper at ICLR 2021
GT
SG2IM
Ours
Figure 10: Comparing Sg2lm and Ag2Vid results CATER. Each column is different sample. Click image to play the video clip in a browser.
19


Reviews:
[{"review_id": "e3JmR7Z3hLD", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview:\n\nThe paper proposes a hierarchical approach to video synthesis based on Action Graph. Action Graph is a graph representation to describe the dynamics of individual objects. Based on this, the authors proposes an action scheduling mechanism to track the progress of action and then generate the scene layout at each timestamp. Finally, the pixels are generated based on the predicted scene layout. Experiments show that such AG2Vid paradigm can generate images on CATER and Something-Something dataset with a better quality compared to the baselines. It can also generate novel actions, treated by a composition of seen actions.\n\nStrengths:\n\n++ The idea of three-level abstraction (action schedule + scene layout generation + pixel generation) is sound. The formulation based on the idea is neat and straightforward.\n\n++ The experiments on generating multiple actions, single action, as well as novel actions indicate that the method is capable of disentangling and composing atomic action for individual object.\n\n\nWeaknesses:\n\n-- In the stage of pixel generation, it seems that the frame generation function cannot handle the overlap case very well. For example, in the \"pick place\" video in Figure 4, when the green cone is picked up and moved towards, it should have occluded the yellow cone at certain steps. However, it is occluded **by** the yellow one instead in the generated video. But since the masks \n$ M_{t-1}, M_t $ are given individually, I personally think it should not be the case: since there is only one object moving (i.e. green cone), you can warp that specific mask and simply stick that on top of the original frame. Can the authors explain why does it fail?\n\n-- The results on Something-Something indicate that flow-warping method might not be a good way to preserve the structure of the object/hand. I think the authors could spend some space on this limitation and discuss possible ways for improvement.", "extra_scores": {}, "extra_text": {}}, {"review_id": "TBsQHrJEGcw", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a generative method (AG2Vid) that generates video conditioned by the first frame, first layout and an action graph. An action graph is defined such that nodes represent objects in the scene and edges represent actions. To capture the temporal dynamics, each pairwise connection is enriched with a time interval to indicate the temporal segment when the action happens. For each time step, the method consists of several stages: First, it creates the layout corresponding to the current time step based on the current graph and previous layout. Then it extracts the optical flow based on the last two layouts and the previous generated frame and finally, it generates the current frame (at pixel level) based on the predicted optical flow and the previous frame. Several metrics, including human evaluation, indicates that the method outperforms powerful baselines on two datasets: CATER and Something-Something v2.\n\nPro:\n- Generating video content is a difficult task and the idea of generating frames based on action graphs to more explicitly focus on the activity class is interesting and naturally integrated into the proposed architecture.\n- The method clearly outperforms the baselines and produces high-quality videos. \n- The experiments regarding the generalization to novel compositions of actions are interesting and show promising results for generating videos beyond the training domain.\n- The paper is generally well written, with clear explanations on the main aspects and a good balance between quantitative evaluation and qualitative examples.\n\nCons:\n- Since Action Genome [1] dataset provides more complex scene-graph annotations for videos in Charades, quite similar to the one required in this work (especially the contact subset of Action Genome), why do the authors choose Something-Something dataset instead of Charades? \n- From the paper, it seems to me that the subset of videos picked from Smt-Smt dataset only contains 2 objects (nodes), thus the action graph is very simple and the temporal segment covers the entire video. This aspect is only briefly mentioned in the main paper. Moreover, I think the ability of the method would be more clearly demonstrated in classes that have more than 2 objects if the extraction of the AG would be possible in that case. \n- The RNN ablation is interesting, showing the necessity of GNN processing. However, more details about the motivation and intuition behind these experiments should be added in section 5.\n\nMinor:\n- I think there is a typo in Eq (4). Shouldn’t VGG be applied also on the predicted v_t?\n- In the same manner, as the compositional experiment, it would be interesting to test the model using the same first frame from training videos, but changing the action labels from the Action Graphs (on Smt-Smt). In this way, it would be clearer that the model doesn’t use any kind of biases in the dataset. \n\nSince video generation could be a sensitive task, especially when conditioned on a set of actions, the ethical aspect of that work should be taken into consideration and discussed. \n\n[1] Action Genome: Actions as Compositions of Spatio-temporal Scene Graphs, Ji et. al, CVPR 2020\n\nI found the proposed method interesting and suitable for the video generation tasks. Moreover, both the ablation study and the quantitative evaluation show good performance, so I recommend the acceptance.\n\n########### UPDATE #########\n\nI thank the authors for their responses and for updating the paper. I think this work introduces some new and valuable ideas for generating videos conditioned by an action graph and I recommend the acceptance.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ENi1PfOPg69", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes a method to generate videos from an initial image and a graph based description of how the scene should evolve.\nThe description is called \"Action Graphs\" and consists of a nodes representing objects, along with edges that describe actions between objects that should occur over given time intervals.\nThe method for producing a video is split into 4 parts:\n1) An initial image frame and set of objects with bonding boxes are provided.\n2) The action graph is \"unrolled\", and a graph is instantiated for each time step. The time intervals for actions are interpolated into 0->1 ranges and this is stored in the state of the edges for each time step. This part is algorithmic with no training.\nThe task of generating the video is then as follows..\n3) The layout generating Function. This is a Graph Convolutional network on a graph consisting of the \"unrolled\" action graph at time t (and the graph at t-1?) and the previous positions of the objects. It's task is to predict positions of objects and their bounding boxes for the next frame.\n4) The Frame generating Function. This takes the predicted layout, and the pixels from the previously generated frame and generates the next frame pixels. This uses previous techniques including a \"flow prediction network\".\n\nReason For Score:\n\nThis is a weak reject. The action graphs are a succinct way of declaring an evolving scene with multiple objects.\nAnd the graph network used to interpolate object positions across a sequence of unrolled AGs is good. My main concern is for the utility of the methods outside the synthetic CATER dataset. And the contribution of the Action Graph to the \"something something\" dataset results is not clear.\n\nPros\n\nThe videos produced for the CATER dataset are nice, with objects in general retaining their boundary shapes and color/texture characteristics as they move. The movements themselves appear accurate and are well timed.\nMost impressive are the compositional actions \"eg. huddle and swap\" in Fig6 which show ability to produce videos of unseen group actions (that are compositions of previously seen individual actions).\n\nThe work is quite thorough. They carry out visual quality assessments and ablation experiments.\n\nIt is possible that others in the community can rally around and develop the action graph description. \nScenario/Storyboard descriptions of evolving scenes are necessary for work on scene understanding, and there aren't any clear candidates to rally around.\n\nCons\n\nWhat is the actual definition of the Action Graphs used?\nFor CATER it is stated that the target positions are provided for some objects, and elsewhere that an angle can be included for rotate.\nCan we see the \"formal\" descriptions of nodes, edge types and edge attributes used?\nIs it possible to have more than one action per object per time step? This seems possible in the novel \"something something\" examples, but I can't see how this would be implemented in the Action Graph for one object without 2 edges to the same object.\n\nIn the something-something dataset, are all the actions in this dataset single actions for one object with an edge from the single node to itself? If so, what is the contribution of the action graph?\nWhat are the object(s) in \"Putting (something) on a surface\"?\n\nThe methods employed: graph neural network and optical flow based video generation not all that original, the contributions are in the Action Graph and the combination and application of known techniques (albeit seemingly well chosen for the CATER dataset).\n\nThe utility of Action Graphs is not clear beyond synthetic data, particularly the CATER dataset which has discrete action/movement time segments with well defined actions to occur within the segments. In the case of this paper, all the object identities and actions over discrete time steps are given directly to the graph model to do \"scene graph interpolation\". Another application would need a similar ground truth dataset for training, and it is not clear one can be made from real data.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "0KS4fwOThqi", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a model for video generation which disentangles the object layout prediction, frame-by-frame, from the actual pixelwise frame generation. A so-called Action Graph (AG) is used as specification of the video to be generated, rather than a sentence. Action graphs model objects as nodes and actions as clocked edges. This way action graphs are \"clocked\" so to take into account the current progress of each action. A Graph Convolutional Network (GCN) is used to process the action graph and predict the next layout. The GCN is fed with the previous layout and the current AG. The final frame is generated by warping the previous frame and predicting an additive signal to the warped output. The network is trained similarly to a GAN.\nExperiments on two datasets are provided human and quantitative evaluation show superior results wrt to the baseline\n\nStrengths\n- well motivated approach for video generation.\n- experimental results show better performance with respect to segmentation based video generation \n\n\nWeaknesses\n\n- one main limitation of the method is the lack of a specific procedure to obtain action graphs. This reflects both at training time and at inference time but it is more critical at training time. The level of detail of annotation required is high and fine grained. How would this work to obtain action annotations for a dataset such as Kinetics, AVA or EPIC-KITCHENS?\n-unclear use of GAN training. GAN-like training is exploited but it is not clear why this is necessary. Moreover, apparently there is  no noise injection allowing to generate multiple videos from the same action graph. What would happen if the loss in Eq (1) is dropped in favor of only the perceptual losses? Is the GAN loss required to impose \"veracity\" on the layout generation? Given that GT layout are available why not adding some layout consistency loss? This could be done by optimizing the IoUs of objects [a].\n- How do you manage similar AG corresponding to different output frames? Is the GAN loss used for this? What happens if you remove it?\n\n\nNovelty and related work.\n\nWhy Nawhal et al., ECCV 2020 is not discussed in the related work section? The proposed approach seems largely inspired by it and HOI-GAN is even used as a baseline, a fair writing of the related work and introduction should mention and discuss in detail how this work improves over it (are the timed edges the main addition?).\nThe concept of action progress has been used before with a very similar definition to the one used here.\n\n\n\n\nRegarding the model:\nLayout are sets of bounding box coordinages + \"features\". The model demands to the \"feature\" to encode all information which is not included in the box coordinates such as pose, lighting, texture, color etc.\nSome of these variables could be modelled explicitely such as 3D or 2D pose. Why not encoding the object pose in the layout? The pose of the object could be modelled as a latent variable if not available as annotation and yet influence the optical flow generation.\n\nRegarding the experiments:\nHOI-GAN, Nawhal et al. 2020, has been tested on EPIC-KITCHENS which is a more complex real-world setting wrt to Something-Something and CATER can AG2Vid be applied in such context? if not why?\n\nA major issue of this work lies in the novelty with respect to HOI-GAN, which is used as a baseline but not discussed in the related work section, plus the comments above make the paper a nice contribution but just marginally above the threshold.\n\nReferences\n\n[a] PolarMask: Single Shot Instance Segmentation with Polar Representation, 2020\n[b] UnitBox: An Advanced Object Detection Network,2016\n[c]  Am I Done? Predicting Action Progress in Videos, 2017\n[d] Temporal Cycle-Consistency Learning, 2019\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper focuses on the task of conditional video synthesis starting from a single image. The authors propose an *Action Graph* to model the configuration of objects, their interactions, and actions. They show promising results on two benchmark datasets (one synthetic and another realistic). 

Based on the reviewers' comments and the limited discussion that ensued, it seems that some concerns in the paper were addressed by the authors; however, a main concern persists, namely the applicability of this Action Graph representation to more complex realistic videos (*e.g.* for datasets such as Kinetics and AVA). The authors do mention a manner in which an automated extraction of Action Graphs can be done, specifically with off-the-shelf (spatial or spatiotemporal) detectors for actions, objects, and object-object interactions. However, these are complicated tasks in their own right and still open problems in the field. Given that the Action Graph computable from this automated pipeline will undoubtedly contain noise (compounded by the errors of each component of this pipeline), the paper could have made a stronger argument for its contributions in realistic video, if for example an ablation study was done where *noisy* action graphs were used in training. Without more evidence that this representation will be applicable to more realistic scenarios of interest, it is difficult to gauge the impact it will have on the community. Despite its merits and promising initial results, the authors are encouraged to address this persisting concern and the other reviewers' comments to produce a stronger submission in the future. 
Decision: Reject