Compositional Video Synthesis with Action Graphs

1 INTRODUCTION . Learning to generate visual content is a fundamental task in computer vision , with numerous applications from sim-to-real training of autonomous agents , to creating visuals for games and movies . While the quality of generating still images has leaped forward recently ( Karras et al. , 2020 ; Brock et al. , 2019 ) , generating videos is much harder . Generating actions and interactions is perhaps the most challenging aspect of conditional video generation . Actions create long-range spatio-temporal dependencies between people and the objects they interact with . For example , when a player passes a ball , the entire movement sequence of all entities ( thrower , ball , receiver ) must be coordinated and carefully timed . The current paper focuses on this difficult obstacle , the task of generating coordinated and timed actions , as an important step towards generating videos of complex scenes . Current approaches for conditional video generation are not well suited to condition the generation on actions . First , future video prediction ( Ye et al. , 2019 ; Watters et al. , 2017 ) , generates future frames based on an initial input frame , but a first frame can not be used to predict coordinated actions . Second , in video-to-video , the goal is to translate a sequence of semantic masks into an output video . However , segmentation maps contain only class information , and thus do not explicitly capture the action information . As Wang et al . ( 2018a ) notes , this is a limitation that leads to systematic mistakes , such as in the case of car turns . Finally , text-to-video ( Li et al. , 2018 ; Gupta et al. , 2018 ) is potentially useful for generating videos of actions because language can describe complex actions . However , in applications that require a precise description of the scene , language is not ideal due to ambiguities ( MacDonald et al. , 1994 ) or subjectivity of the user ( Wiebe et al. , 2004 ) . Hence , we address this problem with a more structural approach . To provide a better way to condition on actions , we introduce a formalism we call an “ Action Graph ” ( AG ) , propose a new task of “ Action Graph to Video ” ( AG2Vid ) , and present a model for this task . An AG is a graph structure aimed at representing coordinated and timed actions . Its nodes represent objects , and edges represent actions annotated with their start and end time ( Fig . 1 ) . We argue that AGs are an intuitive representation for describing timed actions and would be a natural way to provide precise inputs to generative models . A key advantage of AGs is their ability to describe the dynamics of object actions precisely in a scene . In our AG2Vid task , the input is the initial frame of the video and an AG . Instead of generating the pixels directly , our AG2Vid model uses three levels of abstraction . First , we propose an action scheduling mechanism we call “ Clocked edges ” that tracks the progress of actions in different timesteps . Second , based on this , a graph neural network ( Kipf & Welling , 2016 ) operates on the AGs and predicts a sequence of scene layouts , and finally , pixels are generated conditioned on the predicted layouts . We apply this AG2Vid model to the CATER ( Girdhar & Ramanan , 2020 ) and Something-Something ( Goyal et al. , 2017 ) datasets and show that this approach results in realistic videos that are semantically compliant with the input AG . To further demonstrate the expressiveness of AG representation and the effectiveness of the AG2Vid model , we test how it generalizes to previously unseen compositions of the learned actions . Human raters then confirm the correctness of the generated actions.1 Our contributions are as follows : 1 ) Introducing the formalism of Action Graphs ( AG ) and proposing a new video synthesis task . 2 ) Presenting a novel action-graph-to-video ( AG2Vid ) model for this task . 3 ) Using the AG and AG2Vid model , we show this approach generalizes to the generation of novel compositions of the learned actions . 2 RELATED WORK . Video generation is challenging because videos contain long range dependencies . Recent generation approaches ( Vondrick et al. , 2016 ; Kumar et al. , 2020 ; Denton & Fergus , 2018 ; Lee et al. , 2018 ; Babaeizadeh et al. , 2018 ; Villegas et al. , 2019 ) extended the framework of unconditional image generation to video , based on a latent representation . For example , MoCoGAN ( Tulyakov et al. , 2018 ) disentangles the latent space representations of motion and content to generate a sequence of frames using RNNs ; TGAN ( Saito et al. , 2017 ) generates each frame in a video separately while also having a temporal generator to model dynamics across the frames . Here , we tackle a different problem by aiming to generate videos that comply with AGs . Conditional video generation has attracted considerable interest recently , with focus on two main tasks : video prediction ( Mathieu et al. , 2015 ; Battaglia et al. , 2016 ; Walker et al. , 2016 ; Watters et al. , 2017 ; Kipf et al. , 2018 ; Ye et al. , 2019 ) and video-to-video translation ( Wang et al. , 2019 ; Chan et al. , 2019 ; Siarohin et al. , 2019 ; Kim et al. , 2019 ; Mallya et al. , 2020 ) . In prediction , the goal is to generate future video frames conditioned on few initial frames . For example , it was proposed to train predictors with GANs ( Goodfellow et al. , 2014 ) to predict future pixels ( Mathieu et al. , 2015 ) . However , directly predicting pixels is challenging ( Walker et al. , 2016 ) . Instead of pixels , researchers explored object-centric graphs and perform prediction on these ( Battaglia et al. , 2016 ; Luc et al. , 2018 ; Ye et al. , 2019 ) . While inspired by object-centric representations , our method is different from these works as our generation is goal-oriented , guided by an AG . The video-to-video translation task was proposed by Wang et al . ( 2018a ) , where a natural video was generated from frame-wise semantic segmentation annotations . However , densely labeling pixels for each frame is expensive , and might not even be necessary . Motivated by this , researchers have sought to perform generation conditioned on more accessible signals including audio or text ( Song et al. , 2018 ; Fried 1Our code and models will be released upon acceptance . et al. , 2019 ; Ginosar et al. , 2019 ) . Here , we propose to synthesize videos conditioned on a novel AG , which is easy to obtain compared to semantic segmentation and is a more structured representation compared to natural audio and text . A recent method , HOI-GAN ( HG ) ( Nawhal et al. , 2020b ) , was proposed for the generation task of a single action and object . Specifically , this work addresses the zero-shot setting , and the model is tested on action and object compositions which are first presented at test time . Our focus is on generation of multiple simultaneous actions over time , performed by multiple objects . Our approach directly addresses this challenge via the AG representation and the notion of clocked edges . Various methods have been proposed to generate videos based on input text ( Marwah et al. , 2017 ; Pan et al. , 2017 ; Li et al. , 2018 ) . Most recent methods typically used very short captions which do not contain complex descriptions of actions . For example , ( Li et al. , 2018 ) used video-caption pairs from YouTube , where typical captions are ” playing hockey ” or ” flying a kite ” . Gupta et al . ( 2018 ) proposed the Flinstones animated dataset and introduced the CRAFT model for text-to-video generation . While the CRAFT model relies on text-to-video retrieval , our approach works in an end-to-end manner and aims to accurately synthesize the given input actions . Scene Graphs ( SG ) ( Johnson et al. , 2015 ; 2018 ) are a structured representation that models scenes , where objects are nodes and relations are edges . SGs have been widely used in various tasks including image retrieval ( Johnson et al. , 2015 ; Schuster et al. , 2015 ) , relationship modeling ( Krishna et al. , 2018 ; Schroeder et al. , 2019 ; Raboh et al. , 2020 ) , SG prediction ( Xu et al. , 2017 ; Newell & Deng , 2017 ; Zellers et al. , 2018 ; Herzig et al. , 2018 ) , and image captioning ( Xu et al. , 2019 ) . Recently , SGs have been applied to image generation ( Johnson et al. , 2018 ; Deng et al. , 2018 ; Herzig et al. , 2020 ) , where the goal is to generate a natural image corresponding to the input SG . More generally , spatio-temporal graphs have been explored in the field of action recognition ( Jain et al. , 2016 ; Sun et al. , 2018 ; Wang & Gupta , 2018 ; Yan et al. , 2018 ; Girdhar et al. , 2019 ; Herzig et al. , 2019 ; Materzynska et al. , 2020 ) . For example , a space-time region graph is proposed by ( Wang & Gupta , 2018 ) where object regions are taken as nodes and a GCN ( Kipf & Welling , 2016 ) is applied to perform reasoning across objects for classifying actions . Recently , it was also shown by ( Ji et al. , 2019 ; Yi et al. , 2019 ; Girdhar & Ramanan , 2020 ) that a key obstacle in action recognition is the ability to capture the long-range dependencies and compositionality of actions . While inspired by these approaches , we focus on generating videos which is a very different challenge . Recently , Ji et al . ( 2019 ) presented Action Genome , a new video dataset annotated by SGs . This dataset includes spatio-temporal SG annotations , where for each video , few individual frames were chosen and spatially annotated by SGs . Here , we use the Something-Something V2 ( Goyal et al. , 2017 ) dataset that is larger ( 200K vs. 10K videos ) and more diverse since it includes basic human activities created by a large number of crowd workers . Finally , we propose the Action Graph representation , which we view as a temporal extension of SGs . 3 ACTION GRAPHS . Our goal in this work is to build a model for synthesizing videos that contain a specified set of actions . A key component in this effort is developing a semantic representation to describe the actions performed by different objects in the scene . Towards this end , we introduce a formalism we call Action Graph ( AG ) . In an AG , nodes correspond to objects , and edges correspond to actions that these objects participate in . Objects and actions are annotated by semantic categories , and actions are also annotated by their start and end times . More formally , an AG is a tuple ( C , A , O , E ) described as follows : • An alphabet of object categories C. Categories can be compounded and include attributes . For example “ Blue Cylinder ” or “ Large Box ” . • An alphabet of action categories A . For Example “ Slide ” and “ Rotate ” . Similarly , actions can contain attributes ( e.g. , rotation speed ) . • Object nodes O : A set O ∈ Cn of n objects . • Action edges E : Actions are represented as labeled directed edges between object nodes . Each edge is annotated with an action category and with the time period during which the action is performed . Formally , each edge is of the form ( i , a , j , ts , te ) where i , j ∈ { 1 , ... , n } are object instances , a ∈ A is an action and ts , te ∈ N are action start and end time . Thus , this edge implies that object i ( which has category oi ) performs an action a over object j , and that this action takes place between times ts and te . We note that an AG edge can directly model actions over a single object and a pair of objects . For example , “ Swap the positions of objects i and j between time 0 and 9 ” is an action over two objects corresponding to edge ( i , swap , j , 0 , 9 ) . Some actions , such as “ Rotate ” , involve only one object and will therefore be specified as self-loops .

Summary: This paper proposes a generative method (AG2Vid) that generates video conditioned by the first frame, first layout and an action graph. An action graph is defined such that nodes represent objects in the scene and edges represent actions. To capture the temporal dynamics, each pairwise connection is enriched with a time interval to indicate the temporal segment when the action happens. For each time step, the method consists of several stages: First, it creates the layout corresponding to the current time step based on the current graph and previous layout. Then it extracts the optical flow based on the last two layouts and the previous generated frame and finally, it generates the current frame (at pixel level) based on the predicted optical flow and the previous frame. Several metrics, including human evaluation, indicates that the method outperforms powerful baselines on two datasets: CATER and Something-Something v2.