Action Concept Grounding Network for Semantically-Consistent Video Generation
Authors: Wei Yu
Wenxin Chen
Animesh Garg
Venue:  (2021)
Keywords: action-conditional video prediction
self-supervised learning
counterfactual generation

Abstract: Recent works in self-supervised video prediction have mainly focused on passive forecasting and low-level action-conditional prediction, which sidesteps the problem of semantic learning. We introduce the task of semantic action-conditional video prediction, which can be regarded as an inverse problem of action recognition. The challenge of this new task primarily lies in how to effectively inform the model of semantic action information. To bridge vision and language, we utilize the idea of capsule and propose a novel video prediction model Action Concept Grounding Network (AGCN). Our method is evaluated on two newly designed synthetic datasets, CLEVR-Building-Blocks and Sapien-Kitchen, and experiments show that given different action labels, our ACGN can correctly condition on instructions and generate corresponding future frames without need of bounding boxes. We further demonstrate our trained model can make out-of-distribution predictions for concurrent actions, be quickly adapted to new object categories and exploit its learnt features for object detection. Additional visualizations can be found at https://iclr-acgn.github.io/ACGN/.

Under review as a conference paper at ICLR 2021
ACTION CONCEPT GROUNDING NETWORK FOR SEMANTICALLY--ONSISEENN VIDEO GENERATION
Anonymous authors Paper under double-blind review
ABSTRACT
Recent works in self-supervised video prediction have mainly focused on passive forecasting and low-level action-conditional prediction, which sidesteps the prob- lem of semantic learning. We introduce the task of semantic action-conditional video prediction, which can be regarded an inverse problem of action recogni- tion. The challenge of this new task primarily lies in how to effectively inform the model of semantic action information. To bridge vision and language, we utilize the idea of capsule and propose a novel video prediction model Action Concept Grounding Network (ACGN). Our method is evaluated on two newly designed synthetic datasets, FLEVR-Buildinge show that given different action labels, our ACGN can correctly condition on instructions and generate corresponding future frames without need of bounding boxes. We further demonstrate our trained model can make outto--distribution predictions for concurrent actions, be quickly adapted to new object categories and exploit its learnt features for object detection. Additional visualizations can be found at httttss://ccrraaaootet gi thub. Lo/ACGN/
and Sapien-Kitchen, and experiments
INTRODUCTION
Recently, video prediction and generation has drawn a lot of attention due to its ability to capture meaningful representations learned though self-supervision (Wang et al. (2018b); Yu al. (2019). Although modern video prediction methods have made significant progress in improving predictive accuracy, most of their applications are limited in the scenarios of passive forecasting (Villegas et al.) (2017); Wang et al. (2018a); [yyon et al. (2018), meaning models can only passively observe short period of dynamics and accordingly make a short-term extrapolation. Such settings neglect the fact that the observer can also become an active participant in the environment.
To model interactions between agent and environment, several low-level acciionccondiiion video prediction models have been proposed in the community (Oh et al. (2015); Mathieu et al. (2015); Babaeizadeh et al. 2017); Ebert et al. (2017)) In this paper, we go one step further by introducing the task of semantic ceciin-connitionall video prediction. Instead of using low-level single-entity actions such as action vectors of robot arms as done in prior works (Finn et al. (2016); Kurutach et al. (001)), our task provides semantic descriptions of actions, e.g. "Open the door". " and asks the model to imagine "What I open the door" in the form of future frames. This task requires the model to recognize the object identity, assign correct affordances to objects and envision the long-term expectation by planning a reasonable trajectory toward the target, which resembles how humans might imagine conditional futures. The ability predict correct and semantically consistent future perceptual information is indicative of conceptual grounding of actions, in a manner similar to object grounding in image based detection and generation tasks.
The challenge of semantic action information. Existing low-level counterparts usually achieve this by employing a naive concatenation (Finn et al. (2016): (Babeizzadhh et al. (2017) with action vector of each timestep. While this implementation enables model to move the desired objects, it fails to produce consistent long-term predictions in the multi-entity settings due to its limited inductive bias. To distinguish instances in the image, other related works heavily rely on pre-trained object detectors or ground- -rouhd-truth bounding boxes (Anonymous) (2021): ; Ji et al. (2020)). However, we argue that utilizing a pre- pre-trained detector actually simplifies the task since such a detector already solves the major difficulty by mapping high-dimension inputs to low-dimension groundings. Furthermore, bounding boxes cannot
video prediction primarily lies in how to inform the model of

---

Under review as a conference paper at ICLR 2021
Pick
Pick brown cylinder
green cube
purplo cubo
Initial
Pick purplo cubo
Put yollow cylindor
Pick red cylinder
video prediction. An agent asked to predict what will happen in the form of future frames if takes scries of semantic actions after observing the scene. Each column depicts alternative futures conditioned on the first outcome in its previous column.
Figure I: Semantic
effectively describe complex visual changes including rotations and occlusions. Thus, a more flexible way of representing objects and actions is required.
We present a new video prediction model, ACGN, short for Action Concept Grounding Network. ACGN leverages the idea of attention-based capsule networks (Zhang et al. (2018a)) Sa)) bridge semantic actions and video frame generation. The compositional nature of actions can be efficiently represented by the structure of capsule network in which each group of capsules encodes the spatial representation of specific entity or action. The contributions of this work are summarized as follows:
1. We introduce a task, semantic action-conditional video prediction illustrated in Fig which can be viewed as an inverse problem of action recognition, and create two new video datasets, CLEVR- Buik-- and Sapien-Kitched for evaluation.
2. We propose a novel video prediction model, Action Concept Grounding Network, in which routing between capsules is directly controlled by action labels. ACGN can successfully depict the long-term counterfactual evolution without need of bounding boxes.
We demonstrate that ACGN is capable of making out-of-distribution generation for concurrent actions. We further show that our trained model can be fine-tuned for new categories of objects with very small number of samples and exploit its learnt features for detection.
RELATED WORK
Passive video prediction: ConvLSTM (Shi (2015)) was the first deep learning model that employed hybrid of convolutional and recurrent units for video prediction, enabling it to learn spatial and temporal relationship simultaneously. which was soon followed by studies looking at a similar problem Kalchbrenner et al. (2017); Mathieu et al. (2015). Following ConvLSTM, PredRNN (Wang) al. (2017)) designed novel spatiotemporal LSTM, which allowed memory flow both vertically and horizontally. The same group further improved predictive results by rearranging spatial and temporal memory in a cascaded mechanism in PredRNN++ (Wang et al. (2018)) and by introducing 3D convolutions in E3D-LSTI (Wang al. (2018b)). The latest SOTA, CrevNet (Yu et al. (2019)), utilized the invertible architecture to reduce the memory and computation consumption significantly while preserving all information from input. All above models require multiple frames to warm up and can only make relatively short-term predictions since real-world videos are volatile. Models usually don't t have sufficient information predict the long-term future due partial observation, egomotion and randomness. Also, this setting prevents models from interacting with environment.
Action-conditional video prediction: The low-level action-conditional video prediction task, on the other hand, provides an action vector at each timestep as additional input to guide the prediction (Oh) al.) (2015): Chiappa et al. (2017)). CDNA (Finn al. (2016)) is representative of such an action- conditional video prediction model. In CDNA, the states and action vectors of the robotic manipulator are first spatially tiled and integrated into the model through concatenation. SVG (Denton & Fergus) (2018)) was initially proposed for stochastic video generation but later was extended to action- conditional version in (Babaeizadeh et al. (017)); Villegas et al. (2019); Chiappa et al. (0017)) It is worth noticing that SVG also used concatenation to incorporate action information. Such implementations are prevalent in low-level action- iisiion-conditionall video prediction because the action vector only encodes the spatial information of single entity, usually robotic manipulator (Finn) al. al. (2066) or human being. A common failure case for such models is the presence of multiple affordable entities (Kim al. (2019). scenario that our task definition and datasets focus on.
2

---

Under review as a conference paper at ICLR 2021
Put_ Coniie_oo_Ooen
Action Capsule Module
Put_on
Action labal
Action copsule veciors
Bottle
Actlon capsulo vocters N
Oven
Input
Encoder
Word copsula layer
capsula layar
Prodictor
CNN
ConvLSTM
Feoture
Copsuño vectors
Aggregation
one hot ancoding
Figure 2: The pipeline of ACGN in which the computation of action capsule module is elaborated (Better viewed color). Feature maps extracted by encoder are mapped action capsule then receives one of action labels that controls the collection word capsule vectors and outputs representations encapsulating this action. A shared predictor updates action capsule vectors with different hidden states for each action and finally all vector representations aggregated hefore sent to decoder to yield the final prediction.
word capsule vectors. Each group of
Capsule networks:
The concept of capsule initially proposed in (Sabour al. (2017)). Multiple
layers of capsule units were employed to discover the hierarchical relationships of objects through routing algorithms so that CapsNet could output more robust features. Follow-up works (Kosiorek) et al. (2019). Lenssen et al. (2018) further enhanced capsule structure to better model invariant or equivariant properties. this work, we argue that capsule networks also provide ideal mechanism to map relationships back to pixel space and can be applied in opposite way. As we already learn the whol- relationships of action labels, we will show later that we can utilize capsule networks to control the generative behaviour of model.
OUR APPROACH: ACTION CONCEPT GROUNDING NETWORKS
begin with defining the task of semantic action-conditional video prediction. Given an initial frame zo and sequence of action labels a1:7, the model is required predict the corresponding future frames Z1.T. Each action label is a pre-defined semantic description of a spatiotemporal movement spanning over multiple frames in scene like "take the yellow cup the table". So technically, one can regard this task as an inverse problem of action recognition.
3.1 ACGN: MODULES
The ACGN model is composed of 4 modules including encoder E, decoder D, action capsule module C and recurrent predictor P. The goal of model learn the following mapping:
d ((P(C((((:)))a)))))))
where and h2 are video frame, action labels and hidden states at time t. The overall architecture of our method illustrated Fig
Encoder: At each timestep I, the encoder & receives visual input xy and extracts a set of multi-scale feature maps. We employ a convolutional neural network with an architecture similar to VGG16 (Simonyan & Zisserman (2014)). In order to encode sufficient spatial information of objects, the final down-sampling operations VGG replaced with resolution resolution-preservvin preserving convolution layers.
The action capsule module C is the core module of ACGN. The function of this module to decompose each action label a into words, to learn representations for each word from extracted feature maps, and to reorganize word capsules represent semantic actions. There are two layers present: the word capsule layer and action capsule layer. After the feature maps are obtained from the image, they are fed into K. A convolutional filters, i.e. the word capsule layer, to
Action capsule module:
3

---

Under review as a conference paper at ICLR 2021
create N' word capsule vectors of dimension K. Here, V is the total number of words pre-defined in the vocabulary or dictionary of action labels. Since verbs can be interpreted as spatiotemporal changes of relationships between objects, not only capsules for nouns but also capsules for verbs, like "take' or puff, are computed from the extracted feature maps.
The next layer in action capsule module is action capsule layer. Unlike the original CapsNet (Sabour eet al. (2017) which uses dynamic routing to determine the connections between two consecutive layers, connections between word and action capsule layer are directly controlled by action labels. We don't I need to apply any
algorithms as we already know the part-whole relationships in the case of words and actions. If we consider the following iterative procedure of dynamic routing between capsules i and capsules j of the next layer.
Cj f- softmax(b)). V f a ② Ceg by e big + Ug
where Cj. Cij are coupling coefficients, y, is vector output of capsule j. رز is ju predictor vector computed linearly from capsule and o is the activation function. Our implementation is equivalent to set Cj as one-hot encoding vector determined by action labels and thus we no longer need to update by and cy iteratively. More specifically, we decompose each action label into several meaningful clauses, represent each clause as one-hot encoding and establish connections through capsule- gise multiplication for each clause. Each clause will have its own dictionary recording all predefined words or concepts and one-hot encoding vectors produced based on these dictionaries. Therefore, each action capsule outputs instantiation parameters of some clause in one of ongoing actions. is worth noticing that ACGN is allowed to have multiple concurrent actions in scene. In this case, build additional groups of action capsules to represent other actions.
The recurrent predictor P, implemented then calculate the spatiotemporal evolution for each action label respectively. We concatenate outputs of action capsules from the same action label pixel-wisely before sending them to predictor. The memory mechanism of ConvLSTM is essential for ACGN to obtain the ability to remember its previous actions and recover the occluded objects. To prevent interference between concurrent actions, hidden states are not shared between actions. The outputs of predictor for all action labels are added point-wisely along with pixel-wisely flatten word capsule vectors.
Predictor:
stack of residual ConvL.STM layers, will
Finally, the decoder 'D aggregates the updated latent representations produced by predictor and multi-scale feature maps from encoder to generate the prediction of next frame 1 The decoder is a mirrored version of the encoder with down-sampling operations replaced with spatial up-sampling and additional sigmoid output layer.
Decoder:
We train our model by minimizing the average squared error the between the ground truth frames and the predictions. At the inference phase, the model will use its previous predictions as visual inputs instead except for the first pass. Hence, training strategy called scheduled sampling (Bengio et al. (2015) is adopted to alleviate the discrepancy between training and inference.
Training:
3.2 CONNECTION BETWEEN ACTION CAPSULE AND ACTION GRAPII
The proposed ACGN is related to a concurrent work AG2Vid (Anonymous) (002)) in which the authors provide the formalism of action-graph-basee video synthesis. Action graph can be intuitively viewed as a temporal extension of scene graph, which is another appropriate structure to represent actions. In practice, nodes in both scene and action graphs correspond object bounding boxes with their object categories and actions are represented as labeled directed edges between object nodes.
We would like to argue that the action capsule module also embeds a sparse graph structure into its output. To model the spatiotemporal transition of action graph, a graph convolution neural network (Kipf & Welling (2016)) that outputs bounding box layouts of next frame is employed in AG2Vid. If we treat word capsules as nodes in graph, the routing mechanism controlled by action labels actually carries out a similar computation of A x H part of GCN where A is adjacency matrix, H are node features and x is matrix multiplication. This is because each row in the adjacency matrix of an action graph can be viewed as a concatenation of one-hot encoding vectors. In contrast to AG2Vid, our ACGN adopts three-war way tensors that preserves the spatial structure to represent the entity and actions in the scene instead of using bounding boxes and category embeddings. Also spatiotemporal transformations in our method are mainly modeled by the following recurrent predictor.
4

---

Under review as a conference paper at ICLR 2021
Rotate yellow cylinder transversely
Rotate purple eylinder transversely Put it in the front of yellow cylinder
Rotate grey cylinder nsversely
Input
Put it in the front of purple cylinder
ACGN
Concetenation Baseline
AG2VId
Input
Take pot on oven
Open refrigerator
Put pot inside refrigerator
Close refrigerator
Open refrigerator
E00
ACGN
Concatenation Baseline
AG2Vid
Figure 3: The qualitative comparison of all methods Baii-Buullin--- blockss and Sapien-Kitchen. Kitchen. The first row of each figure is the groundtruth sequence
CLEVR-Buile
4 DATASETS FOR ACTION CONDITIONAL VIDEO GENERATION
In this study, create evaluation instead of using existing datasets. This because most video datasets either don't 't come with semantic action labels (Babaeizadeh al. (2017) fail to provide necessary visual information in their first frames due to egomotions and occlusions (Hundt et al. (2018)) ). especially those inevitable occlusions brought by hands by robotic arms. Although there are several appropriate datasets like Penn Action (Zhang et al. (2013)) and KTH (Schuldt et al. (2044). they all adopt the same single-entity setting which actually indicates they can be solved by much simpler model. To tackle the above issues, we design each video in our datatsets as depiction of certain atomic action performed by an invisible agent with objects which are observable in the starting frame. Furthermore, add functions to generate bounding boxes of all objects for both datasets in order to train AG2Vid.
synthetic datesets, LLEVV-Builddig-- bblocksngg-oo-ssss and Sapien-Kitcher for
4.1 CLEVR-BUILDING BOUILDING-BLOCKS DATASET
As its name would suggest, CLEVR-Buildinge Building-blocke dataset is built upon CLEVR environment (Johnson et al. (2077).. For each video, the data generator first initializes the scene with - randomly positioned and visually different objects. There totally 32 combinations of shapes, colors and materials of objects and at most one instance of each combination is allowed to appear in video sequence. The agent can perform Pick OA. Pick and Rotate OA transversely / longitudinally, Put O on Og, Put O on the left / right side of O B, Put O in the front of behind Op. Each training sample contains video of three consecutive Pick- and Put- action pairs and sequence of semantic action labels of every frame.
of the following actions objects O and On:

---

Under review as a conference paper at ICLR 2021
4.2 KITCHEN DATASET
Compared with LLEEVR-Buldinn--e environment in the sense that: (a). It contains deformable actions like "open" and "close"; (b). The structures of different objects in the same category are highly diverse; (c). Objects can be initialized with randomly assigned relative positions like "along the wall" and "on the dishwasher". We collect totally 21 types of small movable objects in 3 categories, botile, kettle and kitchen pot, and 19 types of large openable appliances in another 3 categories, oven, refrigerator and dishwasher, from Sapien engine (Xiang et al. (2020)). The agent can perform one of the following 6 atomic actions on small object O. and large appliance Oj: Take O, on O₁ Take Og in Or. Put O₂ on Os, Put O₂ in Os Open Oj and Close O₁, Another 3 composite action sequences are defined as follows: "Take on-oont on". "Take tate_onn--e Lini-PPutiin-ossee "Open-Take Saatiukkeiinn-lle in-Close".
Sapien- KitienKittchen Dataset describes a more complicated
5
EXPERIMENTAL EVALUATION
5.1 ACTION-CONDITONAL VIDEO PREDICTION
Baselines and setup: We evaluate the proposed model on LLEVV--uilding--- blockss and Sapien- Kitchen Datasets. AG2Vid (Anonymous) (202I) is re-implemented as the baseline model because it is the most related work. Unlike our method which only needs visual input and action sequence, AG2Vid also requires bounding boxes of all objects and progress meters of actions, i.e. clock edge, for training and testing. Furthermore, with the concatenation of features and tiled action vector, which is commonly used in low-level action-condiitinal video prediction (Finn et al. (2016)) show the effectiveness of our module. To make fair comparison, the number of parameters of ACGN and its concatenation-batt variant are the same.
conduct ablation study by replacing action capsule module
Metrics: To estimate the fidelity of action-condittonal video prediction, MSE, SSIM (Wagg et al. (2004)) and LPIPS (Zhang et al. (2018b)) are calculated between the predictions and groundtruths. We also perform a human study to assess the accuracy of performing the correct action in generated videos for each model. The human judges annotate whether the model can identify the desired objects, perform actions specified by action labels and maintain the consistent visual appearances of all objects in its generations and only videos meeting all three criterions are scored as correct.
The quantitative comparisons of all methods are summarized in Table 1. The ACGN achieves the best scores on all metrics without access to additional information like bounding boxes, showing the superior performance of our action capsule module. The qualitative analysis in Fig 3 further reveals the drawbacks of other baselines. For LLEVR-Building blocks, the concatenation. based variant fails to recognize the right objects due to its limited inductive bias. Although AG2Vid has no difficulty in identifying the desired objects, assumptions made by flow warping are too strong to handle rotation and occlusion. Consequently, the adversarial loss enforces AG2Vid fix these errors by converting them to wrong poses or colors. These limitations of AG2 will be further amplified in more complicated environment, i.c. Sapien-Kitchen. The same architecture used for CLEVR can only learn to remove the moving objects from their starting positions in Sapien-Kitchen because rotation and occlusion occur more often. The concatenation baseline performs better by showing correct generation of open and close actions on large appliance. Yet, it still fails to produce long-tem term consistent predictions as the visual appearances of moving objects are altered. On the contrary, ACGN can authentically depict the correct actions specified by action labels on both datasets.
Results:
5.2 COUNTERFACTUAL GENERATION
Counterfactual generation: The most intriguing application of our ACGN is counterfactual gen- cration. More specifically, counterfactual generation means that our model will observe the same starting frame but receive different valid action labels to generate the corresponding future frames.
The visual results of counterfactual generations on each dataset are displayed in Fig As can see, model successfully identifies the desired objects, plans correct trajectories toward the target places and generates high-quality imaginations of counterfactual futures. It is also worth noticing that all displayed generations long-term generations. i.e. more than 30 frames predicted for each sequence. Our recurrent predictor plays an very important role in sustaining the spatiotemporal consistency and in reconstructing the fully-occluded objects.
Results:

---

Under review as a conference paper at ICLR 2021
Input
Rotate red cylinder transversely
Rotate blue cylinder transversely Put it in the front of redi cylinder
Pick cube Put it behind blue cylinder
red cylinder cube
Pick green cube
Put
Put behind green cube
Pick brown cube blue
Rotate yellow cylinder longitudinally right blue cube
Pick blue cube
Input
Take kettle on floor
Put kettle oven
Take kettle
Open oven
Put kettle
on oven
in oven
Open refrigerator
Put bottle in refrigerator
Open refrigerator refrigerator
Take bottle refrigerator
wall
Figure 4: Counterfactual generations CLEVR-Building blockks and Sapien-Kitchen. Kitchen. Descriptions above images indicate the action labels input. All images are predictions except for input frames.
CLEVR-Buile Buildingg-lookkk LPIPS,
Sapien-Kitchen Kitchen LPIPS./
Model
SSIM
MSEJ
Accuracy1
SSIM
MSEI
Accuracy!
Copy-First-Franee
0.962
251.38
0.1320
0.951
152.87
0.0393
Concatenation Baseline
0.961
226.53
0.1301
50.8%
0.962
23.13
0.0232
52.4%
AG2Vid Anonymous
0.956
58.67
0.0399
78.8%
0.947
270.87
0.0684
5.2%
(22TT)
ACGN
0.983
43,52
0.0303
95.2%
0.971
11.16
0.0178
%
Table 1: Quantitative evaluation of all methods on CLEVR-Buildngg- and Sapien-Kitchen. All metrics are averaged frame-wisely except for accuracy.
5.3 CONCURRENT ACTIONS, ADAPTATION AND DETECTION
We further explore other interesting features of our ACGN on Sapien-Kitchan dataset. We first demonstrate that our ACGN capable of generating video sequences depicting concurrent actions, which can be considered as out-of-distributtinn generations because our model only observes single- action videos during the training. We also try to evaluate how quickly our model can adapted to new objects. It turns out for each new object, our trained ACGN only requires few training video examples to generate decent results. Finally, to verify that our model encodes the spatial information, add SSD (Liu al. 2016)) head after the frozen encoder conduct object detection.
Concurrent actions: Concurrent actions means multiple action inputs at the same time. Generating concurrent-actioo aectinntttatinnnn videos needs to employ copied action capsules and parallel hidden states. As illustrated Fig[5) our ACGN can linearly integrate the action information the latent space and correctly portray concurrent actions in the same scene.

---

Under review as a conference paper at ICLR 2021
Action 1
Take bottle along wall
Put bottle on oven
Take bottle
Action 2
Take bottle
Open refrigerator
on floor
bottle refrigerator
Figure 5: Concurrent-actio video generations on Sapien-K datasets. Descriptions ahove and below images indicate two concurrent action labels input into our ACGN.
Take_cr on
Put in and Take in
Put on
Dispenser
Take_on on
Open, Put_in, Close, Take __
Put_on on
Safe
Figure 6: Generations on new objects, dispenser and safe. Red arrows point to new objects in images
Method
Oven
Refrigerator
Dishwasber
Bottle
Kettle
Kitchen pet
mAP
ACGN encoder + SSD
92.75
94.56
90.89
83.25
77.18
81.32
86.66
Table 2: Quantitative measures object detection on Sapien- in terms of average precision.
Adaptation: We add a new openable category "safe" and a new movable category "dispenser" into Sapien-Kitchen and generate 100 video sequences for each new object showing its interaction with other objects. Approximately. there are about 5 new sequences created for each new action pair between 2 objects. Blank word and action capsule units for new categories ACGN and we finetune it on this small new training set. Fig shows that even with a few training samples, ACGN is accurately adapted new objects and produces reasonable visual evolution. This is because, with the help of action capsules, ACGN can disentangle actions into relatively independent learning new concepts, ACGN can reuse and integrate the prior
attached trained
grounded concepts When knowledge learnt from different scenarios.
The quantitative results of object detection is provided in Table and visualization can be found in 7 in Appendix. We observe that the features learnt by ACGN can be easily transferred for detection as our video prediction task is highly location-dependent. This result indicates that utilizing bounding boxes might be little redundant for some video tasks because videos already provide rich motion information that can be used for salient object detection.
Object detection:
CONCLUSION
In this work, propose the new task of semantic action-conditional video prediction and introduce two new datasets that are meant to bridge the gap towards a robust solution to this task in complex interactive scenarios. ACGN, novel video prediction model, was also designed by utilizing the idea of capsule network to ground action concept for video generation. Our proposed model can generate alternative futures without requiring additional auxiliary data such bounding boxes, and is shown to be both quickly extendible and adaptable to novel scenarios and entities. It is our hope that our contributions will advance progress and understanding within this new task space, and that model robust enough for real-world applications (i.e. in robotic systems) in perception and control will be eventually proposed descendant of this work.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Anonymous. Compositional video synthesis with action graphs. In Submitted International Conference on Learning Representations, 2021. URL hhttps://openrevieww hetpeeiiwwwweeeeeeeeeeeee ffrum?
id=tyd9yxioxgo under review.
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H Campbell, and Sergey Levine. Stochastic variational video prediction. arXiv preprint arXiv: 1252, 2017.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Systems, pp. 1171-1179, 2015.
Wonmin Byeon, Qin Wang, Rupesh Kumar Srivastava, and Petros Koumoutsakos. Contextvp: Fully context-aware video prediction. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 753-769, 2018.
Silvia Chiappa, Sébastien Racaniere, Daan Wierstra, and Shakir Mohamed. Recurrent environment simulators. arXiv preprint wrXiv:1704.02254 2017.
Emily Denton and Rob Fergus. Stochastic video generation with learned prior. arXiv preprini arXiv: 1802.07787 2018.
Frederik Ebert, Chelsea Finn, Alex X Lee, and Sergey Levine. Self-supervised visual planning with temporal skip connections. arXiv preprini arXiv:1710.05265 2017.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. In Advances in neural information processing systems, pp. 64-72, 2016.
Andrew Hundt, Varun Jain, Chia-Hung Lin, Chris Paxton, and Gregory D Hager. The costar block stacking dataset: Learning with workspace constraints. arXiv preprint arXiv: 1810. 1714, 2018.
Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos Niebles. Action genome: Actions as compositions of spatio-temporal scene graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10236-10247, 2020.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2901-2910, 2017.
Nal Kalchbrenner, Aäron van den Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Video pixel networks. Proceedings of Machine Learning Research, 2017. URL http:htppp/prccceengggeooooo
morpsmmmrr. wrw.s/vesssv.... wwwp/////kaaccccceeerre..e. html
Yunji Kim, Seonghyeon Nam, Cho, and Seon Joo Kim. Unsupervised keypoint learning for guiding class-conditional video prediction. In Advances in Neural Information Processing Systems, pp. 38 14-24,,,4, 2019.
Thomas Kipf and Max Welling, Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv: (e09: (0099009077 2016.
Adam Kosiorek, Sara Sabour, Yee Whye Teh, and Geoffrey E Hinton. Stacked capsule autoencoders. In Advances in Neural Information Processing Systems, pp. 15512-15522, 2019.
Thanard Kurutach, Aviv Tamar, Ge Yang, Stuart J Russell, and Pieter Abbeel. Learning plannable representations with causal infogan. In Advances in Neural Information Processing Systems, pp. 8733-8744, 2018
Jan Eric Lenssen, Matthias Fey, and Pascal Libuschewski. Group equivariant capsule networks. In Advances in Neural Information Processing Systems, pp. 8844 8853, 2018.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander Berg. Ssd: Single shot multibox detector. European conference on computer vision, pp. 21-37. Springer, 2016.

---

Under review as a conference paper at ICLR 2021
Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error, arXiv preprint FrXii:1511.0544 2015.
Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L Lewis, and Satinder Singh. Action-conditional video prediction using deep networks in atari games. In Advances in neural information processing systems, pp. 2863-2871, 2015.
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. In Advances in neural information processing systems, pp. 3856-3866 2017.
Christian Schuldt, Ivan Laptev, and Barbara Caputo. Recognizing human actions: local svm approach. In Pattern Recognition, 2004. ICPR 2004. Proceedings of the 17th International Conference on, volume 3, pp. 32-36. IEEE, 2004.
Xingjian Shi, Zhourong Chen, Hao Wang. Din
Yan Yeung. Wai-Kin Wong, and Wang-chun Woo. Convolutional Istm network: A machine learning approach for precipitation nowcasting. In Advances in neural information processing systems, pp. 802-810, 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint deXiv:1409 1556, 2014.
Ruben Villegas, Jimei Yang, Seunghoon Hong. Xunyu Lin, and Honglak Lee. Decomposing motion and content for natural video sequence prediction. arXiv preprint 17Xx::7706.80033, 2017.
Ruben Villegas, Arkanath Pathak, Harini Kannan, Dumitru Erhan, Quoe V Le, and Honglak Lee. High fidelity video prediction with large stochastic recurrent neural networks. In Advances in Neural Information Processing Systems, pp. 81 81-91, 2019.
Yunbo Wang, Mingsheng Long, Jianmin Wang. Zhifeng Gao, and Yu Philip. Predrnn: Recur- rent neural networks for predictive learning using spatiotemporal Istms. In Advances in Neural Information Processing Systems, pp. 879-888, 2017.
Yunbo Wang, Zhifeng Gao, Mingsheng Long, Jianmin Wang, and Philip S Yu. Predrnn++: Towards
resolution of the deep-in-time dilemma in spatiotemporal predictive learning. arXiv preprint arXiv:1804.006300, 2018a
Yunbo Wang, Lu Jiang, Ming-Hsuan Yang, Li-Jia Li, Mingsheng Long, and Li Fei-Fei. Eidetic 3d Istm: A model for video prediction and beyond. In International Conference on Learning Representations, 2018b.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Ecro P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4)6600-612, 2004.
Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, et al. Sapien: A simulated part-based interactive environment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
11097-11107, 2020.
Wei Yu, Yichao Lu, Steve Fasterbrook, and Sanja Fidler. Efficient and information future frame prediction and beyond. In International Conference on Learning Representations, 2019.
Ningyu Zhang, Shumin Deng, Zhanlin Sun, Xi Chen, Wei Zhang, and Huajun Chen. Attention-basee capsule networks with dynamic routing for relation extraction. arXiv preprint 18X2:18122.13211, 8a.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and puttern recognition, pp. 586-595, 2018b.
Weiyu Zhang, Menglong Zhu, and Konstantinos G Derpanis. From actemes to action: A strongly- supervised representation for detailed action understanding. In Proceedings of the IEEE Interna- tional Conference on Computer Vision, pp. 2248 2188-225,,20 2013.
10

---

Under review as a conference paper at ICLR 2021
A VISUALIZATION OF OBJECT DETECTION
Figure 7: Visualization of 2D Object Detection Sapien- Kitchen.

---

Under review as a conference paper at ICLR 2021
B MORE ACTION-CONDITIONA, VIDEO PREDICTIONS OF ACGN ON BEEWR-BILDING-BLOCC DATASET
Input
Rotate grey cyllnder longitudinally
Pick yellow cube Put it the left side of grey cylinder
Rotate green cylinder transversely Put in front of yellow cube
Input
Rotate green cylinder transversely
Rotate cylinder transversely Put behind green cylinder
Pick brown cube Put it in front of green cylinder
Input
Pick purple cylinder cyan cube
Rotate yellow cylinder transversely
cyan
behind cyan cube
Input
Rotate yellow cylinder longitudinally
Pick green cube
Pick cyan cube behind green
cylinder
Input
Rotate purple cylinder longitudinally
Rotate yellow cylinder longitudinally purple cylinder
Rotate red cylinder longitudinally right purple cylinder
Figure 8: Action-conditional video prediction of ACGN CLEVR--Bilding- dataset.
12

---

Under review as a conference paper at ICLR 2021
C MORE ACTION-CONDITIONA VIDEO PREDICTIONS OF ACGN ON -APHEN-KITCHEN DATASET
Input
Take on floor
Put pot on dishwasher on dishwasher
Take
Open fridge
pot In fridge
Open fridge
Take pot fridge
fridge
1B
Input
Take pot floor
Put pot dishwasher
Take pot dishwasher
Open
Put pot
Close oven
Open oven
Take
in
in oven
01
Input
kettle
kettle fridge
kettle fridge
Open fridge
Put kettle fridge
Close fridge
Open fridge
Take kettle in fridge
11
H1
B
H1
Input
Take bottle on floor
Put bottle on dishwasher
Take bottle on dishwasher
Open fridge
Take bottle
fridge
in fridge
fridge
I
iF
10
TD
IB
Input
Take on floor
Take on oven
pot
Open fridge
Take pot in fridge
on oven
fridge
In fridge
fridge
Figure Action-conditional video prediction of ACGN on Sapien-Kitchen Kitchen dataset.
13


Reviews:
[{"review_id": "zqPDfckiTvX", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "###Summary###\n\nThe paper proposes the video prediction model that can handle multiple objects.\nThe proposed model is built from attention-based capsule network, and it generates video in a hierarchical feature computation.\nSpecifically, it first computes features for each concept(objects, actions), and then generate features for future image sequence corresponding to given action labels.\n\n\n\n###Pros###\n\nBased on the capsule network, the model learns to discern each concept without any need for labels such as bounding boxes.\n\n\n\n###Questions###\n\n-\nThe explanation of the method is insufficient.\nIs the length of the action label sequence(a1:T) same with the length of the future frame sequence(x1:T)?\nCould you explain more about the one-hot encoding vector? Is code for bottle and oven same? (Figure 2)\nIt would help in understanding if more equations are specified.\nSpecifying the dimension of each feature in the figure 2 also might be helpful too.\n\n-\nSince the datasets used are relatively simple, it is possible that the trained model was overfit.\nAs it can be seen in the figure 3, prediction results from ACGN is quite similar to the ground truth sequence.\nCould you represent some results which prove that model has not just memorized the dataset?\n\n\n###Minor comments###\n\n-\nThere are some typos\n\n(1) 8th line in abstract : AGCN -> ACGN\n\n(2) title of 3.1 section : AGCN -> ACGN\n\n(3) 3rd line in Results paragraph of 5.1 section : quantitative -> qualitative\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Y4yx6RAY67R", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new task of semantic action-conditioned video prediction. The purpose of this task is to generate future frames given objects and action pairs. The paper presents a new model for this task, which makes use of the capsule networks to learn hierarchical relationships between objects. The object-action relations are represented with three-way tensors. In general, I think it is an interesting paper. But I still have some concerns about the significance of the new task, the novelty of the technical contributions, and the experiments as well.\n1. The proposed task emphasizes the use of semantic actions instead of low-level actions in previous video prediction work. As we have seen the success of low-level action-conditioned video prediction in the field of visual planning [Ebert et al., 2017], how significant is the proposed task in practical applications? \n[Ebert et al., 2017] Self-Supervised Visual Planning with Temporal Skip Connections.\n2. The proposed model is mainly based on the previous Capsule networks. What is the most challenging part of the new task? In addition to using Capsule networks to capture three-way object-action relations, what is the key insight of the proposed model?\n3. Different from the low-level action-conditioned video prediction task, the action labels are ambiguous sometimes. As shown by the case in Figure 2, the action “put pot on oven” may have multiple plausible moving trajectories over several timestamps, and may finally lead to many plausible positions of the pot on the oven. All these future trajectories and final positions are reasonable since they reflect the uncertainty of the action labels. The question is how does the proposed model cope with this uncertainty? Can the model produce diverse future frames?\n4. How does the model perform under more action labels and object labels? On the kitchen data set, the authors used only six categories of objects and six categories of actions, which means that the model only needs to consider 36 types of input actions. This may not be a very challenging problem in my view.\n5. It would be nice to have more discussion about the generalization ability of the model. Can the model be generalized to action descriptions with more than two objects? \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Bdc-VJZv6--", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\n\nPros\n+ New interesting task for video synthesis\n+ New capsule based neural network to learn relationships\n+ Outperforms baselines in pixel based metrics and user study\n\nComments / Questions:\n- Stopping criteria:\nI may have missed it, but I cannot find what is the stopping criteria of the video generation. Is it when the generated frame stops changing? If not, what happens if you let the network generate frames infinitely, does the frame become static once the task is accomplished?\n\n- Number of parameters:\nThe proposed method needs additional parameters for each of the capsule blocks feature computation. Does the concatenation match the number of parameters of the proposed method? Or is everything kept the same except for the concatenation replacing the action capsule?\n\n- Difference between the testing and training data:\nIt’s obvious that for the input instructions, the network must be able to see the before to generalize, however, the object arrangement is a different story. How much does the training and testing data differ in object spatial arrangement? Is it safe to assume that the same exact test image won’t be in the training set?\n\n- Capsule module embeds sparse graph structure:\nThe authors mention that they argue about sparse graph structure happening in their network. Is there a way to visualize the internal state of the network to verify this?\n\n\nConclusion:\nThis paper presents an interesting task that has the potential of sparking research in this direction. In addition, the authors propose an interesting architecture that generates extremely good video given some simple instructions. There are a few questions I have listed above, however, the paper to me seems interesting enough consider for acceptance.\n\n###########################\n POST REBUTTAL DECISION\n###########################\n\nAfter reading other reviews and the rebuttal, I have concerns related to the stochastic trajectories mentioned by R2. The fact that the authors confirmed that all semantic actions have the same number of steps makes me question potential overfitting. I would imagine that some actions should take less steps than others based on the objects that are being interacted with, and potential multiple trajectories when performing the same semantic action. I still think the task is interesting, but the setup seems not appropriate to claim a general concept from the current method. Therefore, I have decided to lower my score.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents work on semantic action-conditioned video prediction.  The reviewers appreciated the interesting task and use of capsule networks to address it.  Concerns were raised over generalization ability of the proposed approach, points on clarity, scalability, and handling of uncertainty/diversity by the method.  After reading the authors' response, the reviewers engaged in discussion.  Over the course of this discussion, the reviewers converged on a reject rating, noting that the concerns raised above were not sufficiently addressed to warrant publication at this stage.
Decision: Reject