Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Overview:

The paper proposes a hierarchical approach to video synthesis based on Action Graph. Action Graph is a graph representation to describe the dynamics of individual objects. Based on this, the authors proposes an action scheduling mechanism to track the progress of action and then generate the scene layout at each timestamp. Finally, the pixels are generated based on the predicted scene layout. Experiments show that such AG2Vid paradigm can generate images on CATER and Something-Something dataset with a better quality compared to the baselines. It can also generate novel actions, treated by a composition of seen actions.

Strengths:

++ The idea of three-level abstraction (action schedule + scene layout generation + pixel generation) is sound. The formulation based on the idea is neat and straightforward.

++ The experiments on generating multiple actions, single action, as well as novel actions indicate that the method is capable of disentangling and composing atomic action for individual object.


Weaknesses:

-- In the stage of pixel generation, it seems that the frame generation function cannot handle the overlap case very well. For example, in the "pick place" video in Figure 4, when the green cone is picked up and moved towards, it should have occluded the yellow cone at certain steps. However, it is occluded **by** the yellow one instead in the generated video. But since the masks 
$ M_{t-1}, M_t $ are given individually, I personally think it should not be the case: since there is only one object moving (i.e. green cone), you can warp that specific mask and simply stick that on top of the original frame. Can the authors explain why does it fail?

-- The results on Something-Something indicate that flow-warping method might not be a good way to preserve the structure of the object/hand. I think the authors could spend some space on this limitation and discuss possible ways for improvement.
review_title: Reviewer #3

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper describes a method to generate videos from an initial image and a graph based description of how the scene should evolve.
The description is called "Action Graphs" and consists of a nodes representing objects, along with edges that describe actions between objects that should occur over given time intervals.
The method for producing a video is split into 4 parts:
1) An initial image frame and set of objects with bonding boxes are provided.
2) The action graph is "unrolled", and a graph is instantiated for each time step. The time intervals for actions are interpolated into 0->1 ranges and this is stored in the state of the edges for each time step. This part is algorithmic with no training.
The task of generating the video is then as follows..
3) The layout generating Function. This is a Graph Convolutional network on a graph consisting of the "unrolled" action graph at time t (and the graph at t-1?) and the previous positions of the objects. It's task is to predict positions of objects and their bounding boxes for the next frame.
4) The Frame generating Function. This takes the predicted layout, and the pixels from the previously generated frame and generates the next frame pixels. This uses previous techniques including a "flow prediction network".

Reason For Score:

This is a weak reject. The action graphs are a succinct way of declaring an evolving scene with multiple objects.
And the graph network used to interpolate object positions across a sequence of unrolled AGs is good. My main concern is for the utility of the methods outside the synthetic CATER dataset. And the contribution of the Action Graph to the "something something" dataset results is not clear.

Pros

The videos produced for the CATER dataset are nice, with objects in general retaining their boundary shapes and color/texture characteristics as they move. The movements themselves appear accurate and are well timed.
Most impressive are the compositional actions "eg. huddle and swap" in Fig6 which show ability to produce videos of unseen group actions (that are compositions of previously seen individual actions).

The work is quite thorough. They carry out visual quality assessments and ablation experiments.

It is possible that others in the community can rally around and develop the action graph description. 
Scenario/Storyboard descriptions of evolving scenes are necessary for work on scene understanding, and there aren't any clear candidates to rally around.

Cons

What is the actual definition of the Action Graphs used?
For CATER it is stated that the target positions are provided for some objects, and elsewhere that an angle can be included for rotate.
Can we see the "formal" descriptions of nodes, edge types and edge attributes used?
Is it possible to have more than one action per object per time step? This seems possible in the novel "something something" examples, but I can't see how this would be implemented in the Action Graph for one object without 2 edges to the same object.

In the something-something dataset, are all the actions in this dataset single actions for one object with an edge from the single node to itself? If so, what is the contribution of the action graph?
What are the object(s) in "Putting (something) on a surface"?

The methods employed: graph neural network and optical flow based video generation not all that original, the contributions are in the Action Graph and the combination and application of known techniques (albeit seemingly well chosen for the CATER dataset).

The utility of Action Graphs is not clear beyond synthetic data, particularly the CATER dataset which has discrete action/movement time segments with well defined actions to occur within the segments. In the case of this paper, all the object identities and actions over discrete time steps are given directly to the graph model to do "scene graph interpolation". Another application would need a similar ground truth dataset for training, and it is not clear one can be made from real data.

review_title: Action Graphs Assist Video Generation.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a model for video generation which disentangles the object layout prediction, frame-by-frame, from the actual pixelwise frame generation. A so-called Action Graph (AG) is used as specification of the video to be generated, rather than a sentence. Action graphs model objects as nodes and actions as clocked edges. This way action graphs are "clocked" so to take into account the current progress of each action. A Graph Convolutional Network (GCN) is used to process the action graph and predict the next layout. The GCN is fed with the previous layout and the current AG. The final frame is generated by warping the previous frame and predicting an additive signal to the warped output. The network is trained similarly to a GAN.
Experiments on two datasets are provided human and quantitative evaluation show superior results wrt to the baseline

Strengths
- well motivated approach for video generation.
- experimental results show better performance with respect to segmentation based video generation 


Weaknesses

- one main limitation of the method is the lack of a specific procedure to obtain action graphs. This reflects both at training time and at inference time but it is more critical at training time. The level of detail of annotation required is high and fine grained. How would this work to obtain action annotations for a dataset such as Kinetics, AVA or EPIC-KITCHENS?
-unclear use of GAN training. GAN-like training is exploited but it is not clear why this is necessary. Moreover, apparently there is  no noise injection allowing to generate multiple videos from the same action graph. What would happen if the loss in Eq (1) is dropped in favor of only the perceptual losses? Is the GAN loss required to impose "veracity" on the layout generation? Given that GT layout are available why not adding some layout consistency loss? This could be done by optimizing the IoUs of objects [a].
- How do you manage similar AG corresponding to different output frames? Is the GAN loss used for this? What happens if you remove it?


Novelty and related work.

Why Nawhal et al., ECCV 2020 is not discussed in the related work section? The proposed approach seems largely inspired by it and HOI-GAN is even used as a baseline, a fair writing of the related work and introduction should mention and discuss in detail how this work improves over it (are the timed edges the main addition?).
The concept of action progress has been used before with a very similar definition to the one used here.




Regarding the model:
Layout are sets of bounding box coordinages + "features". The model demands to the "feature" to encode all information which is not included in the box coordinates such as pose, lighting, texture, color etc.
Some of these variables could be modelled explicitely such as 3D or 2D pose. Why not encoding the object pose in the layout? The pose of the object could be modelled as a latent variable if not available as annotation and yet influence the optical flow generation.

Regarding the experiments:
HOI-GAN, Nawhal et al. 2020, has been tested on EPIC-KITCHENS which is a more complex real-world setting wrt to Something-Something and CATER can AG2Vid be applied in such context? if not why?

A major issue of this work lies in the novelty with respect to HOI-GAN, which is used as a baseline but not discussed in the related work section, plus the comments above make the paper a nice contribution but just marginally above the threshold.

References

[a] PolarMask: Single Shot Instance Segmentation with Polar Representation, 2020
[b] UnitBox: An Advanced Object Detection Network,2016
[c]  Am I Done? Predicting Action Progress in Videos, 2017
[d] Temporal Cycle-Consistency Learning, 2019

review_title: Interesting method, some issues regarding the novelty and training settings.

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a generative method (AG2Vid) that generates video conditioned by the first frame, first layout and an action graph. An action graph is defined such that nodes represent objects in the scene and edges represent actions. To capture the temporal dynamics, each pairwise connection is enriched with a time interval to indicate the temporal segment when the action happens. For each time step, the method consists of several stages: First, it creates the layout corresponding to the current time step based on the current graph and previous layout. Then it extracts the optical flow based on the last two layouts and the previous generated frame and finally, it generates the current frame (at pixel level) based on the predicted optical flow and the previous frame. Several metrics, including human evaluation, indicates that the method outperforms powerful baselines on two datasets: CATER and Something-Something v2.

Pro:
- Generating video content is a difficult task and the idea of generating frames based on action graphs to more explicitly focus on the activity class is interesting and naturally integrated into the proposed architecture.
- The method clearly outperforms the baselines and produces high-quality videos. 
- The experiments regarding the generalization to novel compositions of actions are interesting and show promising results for generating videos beyond the training domain.
- The paper is generally well written, with clear explanations on the main aspects and a good balance between quantitative evaluation and qualitative examples.

Cons:
- Since Action Genome [1] dataset provides more complex scene-graph annotations for videos in Charades, quite similar to the one required in this work (especially the contact subset of Action Genome), why do the authors choose Something-Something dataset instead of Charades? 
- From the paper, it seems to me that the subset of videos picked from Smt-Smt dataset only contains 2 objects (nodes), thus the action graph is very simple and the temporal segment covers the entire video. This aspect is only briefly mentioned in the main paper. Moreover, I think the ability of the method would be more clearly demonstrated in classes that have more than 2 objects if the extraction of the AG would be possible in that case. 
- The RNN ablation is interesting, showing the necessity of GNN processing. However, more details about the motivation and intuition behind these experiments should be added in section 5.

Minor:
- I think there is a typo in Eq (4). Shouldn’t VGG be applied also on the predicted v_t?
- In the same manner, as the compositional experiment, it would be interesting to test the model using the same first frame from training videos, but changing the action labels from the Action Graphs (on Smt-Smt). In this way, it would be clearer that the model doesn’t use any kind of biases in the dataset. 

Since video generation could be a sensitive task, especially when conditioned on a set of actions, the ethical aspect of that work should be taken into consideration and discussed. 

[1] Action Genome: Actions as Compositions of Spatio-temporal Scene Graphs, Ji et. al, CVPR 2020

I found the proposed method interesting and suitable for the video generation tasks. Moreover, both the ablation study and the quantitative evaluation show good performance, so I recommend the acceptance.

########### UPDATE #########

I thank the authors for their responses and for updating the paper. I think this work introduces some new and valuable ideas for generating videos conditioned by an action graph and I recommend the acceptance.


review_title: Sound method for video generation based on action graphs

Meta-review: The paper focuses on the task of conditional video synthesis starting from a single image. The authors propose an *Action Graph* to model the configuration of objects, their interactions, and actions. They show promising results on two benchmark datasets (one synthetic and another realistic). 

Based on the reviewers' comments and the limited discussion that ensued, it seems that some concerns in the paper were addressed by the authors; however, a main concern persists, namely the applicability of this Action Graph representation to more complex realistic videos (*e.g.* for datasets such as Kinetics and AVA). The authors do mention a manner in which an automated extraction of Action Graphs can be done, specifically with off-the-shelf (spatial or spatiotemporal) detectors for actions, objects, and object-object interactions. However, these are complicated tasks in their own right and still open problems in the field. Given that the Action Graph computable from this automated pipeline will undoubtedly contain noise (compounded by the errors of each component of this pipeline), the paper could have made a stronger argument for its contributions in realistic video, if for example an ablation study was done where *noisy* action graphs were used in training. Without more evidence that this representation will be applicable to more realistic scenarios of interest, it is difficult to gauge the impact it will have on the community. Despite its merits and promising initial results, the authors are encouraged to address this persisting concern and the other reviewers' comments to produce a stronger submission in the future. 
Decision: Reject