Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a new task of semantic action-conditioned video prediction. The purpose of this task is to generate future frames given objects and action pairs. The paper presents a new model for this task, which makes use of the capsule networks to learn hierarchical relationships between objects. The object-action relations are represented with three-way tensors. In general, I think it is an interesting paper. But I still have some concerns about the significance of the new task, the novelty of the technical contributions, and the experiments as well.
1. The proposed task emphasizes the use of semantic actions instead of low-level actions in previous video prediction work. As we have seen the success of low-level action-conditioned video prediction in the field of visual planning [Ebert et al., 2017], how significant is the proposed task in practical applications? 
[Ebert et al., 2017] Self-Supervised Visual Planning with Temporal Skip Connections.
2. The proposed model is mainly based on the previous Capsule networks. What is the most challenging part of the new task? In addition to using Capsule networks to capture three-way object-action relations, what is the key insight of the proposed model?
3. Different from the low-level action-conditioned video prediction task, the action labels are ambiguous sometimes. As shown by the case in Figure 2, the action “put pot on oven” may have multiple plausible moving trajectories over several timestamps, and may finally lead to many plausible positions of the pot on the oven. All these future trajectories and final positions are reasonable since they reflect the uncertainty of the action labels. The question is how does the proposed model cope with this uncertainty? Can the model produce diverse future frames?
4. How does the model perform under more action labels and object labels? On the kitchen data set, the authors used only six categories of objects and six categories of actions, which means that the model only needs to consider 36 types of input actions. This may not be a very challenging problem in my view.
5. It would be nice to have more discussion about the generalization ability of the model. Can the model be generalized to action descriptions with more than two objects? 

review_title: An interesting paper with limited technical contributions

final_score: rating: 5: Marginally below acceptance threshold
review_content: ###Summary###

The paper proposes the video prediction model that can handle multiple objects.
The proposed model is built from attention-based capsule network, and it generates video in a hierarchical feature computation.
Specifically, it first computes features for each concept(objects, actions), and then generate features for future image sequence corresponding to given action labels.



###Pros###

Based on the capsule network, the model learns to discern each concept without any need for labels such as bounding boxes.



###Questions###

-
The explanation of the method is insufficient.
Is the length of the action label sequence(a1:T) same with the length of the future frame sequence(x1:T)?
Could you explain more about the one-hot encoding vector? Is code for bottle and oven same? (Figure 2)
It would help in understanding if more equations are specified.
Specifying the dimension of each feature in the figure 2 also might be helpful too.

-
Since the datasets used are relatively simple, it is possible that the trained model was overfit.
As it can be seen in the figure 3, prediction results from ACGN is quite similar to the ground truth sequence.
Could you represent some results which prove that model has not just memorized the dataset?


###Minor comments###

-
There are some typos

(1) 8th line in abstract : AGCN -> ACGN

(2) title of 3.1 section : AGCN -> ACGN

(3) 3rd line in Results paragraph of 5.1 section : quantitative -> qualitative

review_title:   

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary


Pros
+ New interesting task for video synthesis
+ New capsule based neural network to learn relationships
+ Outperforms baselines in pixel based metrics and user study

Comments / Questions:
- Stopping criteria:
I may have missed it, but I cannot find what is the stopping criteria of the video generation. Is it when the generated frame stops changing? If not, what happens if you let the network generate frames infinitely, does the frame become static once the task is accomplished?

- Number of parameters:
The proposed method needs additional parameters for each of the capsule blocks feature computation. Does the concatenation match the number of parameters of the proposed method? Or is everything kept the same except for the concatenation replacing the action capsule?

- Difference between the testing and training data:
It’s obvious that for the input instructions, the network must be able to see the before to generalize, however, the object arrangement is a different story. How much does the training and testing data differ in object spatial arrangement? Is it safe to assume that the same exact test image won’t be in the training set?

- Capsule module embeds sparse graph structure:
The authors mention that they argue about sparse graph structure happening in their network. Is there a way to visualize the internal state of the network to verify this?


Conclusion:
This paper presents an interesting task that has the potential of sparking research in this direction. In addition, the authors propose an interesting architecture that generates extremely good video given some simple instructions. There are a few questions I have listed above, however, the paper to me seems interesting enough consider for acceptance.

###########################
 POST REBUTTAL DECISION
###########################

After reading other reviews and the rebuttal, I have concerns related to the stochastic trajectories mentioned by R2. The fact that the authors confirmed that all semantic actions have the same number of steps makes me question potential overfitting. I would imagine that some actions should take less steps than others based on the objects that are being interacted with, and potential multiple trajectories when performing the same semantic action. I still think the task is interesting, but the setup seems not appropriate to claim a general concept from the current method. Therefore, I have decided to lower my score.

review_title: Review

Meta-review: This paper presents work on semantic action-conditioned video prediction.  The reviewers appreciated the interesting task and use of capsule networks to address it.  Concerns were raised over generalization ability of the proposed approach, points on clarity, scalability, and handling of uncertainty/diversity by the method.  After reading the authors' response, the reviewers engaged in discussion.  Over the course of this discussion, the reviewers converged on a reject rating, noting that the concerns raised above were not sufficiently addressed to warrant publication at this stage.
Decision: Reject