Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper presents an approach to extract a character from a video and then maneuver that character in the plane, optionally with other backgrounds. The character is then redrawn into the background with a neural net, and all of this is done in real time.

All in all, this paper was well structured and extensively detailed wrt how it engineered this solution (and why). If I had a complaint, it would be that I did not learn anything scientifically from the paper. There isn't a tested hypothesis, but rather it's a feat of engineering to get this to work. Those are important as well for the field, and I suspect that this direction could be pushed a lot more. For example, it's not close to getting realistic spatial movement relative to the plane nor is the control that impressive wrt limbs. However, as a next-contribution, this work deserves to be seen more widely.

Hence, I rate it as a weak accept.
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: This paper presents  a controllable model from a video of a person performing a certain
activity. It generates novel image sequences of that person, according
to user-defined control signals, typically marking the displacement of the moving
body. The generated video can have an arbitrary background, and effectively
capture both the dynamics and appearance of the person. It has two networks, Pose2Pose, and Pose2Frame. The overall pipeline makes sense; and the paper is well written.

The main problems come from the experiments, which I would ask for more things. It has two components, i.e., Pose2Pose and Pose2Frame. So how importance of each component to the whole framework? I would ask for the ablation study/additional experiments of using each component.  How about combining only Pose2Pose/ Pose2Frame  with pix2pixHD? Whether the performance can get improved?

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper proposes a method to address the interesting task, i.e. controllable human activity synthesis, by conditioning on the previous frames and the input control signal. To synthesis the next frame, a Pose2Pose network is proposed to first transfer the input information into the next frame body structure and object. Then, a Pose2Frame network is applied to generate the final result. The results on several video sequences look nice with more natural boundaries, object, and backgrounds compared to previous methods.

Pros:
1. The proposed Pose2Pose successfully transfer the pose conditioned on the past pose and the input control signal. The proposed conditioned residual block, occlusion augmentation and stopping criteria seem to help the Pose2Pose network work well. Besides, the object is also considered in this network, which makes the method generalized well to the videos where human holds some rigid object.
2. The Pose2Frame network is similar to previous works but learns to predict the soft mask to incorporate the complex background and to produce shallow. The mask term in Eq. (7) seems to work well for the foreground (body+object) and the shallow regions.
3. The paper is easy to follow.

Cons:
1. Since the method is only evaluated on several video sequences, I am not sure how the method will perform on other different scenes. Results on more scenes will make the performance more convincing. I also wonder if the video data will be released, which could be important for the following comparisons.
2. As to the results of the Pose2Pose network, I wonder if there are some artifacts that will affect the performance of the Pose2Frame network. Then, there will be another question: how the two networks are trained? Are they trained separately or jointly? I assume the authors first train the Pose2Pose network, then use the output to train the Pose2Frame network. Otherwise, the artifacts from Pose2Pose will affect the testing performance of the Pose2Frame network.
3. The mask term seems to work well for the shallow part. I wonder how the straightforward regression term plus the smooth term will perform for the mask. Here, the straightforward regression term means directly regress the output mask to the target densepose mask. Will the proposed mask term perform better?

review_title: Official Blind Review #2

Meta-review: This paper proposes to extract a character from a video, manually control the character, and render into the background in real time.  The rendered video can have arbitrary background and capture both the dynamics and appearance of the person. All three reviewers praises the visual quality of the synthesized video and the paper is well written with extensive details. Some concerns are raised. For example, despite an excellent engineering effort, there is few things the reader would scientifically learn from this paper. Additional ablation study on each component would also help the better understanding of the approach. Given the level of efforts, the quality of the results and the reviewers’ comments, the ACs recommend acceptance as a poster.
Decision: Accept (Poster)