Vid2Game: Controllable Characters Extracted from Real-World Videos
Authors: Oran Gafni
Lior Wolf
Yaniv Taigman
Venue:  (2020)

Abstract: We extract a controllable model from a video of a person performing a certain activity. The model generates novel image sequences of that person, according to user-defined control signals, typically marking the displacement of the moving body. The generated video can have an arbitrary background, and effectively capture both the dynamics and appearance of the person. 

The method is based on two networks. The first  maps a current pose, and a single-instance control signal to the next pose. The second maps the current pose, the new pose, and a given background, to an output frame. Both networks include multiple novelties that enable high-quality performance. This is demonstrated on multiple characters extracted from various videos of dancers and athletes.

Published as a conference paper at ICLR 2020
VID2GAME: CONTROLLABLE CHARACTERS Ex- TRACTED FROM REAL-WORDE VIDEOS
Oran Gafni Facebook Al Research e fb. com
Lior Wolf
Yaniv Taigman Facebook Al Research yaní vê fb. com
Facebook Al Research & Tel Aviv Uni. wol 1 f fb. com
ABSTRACT
We extract controllable model from video of person performing certain activity. The model generates novel image sequences of that person, according user-defined control signals, typically marking the displacement of the moving body. The generated video can have an arbitrary background, and effectively capture both the dynamics and appearance of the person.
The method is based on two networks. The first maps a current pose, and a single- instance control signal to the next pose. The second maps the current pose, the new pose, and given background, to an output frame. Both networks include multiple novelties that enable high-quality performance. This is demonstrated on multiple characters extracted from various videos of dancers and athletes.
INTRODUCTION
We propose a new video generation tool that is able to extract a character from video, reanimate it, and generate novel video of the modified scene, see Fig. []] Unlike previous work, the reanimation is controlled by low-dimensional signal, such as the one provided by joystick, and the model has to complete this signal a high-dimensional full-body signal, in order generate realistic motion sequences. In addition, our method is general enough to position the extracted character in new background, which is possibly also dynamic. video containing a short explanation of our method, samples of output videos, and comparison to previous work, is provided in https //youtu.be// No 6HskavBE
Our work provides general and convenient way for human users control the dynamic development of a given video. The input is a video, which contains one or more characters. The characters are extracted, and each is associated with sequence of displacements. In the current implementation, the motion is taken as the trajectory of the center of mass of that character the frame. This can be readily generalized to separate different motion elements. Given user-defined trajectory, realistic video of the character, placed in front of an arbitrary background, is generated.
The method employs networks, applied in sequential manner. The first is the Pose2Pose (P2P) network, responsible for manipulating given pose in an autoregressive manner. based on an input stream of control signals. The second the Pose2Frame (P2F) network, accountable for generating a high-resolution realistic video frame, given an input pose and a background image.
Each network addresses a computational problem not previously fully met, together paving the way for the generation of video games with realistic graphics. The Pose2Pose network enables guided pose generation for a specific trained domain (e.g., tennis player, a dancer, etc.), where guiding takes the form of 2D motion controls, while the 2rsmm network allows the incorporation of reato-rea generated character into a desired environment.
In order to enable this, the following challenges are to be addressed: (1) replacing the background requires the system to separate the character from the surroundings, which is not handled by previous work, since they either embed the character into the same learned background, or paste the generated character into the background with noticeable artifacts, (2) the separation is not binary, and some effects, such shadows, blend the character's 's motion effect with that background information, (3) the control signal is arbitrary, and can lead the character to poses that are not covered by the training set, and (4) generated sequences may easily drift, by accumulating small errors over time.

---

Published as a conference paper at ICLR 2020
Our method extracts a character from an uncontrolled video, and enables us to control its motion. The pose of the character, shown in the first row, is created by our Pose2Pose network in an autoregressive way, so that the motion matches the control signal illustrated by the joystick. The second row depicts the character's appearance, as generated by the Pose2Frame network, which also generates the masks shown in the third row. The final frame (last row) blends given background and the generated frames, accordance with these masks.
Figure 1:
C
(a)
(b)
(c)
(d)
(b)
(c)
(d)
Figure 2: Comparison with Esser al. (018b).
Figure Comparison with Esser et al.) (2018)).
(a) Their input, (b) their output, (c) a frame from
(a) Their input, (b) their generated output, (c) our pose input, (d) the output generated by our P2F network. In contrast to method, [Esser et al. (2018a) do not render environmental effects, resulting in unnatural blending of the character, undesired residues (e.g. source clothing), and works in lower resolution.
training video, (d) generated frame. With different objectives and dataset types, a direct comparison is not applicable. Qualitatively, FEsse eet al. (0188) output a low-res image with notice- able artifacts, and cannot model the racket, while ours is indistinguishable from the source.
Both the Pose2Pose and Pose2Frame networks adopt the pix2pixHD framework of Wang et al. (2018b) as the generator and discriminator backbones, yet add many contributions in order to address the aforementioned challenges. As building block, we use the pose representation provided by the DensePose framework by Riza Alp Güler (2018), unmodified. Similarly, the hand-held object is extracted using the semantic segmentation method of Zhou et al. (2019), which incorporates elements from Maninis et al. 2018 Law & Deng (2018).
In addition to the main application of generating realistic video from 2D trajectory, the learned Pose2Frame network can be used for other applications. For example, instead of predicting the pose, it can be extracted from existing video. This allows to compare the Pose2Frame network directly with recent video-to-video solutions.
2 RELATED WORK
Novel view synthesis is task where unseen frames, camera views, or poses, synthesized given prior image. Recent approaches have also shown success in generating detailed images of human subjects in different poses (Balakrishnan et al. 2018 Kanazawa et al. 2018), where some of them
2

---

Published as a conference paper at ICLR 2020
also condition on pose (Chan et al. 2018 Yang et al. 2018 Li et al. 2019) to guide the generation. These approaches do not build a movable character model, but transfers one image to target poses. The pose variability in these images is smaller than required for application, the handling of the background is limited, and these were also not demonstrated on video. For example, much of the literature presents results fashion dataset, in which the poses are limited and white background is used. Another common benchmark is gathered from surveillance cameras, where the resolution is low, and background generation lacking due an inherent lack of supervision.
A method for learning motion patterns by analyzing YouTube videos is demonstrated by Peng et al. (2018). where synthetic virtual characters are set perform complex skills in physically simulated environments, leveraging data duta-driven Reinforcement Learning method that utilizes reference motion. This method outputs a control policy that enables the character to reproduce a particular skill observed in video, which the rendered character then imitates. Unlike our method, the control signal is not provided online, one frame at a time. In addition, rendering is performed using simulated characters only, and the character in the video is not reanimated.
Autoregressive models, which can controlled one step at a time, are suitable for the dynamic nature of video games. However, such models, including RNNs, can easily drift with long range sequences (Fragkiadaki et al., 2015), and training RNN models for long sequences suffers from vanishing exploding gradients. Holddn al. (2017) propose a more stable model by generating the weights of a regression network at each frame as a function of the motion phase. However, this is mostly practical to apply given a limited number of keypoints, whereas dense pose models contain more information.
Generative Adversarial Networks (GANs) (Goodfellow al. 2014) and conditional GANs (Mirza & Osindero 2014), have been used for video synthesis by Vondrick et al. (2016) who separately generates the static background and the foreground motion. Frameworks such as vid2vid (Wang et al. 2018a Chan et al. 2018) learn mappings between different videos, and demonstrate motion transfer between faces, and from poses body. In these contributions, the reference pose is extracted from a real frame, and the methods are not challenged with generated poses. Working with generated poses, with the accompanying artifacts and the accumulated error, is considerably more challenging. In order to address this, we incorporate few modifications, such relying on second input pose, in case one of the input poses is of lesser quality, and add additional loss terms to increase the realism of the generated image. In addition, these approaches model the entire frame, including both the character and the background, which usually leads to blurry results (Pumarola et al., 2018 Chao et al. 2018 8), particularly near the edges of the generated pose, and with complex objects, such as faces. It also leads to a loss of details from the background, and to unnatural background motion.
A method for mixing the appearance of figure seen in image with arbitrary pose is presented by |Esserr et al.] (2018b). While it differs greatly in the performed task, we can compare the richness of the generated images, as shown in Fig. 2 Their method results in low-resolution output with noticeable artifacts, and cannot model the object, while our result is indistinguishable from the source. The same true for the follow-up work (Esser et al. (018a). We work a higher resolution of 1024p, while their work is limited to low-resolution characters, see Fig. [ Similarly, the work of Balakkishhan et al. 2018) provides lower resolution outputs, limited to the same hackground, and does not handle shadows (as seen in Fig. 9-10 of that work).
In another set of experiments, (Esserr al. (2018a) also present a step toward our task and show results for generating controllable figure, building upon the based neural network Holden et al. (017)). Their work keypoint based and does not model environmental factors, such as shadows. The videos presented by Esser et al. (2018a) for a controllable figure are displayed only synthetic background with a checkerboard floor pattern in an otherwise empty scene. These examples are limited to either walking running, and the motion patterns are of an existing animation model.
3 METHOD OVERVIEW
The method's 's objective is to learn the character's 's motion from a video sequence, such that new videos of that character can be rendered, based user- provideed motion sequence. The input of the training procedure is a video sequence of a character performing an action. From this video, the pose and an approximated foreground mask are extracted by the DensePose network, augmented by the semantic
3

---

Published as a conference paper at ICLR 2020
RecBlock
Cradiione Res Blnck
Cooditoee Rec Bladk
Bex Black
Blakk
k
Doves siagge Cues Blokk
Up saxple (herpett Care Blok
Coater Mos
Uisiing
Infermnc
Ruly Conaeced
Figure 4: The architecture of the Pose2Pose generator. During training. the middle n, - residual blocks are conditioned by a linear projection (FC layer) of the center-mass differences between con- secutive frames (in the x and y axes). For each concatenation of input pose and object [p;- obj;- 1.. the network generates the next consecutive pose and object [p,, obj.]. At inference time, the network generates the next pose-object pair in an autoregressive manner, conditioned on input directions.
segmentation of the hand-held object, for each frame. The trajectory of the center of mass is taken to be the control sequence. At test time, the user provides a sequence 2D displacements, and a video is created, in which the character moves in accordance with this control sequence. The background can be arbitrary, and is also selected by the user. The method then predicts the sequence of poses based the given control sequence (starting with an arbitrary pose), and synthesizes a video in which the character extracted from the training video rendered in the given background.
The following notation is used: video sequence with frames ʃa is generated, based on sequence of poses Pt and a sequence of background images 6,, where = 1, 2.... is the frame index. The frame generation process also involves sequence of spatial masks m, that determine which regions of the background are replaced by synthesized image information 2;.
To generate video, the provides the pose at time zero: Pu, the sequence of background images by (which can be static, i.e., Vi b, = b) and a sequence of control signals 8;. In our experiments, the control signal is typically comprised of the desired 2D displacement of the animated character.
Our method is an autoregressive pose model, coupled with a frame-rendering mechanism. The first aspect of our method creates sequence of poses, and optionally of hand-held objects. Each pose and object pair [pi. obji] is dependent on the previous pair 1p:-1, objt-1], 1]] as well as on the current control signal sy. The second aspect generates the current frame ft. based
the current background image b, the previous combined pose and object Pi-1 + obji-1- and the current combined pose and object p; + obj.. The pose and object are combined by simply summing the object channel with each of the three RGB channels that encode the pose. This rendering process includes the generation of both a raw image output z, and a blending mask m4. my has values between 0 and 1, with m, denoting the inverted mask.
Formally, the high-level processing is given by the following three equations:
[pt, obji] P2P obji-1), 8i)
(1)
(z,, m.) = P2F ([p obje-1,4 + obj.))
f. 2; 3 m; + b, o (1 -1-10 m,)
m1)
where P2P and P2F are the Pose2Pose and the Pose2Frame networks. As stated, P2F returns pair of outputs that are then linearly blended with the desired background, using the per-pixel multiplication operator
THE POSE2POSE NETWORK
As mentioned, the P2P network is an evolution of the pix2pixHI architecture. Although the primary use of the pix2pixHD framework in the literature for unconditioned image-to-image translation, we show how to modify it to enable conditioning scaled-dowd frame (512 pixels wide), allowing the network to focus on pose representation, rather than high-resolution image generation. Generation of a high-res output is deferred to the P2F network. This enables us to train the P2P network much more effectively, resulting in stable training process that generates natural dynamics, and leads to significantly reduced inference time (post-training).
control signal. The P2P network generates
4

---

Published as a conference paper at ICLR 2020
The generator's architecture is illustrated in Fig. The encoder is composed of convolutional layer, followed by convolutions with batch normalization loffe & Szegedy (2015) and ReLU Nair & Hinton (2010) activations. The latent space combines sequence of n, residual blocks. The decoder is composed of fractional strided convolutions with instance normalization [Ulyanov et al. (2016) and ReI.U activations, followed by single convolution terminated by Tanh activation for the generated frame output.
Recall that the P2P network also receives the control signal as second input (Eqq. D. In our experiment, the control signal is a vector of dimension ny representing displacements along the and y axes. This signal incorporated into the network, by conditioning the center n, blocks of the latent space.
The conditioning takes place by adding to the activations of each residual block, similarly sized tensor that is obtained by linearly projecting the 2D control vector 8;-
Modified conditional block Rather than applying a conditioning block based on a traditional ResNet block, we apply a modified one that does not allow for a complete bypass of the convolutional layers. This form of conditioning increases the motion naturalness, as seen in our ablation study.
The specific details are follows. The P2P network contains a down-sampling encoder e, latent space transformation network r, and an up-sampling decoder u. The r network is conditioned on the control signal and contains n, blocks of two types: vanilla residual blocks (r), and conditioned blocks w. P2P(p, s) u(((e()) s))
T = U. W... wo
(5)
times
The architecture and implementation details of the P2P network can be found in appendix A Briefly, let x denote the activations of the previous layer, and fi (z), f2(x) be two consecutive convolutional layers. Let be 2D displacement vector, and a fully-connected layer with a number of output neurons that equals the product of the dimensions of the tensor x. The two block types take the form:
v(x) (z)) +
(6)
w(x,s) (2(f(()) + 9(8)) fi()) + g(s)
4.1 TRAINING THE POSE PREDICTION NETWORK
Following Wang al. 2018b), employ two discriminators (low-res and high-res), indexed by = 1, During training. the LSGAN (Mao et al., (017) loss is applied to the generator and discriminator. An LI feature-matching loss is applied over the discriminators' activations, and trained VGG (Simonyan & Zisserman 20145) network. The loss applied to the generator can then be formulated as:
2
Lppp
CLS + *DDFMM)) + +AvcoLFMvee
(8)
A=1
where the networks are trained with Ao = AvGo = 10. The LSGAN generator loss is (the objs elements are omitted for brevity):
Crss 1..) (D PPP(0--11PPPPp111.1111 11)²
(9)
The expectation is computed per mini-batch, over the input pose P;-1 and the associated 8;. The eegriminnttorfeatuur matching-loss compares the predicted pose with that of the generated pose, using the activations of the discriminator, and is calculated as:
M
LFMB = E(p..
DID p.)) DC (pi-1, P2P(p:11,
(10)
jml
with M being the number of layers, N, the number of elements in each layer, Pi-1 the input (previous) pose, Pt the current (real) pose, P2P(p:-1, the estimated pose, and D( the activations of discriminator à in layer j.
The VGG feature-matching loss is calculated similarly, acting as a perceptual loss over trained VGG classifier:
LFMvoo Σ
1 -ccc
(11)
5

---

Published as a conference paper at ICLR 2020
with N being the number of elements in the j-th layer, and VGGU) the VGG classifier activations at the j-th layer. The loss applied to the discriminator is formulated as:
1 D 2 E6 (.)))))))) P2PP + 2P (D((DiL))))) m)²] (12)
The training sequences are first processed by employing the DensePose network, in order to extract the pose information from each frame. This pose information takes the form of an RGB image, where the 2D RGB intensity levels are a projection of the 3D (I)UV mapping.
By applying a binary threshold over the DensePose RGB image, we are able to create a binary mask for the character in the video. From the binary mask t; of each frame i, we compute the center of mass of the character p.. The control signal during training is denoted as s; = pi Pi-1-
Due to the temporal smoothness in the videos, the difference between consecutive frames in the full frame-rate videos (30fps) is too small to observe significant motion. This results in learned networks that are biased towards motionless poses. Hence, train with A = 2 inter-frame intervals (where A = describes using consecutive frames). During inference, we sample at 30fps and apply directional conditioning signal that has half of the average motion magnitude during training.
Stopping criteria We use the Adam optimizer Kingmaa & Ba 2016) with a learning rate of 2 10-4, 31 = 0.5 and B2- = 0.999. We observe that training the P2P network does not provide for monotonic improvement in output quality. We stipulate the P2P network final model to be that which yields the lowest loss, in terms of discriminator while training the P2P network, the discriminator feature- feature-matching loss is the only one that holds both motion context (i.e. receives both the previous and current pose). and information of different abstraction levels (i.e. Seature-matching is applied over different levels of activations). This results in improved motion naturalness, and reduced perceptual distance, as evident from the ablation study.
matching. While there are several losses applied
Random occlusions To cope with pose detection imperfections that occasionally occur, which in turn impair the quality of the generated character, we employ a dedicated data augmentation method, in order to boost the robustness of the P2P network. A black ellipse of random size and location is added to each input pose frame within the detection bounding box, resulting in an impaired pose (see appendix Fig. 8). with characteristics that are similar to "naturally" occurring imperfections.
5 THE POSE2FRAME NETWORK
While the original pix2pixHD network transforms entire image to an output image of the same size from a specified domain, our Pose2Frame network transforms a pose to a character that is localized in specific part of the output image and embedded in given, possibly dynamic, background. This is done by both refocusing the discriminators' receptive field, and applying a learned blending mask over the raw image output. The DensePose network plays a crucial role, as it provides both the relevant image region and a prior over the blending mask.
Focusing the discriminator on the character eliminates the need for feature-enhancing techniques, such as the introduction of a face-GAN, as done by Chan et al. (2018). ). or adding a temporal loss (which is useful for reducing irrelevant background motion) done by Wang al. (2018a).
The generator architecture is illustrated in Fig. aa). The P2F low-level network architecture details are somewhat similar to those of the P2P network, with the following modifications: (1) the P2F network generates frames with resolution width of 1024, (2) no conditioning is applied, i.e., the w blocks are replaced by u blocks, (3) the network generates two outputs: the raw image data 2 and a separate blending mask m, (4) the discriminators are altered to reflect the added focus, and (5) new regularization terms are added to ensure that the masking takes place at the relevant regions (Eq. see Fig. 2
in the appendix.
The generated mask m blends the raw output z with the desired background b, rendering the final output frame f. according to Eq. (omitting the index for brevity). Note that the blending mask is not binary. since various effects such as shadows, contain both character-derin background information, see Fig. Nevertheless, the background in regions external to the character, and discourage the generator from rendering meaningful representations outside the character. This is done by employing several regularization
information and softly encourage the blending mask to favor

---

Published as a conference paper at ICLR 2020
Aucidtio
so bobl
No Hlaes
Kox Bllcct
A
Dowe cample Cono flokk
coople Dimoowe Can Rook
Oepor dodiowdd Ageen
(a)
Figure The Pose2Frame network. (a) For each two combined input pose and object (p [pi-1 + obj;-1,, Pi + obj+]) the network generates an RGB image (2;) and mask (mn:). The RGB and background images are then linearly blended by the generated mask to create the output frame fi- (b) The P2F discriminator setup. The multi-scale discriminator focuses the binary-threse threshhlded character, obtained with the binary mask t, as it appears in both the ground truth image o and the output of the P2F network, for given pose p = (pi, P;-1). The & operator denotes downscaling factor of two, obtained by average pooling, as applied before the low-resolution discriminator. The VGG feature-matching loss term engages with the full frame, covering perceptual context in higher abstraction layers (e.g. generated shadows).
Figure 6: Samples of masks that model both the character, and places the scene in which appear- ance is changed by the character. (a) The shadow and the tennis racket of the character are captured by the mask, (b) the dancer's 's shadow appears as part of the mask.
(b)
terms over the generated mask. As a side effect of these added losses, the network is required to perform higher-level reasoning and not rely memorization. In other words, instead of expanding the mask to include all background changes, the network separates between character dependent changes, such shadows, held items, and reflections, and those that are independent.
The discriminator setup is illustrated in Fig. b[.). The discriminator's attention is predominantly shifted towards the character, by applying inverse binary mask over the character. The masked character image is fed into the discriminators, affecting both the multi-scale loss, and the feature- matching loss applied over the discriminators' activations. In parallel, the fully generated frame fed into the VGG network, allowing the VGG meature-matching loss to aid in the generation of desired structures external to the character.
TRAINING THE POSE TO FRAME NETWORK
The P2F generator loss is formulated
LF2F Σ (Crst + ADDrMI) + >a Dr + Xxhmaak
(13)
where A and d 1. The LSGAN generator loss calculated as:
Cr.sk = Eppt) (D
(14)
where = [p:-1 obji + obji] denotes the two pose images, and is the binary mask obtained by thresholding the DensePose image at time i. The discriminator-fcature matching-loss is calculated as:
1 N
FFMB Epp.o,t
(15)
7

---

Published as a conference paper at ICLR 2020
VOTQUET
Figure 7:
Generated frames for the controllable tennis character, blended into different backgrounds.
with M being the number of layers, Nj the number of elements in each layer, and the real (ground truth) frame. The VGG feature- feature-matching loss is calculated over the full ground truth frame, rather than the one masked by
M
LFMV
- (V)
(16)
with being the ground truth frame, N, being the number of elements the j-th layer, and, as before, VGG the VGG activations of the j-th layer.
The mask term penalizes the mask (see appendix Fig. 9 for visual illustration):
Lmask = m @ (1 f) + I|m @ (1 - f)]l + I|m, O (1 - -)]l + ||1 m 3 t/11
(17)
where m the generated mask, and my and my the mask derivatives in the and y axes respectively. The first term acts reduce the mask's activity outside the regions detected by DensePose. The mask, however, is still required to function in such regions, e.g., to render shadows. Similarly, we suppress the mask derivative outside the pose-detected region, order climinate secluded points, and other figg-ffreueecy patterns. Finally, a term is added to encourage the mask to be active in the image regions occupied by the character.
The loss applied to the two discriminators is given by:
1 2 B(rA)) (DNpot.fot)] + (poo)) 2 (Dx(ptt.ot)) 1)²]
1
Lp
(18)
The Adam optimizer is used for P2F similar to the P2P. The training progression across the epochs is visualized the appendix (Fig. 10P.
EXPERIMENTS
The method was tested on multiple video sequences, see the supplementary video (https // youtu.be kaee/sNpoaaaavEBE) The first video shows tennis player outdoors, the second video, person swiping a sword indoors, and the third, person walking. The part of the videos used for training consists of 5.5min, 3.5min, and 7.5min, respectively. In addition, for comparative purposes, we trained the P2F network on a three min video of a dancer, which was part of the evaluation done by Wang et al. (2018)).
The controllable output of the tennis player is shown in Fig. which depicts the controller signal used to drive the pose, well the generated pose pt, object objt, mask my frame 2;, and
8

---

Published as a conference paper at ICLR 2020
Dataset
Method
SSIM
LPIPS
LPIPS
LPIPS
Table 1: Comparison with HiX2pixHD (see also Fig. 14). SSIM and LPIPS (multiplied by T000) are shown for three scenarios: (1) tennis (contains dynamic elements, e-g. other players, crowd, difference in camera angle), (2) walking (different character clothing, lighting, and camera angle), (3) fenc- ing (fixed background and view).
(SqzNet) (AlexNet)
(VGG)
Tennis
ours
240=22
265+
400
4744
pix2pixHD
301=26
379#35
533+42
589432
ours 193
146
365-
3741:258
Walking pix2pixHD
254
308
485=3447
4344:03
Ours
45++
41
52::11
150+:15
Fencing
pix2pixHD 308+
5314-199 670-1168
642
output frame ft. A realistic character is generated with some artifacts (see supplementary video) around the tennis racket, for which the segmentation of the training video is only partially successful. Fig. 7 depicts additional results. in which the character is placed on diverse set of backgrounds containing considerable motion. Appendix B also present controlled walking character, and a controlled fencing character, which also appear the supplementary video.
A comparison of the P2F network with the pix2pixHI method of Wang et al. (0188) is provided in T and as a figure in appendix Fig. [4 We compare by Structural Similarity (SSIM) (Wang et al.) 2004) and Learned Perceptual Image Patch Similarity (LPIPS) Zhang et al. (2018) distance methods. The mean and standard deviation are calculated for each generated video. The LPIPS method provides perceptual distance metric. by comparing the activations of three different network architectures, VGG (Simonyan & Zisserman 2014a), AlexNet (Krizhevsky) 2014), and SqueezeNet (Iandola et al.) 2016), with additional linear layer set top of each network. For each dataset, select Test set that was not used during training. Although this test set is evaluated as the ground-truth, there is domain shift between the training and the test video: the tennis test set contains dynamic elements, such as other players, crowd, and a slight difference in camera angle; the walking test set contains different character clothing, background lighting, and camera angle. The fencing test set is more similar to the training set. As seen in appendix Fig. [44] the baseline method results in many background and character artifacts, and degradation in image and character quality, is forced to model the entire scene, rather than focus solely on the character and its shadow, as our method does. This is also apparent in the statistics reported in the table.
Another experiment dedicated to the P2F network (other methods do not employ P2P), compares it with the vid2vid method of Wang et al. (2018a). The results are reported in the supplementary video, and in appendix C Our method produces far fewer background distortions, can better handle variation in the character's s location, and has the ability to embed the character into novel backgrounds.
An ablation study is presented in appendix D showing the contribution of the various components of the system both quantitatively and qualitatively. In addition, we describe the unfavorable results obtained when replacing the autoregressive model with a concatenative model.
7 CONCLUSIONS
Generating smooth motion that combines unpredictable control, the current pose, and previous motion patterns is a challenging task. The proposed novel method employs two autoencoders: one generates autoregressive motion for specific learned style, and the other generates realistic frame for blending with a dynamic background.
Our work paves the way for new types of realistic and personalized games, which can be casually created from everyday videos. In addition, controllable characters extracted from YouTube- like videos can find their place in the virtual worlds and augmented realities. The work is still limited in various aspects, such as not allowing control over the illumination of the character, the lack of support for novel views, and not modeling the character's interaction with scene objects.
ACKNOWLEDGMENTS
The authors would like to thank Lisa Rhee, llkka Hartikainen, and Adrian Bryant for allowing us to their videos for training.

---

Published as a conference paper at ICLR 2020
REFERENCES
Guha Balakrishnan, Amy Zhao. Adrian V Dalca, Fredo Durand, and John Guttag, Synthesizing images of humans unseen poses. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. -340- 2018.
Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexci A Efros. Everybody dance now. arXiv preprint arXiv: 07371, 2018.
Patrick Chao. Alexander Li, and Gokul Swamy, Generative models for pose transfer. arXiv preprint 18X6v:800.09900 2018.
Patrick Esser, Johannes Haux, Timo Milbich, and Bjöm Ommer. Towards learning a realistic rendering of human behavior. In ECCV WORKSHOP, 2018a.
Patrick Esser, Ekaterina Sutter, and Bjöm Ommer. A variational u-net for conditional appearance and shape generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8857-8866, 20.
Katerina Fragkiadaki, Sergey Levine, and Jitendra Malik. Recurrent network models for kinematic tracking. arXiv preprint 1508: 00271, 2015.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems. 2014.
Daniel Holden, Taku Komura, and Jun Saito. functionnd neural networks for character control. ACM Trans. Graph., 36(4):42:1-42:13, July 2017. ISSN 0730-0301. doi: 10.1145/3072959, 3073663. h acmp:://ddiccc...oo..mmmmmmmmm acm. 145...........111111111.3.5.55.. 3207373377299933366 3073663
Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprint arXiv: 160vv:160.007366 2016.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint 150.00360. 2015.
Phillip Isola, Jun Zhu, Tinghui Zhou, and Alexci A Efros. Image-to-image translation with conditional adversarial networks. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Justin Johnson, Alexandre Alahi, and Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European Conference Computer Vision (ECCV), 2016.
Angjoo Kanazawa, Jason Y. Zhang, Panna Felsen, and Jitendra Malik. Learning 3d human dynamics from video. arXiv preprint 11Xiv=1812.01611, 2018.
D.P. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR, 2016.
Alex Krizhevsky, One weird trick for parallelizing convolutional neural networks. arXiv preprint 2014.
Hei Law and Jia Deng. Cornernet: Detecting objects as paired keypoints. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 734-750, 2018.
Yining Li, Chen Huang, and Chen Change Loy. Dense intrinsic appearance flow for human pose transfer. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 3693-3702, 2019.
K.K. Maninis, S. Caelles, J. Pont-Tuset, and I.. Van Gool. Deep extreme cut: From extreme points to object segmentation. In Computer Vision and Pattern Recognition (CVPR), 2018.
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In ICCV, 2017.
10

---

Published as a conference paper at ICLR 2020
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv:1411.17888 2014.
arXiv preprint
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML), pp. 807-814, 2010.
Xue Bin Peng. Angjoo Kanazawa, Jitendra Malik, Pieter Abbeel, and Sergey Levine. Sfv: Reinforce- ment learning of physical skills from videos. ACM Trans. Graph., 37(6), November 2018.
Albert Pumarola. Antonio Agudo, Alberto Sanfeliu, and Francesc Moreno-Noguer. Unsupervised person image synthesis in arbitrary poses. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
lasonas Kokkinos Riza Alp Güllrr, Natalia Neverova. Densepose: Dense human pose estimation in the wild. 2018
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014a.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014b.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint wrXiv:167.080022 2016.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. arXiv preprint arXiv: wtt.:/609...120 2016
Ting-Chun Wang, Ming-Yi Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Video-to-video synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2018a.
Ting-Chun Wang, Ming- Yu Liu, Jun- Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018b.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli, et al. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 1304)6600-112, 2004.
Ceyuan Yang, Zhe Wang, Xinge Zhu, Chen Huang, Jianping Shi, and Dahua Lin. Pose guided human video generation. arXiv preprint /807. /1/52, 2018.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as perceptual metric. In CVPR, 2018.
Xingyi Zhou, Jiacheng Zhuo, and Philipp Krähenbühl. Bottom-up object detection by grouping extreme and center points. In CVPR, 2019.
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-mmage translation using cycle-cone cycle-consistent adversarial networkss. arXiv preprint arXiv: 10593, 2017.

---

Published as a conference paper at ICLR 2020
(a)
(b)
(c)
Figure 8: The occlusion-based augmentation technique used to increase robustness during training the P2P network. Each row a single sample. (a) P:-1 with part of it occluded by random ellipse, (b) the predicted pose P. (c) the ground truth pose The generated output seems to "fill in" the missing limbs, as well as predict the next frame. In this figure and elsewhere, the colors represent the 3D UV mapping.
A ADDITIONAL POSE2POSE NETWORK ARCHITECTURE AND IMPLEMENTATION DETAILS
We follow the naming convention of (Wang et al., 2018b Zhu et al., 2017: Johnson et al. 2016). Let Ck denote a Conv-IntaanceNorm-Rer Rel.U layer with k filters, each with kernel size of 7x7, with stride of 1. Dk denotes Convolution- caavolution-Isttaneeoonn ReLU layer with & filters and stride of 2, where reflection padding is used. Vk denotes vanilla residual block with two 3x3 convolutional layers with the same number of filters both layers. Wk denotes conditioned residual block. Uk denotes a 3x3 Fractional -ettiiion--SSieeeddeeee Convolution Convolution--nstaaeeNoor layer with & filters, and a stride of 0.5.
The generator, ie.. the P2P network, can then be described C64, D128, D256, D512, D1024, V1024, W1024, W1024, W1024, W1024, W1024, W1024, W1024, V1024, U512, U256, U128, U64, C3.
The input images are scaled to a width size of 512 pixels, with the height scaled accordingly.
The discriminators two PatchGANs Isola al. 2017) with identical architecture of C64,C128,C256 C256, working at the input resolution and lower resolution, down-sampled by an average-2D-ppolling operation with kernel size of 3, and stride of
The architecture of the P2F network is similar to that of the P2P network, with the following adjustments: (i) the conditional residual blocks are replaced by non residual ones, (ii) the input of P2F has channels for and p;-1, (iii) there is an additional head generating the mask output, which uses sigmoid activation function.
B ADDITIONAL IMAGES
Fig. ddepict the random occlusion process (P2P training). in which a black ellipse of random size and location is added each input pose frame within the detection bounding box. This results in an impaired pose, with characteristics that are similar to "naturally" occurring imperfections.
The mask loss term Liassk of P2F (Sec. 5) is illustrated in Fig.
Fig. 10 depicts the progression during training of the P2F dancer model. As training progresses, the details of the dancer become sharper and the hair becomes part of the mask, despite being outside the DensePose detected area (i.e., off pixels in l).
Fig. 11 depicts controlled walking character along with the control signal and the generated poses.
12

---

Published as a conference paper at ICLR 2020
② 21, ,,..7 01-
X
Figure 9: Mask losses applied during the P2F network training. An inverse binary- threhoddree mask is used to penalize pixel intensity for the generated mask, in the regions excluding the character of interest. For the generated mask, we apply regularization over the derivatives in the and axes as well, to encourage smooth mask generation, and discourage high-frequency pattern generation.
[c]
Figure 10: Training the P2F network. (a) A sample pose, (b) the target frame, (c) the generated raw frame, the mask, and the output frame different epochs: 10, 30, 50, and 200 (final).
The fencing character shown in Fig. 12
The mask for various frames the controlled sequence is shown, well as two backgrounds: the background of the reference video, and animated background. Fig. [3 depicts an additional controlled walking character, along with the control signal and the generated poses.
Fig 14 compares visually with the baseline method of pix2pixHI (Wag et al. 2018b). As can be seen, the baseline method results in many background and character artifacts, degradation in image and character quality, as is forced to model the entire scene, rather than focus solely on the character and the environmental factors, such our method.
C COMPARISON WITH VID2VID
Fig. 15(a-c) presents a comparison with the method of Wang et al. (2018a). Shown are the target image from which the driving pose extracted, the extracted pose, the results of the baseline method, and our result. As can be seen, our method handles the background a way that creates far fewer distortions, as apply learned mask, thus background generation is not required.
The characters themselves are mostly comparable in quality, despite our choice not to add a dedicated treatment to the face. In addition, despite not applying considerable emphasis temporal consistency
13

---

Published as a conference paper at ICLR 2020
Figure 11: Synthesizing a walking character, emphasizing the control between the frames. Shown are the sequence of poses generated in an autoregressive manner, as well as the generated frames.
Figure 12: Generated frames for the controllable fencing character. Each column is a different pose.
The rows are the obtained mask, and the placement on two different backgrounds: the one obtained by applying median filter to the reference video, and one taken from a motion picture.
Figure 13: Synthesizing an additional walking character. Shown are the sequence of poses generated in an autoregressive manner, as well as the generated frames.
14

---

Published as a conference paper at ICLR 2020
Figure 14: A comparison of the P2F network with the pix2pixHI method of Wang al. (008h)). (a) Ground truth image used as the pose source, (b) our result, (c) The results of pix2pixHID. The baseline method results in many background artifacts, as generates the entire frame. The degradation in image quality is apparent as well, and that of the character in particular.
(a)
(b)
(c)
(d)
(e)
(f)
(h)
(k)
Figure 15: A comparison of the P2F network with the vid2vid method of Wang et al. (2018a). (a)
The target-pose image, (b) the pose extracted from this image, (c) the result of vid2vid, (d) our result, (e) frame from the reference video. Many artifacts apparent in the background produced by vid2vid. vid2vid also distorts the character's appearance and dimensions to better match the pose. (f-k) The same pose, displayed by two characters three different backgrounds, demonstrates advantage over vid2vid in replacing backgrounds.
during training (e.g. optical flow, temporal discriminator), our method produces videos that are as smooth. Finally, the proportions of the character in video are better maintained, while in the baseline model, the character is slightly distorted toward the driving pose.
In addition, demonstrate in Fig. 15 f-k), method has the ability replace the background.
15

---

Published as a conference paper at ICLR 2020
Network Component
SSIM
LPIPS (SqzNet) (AlexNet)
LPIPS
LPIPS (VGG)
Base Conditioning
15.0+t4
20.5+114
39.8+22
37.0+14
+ Conditioning Block
14.7#:3
15.6=17
29.8+11
30.6488
Stopping Criteria
14.0=:
14.947
28.1
29.548
+ Object Channel
14.113 13.3:16
24.9112 28.617
Table 2: Ablation study of the P2P network the tennis sequence. The results are multiplied by factor of 1000 for readability.
onMobil
onMobil
Pen
onMobil
pen
onnMobil
Opan
oonn
WO
UIR
Wo
uR
Wo a
WO
a
(b)
dd
(f)
Figure 16: P2F ablation. (a) Ours, (b) no VGG FM on the full-frame (no shadows generated), (c) no mask regularization (background artifacts), (d) input pose (no racket generation due semantic segmentation mis-detections (e) no discriminator FM (character/racket heavily distorted), (f) no mask, i.e. background fully-generated (excessive distortion background character).
Figure 17: P2P baseline method comparison. Temporal consistency of P2P generated motion (row 1) is apparent, as opposed to the baseline method (row 2), that results in temporal inconsistency.
D ABLATION STUDY
We test the effect of several novel P2P network components, both by SSIM and LPIPS. The test performed when predicting one frame into the future (in "teacher forcing" mode). The results in Tab. 2 demonstrate that our conditioning block preferable to the conventional one, and that adding the object channel is beneficial. Selecting the model based on the minimal discriminator feature-matching loss is helpful well.
qualitative ablation study for the P2F network is provided in Fig. 16 component contributes to the naturalness of the results.
As can be seen, each
To validate the need for an autoregressive motion generation, as done by the P2P network, we implemented baseline method that copies motion patterns from the training set, matching the displacement, and verified that such naive approach fails to produce natural motion. A sequence of frames from the experiment video can be seen in Fig.
16


Reviews:
[{"review_id": "rkeqvIaJor", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an approach to extract a character from a video and then maneuver that character in the plane, optionally with other backgrounds. The character is then redrawn into the background with a neural net, and all of this is done in real time.\n\nAll in all, this paper was well structured and extensively detailed wrt how it engineered this solution (and why). If I had a complaint, it would be that I did not learn anything scientifically from the paper. There isn't a tested hypothesis, but rather it's a feat of engineering to get this to work. Those are important as well for the field, and I suspect that this direction could be pushed a lot more. For example, it's not close to getting realistic spatial movement relative to the plane nor is the control that impressive wrt limbs. However, as a next-contribution, this work deserves to be seen more widely.\n\nHence, I rate it as a weak accept.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "H1gcRli3YB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method to address the interesting task, i.e. controllable human activity synthesis, by conditioning on the previous frames and the input control signal. To synthesis the next frame, a Pose2Pose network is proposed to first transfer the input information into the next frame body structure and object. Then, a Pose2Frame network is applied to generate the final result. The results on several video sequences look nice with more natural boundaries, object, and backgrounds compared to previous methods.\n\nPros:\n1. The proposed Pose2Pose successfully transfer the pose conditioned on the past pose and the input control signal. The proposed conditioned residual block, occlusion augmentation and stopping criteria seem to help the Pose2Pose network work well. Besides, the object is also considered in this network, which makes the method generalized well to the videos where human holds some rigid object.\n2. The Pose2Frame network is similar to previous works but learns to predict the soft mask to incorporate the complex background and to produce shallow. The mask term in Eq. (7) seems to work well for the foreground (body+object) and the shallow regions.\n3. The paper is easy to follow.\n\nCons:\n1. Since the method is only evaluated on several video sequences, I am not sure how the method will perform on other different scenes. Results on more scenes will make the performance more convincing. I also wonder if the video data will be released, which could be important for the following comparisons.\n2. As to the results of the Pose2Pose network, I wonder if there are some artifacts that will affect the performance of the Pose2Frame network. Then, there will be another question: how the two networks are trained? Are they trained separately or jointly? I assume the authors first train the Pose2Pose network, then use the output to train the Pose2Frame network. Otherwise, the artifacts from Pose2Pose will affect the testing performance of the Pose2Frame network.\n3. The mask term seems to work well for the shallow part. I wonder how the straightforward regression term plus the smooth term will perform for the mask. Here, the straightforward regression term means directly regress the output mask to the target densepose mask. Will the proposed mask term perform better?\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Byxe5udgcS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents  a controllable model from a video of a person performing a certain\nactivity. It generates novel image sequences of that person, according\nto user-defined control signals, typically marking the displacement of the moving\nbody. The generated video can have an arbitrary background, and effectively\ncapture both the dynamics and appearance of the person. It has two networks, Pose2Pose, and Pose2Frame. The overall pipeline makes sense; and the paper is well written.\n\nThe main problems come from the experiments, which I would ask for more things. It has two components, i.e., Pose2Pose and Pose2Frame. So how importance of each component to the whole framework? I would ask for the ablation study/additional experiments of using each component.  How about combining only Pose2Pose/ Pose2Frame  with pix2pixHD? Whether the performance can get improved?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to extract a character from a video, manually control the character, and render into the background in real time.  The rendered video can have arbitrary background and capture both the dynamics and appearance of the person. All three reviewers praises the visual quality of the synthesized video and the paper is well written with extensive details. Some concerns are raised. For example, despite an excellent engineering effort, there is few things the reader would scientifically learn from this paper. Additional ablation study on each component would also help the better understanding of the approach. Given the level of efforts, the quality of the results and the reviewers’ comments, the ACs recommend acceptance as a poster.
Decision: Accept (Poster)