Vid2Game: Controllable Characters Extracted from Real-World Videos

1 INTRODUCTION . We propose a new video generation tool that is able to extract a character from a video , reanimate it , and generate a novel video of the modified scene , see Fig . 1 . Unlike previous work , the reanimation is controlled by a low-dimensional signal , such as the one provided by a joystick , and the model has to complete this signal to a high-dimensional full-body signal , in order to generate realistic motion sequences . In addition , our method is general enough to position the extracted character in a new background , which is possibly also dynamic . A video containing a short explanation of our method , samples of output videos , and a comparison to previous work , is provided in https : //youtu.be/sNp6HskavBE . Our work provides a general and convenient way for human users to control the dynamic development of a given video . The input is a video , which contains one or more characters . The characters are extracted , and each is associated with a sequence of displacements . In the current implementation , the motion is taken as the trajectory of the center of mass of that character in the frame . This can be readily generalized to separate different motion elements . Given a user-defined trajectory , a realistic video of the character , placed in front of an arbitrary background , is generated . The method employs two networks , applied in a sequential manner . The first is the Pose2Pose ( P2P ) network , responsible for manipulating a given pose in an autoregressive manner , based on an input stream of control signals . The second is the Pose2Frame ( P2F ) network , accountable for generating a high-resolution realistic video frame , given an input pose and a background image . Each network addresses a computational problem not previously fully met , together paving the way for the generation of video games with realistic graphics . The Pose2Pose network enables guided human-pose generation for a specific trained domain ( e.g. , a tennis player , a dancer , etc . ) , where guiding takes the form of 2D motion controls , while the Pose2Frame network allows the incorporation of a photo-realistic generated character into a desired environment . In order to enable this , the following challenges are to be addressed : ( 1 ) replacing the background requires the system to separate the character from the surroundings , which is not handled by previous work , since they either embed the character into the same learned background , or paste the generated character into the background with noticeable artifacts , ( 2 ) the separation is not binary , and some effects , such as shadows , blend the character ’ s motion effect with that background information , ( 3 ) the control signal is arbitrary , and can lead the character to poses that are not covered by the training set , and ( 4 ) generated sequences may easily drift , by accumulating small errors over time . Both the Pose2Pose and Pose2Frame networks adopt the pix2pixHD framework of Wang et al . ( 2018b ) as the generator and discriminator backbones , yet add many contributions in order to address the aforementioned challenges . As a building block , we use the pose representation provided by the DensePose framework by Rĩza Alp Güler ( 2018 ) , unmodified . Similarly , the hand-held object is extracted using the semantic segmentation method of Zhou et al . ( 2019 ) , which incorporates elements from Maninis et al . ( 2018 ) ; Law & Deng ( 2018 ) . In addition to the main application of generating a realistic video from a 2D trajectory , the learned Pose2Frame network can be used for other applications . For example , instead of predicting the pose , it can be extracted from an existing video . This allows us to compare the Pose2Frame network directly with recent video-to-video solutions . 2 RELATED WORK . Novel view synthesis is a task where unseen frames , camera views , or poses , are synthesized given a prior image . Recent approaches have also shown success in generating detailed images of human subjects in different poses ( Balakrishnan et al. , 2018 ; Kanazawa et al. , 2018 ) , where some of them also condition on pose ( Chan et al. , 2018 ; Yang et al. , 2018 ; Li et al. , 2019 ) to guide the generation . These approaches do not build a movable character model , but transfers one image to target poses . The pose variability in these images is smaller than required for our application , the handling of the background is limited , and these were also not demonstrated on video . For example , much of the literature presents results on a fashion dataset , in which the poses are limited and a white background is used . Another common benchmark is gathered from surveillance cameras , where the resolution is low , and background generation is lacking due to an inherent lack of supervision . A method for learning motion patterns by analyzing YouTube videos is demonstrated by Peng et al . ( 2018 ) , where synthetic virtual characters are set to perform complex skills in physically simulated environments , leveraging a data-driven Reinforcement Learning method that utilizes a reference motion . This method outputs a control policy that enables the character to reproduce a particular skill observed in video , which the rendered character then imitates . Unlike our method , the control signal is not provided online , one frame at a time . In addition , rendering is performed using simulated characters only , and the character in the video is not reanimated . Autoregressive models , which can be controlled one step at a time , are suitable for the dynamic nature of video games . However , such models , including RNNs , can easily drift with long range sequences ( Fragkiadaki et al. , 2015 ) , and training RNN models for long sequences suffers from vanishing or exploding gradients . Holden et al . ( 2017 ) propose a more stable model by generating the weights of a regression network at each frame as a function of the motion phase . However , this is mostly practical to apply given a limited number of keypoints , whereas dense pose models contain more information . Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) and conditional GANs ( Mirza & Osindero , 2014 ) , have been used for video synthesis by Vondrick et al . ( 2016 ) who separately generates the static background and the foreground motion . Frameworks such as vid2vid ( Wang et al. , 2018a ; Chan et al. , 2018 ) learn mappings between different videos , and demonstrate motion transfer between faces , and from poses to body . In these contributions , the reference pose is extracted from a real frame , and the methods are not challenged with generated poses . Working with generated poses , with the accompanying artifacts and the accumulated error , is considerably more challenging . In order to address this , we incorporate a few modifications , such as relying on a second input pose , in case one of the input poses is of lesser quality , and add additional loss terms to increase the realism of the generated image . In addition , these approaches model the entire frame , including both the character and the background , which usually leads to blurry results ( Pumarola et al. , 2018 ; Chao et al. , 2018 ) , particularly near the edges of the generated pose , and with complex objects , such as faces . It also leads to a loss of details from the background , and to unnatural background motion . A method for mixing the appearance of a figure seen in an image with an arbitrary pose is presented by Esser et al . ( 2018b ) . While it differs greatly in the performed task , we can compare the richness of the generated images , as shown in Fig . 2 . Their method results in a low-resolution output with noticeable artifacts , and can not model the object , while our result is indistinguishable from the source . The same is true for the follow-up work ( Esser et al. , 2018a ) . We work at a higher resolution of 1024p , while their work is limited to low-resolution characters , see Fig . 3 . Similarly , the work of Balakrishnan et al . ( 2018 ) provides lower resolution outputs , limited to the same background , and does not handle shadows ( as seen in Fig . 9-10 of that work ) . In another set of experiments , Esser et al . ( 2018a ) also present a step toward our task and show results for generating a controllable figure , building upon the phase-based neural network of Holden et al . ( 2017 ) . Their work is keypoint based and does not model environmental factors , such as shadows . The videos presented by Esser et al . ( 2018a ) for a controllable figure are displayed only on a synthetic background with a checkerboard floor pattern in an otherwise empty scene . These examples are limited to either walking or running , and the motion patterns are of an existing animation model . 3 METHOD OVERVIEW . The method ’ s objective is to learn the character ’ s motion from a video sequence , such that new videos of that character can be rendered , based on a user-provided motion sequence . The input of the training procedure is a video sequence of a character performing an action . From this video , the pose and an approximated foreground mask are extracted by the DensePose network , augmented by the semantic segmentation of the hand-held object , for each frame . The trajectory of the center of mass is taken to be the control sequence . At test time , the user provides a sequence of 2D displacements , and a video is created , in which the character moves in accordance with this control sequence . The background can be arbitrary , and is also selected by the user . The method then predicts the sequence of poses based on the given control sequence ( starting with an arbitrary pose ) , and synthesizes a video in which the character extracted from the training video is rendered in the given background . The following notation is used : a video sequence with frames fi is generated , based on a sequence of poses pi and a sequence of background images bi , where i = 1 , 2 , . . . is the frame index . The frame generation process also involves a sequence of spatial masks mi that determine which regions of the background are replaced by synthesized image information zi . To generate a video , the user provides the pose at time zero : p0 , the sequence of background images bi ( which can be static , i.e. , ∀i bi = b ) and a sequence of control signals si . In our experiments , the control signal is typically comprised of the desired 2D displacement of the animated character . Our method is an autoregressive pose model , coupled with a frame-rendering mechanism . The first aspect of our method creates a sequence of poses , and optionally of hand-held objects . Each pose and object pair [ pi , obji ] is dependent on the previous pair [ pi−1 , obji−1 ] , as well as on the current control signal si . The second aspect generates the current frame fi , based on the current background image bi , the previous combined pose and object pi−1 + obji−1 , and the current combined pose and object pi + obji . The pose and object are combined by simply summing the object channel with each of the three RGB channels that encode the pose . This rendering process includes the generation of both a raw image output zi and a blending mask mi . mi has values between 0 and 1 , with 1−mi denoting the inverted mask . Formally , the high-level processing is given by the following three equations : [ pi , obji ] = P2P ( [ pi−1 , obji−1 ] , si ) ( 1 ) ( zi , mi ) = P2F ( [ pi−1 + obji−1 , pi + obji ] ) ( 2 ) fi = zi mi + bi ( 1−mi ) ( 3 ) where P2P and P2F are the Pose2Pose and the Pose2Frame networks . As stated , P2F returns a pair of outputs that are then linearly blended with the desired background , using the per-pixel multiplication operator .

Summary: This paper proposes a method to address the interesting task, i.e. controllable human activity synthesis, by conditioning on the previous frames and the input control signal. To synthesis the next frame, a Pose2Pose network is proposed to first transfer the input information into the next frame body structure and object. Then, a Pose2Frame network is applied to generate the final result. The results on several video sequences look nice with more natural boundaries, object, and backgrounds compared to previous methods.