MetaPix: Few-Shot Video Retargeting

We address the task of retargeting of human actions from one video to another . We consider the challenging setting where only a few frames of the target is available . The core of our approach is a conditional generative model that can transcode input skeletal poses ( automatically extracted with an off-the-shelf pose estimator ) to output target frames . However , it is challenging to build a universal transcoder because humans can appear wildly different due to clothing and background scene geometry . Instead , we learn to adapt – or personalize – a universal generator to the particular human and background in the target . To do so , we make use of meta-learning to discover effective strategies for on-the-fly personalization . One significant benefit of meta-learning is that the personalized transcoder naturally enforces temporal coherence across its generated frames ; all frames contain consistent clothing and background geometry of the target . We experiment on in-the-wild internet videos and images and show our approach improves over widely-used baselines for the task . 1 INTRODUCTION . One of the hallmarks of human intelligence is the ability to imagine . For example , given an image of a never-before-seen person , one can easily imagine them performing different actions . To do so , we make use of years of experience watching humans act and interact with the world . We implicitly encode the rules of physical transformations of humans , objects , clothing and so on . Crucially , we effortlessly adapt or retarget those universal rules to a specific human and environment - a child on a playground will likely move differently than an adult walking into work . Our goal in this work is to develop models that similarly learn to generate human motions by specializing universal knowledge to a particular target human and target environment , given only a few samples of the target . It is attractive to tackle such video generation tasks using the framework of generative ( adversarial ) neural networks ( GANs ) . Past work has cast the core computational problem as one of conditional image generation where input source poses ( automatically extracted with an off-the-shelf pose estimator ) are transcoded into image frames ( Balakrishnan et al. , 2018 ; Siarohin et al. , 2018 ; Ma et al. , 2017 ) . However , it is notoriously challenging to build generative models that are capable of synthesizing diverse , in-the-wild imagery . Notable exceptions make use of massively-large networks trained on large-scale compute infrastructure ( Brock et al. , 2019 ) . However , modestly-sized generative networks perform quite well at synthesis of targeted domains ( such as faces ( Bansal et al. , 2018 ) or facades ( Isola et al. , 2017 ) ) . A particularly successful approach to generating from pose-to-image is training of specialized – or personalized – models to particular scenes . These often require large-scale target datasets , such as 20 minutes of footage in a target lab setting ( Chan et al. , 2018 ) The above approaches make use of personalization as an implicit but crucial ingredient , by on-the-fly training of a generative model tuned to the particular target domain of interest . Often , personalization is operationalized by fine-tuning a generic model on the specific target frames of interest . Our key insight is recasting personalization as an explicit component of a video-retargeting engine , allowing us to make use of meta-learning to learn how best to fine-tune ( or personalize ) a generic model to a particular target domain . We demonstrate that ( meta ) learning-to-fine-tune is particularly effective in the few-shot regime , where few target frames are available . From a technical perspective , one of our contributions is extending meta-learning to GANs , which is nontrivial because both a generator and discriminator need to be adversarially fine-tuned . To that end , we propose MetaPix , a novel approach to personalization for video retargeting . Our formulation treats personalization as a few-shot learning problem , where the task is to adapt a generic generative model of human actions to a specific person given a few samples of their appearance . Our formulation is agnostic to the actual generative model used , and is compatible with both poseconditioned transfer ( Balakrishnan et al. , 2018 ) or generative ( Chan et al. , 2018 ) approaches . Taking inspiration from the recent successes of meta-learning approaches for few-shot tasks ( Nichol et al. , 2018 ; Finn et al. , 2017 ) , we propose a novel formulation by adapting the popular first-order metalearning algorithm Reptile ( Nichol et al. , 2018 ) for jointly learning initial weights for both the generator and discriminator . Hence , our model is optimized for efficient adaptation ( personalization ) , given only a few samples and on a computational budget , and obtains stronger performance compared to a model not optimized in this form . Interestingly , we find this personalized model naturally enforces strong temporal coherence in the generated frames , even though it is not explicitly optimized for that task . 2 RELATED WORK . Deep generative modeling . There has been a growing interest in using deep networks for generative modeling of visual data , particularly images . Popular techniques include Variational Auto-Encoders ( VAEs ) ( Kingma & Welling , 2014 ) and Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) . Particularly , GAN based techniques have shown strong performance for various tasks such as conditional image generation ( Brock et al. , 2019 ) , image-to-image translation ( Isola et al. , 2017 ; Wang et al. , 2018b ; Zhu et al. , 2017 ; Balakrishnan et al. , 2018 ) , unsupervised translation ( Zhu et al. , 2017 ) and domain adaptation ( Hoffman et al. , 2018 ) . More recently , these techniques have been extended to video tasks , such as generation ( Vondrick et al. , 2016 ) , future prediction ( Finn et al. , 2016 ) and translation ( Bansal et al. , 2018 ; Wang et al. , 2018a ) . Our work explores generative modeling from a few samples , with our main focus being the task of video translation . There has been some prior work in this direction ( Zakharov et al. , 2019 ) , though is largely limited to faces and portrait images . Motion transfer and video retargeting . This refers to the task of driving a video of a person or a cartoon character given another video ( Gleicher , 1998 ) . While there exist some unsupervised techniques ( Bansal et al. , 2018 ) to do so , most successful approaches for articulated bodies involve using pose as an intermediate supervision . Recently , there have been two broad categories of approaches that have been employed for this task : 1 ) Learning to transform an image into another , given pose as input , either in 2D ( Zhou et al. , 2019 ; Balakrishnan et al. , 2018 ; Siarohin et al. , 2018 ; Ma et al. , 2017 ) or 3D ( Liu et al. , 2018 ; Neverova et al. , 2018 ; Walker et al. , 2017 ) . And 2 ) Learning a model to directly generate images given a pose as input ( or , Pose2Im ) ( Chan et al. , 2018 ) . The former approaches tend to be more sophisticated , separately generating foreground and background pixels , and tend to perform slightly better than the latter . However , they typically learn a generic model across datasets that can transfer from a single frame , whereas the latter can learn a more holistic reconstruction by learning a specific model for a video . Our approach is complementary to such transfer approaches , and be applied on top of either , as we discuss in Section 3 . Few-shot learning . Low shot learning paradigms attempt to learn a model using very small amount of training data ( Thrun , 1996 ) , typically for visual recognition tasks . Classical approaches build generative models that share priors across the various categories ( Fei-Fei et al. , 2006 ; Salakhutdinov et al. , 2012 ) . Another category of approaches attempt to learn feature representations invariant to intra-class variations by using hallucinated data ( Hariharan & Girshick , 2017 ; Wang et al. , 2018c ) or specialized training procedures/loss functions ( Wang & Hebert , 2016a ; Bart & Ullman , 2005 ) . More recently , it has been framed as a ‘ learning-to-learn ’ or a meta-learning problem . The key idea is to directly optimize the model , for the eventual few-shot adaptation task , where the model is finetuned using a few examples ( Finn et al. , 2017 ) . Alternatively , it has also been explored in form of directly predicting classifier weights ( Bertinetto et al. , 2016 ; Wang & Hebert , 2016b ; Wang et al. , 2017 ; Misra et al. , 2017 ) . Meta Learning . The goal of metalearning is to learn models that are good at learning , similar to how humans are able to quickly and efficiently learn to do a new task . Many different approaches have been explored to that end . One direction involves learning weights through recurrent networks like LSTM ( Hochreiter et al. , 2001 ; Santoro et al. , 2016 ; Duan et al. , 2016 ) . More commonly , meta-learning has been used as a way to learn an initialization for a network , that is finetuned at test time on a new task . A popular approach in this direction is MAML ( Finn et al. , 2017 ) , where the parameters are directly optimized for the test time performance of the task it needs to adapt to . This is performed by backpropagating through the finetuning process by computing second order gradients . They and others ( Andrychowicz et al. , 2016 ) have also proposed first-order methods like FOMAML that forego the need to compute second order gradients , making it more efficient at empirically small drop in performance . However , most of these works still tend to have the requirement of SGD to be used as the task optimizer . A recently proposed meta-learning algorithm , Reptile ( Nichol et al. , 2018 ) , forgoes that constraint by proposing a much simpler first order meta learning algorithm that is compatible with any black box optimizer . 3 OUR APPROACH . We now describe MetaPix in detail . To reiterate , our goal is to learn a generic model of human motion , parameterized by θ , that can quickly and efficiently be personalized for a specific person . We define speed and efficiency requirements in terms of two parameters : computation/iterations ( T ) and the number samples required for personalization ( K ) , respectively . We now describe the base architecture , MetaPix training setup , and the implementation details . Base retargeting architecture . We build upon popular video retargeting architectures . Notably , there are two common approaches in literature:1 ) Learning a transformation from one image to another , conditioned on the pose ( Zhou et al. , 2019 ; Balakrishnan et al. , 2018 ) and 2 ) Learning a mapping from pose to RGB ( Pose2Im ) , like ( Chan et al. , 2018 ) . Both obtain strong performance and amenable to the speed and efficiency constraints we are interested in . For example in K-shot setting ( i.e . to learn a model using K frames ) , one can train the Pose2Im mapping using the K frames in the former case , or use the CK2 pairs from K frames to learn a transformation function from one of the Algorithm 1 Meta-learning for video re-targeting for the Pose2Im setup . Initialize θD , θG from pretrained weights for iteration = 1 , 2 , ... do Sample K pose image pairs from the same shot randomly Compute θ̃D , θ̃G = Pix2PixHDTK ( θD , θG ) , for K images and T iterations Update θD = θD − ( θ̃D − θD ) Update θG = θG − ( θ̃G − θG ) end for K images to another in the latter case . They are also both compatible with our MetaPix optimization discussed next . Pose2Im ( Chan et al. , 2018 ) approaches essentially build upon image-to-image translation methods ( Isola et al. , 2017 ; Wang et al. , 2018b ) , where the input is a rendering of the body joints , and the output is an RGB image . The model consists of an encoder-decoder style generator G. It is trained using a combination of perceptual reconstruction losses ( Johnson et al. , 2016 ) , implemented using an L1 penalty over VGG ( Simonyan & Zisserman , 2015 ) features and discriminator losses , where we train a separate discriminator network D that is trained to differentiate the generated images from real images . The reconstruction loss forces it to be close to the ground truth , potentially leading to blurry outputs . Adding the discriminator helps fix that , as it forces the output onto the manifold of real images . Given its strong performance , we use Pix2PixHD ( Wang et al. , 2018b ) as our base architecture for Pose2Im . For brevity , we skip a complete description of the model architecture , and refer the reader to ( Wang et al. , 2018b ) for more details . Pose Transfer ( Balakrishnan et al. , 2018 ; Zhou et al. , 2019 ) , on the other hand , takes a source image of a person and a target pose , and generates an image of the source person in that target pose . These approaches typically segment the limbs , transform their position as in the target pose , and generate the target image by combining the transformed limbs and segmented background by using a generative network like a U-Net ( Ronneberger et al. , 2015 ) . These approaches can leverage learning to move pixels instead of having to generate color and background image from a learned representation . We utilize the Posewarp method ( Balakrishnan et al. , 2018 ) as our base Pose Transfer architecture due to available implementation . MetaPix . MetaPix builds upon the base retargeting architecture by optimizing it for few-shot and fast adaptation for personalization . We achieve that by taking inspiration from the literature on few-shot learning , where meta-learning has shown promising results . We use a recently introduced first-order meta-learning technique , Reptile ( Nichol et al. , 2018 ) . As compared to the more popular technique , MAML ( Finn et al. , 2017 ) , it is more efficient as it does not compute a second gradient and is amenable to work with arbitrary optimizers as it does not need to backpropagate through the optimization process . Given that GAN architectures are hard to optimize , Reptile suits our purposes of its ability to use Adam ( Kingma & Ba , 2015 ) , the default optimizer for Pix2PixHD , as our task optimizer . Figure 2 illustrates the high level idea of our approach , which we describe in detail next . We start with either a Pose2Im or a Pose Transfer trained base model . We then finetune this model as described in Algorithm 1 . Note that Pix2PixHD is based on a GAN , so has two network weights to be optimized , the generator ( θG ) and discriminator ( θD ) . In each meta-iteration , we sample a task : in our case a set of K frames from a new video to personalize to . We then finetune the current model parameter to that video over T iterations , and update the model parameters in the direction of the personalized parameters using a meta learning rate . We optimize both θD and θG jointly at each step . Note that Posewarp employs a more complicated two-stage training procedure , and we metalearn only the first stage ( which has no discriminator ) for simplicity . Implementation Details . We implement MetaPix for the Pose2Im base model by building upon a public Pix2PixHD implementation1 in PyTorch , and perform all experiments on a 4 TITAN-X or GTX 1080Ti GPU node . We follow the hyperparameter setup as proposed in ( Wang et al. , 2018b ) . We represent the pose using a multi-channel heatmap image , and input and output are 512× 512px RGB 1https : //github.com/NVIDIA/pix2pixHD/ images . The generator consists of 16 convolutional and deconvolutional layers , and is trained with a equally weighted combination of GAN , Feature Matching , and VGG losses . Initially , we pretrain the model on a large corpus of videos to learn a generic Pose2Im model as described in Section 4 . During this pretraining stage , the model is trained on all of the training frames for 10 epochs using learning rate of 0.0002 and batch size of 8 distributed over the 4 GPUs . We experimented with multiple learning rates including 0.2 , 0.02 , 0.002 ; however , we observed that higher learning rates caused the training to diverge . When finetuning for personalization , given K frames and a computational budget T , we train the first T2 iterations using a constant learning rate of 0.0002 , and the remaining iterations using a linear decay to 0 , following ( Wang et al. , 2018b ) . The batch size is fixed to 8 , and for K < 8 , we repeat the frames to get 8 images for the batch . For the metalearning , we set the meta learning rate , = 1 with a linear decay to 0 , and train 300 meta-iterations . We also experiment with meta learning rate , = 0.1 , however , was much slower to converge . To potentially stabilize metatraining , we experiment with differing numbers of updates to the generator and discriminator during iterations of Alg . 1 , as well as simplified objective functions . Recall that the GAN loss adds significant complexity due to the presence of a discriminator that need also be adversarially finetuned . In total , our metalearning takes 1 day of training time on 4 GPUs . For the Pose Transfer base model , we apply MetaPix in a similar fashion on top of Posewarp2 , using the author provided pretrained weights . We will release the MetaPix source code for details .

Summary: This submission proposes an application of meta-learning to video frame generation modeling conditioned on human pose information, in order to allow the model to adapt to the context of each video. This context is provided in the form of a support set of K pairs of pose/frame images for the video. Reptile is used as the meta-learning method, and applied to two recently proposed video-frame generative networks (Pix2PixHD and Posewarp). In both cases, results show that Reptile is able to produce better adaptive models, i.e. models that when fine-tuned on the support set produce better image frames.