MetaPix: Few-Shot Video Retargeting
Authors: Jessica Lee
Deva Ramanan
Rohit Girdhar
Venue:  (2020)
Keywords: Meta-learning
Few-shot Learning
Generative Adversarial Networks
Video Retargeting

Abstract: We address the task of unsupervised retargeting of human actions from one video to another. We consider the challenging setting where only a few frames of the target is available. The core of our approach is a conditional generative model that can transcode input skeletal poses (automatically extracted with an off-the-shelf pose estimator) to output target frames. However, it is challenging to build a universal transcoder because humans can appear wildly different due to clothing and background scene geometry. Instead, we learn to adapt – or personalize – a universal generator to the particular human and background in the target. To do so, we make use of meta-learning to discover effective strategies for on-the-fly personalization. One significant benefit of meta-learning is that the personalized transcoder naturally enforces temporal coherence across its generated frames; all frames contain consistent clothing and background geometry of the target. We experiment on in-the-wild internet videos and images and show our approach improves over widely-used baselines for the task.


Published as a conference paper at ICLR 2020
METAPIX: FEW-SHOT VIDEO RETARGETING
Jessica Lee
Deva Ramanan
Rohit Girdhar
Carnegie Mellon University
https://ing htw.aaagaaaaua.a.am.aa..aaa 1o/ Lo/MctaPix
ABSTRACT
We address the task of retargeting of human actions from one video to another. We consider the challenging setting where only few frames of the target is available. The core of our approach is a conditional generative model that can transcode input skeletal poses (automatically extracted with off-the-shelf pose estimator) out- put target frames. However, it is challenging to build universal transcoder because humans can appear wildly different due clothing and background scene geometry. Instead, we learn to adapt or personalize a universal generator to the particular human and background in the target. To do so, we make use of meta-learning to discover effective strategies for on-the-fly personalization. One significant benefit meta-learning is that the personalized transcoder naturally enforces temporal coherence across its generated frames; all frames contain consistent clothing and background geometry of the target. We experiment on in-the- wild internet videos and images and show our approach improves over widely-use baselines for the task.
INTRODUCTION
One of the hallmarks of human intelligence is the ability to imagine. For example, given an image of never-before-seen person, one can easily imagine them performing different actions. To do so, we make use of years of experience watching humans act and interact with the world. We implicitly encode the rules of physical transformations of humans, objects, clothing and so on. Crucially, we effortlessly adapt relargef those universal rules to specific human and environment child on playground will likely move differently than an adult walking into work. Our goal in this work is to develop models that similarly learn to generate human motions by specializing universal knowledge to particular target human and target environment, given only a few samples of the target.
It is attractive to tackle such video generation tasks using the framework of generative (adversarial) neural networks (GANs). Past work has cast the core computational problem as one conditional image generation where input source poses (automatically extracted with an off-the-shelf pose estimator) are transcoded into image frames (Balakrishnan et al., 2018; Siarohin et al., 2018; Ma al., 2017). However, it is notoriously challenging to build generative models that are capable of synthesizing diverse, in-the-wild imagery. Notable exceptions make use of massively-large networks trained large-scale compute infrastructure (Brock et al., 2019). However, modestly-sizee generative networks perform quite well at synthesis of targeted domains (such as faces (Bansal et al., 2018) or facades (Isola et al., 2017)). A particularly successful approach generating from pose-to-image is training of specialized - or personalized models to particular scenes. These often require large-scale target datasets, such as 20 minutes of footage target lab setting (Chan et al., 2018)
The above approaches make use of personalization as an implicit but crucial ingredient, by on-the-fty training a generative model tuned to the particular target domain of interest. Often, personalization is operationalized by fine-tuning generic model on the specific target frames of interest. Our key insight is recasting personalization as an explicit component of a video-retargeting engine, allowing us to make use of meta-learning to learn how best to fine-tune (or personalize) a generic model to particular target domain. We demonstrate that meatanlermnngg-to to--ine-ttuee is particularly effective in the few-shot regime, where few target frames are available. From technical perspective, of our contributions is extending meta-learning to GANs, which is nontrivial because both a generator and discriminator need be adversarially fine-tuned.

---

Published as a conference paper at ICLR 2020
K frimes from farget video
MetaPix personalization
Source, Driver Video
T
0
Output
Figure I:
Video retargeting on a budget. Our goal is to retarget source video to a target, quickly (running a few, T, iterations for adaptation) and efficiently (given few, K, frames from the target domain). We achieve that by (meta)learning model that is able quickly and efficiently adapt given target video.
To that end, we propose MetaPix, a novel approach to personalization for video retargeting, Our formulation treats personalization as few-shot learning problem, where the task is to adapt generic generative model of human actions to specific person given few samples of their appearance. Our formulation agnostic the actual generative model used, and is compatible with both pose- conditioned transfer (Balakrishnan et al., 2018) or generative (Chan et al., 2018) approaches. Taking inspiration from the recent successes meta-learning approaches for few-shot tasks (Nichol et al., 2018; Finn et al., 2017), we propose novel formulation by adapting the popular first-order meta- learning algorithm Reptile (Nichol et al., 2018) for jointly learning initial weights for both the generator and discriminator. Hence, our model is optimized for efficient adaptation (personalization) given only a few samples and on a computational budget, and obtains stronger performance compared a model not optimized in this form. Interestingly, find this personalized model naturally enforces strong temporal coherence in the generated frames, even though it is not explicitly optimized for that task.
RELATED WORK
Deep generative modeling. There has been growing interest in using deep networks for generative modeling of visual data, particularly images. Popular techniques include Variational Auto-Encoders (VAEs) (Kingma & Welling, 2014) and Generative Adversarial Networks (GANs) (Goodfellow et al., 2014). Particularly. GAN based techniques have shown strong performance for various tasks such as conditional image generation (Brock al., 2019), image-to-image translation (Isola al., 2017; Wang et al., 2018b; Zhu et al., 2017; Balakrishnan et al., 2018), unsupervised translation (Zhu et al., 2017) and domain adaptation (Hoffman et al., 2018). More recently, these techniques have been extended to video tasks, such as generation (Vondrick et al., 2016), future prediction (Finn et al., 2016) and translation (Bansal et al., 2018; Wang et al., 201 2018a). Our work explores generative modeling from few samples, with main foeus being the task of video translation. There has been some prior work this direction (Zakharov et al., 2019), though is largely limited to faces and portrait images.
Motion transfer and video retargeting. This refers to the task of driving a video of a person or cartoon character given another video (Gleicher, 1998). While there exist some unsupervised techniques (Bansal et al., 2018) to do so, most successful approaches for articulated bodies involve using pose as an intermediate supervision. Recently, there have been two broad categories of approaches that have been employed for this task: 1) Learning to transform an image into another, given pose as input, either in 2D (Zhou et al., 2019; Balakrishnan et al., 2018; Siarohin al., 2018; Ma et al., 2017) or 3D (Liu et al., 2018; Neverova et al., 2018; Walker et al., 2017). And 2) Learning model to directly generate images given a pose as input (or, Pose2Im) (Chan al., 2018). The former approaches tend to be more sophisticated, separately generating foreground and background pixels, and tend to perform slightly better than the latter. However, they typically learn a generic model across datasets that transfer from single frame, whereas the latter can learn more holistic reconstruction by learning a specific model for a video. Our approach complementary to such transfer approaches, and be applied top of either, discuss in Section 3.
2

---

Published as a conference paper at ICLR 2020
Legend
θ
VL3
03
VL2
θ
0 = (0P,00 )
V L
Pix2PixHD with weights
0
Oz 2
Meta-learning
01
2
01
Finctuming
Figure 2: Meta-learning for video retargeting, Our goal is to learn a generic retargeting model @ (parameters of a Pix2PixHD (Wang al., 2018b) in our case), such that it can be efficiently personalized for specific person given only few samples/frames of their appearance. We achieve it using meta-learning, where our model optimized for personalization to a new person, given only K samples of their appearance, and being trained for T iterations.
Few-shot learning, Low shot learning paradigms attempt learn model using very small amount of training data (Thrun, 1996), typically for visual recognition tasks. Classical approaches build generative models that share priors across the various categories (Fei-Fei et al., 2006; Salakhutdinoy et al., 2012). Another category of approaches attempt to learn feature representations invariant to intra-class variations by using hallucinated data (Hariharan & Girshick, 2017; Wang et al., 2018c) or specialized training procedures/loss functions (Wang & Hebert, 201 Bart & Ullman, 2005). More recently, it has been framed as a 'learning-to-le to-aaarnn' or a meta-leaming problem. The key idea is to directly optimize the model, for the eventual few-shot adaptation task, where the model is finetuned using a few examples (Finn ct al., 2017). Alternatively, it has also been explored in form of directly predicting classifier weights (Bertinetto et al., 2016; Wang & Hebert, 2016b; Wang al., 2017; Misra et al., 2017).
Meta Learning. The goal of metalearning is to learn models that are good at learning, similar to how humans are able to quickly and efficiently learn to do a new task. Many different approaches have been explored that end. One direction involves learning weights through recurrent networks like LSTM (Hochreiter et al., 2001; Santoro et al., 2016; Duan et al., 2016). More commonly, meta-learning has been used as a way learn an initialization for a network, that is finetuned at test time on a new task. A popular approach in this direction is MAML (Finn et al., 2017), where the parameters are directly optimized for the test time performance of the task needs to adapt to. This performed by backpropagating through the finetuning process by computing second order gradients. They and others (Andrychowicz et al., 2016) have also proposed first-order methods like FOMAML that forego the need to compute second order gradients, making it more efficient at empirically small drop in performance. However, most of these works still tend to have the requirement of SGD to be used the task optimizer. A recently proposed meta-learning algorithm, Reptile (Nichol et al., 2018), forgoes that constraint by proposing a much simpler first order meta learning algorithm that is compatible with any black box optimizer.
OUR APPROACH
We now describe MetaPix in detail. To reiterate, our goal is learn a generic model of human motion, parameterized by 6, that can quickly and efficiently be personalized for a specific person. We define speed and efficiency requirements terms of two parameters: computation//terations (T" and the number samples required for personalization (K), respectively. We now describe the base architecture, MetaPix training setup, and the implementation details.
Base retargeting architecture. We build upon popular video retargeting architectures. Notably, there are two common approaches in literature: 1) Learning a transformation from one image to another, conditioned on the pose (Zhou et al., 2019; Balakrishnan et al., 2018) and 2) Learning mapping from pose to RGB (Pose2lm), like (Chan et al., 2018). Both obtain strong performance and interested in. For example in K K-shot setting
amenable to the speed and efficiency constraints (i.e. to learn a model using K frames), one can train the Pose2lm mapping using the K frames in the former case, use the C/K pairs from K frames to learn transformation function from one of the
3

---

Published as a conference paper at ICLR 2020
Algorithm 1 Meta-learning for video re-targeting for the Pose2Im setup.
Initialize 0D, f from pretrained weights
for iteration = 2...... do
Sample K pose image pairs from the same shot randomly
Compute p 0 = Pix2PixHD (Op. 0o), for K images and 7 iterations
Update 0D = 0D c(0p 0D)
Update fg = 0g c(OG - -OG)
end for
K' images to another in the latter case. They are also both compatible with our MetaPix optimization discussed next.
Pose2Im (Chan et al., 2018) approaches essentially build upon image-to-image translation meth- ods (Isola et al., 2017; Wang et al., 2018b), where the input is rendering of the body joints, and the output is an RGB image. The model consists of an encoder-decoder style generator G. It is trained using a combination of perceptual reconstruction losses (Johnson et al., 2016), implemented using an L penalty over VGG (Simonyan & Zisserman, 2015) features and discriminator losses, where we train separate discriminator network D that is trained differentiate the generated images from real images. The reconstruction loss forces it to be close to the ground truth, potentially leading to blurry outputs. Adding the discriminator helps fix that, as forces the output onto the manifold of real images. Given its strong performance, we use Pix2PixHD (Wang et al., 201 as our base architecture for Pose2Im. For brevity, we skip a complete description of the model architecture, and refer the reader to (Wang et al., 2018b) for more details.
Pose Transfer (Balakrishnan et al., 2018; Zhou et al., 2019), on the other hand, takes a source image of person and target pose, and generates an image of the source person in that target pose. These approaches typically segment the limbs, transform their position as in the target pose, and generate the target image by combining the transformed limbs and segmented background by using generative network like a U-Net (Ronneberger et al., 2015). These approaches can leverage learning to move pixels instead of having to generate color and background image from learned representation. We utilize the Posewarp method (Balakrishnan et al., 2018) as our base Pose Transfer architecture due to available implementation.
MetaPix. MetaPix builds upon the base retargeting architecture by optimizing it for few-shot and fast adaptation for personalization. We achieve that by taking inspiration from the literature on few-shot learning, where meta-learning has shown promising results. We use a recently introduced first-order meta-lcarning technique. Reptile (Nichol et al., 2018). As compared to the more popular technique, MAML (Finn et al., 2017), it more efficient as it does not compute a second gradient and is amenable to work with arbitrary optimizers as it does not need to backpropagate through the optimization process. Given that GAN architectures are hard to optimize, Reptile suits our purposes of its ability to use Adam (Kingma & Ba, 2015), the default optimizer for Pix2PixHD, as our task optimizer. Figure? illustrates the high level idea of our approach, which we describe in detail next.
We start with either Pose2Im or Pose Transfer trained base model. We then finetune this model as described in Algorithm 1. Note that Pix2PixHI based on GAN, so has two network weights to be optimized, the generator (0c) and discriminator (0p). In each meta-iteration, we sample task: in our case a set of K frames from a new video to personalize to. We then finetune the current model parameter to that video over T iterations, and update the model parameters in the direction of the personalized parameters using a meta learning rate c. We optimize both 0n and AG jointly at each step. Note that Posewarp employs more complicated two-stage training procedure, and metalearn only the first stage (which has no discriminator) for simplicity.
Implementation Details. We implement MetaPix for the Pose2Im base model by building upon a public Pix2PixHD implementation' in PyTorch, and perform all experiments on a TITAN-X or GTX 1080Ti GPU node. We follow the hyperparameter setup as proposed in (Wang et al., 2018b). We represent the pose using multi chulni- heatmap image, and input and output are 512 x 512px RGB
hhttps://gi thub. A/ //pppp//// x2pi XHD/
4

---

Published as a conference paper at ICLR 2020
Figure 3: Training data. Frames from the additional training data we collected. We download 10 videos from YouTube, distinct from the ones used for personalization and evaluation.
images. The generator consists of 16 convolutional and deconvolutional layers, and is trained with equally weighted combination of GAN, Feature Matching, and VGG losses. Initially, we pretrain the model large corpus of videos to learn generic Pose2lm model as described in Section 4. During this pretraining stage, the model is trained on all of the training frames for 10 epochs using learning rate of 0.0002 and batch size of 8 distributed over the 4 GPUs. We experimented with multiple learning rates including 0.2, 0.02, 0.002; however, we observed that higher learning rates caused the training diverge. When finetuning for personalization, given K frames and a computational budget T,, we train the first 2 iterations using a constant learning rate of 0.0002, and the remaining iterations using a linear decay to O, following (Wang et al., 2018b). The batch size is fixed 8, and for K < 8, we repeat the frames to get 8 images for the batch. For the metalearning, we set the meta learning rate, 6 with linear decay to 0, and train 300 meta-iterations We also experiment with meta learning rate, 0.1, however, was much slower to converge. To potentially stabilize metatraining, we experiment with differing numbers of updates to the generator and discriminator during iterations of Alg. 1, as well as simplified objective functions. Recall that the GAN loss adds significant complexity due to the presence of discriminator that need also be adversarially finetuned. In total, our metaleaming takes 1 day of training time on 4 GPUs. For the Pose Transfer base model, we apply MetaPix in similar fashion on top of Posewarp', using the author provided pretrained weights. We will release the MetaPix source code for details.
1
EXPERIMENTS
We now experimentally evaluate MetaPix. We start by describing the datasets used and evaluation metrics. We then describe our base Pose2lm and Pose Transfer setup, followed by training that model using MetaPix. Finally, we analyze and ablate the various design choices in MetaPix.
4.1 DATASETS AND EVALUATION
We train and evaluate our approach on in-the-wild internet videos. Due the lack of standard benchmark for such retargeting tasks, we use the dataset as described in (Zhou et al., 2019) as our test set. These are a set of videos downloaded from youtube, each 4-12 minutes long. We refer the reader to Figure 1 in (Zhou et al., 2019) for sample frames from this dataset. Additionally, we collect set of 10 more dance videos from YouTube (distinct from the above 8), as our pre-training and meta-learning corpus. We provide the list of YouTube video IDs for both in the supplementary. Our models are only trained on these videos, and videos from (Zhou et al., 2019) are only used for personalization (using K frames) and evaluation. Figure 3 shows sample frames from these newly collected videos.
Evaluation and Metrics: Similar to (Zhou et al., 2019), split each of the 8 test videos into training and test sequence in 0.85:0.15 ratio, and sample K training and 2000 test frames from the test sequence. We use the same metrics as in (Zhou et al., 2019) for ease of comparison: Mean Squared Error (MSE), Structured Similarity Index (SSIM) and Peak Signal-to-Noise Ratio (PSNR). Each of these are averaged over the 2000 test frames from each of the test videos. To compare our baselines and our method, pose retargeting as a task aims to minimize MSE and maximize SSIM and PSNR.
5

---

last four Published as a conference paper at ICLR 2020
Ground Truth
Px22PsHD K
Prerrained Pix2FixHI>
MetaPix Pix2PisHD)
Pretmined Posewarp
Meta Pix Posewarp
WOR
DAN
Figure 4: Qualitative comparison. Here compare MetaPix with baselines and network architectures. The Pix2PixHD K ac model an upper bound tuned columns compare the constrained setting, where at test time only K = 5 frames are used for personalization, over 7 = 20 iterations of fine- tuning. We visualize results of challenging pose-image test pairs that include poses not commonly backgrounds from the training set, while MetaPix is much better at personalizing the fine- tuning frames. The two columns show that Posewarp more accurate base architecture than Pix2PixHD, meta-learning on top of it still produces more accurate images with less blur.
available frames for personalization.
in training. In this
the pre- trained baseline tends to copy clothing and

---

Published as a conference paper at ICLR 2020
Table 1: Adding MetalPx. We compare the Pix2PixHD and Posewarp models' performance. The models are initialized either at random, pretrained on entire training set of videos, or meta-learned with MetaPix. Constraining the amount of frames and compute used at test time leads to a drop in performance as expected. In general, Pix2PixHD performs well with large training sets hecause it can learn to model subtle dependancies between pose and appearance. In contrast, Posewarp performs better with smaller training sets becausc it directly transfers pixels from the background and foreground. However, in both cases, meta-lcarning with MetaPix produces better performance, given the exact same test time constraints. We compare these methods qualitatively in Figure 4.
Method
Init
K'
T
SSIM
PSNR
MSE
Random
oo
0C
0.68
19.56
2, 427.
Pretrain
00
0.69
19.31
2, 6.33.40
Pix2PixHD
Random
5
20
0.08
9.51
7, 801.19
Pretrain
20
0.35
12.00
5, 576.28
MetaPix
5
20
0.39
13.73
4, 696.00
Pretrain
0
0C
0.58
17.51
2, 901.13
Posewarp
Random
5
20
0.40
12.25
4, 670.81
Pretrain
5
20
0.55
16.53
3, 140.91
MetaPix
5
20
0.56
16.94
2, 962.67
4.2 EVALUATING METAPIX
We start by building our baseline retargeting model, based on Pix2PixHD (Wang et al., 2018b; Chan et al., 2018). To get sense of the upper bound performance of our model, we train the model for each test video with no constraints on T or K, starting from the model pre-trained on our train set. Specifically, we use all the frames from the first 85% of each video, and train it for 10 epochs. We report the performance of this model in first section of Table and show sample generations in second column of Figure 4. Since this model gets strong quantitative and qualitative performance, we stick with it as our base retargeting architecture through the rest of the experiments. We also employ baseline retargeting model based on Posewarp for evaluation, but we focus on Pose2Im for further experimentation due to its relative simplicity.
Now we evaluate the performance of our model in constrained settings, where we want learn to personalize given few samples and in constrained computational budget. Hence, we use pretrained model on train set and a random model, and we personalize them by finetuning on each test video. As Table 1 shows, applying constraints leads to drop in performance in all methods, as expected from using only 5 frames finetuned over 20 iterations. Finally, we compare that to the MetaPix model: in that case, we start from the pre- pre-trained model, and do meta-learning on top of those parameters to optimize them for the transfer task as described in Section 3. That leads to a significant improvement
the pretrained model, showing the strength of MetaPix for this task.
In Figure 43, we visualize the predictions using the unconstrained model, as well as the constrained models trained using MetaPix and without, i.e. with simple pretraining. It is interesting note that the meta-learned model is able to adapt to the color of the clothing and the background much better than a pretrained model, given the same frames for personalization. This reinforces MetaPix is much better initialization for few-shot personalization than directly finetuning from a generic pretrained model. We further explore this quality of coherence in the next section.
4.3 ABLATIONS
We now ablate the key design choices in our MetaPix formulation. One of the strengths of our formulation is the explicit control on the supervision provided and computation the model allowed to perform, and depending on the use-case, those parameters can easily be tweaked. We explore the effect of MetaPix on those parameters next on the Pose2Im base retargeting architecture.
Variation in K: We vary the amount of supervision for personalization, K, and evaluate its effect on the metrics in Figure We compare the following models: a) Randomly initialized, b) Pretrained
httpps: / / thub, com/ba 1 lakg/ pwwpp/pp-wwpppppeeoe.eep 201 Video visualization at https: /youut. UnUddUnUUUUU---
7

---

Published as a conference paper at ICLR 2020
8000
0.4
7000
5SIM
0.3
PSNR
13
Legend
Pretrained
0.2
MelaPix
5000
MetaPix (K 5)
Random
10
4000
K
Figure 5: Personalization using frames. We find that while all initializations get better with increasing K, and using MctaPix consistently outperforms simple pretraining. Moreover we note that even using model trained with MetaPix for K = 5 works well at any K value used at test time, showing the generalizability of MetaPix. It is worth noting that the biggest gap is seen lower values of K, showing our method is most useful cases where has little data for personalization.
9030
0.5
7000
13
MSE
8000
Legend
Pretrained
12
5000
MetaPix
11
MotaPix =
4000
Random
0.1
10
0
90
120
160
200
40
80
120
160
200
40
80
120 llerations
160
200
llerations
Iterations
Figure 6: Personalization after T iterations. We compare performance on increasing the computational budget for personalization. As expected all initializations improve with T, though MetaPix consistently outperforms random pretraining. Again strong generalizability, MetaPix model trained for T 20 performs well at other values used lest time.
on the train set, c) Trained using MetaPix for each value of K and tested with the same K, and d) Trained using MetaPix for K = 5 and tested at each value of K. The last one tests the generalizability of MetaPix to different values K at train and test time. We find that MetaPix trained models consistently perform better than simple pretrained model on all metrics. Notably, the model only values, showing the
trained for K = still able to obtain strong performance at different MetaPix trained model can generalize beyond the specific setup it is optimized for. The gap between the MetaPix trained model and the pretrained model tends to reduce with higher K, which as expected: more data for personalization would likely reduce the importance of the initialization. However, there is clear and significant gap for lower values of K, showing that MetaPix is highly effective for retargeting from few samples. fact, we find that meta-learning most effective for K 1, corresponding to the challenging scenario of video-to-image retargeting.
Variation in T: Similar variation in supervision, we experiment with varying the computation, or T. in Figure 6. We experiment with similar set of baselines as in the case for K, and again observe that the MetaPix model consistently outperforms random initialization or pretraining on all metrics. Also, we see similar generalizability. as the model metatrained for T = 20 is able to perform well for other 7 values test time too. The ability for MetaPix to generalize across and implies cost-effective strategies for training. The computational cost for training meta-learner is dominated by fine-tuning, which scales linearly with К and Training with smaller values of both can result in significant speedups up 10x in experiments.
Variation meta learning rate e: We also experimented with changing the meta learning rate. At 0.1 (K 5, T 200), we obtained SSIM=0.47, similar to what the pretrained model gets. Using our default e = 1.0, improves performance to 0.51. Hence, a higher meta learning rate was imperative improvements with MetaPix.
8

---

Published as a conference paper at ICLR 2020
Pretrained
MetaPix
GT 8
8g
Figure 7: Visualizing finetuning between MetaPix and Pretrained. We compare MetaPix's initialization for the K T' task to our pretrained model initialization. visualize models obtained during iterations of finetuning, at 0, 10, 20, 40, 80 and 200 iterations for random test pose-image pairs. The images generated by MetaPix's initialization are temporally coherent, whereas the pretrained weights produce various training images depending on the pose. As observed in the intermediate iterations, the initialization translates its temporal coherence properties across finetuning iterations as well. This further reinforces our belief that MetaPix learns an initialization that able quickly adapt the actor and background appearance from the few samples provided at test time.
Only training the generator: We apply Reptile in GAN setting, where we jointly meta-optimize two networks. We also experimented with freezing one of the networks, specifically the discriminator, to the weights learned during pretraining. For our K 5, 200, e 1.0 setup, we obtain similar performance as optimizing both, suggesting that 'universal' discriminator might suffice for meta-learning on GANs.
Visualizing the dynamics of personalization: In order to examine the process of personalization, we visualize models obtained during iterations finetuning, at 10, 20, 40, and 200 iterations for random test pose-image pairs. We compare both the pretrained and metalearned model, trained for = 5, = 200. Figure 7 shows images generated by these intermediate iterations. Both methods learn clothing details and background colors after 20 iterations. Interestingly, MetaPix produces images that are temporally coherent, even upon initialization, while the pretrained baseline produces images whose background and clothing vary with pose. This more coherent initialization appears to translate to more coherent generated images after personalization.
CONCLUSION
We have explored the task of quickly and efficiently retargeting human actions from one video to another, given a limited number samples from the target domain. We formalize this as few-shot personalization problem, where we first learn a generic generative model on large amounts of data, and then specialize it to small amount of target frames via finetuning. We further propose a novel meta-learnan meta-learning based approach, MetaPix, to leam this generic model in a way that is more amenable to personalization fine-tuning.
so, we repurpose a first-order meta-learning algorithm, Reptile, to adversarially meta-optimize both the generator and discriminator of generative adversarial network. We experiment with on in-the-wild YouTube videos, and find that MetaPix outperforms widely-used approaches for pretraining, while generating temporally coherent videos.
Acknowledgements: This research based upon work supported in part by NSF Grant 1618903, the Intel Science and Technology Center for Visual Cloud Systems (ISTC-VCS), and Google.
REFERENCES
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffmman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient
Viidoo visualization httptttt///// //yutuu.uuuuu///oo be/bxJJXXK4100

---

Published as a conference paper at ICLR 2020
descent. In NeurIPS, 2016.
Guha Balakrishnan, Amy Zhao, Adrian V Dalca, Fredo Durand, and John Guttag. Synthesizing images of humans in unseen poses. In CVPR, 2018. 1, 3,4
Aayush Bansal, Shugao Ma, Deva Ramanan, and Yaser Sheikh. Recycle-gan, Unsupervised video retargeting. In ECCV, 2018. 1,2
Evgeniy Bart and Shimon Ullman. Cross-generalization: Learning novel classes from a single example by feature replacement. In CVPR, 2005. 3
Luca Bertinetto, João F Henriques, Jack Valmadre, Philip Torr, and Andrea Vedaldi. Learning feed-forward one-shot learners. In NeurIPS, 2016.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. ICLR, 2019. 1,2
Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A Efros. Everybody dance now. arXiv preprint arXiv:1808.0733: 2018. 1, 3, 7,7
Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. RI2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint / 2016. 3
Li Fei-Fei, Rob Fergus, and Pietro Perona. One-shot learning of object categories. TPAMI, 2006. 3
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. In NeurIPS, 2016. 2
Chelsea Finn, Pieter Abbeel, and Sergey Levine. agdel-agnostic meta-learning for fast adaptation of deep networks. /CML, 2017. 2, 3,
Michael Gleicher, Retargetting motion to new characters. In SIGGRAPH, 1998.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS, 2014.
Bharath Hariharan and Ross Girshick. Low-shot visual recognition by shrinking and hallucinating features. In ICCV, 2017. 3
Sepp Hochreiter, A Steven Younger, and Peter R Conwell. Learning to learn using gradient descent. In ICANN, 2001.
Judy Hoffmman, Eric Tzeng, Taesung Park, Jun- Yan Zhu, Phillip Isola, Kate Saenko, Alexci A. Efros, and Trevor Darrell. Cycada: Cycle consistent adversarial domain adaptation. In ICML, 2018.
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In CVPR, 2017. 1,
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, 2016. 4
Diederik P. Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. ICLR, 2014.
Lingjie Liu, Weipeng Xu, Michael Zollhoefer, Hyeongwoo Kim, Florian Bernard, Marc Habermann, Wenping Wang, and Christian Theobalt. Neural animation and reenactment of human actor videos. arXiv preprint 1809:1889900358 2018. 2
Liqian Ma, Xu Jia, Qianuu Sun, Bernt Schiele, Tinne Tuytelaars, and Luc Van Gool. Pose guided
person image generation. NewrIPS, 2017. 1,2
Ishan Misra, Abhinav Gupta, and Martial Hebert. From red wine to red tomato: Composition with context. In CVPR, 2017.
10

---

Published as a conference paper at ICLR 2020
Natalia Neverova, Riza Alp Guler, and lasonas Kokkinos. Dense pose transfer. In ECCV, 2018.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-leamingg algorithms. arXiv preprint arXiv:1803.02900 02999, 2018. 2, 3,4
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. CoRR, abs/1505 0009700 2015. URL 04597. 4
1505.
Ruslan Salakhutdinov, Joshua Tenenbaum, and Antonio Torralba. One-shot learning with a hierarchi- cal nonparametric bayesian model. In ICML Workshop, 2012. 3
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with memory memory-augmentted neural networks. In ICMI., 2016. 3
Aliaksandr Siarohin, Enver Sangineto, Stéphane Lathuilière, and Nicu Sebe. Deformable gans for pose-based human image generation. In CVPR, 2018. 1,2
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015.
Sebastian Thrun. learning the n-th thing any easier than learning the first? In NeurlPS, 1996. 3
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In NeurlPS, 2016.
Jacob Walker, Kenneth Marino, Abhinav Gupta, and Martial Hebert. The pose knows: Video forecasting by generating pose futures. In ICCV, 2017. 2
Ting-Chun Wang, Ming-Yu Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Video-to-vin video synthesis. In NewrlPS, 2018
Ting-Chun Wang, Ming- Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In CVPR, 2018b. 3, 7,7
Yu-Xiong Wang and Martial Hebert. Learning from small sample sets by combining unsupervised meta-training with cnns. NeurlPS, 2016a. 3
Yu-Xiong Wang and Martial Hebert. Learning to learn: Model regression networks for easy small sample learning. In ECCV, 2016b. 3
Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Learning to model the tail. In NeurlPS, 2017.
Yu-Xiong Wang, Ross Girshick, Martial Hebert, and Bharath Hariharan. Low-shot learning from imaginary data. In CVPR, 2018c.
Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor Lempitsky. Few-shot adversarial learning of realistic neural talking head models. arXiv preprint 1rXiv:190S.08233 2019. 2
Yipin Zhou, Zhaowen Wang, Chen Fang, Trung Bui, and Tamara L. Berg. Dance dance generation: Motion transfer for internet videos. arXiv preprint arXiv: 0e0:::/44000999 2019. 2, 4,
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-imann translation using cycle-consistent adversarial networks. In ICCV, 2017. 2
A APPENDIX
11

---

Published as a conference paper at ICLR 2020
GT
K = 3
K = 5
K 10
= 20
PT
MT
PT
MT
PT
MT
Figure 8: Qualitative Variation in K. We compare the MetaPix-trained models (MT) with their pretrained
counterparts (PT) for K = [3, 10, 20). We fix the base architecture to Pix2PixHD and T = 20. With higher K, both methods generate good images, but with lower K, MetaPix generates backgrounds and clothing that better match the ground-truth. Our results illustrate that meta-learning excels in the few-shot regime.
12

---

Published as a conference paper at ICLR 2020
Ground Truth
PT
MTT -
MTT T 40
MT T 100
MT T 200
Figure 9: Visualizing Initializations." We visualize knowledge captured by meta-lcarning by running the Eittaix-trained model (MT) without finetuning test vidco. We fix K vary T [20, 40, 100, 200]. The pre-trained model (PT) generates image from its training As increase T. MetaPix learns factor pose appearance, generating consistent background clothing appearance regardless of the human pose. Figure demonstrates that such factored representations are easier to fine-tune, and result in more temporally stable generated videos.
VVidoo visualization is available htttps://yout..be//eeet
13


Reviews:
[{"review_id": "HyePk2nK5r", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel and interesting task that learn to retarget human actions with few-shot samples. The overall pipeline is built by applying  meta-learning strategy on pre-trained retargeting module. It follows a conditional generator and discriminator structure that leverages few-shot frames to retarget the action of source video. The approach is technically sound.\n\nThe evaluations are compared with two baseline methods,  Pix2PixHD and Posewarp. Their evaluations are satisfactory and convincing, the results demonstrate some improvements over baseline models regarding both the selected metrics and the visualizing performance.\n\nThough the proposed problem is novel and somewhat interesting, there are also several weaknesses of this work:\n- The novelty of methodology is somewhat limited. It is more about merging several state-of-the-art modules in different tasks to tackle the few-shot retargeting problem. Though efforts may be needed to make the pipeline work, the overall contribution is not significant. \n- The improvement obtained with proposed MetaPix module is not significant in few-shot setting according to Table 1. Additionally, could the authors provide some visualizing results for different K number, which would be interesting for analysis.\n\n\n\n\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "ByezyYV9Fr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submission proposes an application of meta-learning to video frame generation modeling conditioned on human pose information, in order to allow the model to adapt to the context of each video. This context is provided in the form of a support set of K pairs of pose/frame images for the video. Reptile is used as the meta-learning method, and applied to two recently proposed video-frame generative networks (Pix2PixHD and Posewarp). In both cases, results show that Reptile is able to produce better adaptive models, i.e. models that when fine-tuned on the support set produce better image frames.\n\nThough the originality of the work is somewhat weak (it's a relatively straightforward application of Reptile to Pix2PixHD and Posewarp), the problem setting is novel and I find the demonstration that Reptile works well in this setting interesting and valuable. The paper is also clearly written and easy to follow. For these reasons, I'm personally leaning towards recommending to accept this submission.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BkeVb25Otr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, authors propose to address few shot video retargeting, where one should adapt a generic generative model of human actions to a specific person given a few samples of their appearance. \n\nOverall, the paper is written with a good structure. I do like the problem setting and motivations in this paper. However, the solution is not quite novel for me. Both base model (Pix2PixHD) and few-shot adaptation (Reptile) come from the previous works. Their combination is somewhat incremental.  ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three reviewers have assessed this paper and they have scored it 6/6/6 after rebuttal. Nonetheless, the reviewers have raised a number of criticisms and the authors are encouraged to resolve them for the camera-ready submission.
Decision: Accept (Poster)