Diverse Video Generation using a Gaussian Process Trigger
Authors: Gaurav Shrivastava
Abhinav Shrivastava
Venue: ICLR 2021 Poster (2021)
Keywords: video synthesis
future frame generation
video generation
gaussian process priors
diverse video generation

Abstract: Generating future frames given a few context (or past) frames is a challenging task. It requires modeling the temporal coherence of videos as well as multi-modality in terms of diversity in the potential future states. Current variational approaches for video generation tend to marginalize over multi-modal future outcomes. Instead, we propose to explicitly model the multi-modality in the future outcomes and leverage it to sample diverse futures. Our approach, Diverse Video Generator, uses a GP to learn priors on future states given the past and maintains a probability distribution over possible futures given a particular sample. We leverage the changes in this distribution over time to control the sampling of diverse future states by estimating the end of on-going sequences. In particular, we use the variance of GP over the output function space to trigger a change in the action sequence. We achieve state-of-the-art results on diverse future frame generation in terms of reconstruction quality and diversity of the generated sequences.

Published as a conference paper at ICLR 2021
DIVERSE VIDEO GENERATION USING A GAUSSIAN PROCESS TRIGGER
Gaurav Shrivastava and Abhinav Shrivastava University of Maryland, College Park
(gauravsh, abhi abhinav) @cs. umd. edu
ABSTRACT
Generating future frames given a few context (or past) frames is a challenging task. It requires modeling the temporal coherence of videos and multi multi-modality in terms of diversity in the potential future states. Current variational approaches for video generation tend marginalize over multi-modal future outcomes. In- stead, we propose to explicitly model the mudli-modality in the future outcomes and leverage it to sample diverse futures. Our approach, Diverse Video Generator, uses a Gaussian Process (GP) to learn priors on future states given the past and maintains probability distribution over possible futures given particular sam- ple. In addition, we leverage the changes in this distribution overtime to control the sampling of diverse future states by estimating the end of on-going sequences. That is, we use the variance of GP over the output function space to trigger a change in an action sequence. We achieve state-of-the-a results on diverse future frame generation in terms of reconstruction quality and diversity of the generated sequences. Webpage http://www.css www.sudddee.o...mmmm......... handdedu/-aae www.uu/aaraas///ygg..kk
INTRODUCTION
Humans often able imagine multiple possible ways that the scene can change over time. Modeling and generating diverse futures is an incredibly challenging problem. The challenge stems from the inherent modalty of the task, i.e., given sequence of past frames, there can be multiple possible outcomes of the future frames. For example, given the image of a "person holding a cup" Figure. most would predict that the next few frames correspond to either the action "drinking from the cup" or "keeping the cup the table." This challenge is exacerbated by the lack of real training data with diverse outputs - all real-world training videos come with a single real future and no "other" potential futures. Similar looking past frames can have completely different futures (e.g., Figure. 1). In the absence of any priors or explicit supervision. the current methods struggle with modeling this diversity. Given similar looking past frames, with different futures in the training data, variational methods, which commonly utilize (Kingma & Welling, 2013), tend to average the results better match all different futures (Denton & Fergus, 2018; Babaeizadeh et al., 2017; Gao et al., 2018; Lee et al., 2018; Oliu et al., 2017). We hypothesize that explicit modeling of future diversity essential for high-quality, diverse future frame generation.
In this paper, we model the diversity of the future states, given past context, using Gaussian Processes (GP) (Rasmussen, 2006), which have several desirable properties. They learn a prior on potential future given past context, in a Bayesian formulation. This allows us to update the distribution of possible futures as more context frames are provided as evidence and maintain list of potential futures (underlying functions in GP). Finally, our formulation provides an interesting property that crucial to generating future frames - the ability to estimate when generate a diverse output vs. continue on-going action, and way to control the predicted futures.
In particular, we utilize the variance of the GP at any specific time step as an indicator of whether an action sequence is on-going finished. An illustration of this mechanism is presented in Figure. 2. When we observe a frame (say at time t) that can have several possible futures, the variance of the GP model is high (Figure. 2 (left)). Different functions represent potential action sequences that can be generated, starting from this particular frame. Once we select the next frame (at t+2), the GP variance of the future states is relatively low (Figure. 2 (center)), indicating that action sequence

---

Published as a conference paper at ICLR 2021
Past
Potential Future
Potential Future 2
Person Holding Cup
Person Keeping Cup the Table
Persen Drinking from the Cup
Figure 1: Given "person holding cup," humans can often predict multiple possible futures ".g..."rinkingg from the cup" or "keeping the cup on the table.").
is on-going, and the model should continue it opposed to trying to sample diverse sample. After the completion of the on-going sequence, the GP variance over potential future states becomes high continue this action (i.e., pick the mean function represented by
again. This implies that the black line in Figure. (center)) or try and sample a potentially diverse sample (i.e., one of the functions that contributes high-variance). This illustrates how trigger diverse actions. An example of using GP trigger is shown in Figure. 2 (right), where after every few frames, we trigger a different action.
use GP decide when to
Now that we have a good way to model diversity, the next step is to generate future frames. Even after tremendous advances in the field of generative models for image synthesis (Denton & Fergus, 2018; Babaeizadeh et al., 2017; Lee et al., 2018; Vondrick & Torralba, 2017; Lu et al., 2017; Von- drick et al., 2016; Saito et al., 2017; Tulyakov al., 2018; Hu & Wang, 2019), the task of generating future frames (not necessarily diverse) conditioned dependent images, the future frames need to obey potential video dynamics that might on-going in the past frames, follow world knowledge (e.g., how humans and objects interact), etc.. We uti- lize a fairly straightforward process to generate future frames, which utilizes two modules: a frame auto-encoder and dynamics encoder. The frame auto-encoder learns to encode frame in latent representation and utilizes to generate the original frame back. The dynamics encoder learns to model dynamics between past and future frames. We learn two independent dynamics encoders: an LSTM encoder, utilized to model on-going actions and the GP encoder (similar to (Srivastava et al., 2015)), and GP encoder, utilized to model transitions to new actions. The variance of this GP encoder can be used as trigger to decide when to sample new actions. We train this framework end-to-end. We first provide an overview of GP formulation and scalable training techniques in $3, and then describe our approach in 54.
past frames is still hard. As opposed to in-
Comprehensively evaluating diverse future frames generation is still an open research problem. Fol- lowing recent state-of thrt we will evaluate different aspects of the approach independently. The quality of generated frames is quantified using image synthesis/reconstruetii
per-frame metries: SSIM (Wang et al., 2019; Sampat et al., 2009), PSNR, and LPIPS (Zhang et al., 2018; Dosovitskiy & Brox, 2016; Johnson et al., 2016). The temporal coherence and quality a short video clip (16 neighboring frames) are jointly evaluated using the FVD (Unterthiner et al., 2018) metric. However, high-quality, temporarily coherent frame synthesis does not evaluate diversity in predicted frames. Therefore, to evaluate diversity, since there no multiple ground-truth futures, we propose alter- native evaluation strategy, inspired by (Villegas et al., 201 7b): utilizing action classifiers to evaluate whether an action switch has occurred not. A change in action indicates that the method was able to sample a diverse future. Together, these metrics can evaluate if an approach can generate multiple high-quality frames that temporally coherent and diverse. Details of these metries and baselines, and extensive quantitative and qualitative results are provided in 85.
To summarize, our contributions are: (a) modeling the diversity of future states using a GP, which maintains priors on future states given the past frames using a Bayesian formulation (b) leveraging the changing gP distribution over time (given new observed evidence) to estimate when an on- on-going action sequence completes and using GP variance to control the triggering of diverse future state. This results state-of-the-art results on future frame generation. We also quantify the diversity of the generated sequences using action classifiers proxy metric.
2 RELATED WORK
Understanding and predicting the future, given the observed past, is fundamental problem in video understanding. The future states are inherently rect applications in many sarety-critical applications (e.g., autonomous vehicles), where it is critical
and capturing their diversity finds di-
2

---

Published as a conference paper at ICLR 2021
New Sequeaie Trigger
New Sequeres Trigger (Paker Yor ane Hight
Nee Scusea Trigarel Suppenoe giiggin
Hodtiee Saquane
Malified
New Action Sequince
Madiffed
Prizze Varinc Lonet
Sequence
Origied
gggng Sequincce
Sequnxe
Figure 2: An illustration of using GP variance to control sampling on-going actions actions,
to model different future modes. Earlier techniques for future prediction (Yuen & Torralba, 2010; Walker al., 2014) relied on searching for matches of past frames in given dataset and transferring the future states from these matches. However, the predictions were limited to symbolic trajecto- ries or retrieved future frames. Given the modeling capabilities of deep representations, the field of future frame prediction tremendous progress in recent years. One of the first works on video gen- cration (Srivastava et al., 2015) used a multi-layer LSTM network to learn representations of video sequences in deterministic way. Since then, wide range of papers (Oliu et al., 2017; Cricri al., 2016; Villegas et al., 2a; Elsayed et al., 2019; Villegas et al., 2019; Wang et al., 2019; Castrejón et al., 2019) have built models that try to incorporate stochasticity of future states. Generally, this stochasticity lacks diverse high-level semantic actions.
Recently, several video generation models have utilized generative models, like variational auto- encoders (VAEs) (Kingma & Welling, 2013) and generative adversarial networks (GANs) (Goodfel- low et al., 2014), for this task. One of the first works by Xue al. (2016) utilized conditional VAE (cVAE) formulation to learn video dynamics. Similar to our approach, their goal was to model the frame prediction problem in a probabilistic way and synthesizing many possible future frames from a single image. Since then, several works have utilized the e cVAE for future generation (Babaeizadeh al., 2017; Denton Fergus, 2018). The major drawback of using the eVAE approach that its objective function marginalizes over the multi-modal future, limiting the diversity in the generated samples (Bhattacharyya et al., 2018). GAN-based models are another important class of synthesis models used for future frame prediction video generation (Vondrick & Torralba, 2017; Lu et al., 2017: Vondrick et al., 2016; Saito et al., 2017: Tulyakov et al., 2018; Hu & Wang. 2019). However, these models are very susceptible mode collapse (Salimans et al., 2016), L.e., the model outputs only one or few modes instead of generating a wide range of diverse output. The problem of mode collapse is quite severe for conditional settings, as demonstrated by (Isola al., 2016; Zhu al., 2017; Mathieu et al., 2015). This problem is worse in the case of diverse future frame generation due to the inherent multi-modaliy of the output space and lack of training data.
Another class of popular video generation models is hierarchical prediction (Walker et al., 2017; Villegas et al., 2017b; Wichers al., 2018: Cai al., 2018). These models decompose the problems into two steps. They first predict high-level structure of a video, like a human pose, and then lever- age that structure to make predictions at the pixel level. These models generally require additional annotation for the high-level structure for training.
Unlike these approaches, Our approach explicitly focuses on learning the distribution of diverse futures using GP prior on the future states using Bayesian formulation. Moreover, such GP approaches have been used in the past for modeling the human dynamics as demonstrated by (Wang et al., 2008; Hardy et al., 2014; Moon & Pavlovic, 2008). However, due to the scalability issues in GP. these models were limited to handling low dimensional data, like human pose, lane switching path planning, etc. To the best of our knowledge, ours is the first approach that can process video sequences to predict when an on-going action sequence completes and control the generation of a diverse state.
3 REVIEW OF GAUSSIAN PROCESS
A Gaussian process (GP) (Rasmussen, 2006) is a Bayesian non- paraamamttiic approach that learns a joint distribution over functions that sampled from multi-variate normal distribution. Consider a data set consisting of n data-points (inputs, targets] in abbreviated as {X, Y YYr, where the inputs denoted by X {x1,..., x,,) and targets by Y {y1,.... ).). The goal of GP is to learn
3

---

Published as a conference paper at ICLR 2021
an unknown function that maps elements from input space to target space. A gP regression formulates the functional form of f by drawing random variables from a multi-variate normal dis- tribution given by (f(x)), (x.)..... f(x,) (u, xx,x), with mean μ, such that μ4, p(x,), and Xx.x is a covariance matrix. (Kx.)); = ,(x,, x;), where k(.) is a kernel function of the GP. Assuming the GP prior f(X) with some additive Gaussian white noise y; f(x) + the conditional distribution at any unseen points X, is given by:
f. X, X,Y Y (V(E,,,), Cov((), where
(I)
Ef = px, - Kx,.x [Kx,x + +o'
Cov[f.] = Kx..x - Kx [Kxx + o
3.1 LEARNING AND MODEL SELECTION
We can derive the marginal likelihood for the GP by integrating out the (() as a function of kernel parameters alone. Its logarithm can be defined analytically
1 (YT (Ko + o²r) -or))' y log Ko + Q²rl) + const, log (p (Y[X) = -
(2)
where 9 denotes the parameters of the covariance function of kernel Xx,x- Notice that the marginal likelihood involves matrix inversion and evaluating determinant for x matrix. A naïve implementation would require cubic order of computations ((n²) and Onn)) of storage, which hinders the of GP for large dataset. However, recent researches have tried ease these constraints under some assumptions.
3.2 SCALABLE GP
The model selection and inference of GP requires cubic order of computations (nn)) and O(?)) of storage which hinders the use of g'P for a large dataset. Titsias (2009) proposed a new variational approach for sparse approximation of the standard GP which jointly infers the inducing inputs and kernel hyperparameters by optimizing a lower bound of the true log marginal likelihood, resulting in O(nm²) computation, where m < n. Hensman et al. (2013) proposed new variational for- mulation of true log marginal likelihood that resulted in a tighter bound. Another advantage of this formulation is that it can be optimized in a stochastic (Hensman et al., 2013) or distributed (Dai et al., 2014; Gal et al., 2014) manner, which is well suited for our frameworks which use stochastic gradient descent. Further, recent works (Wilson Nickisch, 2015; Wilson et al., 2015; 2016) have improved the scalability by reducing the learning to (n)) and test prediction to O(1) under some assumptions.
In this work, for scalability, we leverage the SVGP approach proposed by Hensman et al. (2013). It proposes a tighter bound for the sparse GP introduced by Titsias (2009), which uses pseudo inputs to lower bound the log joint probability over targets and pscudo inputs. SVGP introduces multivariate normal variational distribution q(u) = A M(m, S), where the parameters m and S are optimized using the evidence lower bound ELBO (eq. 3) of true marginal likelihood (eq. 2). where M = dim(u) < N.
The pseudo inputs, u, depend on variational parameters {zm] LM Therefore, the ELBO for SVGP is
m=
Lavgp (Y = Eq(w) [logp u/x, Z)] + H((())),
where the first term was proposed by Titsias (2009), and the modification by Hensman et al. (2013) introduces the second term. For details about the pseudo inputs and variational parameters z, refer the readers to (Titsias, 2009; Hensman et al., 2013).
In this work, build on the sparse GP approach from GPytorch (Gardner al., 2018), which implements (Hensman et al., 2013).
4 OUR APPROACH
Given a set of observed frames, our goal is to generate diverse set future frames. Our model has three modules: (a) frame auto-encoder (or encoder-ggneratorr) (b) LSTM temporal dynamics
4

---

Published as a conference paper at ICLR 2021
Training Frame Eico-er
Inferee Diverse Video Generator (DV G) with Thighr-Switch
15TMI Excoda
Thender
Pcos
LSTM
E
G
Syet
Aest
Ave
Traininy Dynamics Encoders
Switch between θс4m and Eeu-
LSTM
Figure 3: An overview of the proposed Diverse Video Generator (DVG).
encoder, and (c) GP temporal dynamies encoder model priors and probabilities over diverse potential future states.
The frame encoder maps the frames onto latent space, that later utilized by temporal dynamics encoders and frame generator to synthesize the future frames. For inference, we use all three mod- ules together to generate future frames, and use the GP as a trigger to switch to diverse future states. Below we describe each module in detail.
4.1 FRAME
The frame encoder network a convolution encoder which takes frame X, E R Hxx and maps them to the latent space E Idd where H W is the input frame size and d is the dimension of latent space respectively. Therefore, feee : HW + Rd, i.e., 2, = fenc (xt). Similarly, the decoder generator network, utilizes the latent feature generate the image. Therefore, fgen R² " i.e., & fz.) We borrow the architectures for both encoder and generator networks from (Denton & Fergus, 2018), where the frame encoder is convolutional layers from VGG16 network (Simonyan & Zisserman, 2015) and the generator is mirrored version of the encoder with pooling layers replaced with spatial up-sampling, a sigmoid output layer, and skip connections from the encoder network to reconstruct image. Cgen( (Xe, &x) ||x, &, is the reconstruction loss for frame auto-encoder. This auto-encoder is illustrated in Figure. (stage 1).
4.2 LSTM TEMPORAL DYNAMICS ENCODER
The first dynamics want encode of on-going action sequence, i.e., if action sequence is not completed yet, we want to continue generating future frames of the same sequence till finishes. This module fLSTM Rdd - Rdd has fully-comnected layer, followed by two LSTM layers with 256 cells each, and final output fully-connected layer. The output fully-comnected layer takes the last hidden state from LSTM (h++1) and outputs bc+1 after tanh(.) activation. Therefore, 24+1 = (2:) The training loss is given by LISTM- = Ei ||z++1 24+1/12 where T are the total number of frames (both past and future) used for training. This simple dynamics encoder, inspired by (Srivastava et al., 2015) and illustrated in Figure. 3 (stage 2), is effective and performs well on standard metrics.
4.3 GP TEMPORAL DYNAMICS ENCODER
Next, we want to learn the priors for potential future states by modeling the correlation between past and future states using GP layer. Given a past state, this temporal dynamics encoder captures the distribution over future states. This enables us to use the predictive variance to decide when to sample diverse outputs, and provides us with a mechanism to sample diverse future states. The input to the GP layer is and the output is mean and variance, which can be used to sample 2,+1- The loss function for GP dynamics encoder follows from eq. 3, LGp - (2++1, 2t).
Unlike LSTM, the GP layer does not have hidden or transition states; it only mod- els pair-wise correlations between past and future frames (illustrated in Figure. 3 (stage 2)). In this work, we the automatic relevance determination (ARD) kernel, denoted by k(2,z') = 2RDD exp (f wy(2, - =/)²). parameterized by learnable parameters ARD and (w))...,aa). This GP layer is implemented using GPyTorch (Gardner et al., 2018) (refer to $3).
5

---

Published as a conference paper at ICLR 2021
4.4 TRAINING OBJECTIVE
All three modules, frame auto-encoder and the LSTM and GP temporal encoders, are jointly trained using the following objective function:
Frame Auto-Encodea
LSTM Frame Goneration
GP Frame Generation
I
Σ
LIvG =
A1 Aaea (X,, x4) + Xz (xe, geen (2t)) + X Agen (Xt, Sgen (2:))
(4)
(LL1STM(2E+1, (++1) A5 Lapa (
LSTM Dynamics Encoder
GP Dynamies Encoder
where !A,...., As] are hyperparameters. There are three frame generation losses, each utilizing dif- ferent latent code: 2, from frame encoder, ε, from LSTM encoder, and 2, from GP encoder. In additional, there are two dynamics encoder losses, one each for L.STM and GP modules. Empiri- cally, we observed that the model trains better with higher values for \1, A2, X4, possibly because GP is only used to sample a diverse state and all other states are sampled from the LSTM encoder.
4.5 INFERENCE MODEL OF DIVERSE VIDEO GENERATOR (DVG)
During inference, we put together the three modules described above as follows. The output of the frame encoder z is given as imput both LSTM and GP encoders. The LSTM outputs Z4+1 and the GP outputs a mean and a variance. The variance of GP can be used to decide if we want to continue an on-going action or generate new diverse output, process we call trigger switch. we decide to stay with the on-going action, LSTM's output 2,+1 is provided to the decoder to generate the next frame. If we decide switch, we sample 2 from the GP and provide that as input to the decoder. This process is illustrated in Figure. 3 (stage 3). The generated future frame is used as input to the encoder to output the next Z441; this process repeated till we want to generate frames.
4.6 TRIGGER SWITCH HEURISTICS
An important decision for diverse future generation is when to continue the current action and when switch new action. We use the GP to switch new states. We use two heuristics to decide when to generate diverse actions: a deterministic switch and a GP trigger switch. For the deterministie switch, we do not use the variance of the GP as a trigger, and switch every frames. Each switch uses the sample from GP the next future state. This enables to have a consistent sampling strategy across generated samples. For the GP trigger switch, we compare the current GP variance with the mean of the variance of the last 10 states. If the current variance is larger than two standard deviations, we trigger a switch. This variable threshold allows the diverse video generator to trigger switches based on evidence, which can vary widely across samples.
EXPERIMENTS
Next, we describe the experimental setup, datasets we use, qualitative and quantitative results. We evaluate our models on four datasets and compare it with state-of-the-ar baselines. All models use 5 frames as context (past) during training and learn to predict the next 10 frames. However, our model is not limited to generating just 10 frames. All our models are trained using Adam optimizer.
KTH Action Recognition Dataset. The KTH action dataset (Schuldt et al., 2004) consists of video sequences of 25 people performing six different actions: walking, jogging, running, boxing, hand- waving, and hand-clapping. The background is uniform, and a single person is performing actions in the foreground. Foreground motion of the person in the frame is fairly regular.
BAIR pushing Dataset. The BAIR robot pushing dataset (Ebert et al., 2017) contains the videos of table mounted sawyer robotic arm pushing various objects around. The BAIR dataset consists of different actions given to the robotic arm to perform.
Human3.6N, Dataset. Human3.0n 3.M (lonescu et al., 2014) dataset consists of 10 subjects performing 15 different actions. We did not use the pose information from the dataset.
UCF Dataset. This dataset (Soomro et al., 2012) consists of 13,320 videos belonging to 101 differ- ent action classes. We sub-sample small dataset for qualitative evaluation on complex videos. Our

---

Published as a conference paper at ICLR 2021
KTH
BAIR
Human8.6 6M
0.55
SVG
01:55
0.54
RASP
1452
GPASSMM
VRNS
1152
VIRKN
6.97
LPIPS
0.55
TVV (0rmn
VIDELOW
Oun]
LPIPS
0.65
KYG GENRTT VENN
0.59
0.80
DVG [Dun)
0.52
0.94
(Ttegg))
10
10 Framus
30
Frunces
Francs
#
Figure 4: LPIPS Quantitative Results on KTH, BAIR, and Human3.6 6M datascts. All methods use the best matching sample out of 100 random samples. We used fixed trigger heuristic keep trigger point for each sample the same for our approach.
subset consists of 7 classes: Bench press, Bodyweight squats, Clean and Jerk, Pull-ups, Push-ups, Shotput, Tennis-Swing. Lunges and Fencing. The background of this dataset can bias our diversity evaluation metric. Therefore, we only include this dataset only for qualitative evaluation.
5.1 BASELINES
We compared our method with the following prior works. Further, wherever available, used either the official implementation or the pre-trained models for the baselines uploaded by their re- spective authors.
SVG-LP (Denton & Fergus, 2018): Stochastic Video Generation with a Learned Prior (SVG-LP) is VAE-based method that a latent space prior for generating video sequences. It outperforms other VAE-based approaches (e.g., SV2P (Babaeizadeh et al., 2017)). It also uses an LSTM-based dynamics model. This model similar ours except that we use a GP model the prior on future states to aid with modti outputs.
SAVP (Lee et al., Stocchstoii Adversarial Video Prediction (SAVP) VAE-GAN based approach that combines the best of both families of approaches. It also uses the LSTM dynamics model.
Conditional VRNN (Castrejón et al., 2019): Condition variational RNN leverages the flexibility of hierarchical latent variable models to increase the expressiveness of the latent space.
VidFlow (Kumar et al., 2019): VideoFlow model uses normalizing flow approach that enables direct optimization of the data likelihood.
We train model inspired by (AI-Shedivat et al., 2016), where the dynamics model is a GP, which uses recurrent kernels modeled by an LSTM. This method is closest to ours since utilizes the constructs of both GP and LSTM. However, they train single dynamics model and have no way to control the generation of future states.
GP-LSTM:
We refer to our model as Diverse Video Generator (DVG), which uses GP trigger switch. We also study heuristic switching at 15 and 35 frames for ablation analysis. We provide additional ablation analysis in the supplementary material, for models with RNNs and GRUs instead of LSTM.
5.2 METRICS
Evaluation of generated videos itself an open research problem with new emerging metrics. In this work, we tried our best cover all published metrics which have been used for evaluating future frame generation models.
Accuracy of Reconstruction. One way to evaluate a video generation model is to check how close the generated frames are to the ground-truth. Since the models are intended to be stochastic or diverse for variety in prediction, this achieved by sampling a finite number of future sequences from model and evaluating the similarity between the best matching sequence to the ground-truth and the ground truth sequence. Previous works (Denton & Fergus, 2018; Babacizadeh et al., 2017; Lee al., 2018) used traditional image reconstruction metries, SSIM and PSNR, measure the

---

Published as a conference paper at ICLR 2021
Table 1: Quantitative results KTH, BAIR, 1luman3.6M datasets. For the FVD Score, all methods use the best matching sample out 100 random samples and lower numbers are better. For the Diversity Score, we compute the score across 50 generated samples, for 500 starting sequences, and higher numbers are better.
Diversity Score (1)
Diversity Score (t)
Model
Trigger
FVD Score (4)
(frames: [10,25])
(frames: [25,40))
KTH
BAIR
Human3.66
KTH
Human3.6k
KTH
Human3.6M
SVG-LP SAVP
156.35
270.04
718.04
20.10
4.8
21.20
4.6
65.98
126.75
26.60
24.50
GP-LSTM
92.34
197.49
604.75
31.40
5.4
30.90
6.0
VidFlow
124.81
VRNN
67.26
134.81
523.45
32.50
5.6
31.80
5,9
DVG [ours]
@ @15335
65.69
123.08
479,43
48,30
9.3
46.20
9.0
DVG [ours]
GP
69.63
120.03
496.89
47.71
10.8
48.10
10.1
similarity between the generated samples and ground-truth. As shown by (Zhang et al., 2018; Doso- vitskiy & Brox, 2016; Johnson et al., 2016; Unterthiner al., 2018), these metrics are not suited for video evaluation because of their susceptibility towards perturbation like blurring, structural distor- tion, etc.. We include these metries in our supplementary material for the sake of completeness. We also evaluate the similarity of our generated sequences using recently proposed perceptual met- rics, VGG cosine similarity (LPIPS), and Frechet Video Distance (FVD) score. LPIPS Learned Perceptual Image Patch Similarity is metric developed to quantify the perceptual distance between two images using deep features. Several works (Zhang et al., 2018; Dosovitskiy & Brox, 2016; Johnson et al., 2016) show that this metric is much more robust to perturbation like distortion, blur- riness, warping. color shift, lightness shift, etc. Frechet Video Distance (FVD score) (Unterthiner et al., 2018) is deep metric designed to evaluate the generated video sequences. As is standard practice, all methods use the best matching sample out of 100 randomly generated samples.
Diversity of Sequences. Reconstruction accuracy of generated samples only implies that there is at least one generated sequence close to the ground-truth. However, these metrics do not capture the inherent multi-modal nature of the task. Aside from being able to generate samples close to ground truth, a video generation model should be able to represent diversity in its generated video sequences. Therefore, we propose metric inspired by (Villegas et al., 2017b) that utilizes a video classifier to quantify the diversity of generated sequences. The action classifier is trained on the respective datasets (KTH and Humans3.66 Note that we cannot utilize this metrie for the BAIR dataset since we do not have corresponding action labels. For the diversity metric, we use 500 starting sequences of 5 frames, sample 50 future sequences of 40 frames. We ignore the first generated frames since they are likely correlated with the ground-truth and correspond to the on- going sequences. Then, we evaluate the next two clips, made of frames [10, 25] and [25, 40]. Ideally, method that generate diverse sequences will generate more diverse clips as time progresses. For the Diversity Score, we compute the mean number of generated clips that changed from the on-going action as classified by the classifier. More concretely, if N is the total number of generated clips (N = 25000 for us ), e; is the ground-truth class, è is the predicted class, and 1(.) is the indicator function if the the parameter is correct, then Diversity Score 1 ΣN 1(c, # ê.).
N
5.3 RESULTS
5.3.1
QUANTITATIVE RESULTS (RECONSTRUCTION)
We report the quantitative results on KTH, BAIR, and Human3.6M, 3.6M datasets using the LPIPS metric in Figure. 4, and FVD metric in Table For comparisons with baselines in Figure. 4, we see that on the KTH and Human3.6M dataset, our approach generally performs on-par or better than the baselines. In fact, except for SAVP, all methods are very close to each other. For the Human3 6M dataset, the GP-LSTM baseline performs poorly, and all other methods are similar, with ours being better than others. On the BAIR dataset, we notice that our GP-LSTM baseline performs better, closely followed by approach. Again, SAVP performs worse all metrics. For the FVD metric (Table 1), variants of our approach achieve state-of oftth-or-theart the- results on all datasets. Note that using fixed trigger at frames 15 and 35 leads to better FVD scores for KTH and Human3.66 6M dataset,
8

---

Published as a conference paper at ICLR 2021
1-+3
25
6T
SVGLP
SVG.PP
DVG (Cuz) Hon Marchng
GUISTM
Stating
BVG (Dino
VINN
Dorre Sample
Mashing
SVG-LP
DVG (Cus))
Semple
Best Marchig
Figure 5: Qualitative Results BAIR (Icft), KTH (center), Human3.6k (right, top), and UCF (right, bottom) datascts. First row is the ground-truth video each figure (with the last frame of the provided 5 frames shown as "GT",, Every 5th frame is shown.
while GP trigger performs better for the BAIR dataset. All settings, except one, of our approach perform better than the baselines.
5.3.2 QUANTITATIVE RESULTS (DIVERSITY)
We report the quantitative results using the proposed diversity score in Table 1. We notice for the KTH dataset that SVG-LP/SAVP baseline change actions 20.1% %/26.6% of the time in the first clip and 21.2% of the time in the second clip. In comparison, our approach gives the diversity score of 48.53% and 48. 10% for the first and second clips, respectively. As can be observed, the GP trigger results in considerably higher diversity as the sequence progresses. On the Human3.66 6.M dataset, the difference between the scores of baselines and our methods is ~66. The overall score drop between the KTH and the Human 6M datasets on diversity metric can be accounted to actions performed in the videos are very distinct. Besides, cameras are placed far off from the person performing actions making it harder for the models to generate diverse samples.
We further analyze common action triggers for the GP trigger and notice that it separates the moving (walk, jog, run) and still (clap, wave, box) actions, and common action switches are within each cluster; e.g., walk + jog, wave <> clap. More analysis provided the supplementary.
5.3.3 QUALITATIVE RESULTS
Qualitative Results shown in Figure. 5. For KTH results in Figure. 5, we plot randomly selected sample for all methods. As we can see, SAVP and SVG-LP output average or blurry images after 20-30 frames, and method is able switch between action classes (for diverse sample using GP trigger). In supplementary, we show results on KTH with more than 100 sampled frames and best matching samples for basclines and ours. For the BAIR dataset, we show the best LPIPS results for all approaches; where we can see that our method generates samples much closer to the ground-truth. We also included a random sample with a fixed trigger at 15th frame, where we can see a change in the action. For the Human3.66 .6M dataset (after digital zoom), we can see that our best LPIPS sample matches the ground-truth person's 's pose closely as opposed to SVG-LP, demonstrating the effectiveness of our approach. Similar results observed for the UCF example. Note that due to manuscript length constraints, we have provided more qualitative results in the supplementary.
CONCLUSION
We propose method for diverse future video generation. We model the diversity in the potential future states using a GPP, which maintains priors on future states given the past and a probability dis- tribution over possible futures given particular sample. Since this distribution changes with more evidence, we leverage its variance to estimate when to generate from an on-going action and when to switch to new and diverse future state. We achieve state-of-the-art results for both reconstruction quality and diversity of generated sequences.
This work was partially funded by independent grants from Facebook AI, Office of Naval Research (N000141612713), and Defense Advanced Research Projects Agency (DARPA) SAIL-ON program (W91 LNF2020009). We also thank Harsh Shrivastava, Pulkit Kumar, Max Ehrlich, and Pallabi Ghosh for providing feedback the manuscript.
Acknowledgements.
9

---

Published as a conference paper at ICLR 2021
REFERENCES
Maruan Al-Shedivat, Andrew Gordon Wilson, Yunus Saatchi, Zhiting Hu, and Eric Xing. Learn- ing scalable deep kernels with recurrent structure, 2016."
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H. Campbell, and Sergey Levine. Stochastic variational video prediction, 2017. 1, 3,7
Apratim Bhattacharyya, Bernt Schiele, and Mario Fritz. Accurate and diverse sampling of sequences based on a "best of many" sample objective. 2018 JEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun 2018. doi: 10. 10.1109/cvp.22188.000 200.0000...11100088883. URL http: /tp/dxxxoo... doi. org/10 109 CVPR. 2018 00885.
Haoye Cai, Chunyan Bai, Yu-Wing Tai, and Chi-Keung Tang. and completion of human action sequences. Lecture Notes in Computer Science, pp. 374-390, 2018. ISSN 1611-3349.
Deep video generation, prediction
Lluís Castrejón, Nicolas Ballas, and Aaron C. Courville. Improved conditional vrnns for video prediction. CoRR, 12//1904.22655, 2019. URL
1216.
3,7
Francesco Cricri, Xingyang Ni, Mikko Honkala, Emre Aksu, and Moncef Gabhouj. Video ladder networks. CoRR, abs/1612.01156, 2016. URL http: // arxiv. orgg org/abs////eooooooeeo..rr//eeaer abs/ abs/1612... 01756. 3
Zhenwen Dai, Andreas Damianou, James Hensman, and Neil Lawrence. Gaussian process models with parallelization and gpu acceleration, 2014. 4
Emily Denton and Rob Fergus. Stochastic video generation with learned prior, 2018. 1, 2, 3, 5,7
Alexey Dosovitskiy and Thomas Brox. Generating images with perceptual similarity metrics based on deep networks, 2016.
Frederik Ebert, Chelsea Finn, Alex Lee, and Sergey Levine. Self-supervised visual planning with temporal skip connections, 2017. 6
N. Elsayed, A. S. Maida, and M. Bayoumi. Reduced-gate convolutional Istm architecture for next- frame video prediction using predictive coding. 2019 International Joint Conference on Neural Networks (IJCNN), pp. 1-9, July 2019. doi: 10. 111100/JJCNN.221988844444
Yarin Gal, Mark van der Wilk, and Carl E. Rasmussen. Distributed variational inference in sparse gaussian process regression and latent variable models, 2014.
Hang Gao, Huazhe Xu, Qi-Zhi Cai, Ruth Wang. Fisher Yu, and Trevor Darrell. Disentangling propagation and generation for video prediction. ArXiv, abs/1812.00452, 2018.
Jacob R Gardner, Geoff Pleiss, David Bindel, Kilian Q Weinberger, and Andrew Gordon Wilson Gpytorch: Blackbox matrix-matrix gaussian process inference with gpu acceleration. In Advances in Neural Information Processing Systems, 2018. 4.5
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farles Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014.
J. Hardy, F. Havlak, and M. Campbell. Multiple-step prediction using a two stage gaussian process model. 2014 American Control Conference, pp. 1443-3449 2014. doi: 109/AC/4/.. 6859020. 3
James Hensman, Nicolo Fusi, and Neil D. Lawrence. Gaussian processes for big data, 2013. 4
Zhihang Hu and Jason Wang. A novel adversarial inference framework for video prediction with action control. In The IEEE International Conference on Computer Vision (ICCV) Workshops, Oct 2019. 2,
Catalin lonescu, Dragos Papava, Vlad Olaru, and Cristian Sminchiseseu. Human3.6m 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE Transactions Pattern Analysis and Machine Intelligence, 1325)3333,-1339, jul 2014.
10

---

Published as a conference paper at ICLR 2021
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to porsetto-inne -nage-to-image translation with conditional adversarial networks. 2017 IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 5967-5976, 2016.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. Lecture Notes Computer Science, pp. 694-711, 2016. ISSN 1611-3349, 8
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. CoRR, abs/1312.6114, 2013. 3,3
Manoj Kumar, Mohammad Babacizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine, Laurent Dinh, and Durk Kingma. Videoflow: A conditional flow-based model for stochastic video gener- ation, 2019. 7
Alex X. Lee, Richard Zhang, Frederik Ebert, Picter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint arXiv:1804.01523, 2018. 1,
Chaochao Lu, Michael Hirsch, and Bernhard Schölkopf. Flexible spatio-temporal networks for video prediction. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2137-2145, 2017. 2,
Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error, 2015.
K. Moon and V. Pavlovic. 3d human motion tracking using dynamic probabilistic latent semantic analysis. In 2008 Canadian Conference on Computer and Robot Vision, pp. 155-162, 2008. doi: 20.1109//
Mare Oliu, Javier Selva, and Sergio Escalera. Folded recurrent neural networks for future video prediction. CoRR, abs/1712.00311, 2017. URL BBK.tttpp//aaxxv..or////////22/2 wttp://wwww...aa///.......... 1,3
Carl Edward Rasmussen. Gaussian processes for machine learning. MIT Press, 2006. 1,
Masaki Saito, Eiichi Matsumoto, and Shunta Saito. Temporal generative adversarial nets with sin- gular value clipping. 2017 IEEE International Conference Computer Vision (ICCV), Oct 2017. doi: 10.1 10.11009/ccy22017..300 URL http: /xx wwtp.o.gg/...110//00C/0. 2w1.///cc............... 208..308. 2,
Tim Salimans. Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pp. 2234-2242 2016.
M. P. Sampat, Z. Wang, S. Gupta, A. C. Bovik, and M. K. Markey. Complex wavelet structural similarity: A new image similarity index. IEEE Transactions Image Processing, 18(11):2385- 2401, Nov 2009. doi: 10.1 11999TT2220992225522...
C. Schuldt, Laptev, and B. Caputo. Recognizing human actions: local svm approach. In Proceedings of the 17th International Conference Pattern Recognition, 2004. ICPR 2004., volume 3, pp. 32-36 Vol.3, Aug 2004. doi: 10.1 20.1109//CP..204.1133 1334462.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition, 2015. 5
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset of 101 human actions classes from videos in the wild, 2012. 6
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov. Unsupervised learning of video representations using Istms. In Proceedings of the 32Nd International Conference on Interna- tional Conference on Machine Learning Volume 37, ICML'IC...... pp. 843-852. 852. JMLR.org, 2015. URL hwtptttt//cce //1l. wwmm.org/eeee cttggiiiatiiiin..mm cfm? d= 30452099.. 3,5

---

Published as a conference paper at ICLR 2021
Michalis Titsias. Variational learning of inducing variables in sparse gaussian processes. In David van Dyk and Max Welling (eds.), Proceedings of the Twelth International Conference on Artificial Intelligence and Statistics, volume 5 of Proceedings of Machine Learning Research, pp. 567-574, Hilton Clearwater Beach Resort, Clearwater Beach, Florida USA, 16-18 Apr 2009. PMLR.
Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. 2018 IEEE/CVE Conference on Computer Vision and Pattern Recognition, Jun 2018. doi: 10.1 109/cyrr.001800000
URL http: bestp://xx.de doo..i.o.......oooo.....o....... www..doi.org/...o..oo .wr./orgg/.1.
2018. 00118.0016800 65. 3,
Thomas Unterthiner, Sjoerd Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric and challenges, 2018. 2,
Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lec. Decomposing motion and content for natural video sequence prediction. CoRR, fbs//1006.00033 2017a. URL http: //aaxiiv.orgg///oe abs 11006 08033.
Ruben Villegas, Jimei Yang, Yuliang Zou, Sungryull Sohn, Xunyu Lin, and Honglak Lec. Learning to generate long-term future via hierarchical prediction, 2017b. 2, 3, 8
Ruben Villegas, Arkanath Pathak, Harini Kannan, Dumitru Erhan, Quoc V. Le, and Honglak Lee. High fidelity video prediction with large stochastic recurrent neural networks, 2019.
Carl Vondrick and Antonio Torralba. Generating the future with adversarial transformers. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2992-3000, 2017. 2,
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. ArXiv, abs/1609. (00022, 2016.
Jacob Walker, Abhinav Gupta, and Martial Hebert. Patch to the future: Unsupervised visual predic- tion. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 3302-3309, 2014. 3
Jacob Walker, Kenneth Marino, Abhinav Gupta, and Martial Hebert. The pose knows: Video fore- casting by generating pose futures. In International Conference on Computer Vision, 2017. 3
M. Wang, D. Fleet, and Hertzmann. Gaussian process dynamical models for human motion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(2):283-298, 2008. doi: 10.1109/TPAM..2001.1177
Yunbo Wang, Lu Jiang, Ming-Hsuan Yang. Li-Jia Li, Mingsheng Long, and Li Fei-Fei. Eidetic 3d LSTM: model for video prediction and beyond. International Conference on Learning Representations, 2019. URL https: ooppreeveeeeeeeeeeeeeeeeee net
2,
Nevan Wichers, Ruben Villegas, Dumitru Erhan, and Honglak Lee. Hierarchical long-term video prediction without supervision, 2018. 3
Andrew Gordon Wilson and Hannes Nickisch. Kernel interpolation for scalable structured gaussian
processes (kiss-gp). 2015. 4
Andrew Gordon Wilson, Christoph Dann, and Hannes Nickisch. Thoughts massively scalable gaussian processes, 2015.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P. Xing. Stochastic variational deep kernel learning, 2016.
Tianfan Xue, Jiajun Wu, Katherine L Bouman, and William T Freeman. Visual dynamics: Proba- bilistic future frame synthesis via cross convolutional networks. In Advances In Neural Informa- fion Processing Systems, 2016.
12

---

Published as a conference paper at ICLR 2021
Jenny Yuen and Antonio Torralba. A data-driven approach for event prediction. In European Con- ference on Computer Vision, pp. 707-720. Springer, 2010.
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features a perceptual metrie. 2018 IEEE/CVF Conference on Computer Vision and Puttern Recognition, Jun 2018. doi: 10. 10.1109/cpr.2218.0000 8.0068. URL http:/ htttp://dx... /dx. doi. doi.orq/
10. www.../1................ 2018 80068. 8
Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei Efros, Oliver Wang, and Eli Shechtman. Toward multimodal image-to-image translation, 2017.
13

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 ABLATION STUDIES FOR TEMPORAL DYNAMICS ENCODER
We perform ablation studies on our model by trying different variants of recurrent modules for our temporal dynamics encoder networks. These models are: DVG-RNN, our model with an RNN dynamics encoder; DVG-GRU, with an RNN dynamics encoder with GRU units.
Figure. A.1 shows ablation analysis for different variants of our approach. On the KTH dataset, different dynamics models (RNN, GRU, LSTM) all perform the same. On the BAIR dataset, RNN perform poorly and LSTM performs the best among the three. On Human3.6M dataset RNN per- forms higher than our LSTM and GRU models. On the FVD metric in Table A, all variants of our approach perform better than all baselines. In approaches, GRU dynamics model performs better on KTH and LSTM performs better on Human3.66 6. and BAIR dataset.
Average LPIPS on KTTH
Average LPIPS BAIR
Average LPIPS on HUMAN 3.6M
0,925
0.450
Ouns RNN
Oun: GRU
Curs 1STM
0.50
Curs GRU
RNN
ESTM
0.965
0.900
Owas LSTM
Oun GRU
Ous RNN
0,900
0.58
0:975
0.875
0.54
0.970
0.850
0.003
0.825
0.50
0,800
0.78
0,565
10
20
30
Frames #
Frames
Frarnes #
Figure A.1: Ablation results on KTH, Human3.6k 6M6 and BAIR dataset using variants of temporal dynamics model in our method. We report best LPIPS metrie. All methods use the best matching sample out of 100 random samples. We used fixed trigger to keep trigger point for each sample the same. On KTH, all temporal dynamics models have similar performance; and on BAIR, our LSTM model have hest performance.
Table A.1: Quantitative results on KTH, BAIR, Human3.66 6M datasets. For the FVD Score, all the ablation methods use the best matching sample out of 100 random samples and lower numbers are better. For the Diversity Score, we compute the score across 50 generated samples, for 500 starting sequences, and higher numbers are better.
Diversity Score (t)
Diversity Score (1)
Model
Dynamics
Trigger
FVD Score (4)
(frames: [10,25])
(frames: (25,40))
KTH
BAIR
Human3.00 66M
KTH
Human3 6M
KTH
Human3.6M 3.6M
DVG [ours]
LSTM
1-1,35
65
123.08
479.43
48.
9.3
46,20
9.0
DVG [ours]
GRU
@15.35
64.89
124.38
485.00 96
48.53
8.5
44,23
9.1
[ours]
RNN
46.60
7.6
41.50
Post frames das box
86
2.9
3.3
23
2A
3.4
A.2 ANALYSIS: CHANNGSS IN ACTION AFTER GP TRIGGERING
2.6
74
19
0.77
23
0.85
KTH action dataset comprises of action classes: walking, running, jogging, waving, clapping, and boxing. On abstract level, cluster these actions into two categories moving actions and still actions. From the Figure. A.2, it interesting observe that our GP triggering model captures the future trajectories of the videos and clusters them into these two categories moving actions and still actions. Common action switches that are to be expected can be observed from the Figure. A.2; for example, walk and jog, wave and clap interchange frequently after triggering. Still actions seldom change to moving actions.
5.8
5:24
0.24
25
72
3.2
0.63 0.084
0.42
31
60
7.9
box: cep
ware
[50
737 kalc
Future
Figure A.2: Changes in ac- tion from past frames to fu- ture frames on KTH datasct. To- of 25,000 generated videos were used to calculate percentage change shown in the above fig- ure.
14

---

Published as a conference paper at ICLR 2021
A.3 SSIM AND PSNR RESULTS
We evaluated our generated video sequences using the tradition metrics like structural similarity index (SSIM) and peak signal-to-noise ratio (PSNR) for comparison with previous baselines which reported these metrics. We trained all models on 64 x 64-size frames from the KTH, Human3. M.MM, and BAIR datasets. We used the standard training practice of using frames as context (or past) and the model have to predict the next 10 frames. For all methods, SSIM and PSNR is computed by drawing 100 samples from the model for each test sequence and picking the best score with respect to the ground truth. We emphasize that these results are only for completeness and we hope that the community will stop relying on such reconstruction metrics for video prediction.
Results are reported in Figure. A.3 represent the evaluation plots for traditional metrics on KTH, BAIR, and Human3.6M dataset. We follow the experimental setups from the baseline papers.
Averago SSIM KTII
Average SSIM on BAIR
Averagy SSTM on HUMMAN 3.M
0.00
0.30
0.32
SAVP
8VG
GP-15TM
IVG Xiee]
0359
203
SAVP
GINLEES
DNG Donl
1.00
SVC GPARTM
DVG (Duas) (Tigge)
0.300
0197
610-
0.500
4770
0.775
0.30
0.01
Frumes
Frames
Averngo PSNN KTII
Average PSNR on. BAIR
Average PSNR on HUMAN 3.6M
34-
20-
SVC
SAY
31-
SVG
GP-AATM
32-
DVG Klan
Xard
1990 [Dend]
(THozar)
33-
(Mragrr)
(1riggn)
56-
-3-
56-
25
54-
15
20-
25
31
Fromes
Piames
Promess
Quantitative results on KTH, BAIR and Human3.6k 6M dataset. We report average SSIM and PSNR metries. All methods use the hest matching sample out of 100 random samples. We used fixed trigger to keep trigger point for each sample the same.
Figure A.3:
A.4 QUALITATIVE RESULTS
It can be observed from Figure. A.4 that after 15th frame SVG-LP is stuck in the same pose while after 35th frame SAVP starts distorting the human. However, our method (DVG) consistently gener- ates frames that diverse and distortion free for longer period of time. Similarly, in Figure. A.9 it can be observed that after 30th frame SVG-LP and SAVP start generating subpar frames while our method is able to generate visually acceptaale sequences for longer term. Few additional qualitative results on the BAIR dataset are provided in Figures. A.5-A.6, and on the Human 3.6M dataset in Figures. A.7-A.8.
15

---

Published as a conference paper at ICLR 2021
T-5
10
15
20
25
40
45
SS
70
80
55
90
95
103
106
SAVP
SVG-LP
O
Bost Matching
Oun
Diverse Samplel
Diverse Sumple2
Figure A.4: KTH dataset: Qualitative comparison of the generated video sequences (every 5thh frame shown).
First row is the ground-true -rruthhhrrhhhho video (with last frame of the provided 5 frames is shown)
15
25
T=5
10
15
20
25
30
61
SVG LP
SVG LP
SAVP
SAVP
Ours
Ours
Best Matching
Best Matching
Ous
Ours
Trgger 15
Trigger 115
Figure A.5: Qualitative results on BAIR dataset. We show the best LPIPS samples out samples for all methods.
Figure A.6: Qualitative results BAIR dataset. We show the best L.IPS samples out of 100 samples for methods.
T-
20
GT
SVG-LP
SVG.I
GPM1STM
GELSTTM
Match
Best
Ous
Divense Sumple
hautstic tigggeting
Figure A.7: Human3.6M dataset: Qualitative com- parison of the generated video sequences (every 5th frame shown). row is the ground-truth video (with last frame of the provided frames is shown)
Figure A.8: Human3.6M dataset: Qualitative parison of the generated video sequences (every 5t frame shown). First row is the ground truth vidco (with last frame of the provided 5 frames is shown)
16

---

Published as a conference paper at ICLR 2021
T-5
15
35
40
45
60
GT
SAVP
4
1
SVG-LP
Ours
Best Matching
Our
Diverse Samplel
Ours
Diverse Sample2
Figure A.9: KTH dataset: Qualitative comparison of the generated video sequences (every 5th frame shown). First the ground-truth vidco (with last frame of the provided frames shown)
A.5 GAUSSIAN LAYER SPECIFICS
As mentioned in the paper, GPytorch was used for our GP layer implementation. We utilized large-scale variational GP implementation of GPytorch for our multi-dimensional GP regression problem of learning to predict the variance over the future frames in the latent space. For variational GP implementation, 40 inducing points were randomly initialized and learned during the training of GP. We used a RBF kernel along with gaussian likelihood for our GP layer. For optimization of our GPLayer, we employed stochastic optimization technique (Adam optimizer) minimize the variational ELBO for GP.
A.6 I3D NETWORK ARCHITECTURE FOR ACTION CLASSIFIER
For our diversity metric mentioned in 85, we utilized the standard dinetics-pretraiie I3D action recognition classifier. The input to the action classifier is a 15 frames clip and each frame has size of x 64. The action classifier attains accuracy close 100% for KTH dataset and is above 90% accuracy for human3.66 6m dataset.
17


Reviews:
[{"review_id": "xt6RU6emoaT", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work, the authors propose to apply Gaussian Processes to generate future video frames with high diversity. Specifically, they use variance of GP prediction as a trigger to control when we should switch to a new action sequence.\n\nStrength \n\n1 The paper is written well, and the organization is OK\n\n2 The idea of using GP for video generation sounds interesting\n\nWeakness\n\n1 The way of using GP is kind of straightforward and naive. In the GP community, dynamical modeling has been widely investigated, from the start of Gaussian Process Dynamical Model in NIPs 2005. \n\n2 I do not quite get the modules of LSTM Frame Generation and GP Frame Generation in Eq (4). Where are these modules in Fig.3 ? The D in the Stage 3? Using GP to generate Images? Does it make sense? GP is more suitable to work in the latent space, is it? \n\n3 The datasets are not quite representative, due to the simple and experimental scenarios. Moreover, the proposed method is like a fundamental work. But is it useful for high-level research topics, e.g.,  large-scale action recognition, video caption, etc?", "extra_scores": {}, "extra_text": {}}, {"review_id": "_s6FkEfaiat", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes a future frame prediction framework where the video generation can transition between different actions using a Gaussian process trigger. The framework consists of three components: an encoder which encodes the frame to a latent code, an LSTM which predicts the next latent code given the current one, and a Gaussian process which samples a new latent code. The framework can decide whether to switch to the next action by adopting the new latent code, depending on the number of frames passed or the variance of Gaussian.\n\nStrengths:\nThe paper is easy to follow overall. The usage of Gaussian process to trigger the transition to the next action is reasonable and intuitive. Quantitative evaluations show that the method outperforms existing works for both reconstruction and output diversity for various datasets.\n\nWeaknesses and comments:\nThere are quite a few typos in the writing, especially toward the latter part of the paper. I’d encourage the authors to do a thorough check to ensure the paper is typo-free.\nIt seems switching actions at some fixed number of frames beats using the Gaussian variance for FVD, which is quite surprising. Can the authors provide some insights? Is it due to some inherent nature of FVD, or there’s still some room for improvement for the choosing criteria?\nHow important is the heuristic of changing states when using GP? Currently it is triggered when the variance is larger than two standard deviations. How will it affect the performance if a different threshold is used?\nThere’s a mistake in Table 1. The diversity score for DVG@15,35 is the best for KTH frames [10,25] (48.30), but DVG GP is bolded (47.71). This might also be an interesting point to discuss about why fixed number of frames performs better than GP.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Zi0WDzjgDvZ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### SUMMARY\n\nThe authors propose to use a Gaussian Process (GP) to model the uncertainty of future frames in a video prediction setup. In particular, they employ a GP to model the uncertainty of the next step latent in a latent variable model. This allows them to use the GP variance to decide when to change an \"action sequence\", corresponding to a deterministic dynamics function implemented using an LSTM. \n\n### STRENGTHS AND WEAKNESSES\n\n[+] Empirical results\n\n[+] Well-motivated model\n\n[+] Clear presentation\n\n[-] Experimental section could be improved (missing baselines in some tables, results seem to differ from those in the literature)\n\n### DETAILED COMMENTS\n\nThe paper proposes a novel approach for video prediction. Following the standard latent variable model setup used by many VAE-based video prediction models, the authors propose to use a GP to model the uncertainty in the latent space while also learning a deterministic dynamics model (LSTM) on this latent space. Then the GP is used to decide when a future frame has high uncertainty, and in those cases multiple latents can be sampled from the GP. In general the paper is clear and well-written.\n\nThe experimental section could be improved. In particular, more details about how the comparison to some baselines was made would be appreciated. For example,  the results for the VRNN model in Figure 4 and 5 do not follow the results in the literature, where it outperforms SVG and SAVP, and its unclear whether its due to an architectural change, suboptimal hyperparameters, or a different reimplementation. Further this model is missing from some other comparisons such as Table 1. For SAVP the results for Figure 4 seem much worse than those reported in the original paper. On the other hand, the authors did some ablation experiments and included different metrics to analyze the performance of their method.\n\n### SCORE\nI vote for accepting the paper. The model formulation is clear, well-motivated and novel. The results are positive and overall it seems like a valid alternative to current approaches that will be of interest to the video prediction community. I would encourage the authors to provide a more thorough experimental section.\n\n### POST-REBUTTAL UPDATE\nAfter reading the other reviews and the authors' rebuttal, I stand by my rating of 6.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All three reviewers agree on accepting the paper and think that the proposed approach will be of interest for those working in vdieo prediction.  The authors are asked to include the extra discussion with R3 as part of the paper and include the proposed changes by R2 to provide more thorough experimentation.  The paper is recommended as a poster presentation.
Decision: Accept (Poster)