VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generation
Authors: Manoj Kumar
Mohammad Babaeizadeh
Dumitru Erhan
Chelsea Finn
Sergey Levine
Laurent Dinh
Durk Kingma
Venue:  (2020)
Keywords: Video generation
flow-based generative models
stochastic video prediction

Abstract: Generative models that can model and predict sequences of future events can, in principle, learn to capture complex real-world phenomena, such as physical interactions. However, a central challenge in video prediction is that the future is highly uncertain: a sequence of past observations of events can imply many possible futures. Although a number of recent works have studied probabilistic models that can represent uncertain futures, such models are either extremely expensive computationally as in the case of pixel-level autoregressive models, or do not directly optimize the likelihood of the data. To our knowledge, our work is the first to propose multi-frame video prediction with normalizing flows, which allows for direct optimization of the data likelihood, and produces high-quality stochastic predictions. We describe an approach for modeling the latent space dynamics, and demonstrate that flow-based generative models offer a viable and competitive approach to generative modeling of video.

Published as a conference paper at ICLR 2020
VIDEOFLOW: A CONDITIONAL FLOW-BASED MODEL FOR STOCHASTIC VIDEO GENERATION
Manoj Kumar", Mohammad Babaeizadeh, Dumitru Erhan,
Chelsea Finn, Sergey Levine, Laurent Dinh, Durk Kingma
Google Research, Brain Team
(mechcoder, mbz, dumitru, che seaf,, se,,sllvvve slevine, laurentdinh, durk) @google com
ABSTRACT
Generative models that can model and predict sequences of future events can, in principle, learn to capture complex real-world phenomena, such as physical interactions. However, a central challenge in video prediction is that the future is highly uncertain: a sequence of past observations of events can imply many possible futures. Although a number of recent works have studied probabilistic models that can represent uncertain futures, such models are either extremely expensive computationally as in the case of pixel-level autoregressive models, do not directly optimize the likelihood of the data. To our knowledge, our work is video prediction with normalizing flows, which
the first to propose allows for direct optimization of the data likelihood, and produces high-quality stochastic predictions. We describe an approach for modeling the latent space dynamics, and demonstrate that flow-based generative models offer a viable and competitive approach to generative modeling of video.
INTRODUCTION
Exponential progress in the capabilities of computational hardware, paired with relentless effort towards greater insights and better methods, has pushed the field of machine learning from relative obscurity into the mainstream. Progress in the field has translated to improvements in various capabilities, such as classification images (Krizhevsky et al., 2012), machine translation (Vaswani et al., 2017) and super- super-human game-playing agents (Mnih al., 2013; Silver et al., 2017), among others. However, the application of machine learning technology has been largely constrained to situations where large amounts of supervision is available, such in image classification or machine translation, or where highly accurate simulations of the environment are available to the learning agent, such in game-playing agents. An appealing alternative supervised learning is to utilize large unlabeled datasets, combined with predictive generative models. In order for a complex generative model be able effectively predict future events, must build up an internal representation of the world. For example, a predictive generative model that can predict future frames in video would need to model complex weal-world phenomena, such as physical interactions. This provides an appealing mechanism for building models that have rich understanding of the physical world, without any labeled examples. Videos of real-world interactions are plentiful and readily available, and a large generative model can be trained large unlabeled datasets containing many video sequences, thereby learning about a wide range of real-world phenoma. Such a model could be useful for learning representations for further downstream tasks (Mathieu al., 2016), could even be used directly in applications where predicting the future enables effective decision making and control, such as roboties (Finn et al., 2016). A central challenge in video prediction is that the future is highly uncertain: a short sequence of observations of the present can imply many possible futures. Although a number of recent works have studied probabilistic models that can represent uncertain futures, such models are either extremely expensive computationally (as in the case of pixel-level autoregressive models), or do not directly optimize the likelihood of the data.
In this paper, we study the problem of stochastic prediction, focusing specifically on the case of conditional video prediction: synthesizing raw RGB video frames conditioned on short context
A majority of this work was done part of the Google AT Residency Program.

---

Published as a conference paper at ICLR 2020
of past observations (Ranzato al., 2014; Srivastava et al., 2015; Vondrick et al., 2015; Xingjian et al., 2015; Boots et al., 2014). Specifically, we propose a new class of video prediction models that can provide exact likelihoods, generate diverse stochastic futures, and accurately synthesize realistic and high-quality video frames. The main idea behind our approach is to extend flow-based generative models (Dinh al., 2014; 2016) into the setting of conditional video prediction. To our knowledge, flow-based models have been applied only to generation of non-temporal data, such as images (Kingma & Dhariwal, 2018), and audio sequences (Prenger et al., 2018). Conditional generation of videos presents its own unique challenges: the high dimensionality of video sequences makes them difficult to model as individual datapoints. Instead, we learn a latent dynamical system model that predicts future values of the flow model's latent state. This induces Markovian dynamics on the latent state of the system, replacing the standard unconditional prior distribution. We further describe practically applicable architecture for fow-based video prediction models, inspired by the Glow model for image generation (Kingma & Dhariwal, 2018), which we call VideoFlow.
Our empirical results show that VideoFlow achieves results that are competitive with the state-of- the-art in stochastic video prediction on the action-free BAIR dataset, with quantitative results that rival the best VAE-based models. VideoFlow also produces excellent qualitative results, and avoids many of the common artifacts of models that use pixel-level meen-ssqaredd blurry predictions), without the challenges associated with training adversarial models. Compared to models based on pixel-level autoregressive prediction, VideoFlow achieves substantially faster test-time image synthesis 1, making much more practical for applications that require real-time prediction, such as robotic control (Finn & Levine, 2017). Finally, since VidcoFlow directly optimizes the likelihood of training videos, without relying on variational lower bound, we can evaluate its performance directly in terms of likelihood values.
for training (e.g..
2 RELATED WORK
Early work on prediction of future video frames focused on deterministic predictive models (Ranzato et al., 2014: Srivastava et al., 2015; Vondrick et al., 2015; Xingjian et al., 2015; Boots et al., 2014). Much of this research on deterministic models focused architectural changes, such as predicting high-level structure (Villegas et al., 2017b), energy-based models (Xie et al., 2017), generative cooper- ative nets (Xie al., 2020), ABPTT (Xie et al., 2019), incorporating pixel transformations (Finn et al., 2016; De Brabandere et al., 2016; Liu et al., 2017) and predictive coding architectures (Lotter et al., 2017), as well as different generation objectives (Mathieu et al., 2016; Vondrick Torralba, 2017; Walker et al., 2015) and disentangling representations (Villegas et al., 2017a; Denton & Birodkar, 2017). With models that can successfully model many deterministic environments, the next key challenge is to address stochastic environments by building models that can effectively reason over uncertain futures. Real-world videos are always somewhat stochastic, either due to events that are inherently random, events that are caused by unobserved or partially observable factors, such as off-screen events, humans and animals with unknown intentions, and objects with unknown physical properties. In such cases, since deterministic models can only generate one future, these models either disregard potential futures or produce blurry predictions that are the superposition or averages of possible futures.
A variety of methods have sought to overcome this challenge by incorporating stochasticity, via three types of approaches: models based on variational auto-encoders (VAEs) (Kingma & Welling, 2013; Rezende et al., 2014), generative adversarial networks (Goodfellow et al., 2014), and autoregressive models (Hochreiter & Schmidhuber, 1997; Graves, 2013; van den Oord et al., 2016b;e; Van Den Oord et al., 2016).
Among these models, techniques based on variational autoencoders which optimize an evidence lower bound on the log- likeeliooood have been explored most widely (Babaeizadeh et al., 2017: Denton & Fergus, 2018: Lee et al., 2018; Xue al., 2016: Li al., 2018). To our knowledge, the only prior class of video prediction models that directly maximize the log-likelihode of the data are auto- regressive models (Hochreiter & Schmidhuber, 1997; Graves, 2013; van den Oord et al., 201 2016;;e; Van Den Oord et al., 2016), that generate the video one pixel at time (Kalchbrenner et al., 2017). However, synthesis with such models is typically inherently sequential, making synthesis substantially
We generate 64x64 videos of 20 frames in less than 3.5 seconds on a NVIDIA P100 GPU as compared to the fastest autoregressive model for video (Reed et al., 2017) that generates frame every seconds
2

---

Published as a conference paper at ICLR 2020
Figure 1: Left: Multi-scale prior The flow model uses multi-scale architecture using several levels of stochastic variables. Right: Autoregressive latent-dynamic prior The input at cach timestep x. is encoded into multiple levels stochastic variables (z, ,...,,, .2. (.) model those levels through sequential process II. p(z) ()) (21).
inefficient modern parallel hardware. Prior work has aimed speed up training and synthesis with such auto-regressive models (Reed et al., 2017; Ramachandran et al., 2017). However, (Babaeizadeh al., 2017) show that the predictions from these models are sharp but noisy and that the proposed VAE model produces substantially better predictions, especially for longer horizons. In contrast to autoregressive models, we find that our proposed method exhibits faster sampling, while still directly optimizing the oiglliieldhood and producing high-quality long-term predictions.
PRELIMINARIES: FLOW-BASED GENERATIVE MODELS
Flow-bused generative models (Dinh et al., 2014; 2016) have a unique set of advantages: exact latent- variable inference, exact log-likelihood evaluation, and parallel sampling. In flowwow generative models (Dinh al., 2014; 2016), we infer the latent variable corresponding to a datapoint x, by transforming x through composition of invertible functions f = fi o f2 .... o fę- We assume a tractable prior (z) over latent variable z, for eg. Logistic a Gaussian distribution. By constraining the transformations to be invertible, can compute the log-likelihood of x exactly using the change of variables rule. Formally,
K
logPe(x) = log po (z) + log det det(d///dh;-:)))
(1)
where h, x. h, (h; 1). hæ and det(dh, is the Jacobian determinant when h, is transformed to h; by f,. We learn the parameters of f fą by maximizing the log-likelihood, ie Equation (1), over training set. Given g f-1, now generate sample & from the data distribution, by sampling z (z) and computing x = g(z).
PROPOSED ARCHITECTURE
We propose generative flow for video, using the standard multi-scale flow architecture in (Dinh et al., 2016; Kingma Dhariwal, 2018) as a building block. In our model, we break up the latent space into separate latent variables per timestep: t is an invertible transformation of a corresponding frame of video: x, ge (2,). Furthermore, like in (Dinh et al., 2016: Kingma & Dhariwal, 2018), we use multi-scale architecture for (z))) (Fig. I): the latent variable z, is composed of stack of multiple levels: where each level encodes information about frame x, particular scale: 2, {z( (f) )
{z, (2 The latent variable 2, timestep
component 2.(") per level.
3

---

Published as a conference paper at ICLR 2020
MULTI-SCALE 4.1 INVERTIBLE MULTI- SCALE ARCHITECTURE
We first briefly describe the invertible transformations used in the multi-scale architecture to infer {z, (1))z / fo (xx) and refer to (Dinh et al., 2016; Kingma & Dhariwal, 2018) för more details, For convenience, we omit the subscript in this subsection. We choose invertible transformations whose Jacobian determinant in Equation| 1 is simple to compute, that is triangular matrix, diagonal matrix or a permutation matrix as explored in prior work (Rezende & Mohamed, 2015; Deco & Brauer, 1995). For permutation matrices, the Jacobian determinant is one and for triangular and diagonal Jacobian matrices, the determinant is simply the product of diagonal terms.
Actnorm: We apply learnable per-channel scale and shift with data data-dependent initialization.
Coupling: We split the input equally across channels to obtain 31 and y2. We compute 22 f(y1) * y/2 ( where and g are deep networks. We concat y1 and 22 across channels.
SoftPermute: We apply 1xl convolution that preserves the number of channels.
Squeeze: We reshape the input from H x W x C to H/2 x W/2 x 4C' which allows the flow to operate larger receptive field.
We infer the latent variable z(!) at level I using:
Flow(y) Coupling/Sot/eemmute/cctmm///// N Flow((y) = Splic(Flo((Squeeze(()))
(2)
(3)
(h()), z²) Flow Flow)(((2oo) (>(-1))
where N is the number of steps of flow. In Equation (3), via Split, split the output of Flow equally across channels into h (>1), the input to Flowa+1)() and 20), the latent variable at level I. We, thus enable the flows at higher levels to operate on lower number of dimensions and larger scales. When 1 = 1, h(3-11) is just the input frame and for L we omit the Split operation. Finally, our multi-scale architecture fo (xx) is a composition of the flows at multiple levels from / 1.... from which we obtain our latent variables i.e {z, (5) 1==1
4.2 AUTOREGRESSIVE LATENT DYNAMICS MODEL
We use the multi-scale architecture described above infer the set of corresponding latent variables for each individual frame of the video: {z, (1) ) = fx(x;); ); see Figure for an illustration. As in
Equation (1), need choose form of latent prior po (z). We use the following autoregressive factorization for the latent prior:
T
(z) II t=1
po po 12 (Z21)
(5)
where z<イ denotes the latent variables of frames prior to the t-th timestep: {z,,..,, Zt-1}. We specify the conditional prior pe [z (Ze)) as having the following factorization:
po(z|/ze1) = II (z()
(1) (1) 12 (2 2i ((21))
(6)
(1) where z is the set of latent variables previous timesteps and the same level I, while 2; is the set of latent variables at the same timestep and at higher levels. See Figure 1 for graphical illustration of the dependencies.
(>1)
let each po (z. 2 ui (>1))
conditionally factorized Gaussian density:
po (2. 122e, 2 (21)) (1((())
-N(2 0) where (1,, log NNoo 2(21))
(7)
(8)
4

---

Published as a conference paper at ICLR 2020
Model
Fooling rate
SAVP-VAB
16.4 %
VideoFlow
31.8 %
SV2P
17.5 %
Table I: We compare the realism of the generated trajec-
tories using real-vs-fake 2AFC Amazon Mechanical Turk with SAVP-VAB and SV2P.
[-] L+2
Figure 2: We condition the VideoFlow model with the frame == and display generated trajectories at == and = three different shapes.
where (-)) is a deep 3-D residual network (He et al., 2015) augmented with dilations and gated activation units and modified to predict the mean and log-scale. We describe the architecture and our ablations of the architecture in Section D and E of the appendix.
In summary, the log- ligeliiklhood objective of Equation (1) has two parts. The invertible multi-scale architecture contributes Σ log det (dhh/dhh,,1) via the sum of the log Jacobian determinants of the invertible transformations mapping the video {xe) to {zz ) the latent dynamics model contributes log po (z), i.e Equation (5). We jointly learn the parameters of the multi-scale architecture and latent dynamics model by maximizing this objective.
Note that in our architecture we have chosen to let the prior (z),,, as described in eq. (5), model temporal dependencies in the data, while constraining the flow go to act on separate frames of video. We have experimented with using 3-D convolutional flows, but found this to be computationally overly expensive compared to an autoregressive prior; in terms of both number of operations and number of parameters. Further, due to memory limits, we found it only feasible to perform SGD with a small number of sequential frames per gradient step. In case of 3-D convolutions, this would make the temporal dimension considerably smaller during training than during synthesis; this would change the model's input distribution between training and synthesis, which often leads to various temporal artifacts. Using 2-1) convolutions in flow fo with autoregressive priors, allows to synthesize arbitrarily long sequences without introducing such artifacts.
EXPERIMENTS
All our generated videos and qualitative results can be viewed at this website. In the generated videos, border of blue represents the conditioning frame, while a border of red represents the generated frames.
5.1 VIDEO MODELLING WITH THE STOCHASTIC MOVEMENT DATASET
We use VidcoFlow to model the Stochastic Movement Dataset used in (Babaeizadeh al., 7). The first frame of every video consists of a shape placed near the center of a 64x64x3 resolution gray background with its type, size and color randomly sampled. The shape then randomly moves in one of eight directions with constant speed. (Babaeizadeh et al., 2017) show that conditioned on the first frame, a deterministic model averages moves with a uniform speed, we should be able to model the position of the shape at the (I++ 1)th step using only the position of the shape at the th step. Using this insight, we extract random temporal patches of 2 frames from each video of 3 frames. We then use VideoFlow to maximize the log- likelihood the second frame given the first, the model looks back at just one frame. We observe that the bits-per-pixel on the holdout set reduces to a very low 0.04 bits-per-pixel for this model. On generating videos conditioned on the first frame, we observe that the model consistently predicts the future trajectory of the shape to be one of the eight random directions. We compare our model with two state-of-the of-the--rt stochastic video generation models SV2P and SAVP- VAE (Babaeizadeh et al., 2017: Lee et al.. 2018) using their Tensor2Tensee implementation (Vaswani al., 2018). We assess the quality of the generated videos using real vs fake Amazon Mechanical Turk test. In the test, we inform the rater that "real" trajectory is in which the shape is consistent in color and congruent
all eight possible directions in pixel space. Since the shape
5

---

Published as a conference paper at ICLR 2020
Model
Bits pess-per-piell
Foorng
VideoFlow
1.87
SAVP-VAE
< 6.73
SV2P
I 6.78
Table Left:: We report the average bits-per-pixel across 10 target frames with 3 conditioning frames for the BAIR action-free dataset.
Diveraty VSU disiance)
Figure 3: We measure realism using 2AFC test and diversity using mean pairwise cosine distance between generated samples VGG perceptual space.
throughout the video. We show that VideoFlow outperforms the baselines in terms of fooling rate in Table I consistently generating plausible "real" trajectories at a greater rate.
5.2 VIDEO MODELING WITH THE BAIR DATASET
We use the action-free version of the BAIR robot pushing dataset (Ebert et al., 2017) that contain videos of a Sawyer robotic arm with resolution 64x64. In the absence of actions, the task of video generation is completely unsupervised with multiple plausible trajectories due to the partial observability of the environment and stochasticity of the robot actions. We train the baseline models, SAVP-VAE SV2P and SVG-LP to generate 10 target frames, conditioned on 3 input frames. We extract random temporal patches of 4 frames, and train VideoFlow to maximize the log-likelihood of the 4th frame given context of 3 past frames. We, thus ensure that all models have seen a total of 13 frames during training.
We estimated the variational bound of the bits-per-pixel on the test set, via importance sampling, from the posteriors for the SAVP-VAE and SV2P models. We find that VideoFlow outperforms these models on bits-per-pixel and report these values in Table 2. We attribute the high values of bits- bits-per-pinn per-pixell of the baselines to their optimization objective. They do not optimize the variational bound on the log- likelihood directly due to the presence of 3 & 1 term in their objective and scheduled sampling (Bengio et al., 2015).
pits-per-pixel:
Max VGG Cosine Similarity
PSNR
0307
090
34
12
0.31
000
15
Frame number
Frame number
Frame number
Figure 4: For given set of conditioning frames the BAIR action-free sample 100 vidcos from each of the stochastic vidco generation models. We choose the video closest the ground-trune trunh the basis of PSNR, SSIM and VGG perceptual metrics and report the best possible value for each these metrics. All models were trained using ten target frames but are tested to generate 27 frames. For all the reported metrics, higher better.
Accuracy of the best sample: The BAIR robot-pushing dataset is highly stochastic and the number of plausible futures are high. Each generated video can be super realistic, can represent a plausible future in theory but can be far from the single ground truth video perceptually. To partially overcome this, we follow the metrics proposed in prior work (Babaeizadeh et al., 2017: Lee et al., 2018: Denton & Fergus, 2018) to evaluate our model. For given set of conditioning frames in the BAIR action-free test-set, we generate 100 videos from each of the stochastic models. We then compute the closest of these generated videos to the ground truth according three different metrics, PSNR (Peak Signal to

---

Published as a conference paper at ICLR 2020
Noise Ratio), SSIM (Structural Similarity) (Wang et al., 2004) and cosine similarity using features obtained from a pretrained VGG network (Dosovitskiy & Brox, 2016; Johnson et al., 2016) and report our findings in Figure 4. This metric helps understand if the true future lies in the set of all plausible futures according to the video model.
In prior work, (Lee al., 2018; Babaeizadeh et al., 2017; Denton & Fergus, 2018) effectively tune the pixel-level variance as a hyperparameter and sample from a deterministic decoder. They obtain training stabiltiy and improve sample quality by removing pixel-level noise using this procedure. We can remove pixel-level noise in our VideoFlow model resulting in higher quality videos at the cost of diversity by sampling videos at lower temperature, analogous to the procedure in (Kingma & Dhariwal, 2018). For network trained with additive coupling layers, we can sample the th frame 24 from P( (x-ee) with a temperature simply by scaling the standard deviation of the latent gaussian distribution P(=((=<)) by factor of T. We report results with both temperature of 1.0 and the optimal temperature tuned on the validation set using VGG similarity metrics in Figure 4. Additionally, we also applied low-temperature sampling to the latent gaussian priors of SV2P and SAVP-VAE and empirically found it to hurt performance. We report these results in Figure 12
For SAVP-VAE, we notice that the hyperparameters that perform the best on these metrics are the ones that have disappearing arms. For completeness, we report these numbers as well as the numbers for the best performing SAVP models that do not have disappearing arms. Our model with optimal temperature performs better or as well as the SAVP-VAE and SVG-LP models on the VGG-based similarity metrics, which correlate well with human perception (Zhang et al., 2018) and SSIM. Our model with temperature T = 1.0 is also competent with state-of-the-art video generation models on these metrics. PSNR is explicitly pixel-level metric, which the VAE models incorporate as part of its optimization objective. VideoFlow on the other-hand models the conditional probability of the joint distribution of frames, hence as expected underperforms on PSNR.
T=10
T=17
T=24
T=30
T=10
T=17
T=24
T=30
Figure 5: We display three different futures for two sets of conditioning frames (left and right) at T 0.6 showcasing diversity in outcomes
Diversity and quality in generated samples: For each of conditioning frames in the test set, we generate 10 videos and compute the mean distance in VGG perceptual space across these 45 different pairs. We average this across the test-set for T' = 1.0 and T' 0.6 and report these numbers in Figure 3. We also assess the quality of the generated videos at T' = 1.0 and T = 0.6, using a real vs fake Amazon Mechanical Turk test and report fooling rates. We observe that VideoFlow outperforms diversity values reported in prior work (Lee et al., 2018) while being competitive in the realism axis. We also find that VideoFlow at = 0.6 has the highest fooling rate while being competent with -fate-of-the-art VAE models in diversity.
On inspection of the generated videos, we find that at lower temperatures, the arm exhibits less random behaviour with the background objects remaining static and clear achieving higher realism scores. At higher temperatures, the motion of arm is much more stochastic, achieving high diversity scores with the background objects becoming much noisier leading to drop in realism.
Fréchet Video Distance (FVD): We evaluate VideoFlow using the recently proposed Fréchet Video Distance (FVD) metric (Unterthiner et al., 2018), adaptation of the Fréchet Inception Distance (FID) metric (Heusel et al., 2017) for video generation. (Unterthiner et al., 2018) report results with models trained total of 16 frames with 2 conditioning frames; while we train our VideoFlow
7

---

Published as a conference paper at ICLR 2020
Frames Seen: Training
Conditioning
3
2
Total
13
13
13
16
Frames: Evaluation
Ground truth
2
2
Total
13
16
16
16
Model
FVD
VideoFlow (T=0.8)
95+-4
12743
13145
VideoFlow (T=1.0)
14916
22118
251
SAVP
116
SV2P
263
Table 3: Fréchet Video Distance: We report the mean and standard deviation across 5 runs for different frame settings. Results are not directly comparable across models due to the differences between the total number of frames seen during training and the number of conditioning frames.
model on a total of 13 frames with conditioning frames, making our results not directly comparable to theirs. We evaluate FVD for both shorter and longer rollouts in Table 3. We show that, even in the settings that are disadvantageous to VideoFlow, where we compute the FVD on a total of 16 frames, when trained just 13 frames, VideoFlow performs comparable to SAVP.
5.3 LATENT SPACE INTERPOLATION
BAIR robot pushing dataset: We encode the first input frame and the last target frame into the latent space using our trained VideoFlow encoder and perform interpolations. We find that the motion of the arm is interpolated in temporally cohesive fashion between the initial and final position. Further, we use the multi-level latent representation to interpolate representations at a particular level while keeping the representations at other levels fixed. We find that the bottom level interpolates the motion of background objects which are at a smaller scale while the top level interpolates the arm motion.
Figure 6: Left: We display interpolations between a) small blue rectangle and a large yellow rectangle b) small blue circle and a the last target frame of two videos the BAIR robot pushing dataset.
large yellow circle. Right: We display interpolations between the first input frame and
Stochastic Movement Dataset:
We encode two different shapes with their type fixed but a different size and color into the latent space. We observe that the size of the shape gets smoothly interpolated. During training, we sample the colors of the shapes from uniform discrete distribution which is reflected in our experiments. We observe that all the colors in the interpolated space lie in the set of colors the training set.
5.4 LONGER PREDICTIONS
We generate 100 frames into the future using model trained 13 frames with temperature of 0.5 and display our results in Figure 7. On the top, even 100 frames into the future, the generated frames remain in the image manifold maintaining temporal consistency. In the presence of occlusions,
8

---

Published as a conference paper at ICLR 2020
PIX, XiX.a)
233
533
8 2710
200
120
T-22 1-47 T-72 1-97
130
Figure 7: Left: We generate 100 frames into the future with a temperature of 0.5. The top and bottom row correspond to generated videos in the absence and presence of occlusions respectively. Right: We use VideoFlow detect the plausibility of temporally inconsistent frame occur in the immediate future.
the remains super-sharp but the background objects become noisier and blurrier. Our VideoFlow model has a bijection between the 2, and X2 meaning that the latent state 2, cannot store information other than that present in the frame X,- This, in combination with the Markovian assumption in latent dynamics means that the model can forget objects they have been occluded for few frames. In future work, would address this by incorporating longer memory in VideoFlow model; for example by parameterizing N Noo() as a recurrent neural network in our autoregressive prior (eq. 8) using more memory-efficient backpropagation algorithms for invertible neural networks (Gomez et al., 2017).
5.5 OUT-OF-SEQUENCE DETECTION
We use our trained VideoFlow model, conditioned on frames as explained in Section 5.2, detect the plausibility of temporally inconsistent frame to occur in the immediate future. We condition model on the first three frames of test-set video X <4 to obtain a distribution X <4) over its 4th frame X4. We then compute the likelihood of the 1th frame X, of the same video to occur as the 4th time-step using this distribution. i.c. P(X1 = X2 Xxa))) for t = 4... 13. We average the corresponding bits bits-per-pin per-pixel values across the test set and report our findings in Figure 7. We find that our model assigns a monotonically decreasing log- log-likelihood to frames that are more far out in the future and hence less likely occur in the 4th time-step.
OPEN SOURCE CODE AND CHECKPOINTS
We open-source the implementation of our code in the Tensor2Tensor codebase. We additionally open-source various components our trained VideoFlow model, to evaluate log-likelihood, to generate frames and compute latent codes reusable TFHub modules
7 CONCLUSION AND DISCUSSION
We describe a practically applicable architecture for flow-based video prediction models, inspired by the Glow model for image generation Kingma & Dhariwal (2018), which we call VideoFlow. We introduce a latent dynamical system model that predicts future values of the flow model's latent state replacing the standard unconditional prior distribution. Our empirical results show that VideoFlow achieves results that are competitive with the state-of-the-art VAE models stochastic video prediction. Finally, our model optimizes log- likelihooooo directly making easy evaluate while achieving faster synthesis compared to pixel-level autoregressive vidco models, making our model suitable for practical purposes. In future work, we plan to incorporate memory in VideoFlow to model arbitrary long-range dependencies and apply the model to challenging downstream tasks.
ACKNOWLEDGEMENTS
We would like to thank Ryan Sepassi and Lukasz Kaiser for their extensive help in using Ten- ser2Ten,or, Oscar Täckström for finding bug in evaluation pipeline that improved results across all models, Ruben Villegas for providing code for the SVG-LP baseline and Mostafa Dehghani for providing feedback draft of the rebuttal.

---

Published as a conference paper at ICLR 2020
REFERENCES
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H Campbell, and Sergey Levine. Stochastic variational video prediction. arXiv preprint arxiv:/7100 1252, 2017.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. Advances in Neural Information Processing Systems, pp. 1111-117799 2015.
Byron Boots, Arunkumar Byravan, and Dieter Fox. Learning predictive models of depth camera manipulator from raw execution traces. In International Conference on Robotics and Automation (ICRA), 2014.
Bert De Brabandere, Xu Jia, Tinne Tuytelaars, and Luc Van Gool. Dynamic filter networks. In Neural Information Processing Systems (NIPS), 2016.
Gustavo Deco and Wilfried Brauer. Higher order statistical decorrelation without information loss. Advances in Neural Information Processing Systems, pp. 247-254, 1995.
Emily Denton and Vighnesh Birodkar. Unsupervised learning of disentangled representations from video. arXiv preprint arXiv:1705.10915 2017.
Emily Denton and Rob Fergus. Stochastic video generation with learned prior. arXiv preprint 2018.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: non-linear independent components
estimation. arXiv preprint arXiv:/410.cc 2014.
Laurent Dinh, Jascha Dich-Diccstein, and Samy Bengio. Density estimation using Real NVP. arXiv preprint arXiv:1605.08883, 2016.
Alexey Dosovitskiy and Thomas Brox, Generating images with perceptual similarity metrics based on deep networks. In Advances in Neural Information Processing Systems, pp. 658-666, 2016.
Frederik Ebert, Chelsea Finn, Alex X Lee, and Sergey Levine. Self-supervised visual planning with temporal skip connections. arXiv preprint arXiv: 110.0588,, 2017.
Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In International Conference on Robotics and Automation (ICRA), 2017.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. In Advances in Neural Information Processing Systems, 2016.
Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse. The reversible residual network: Backpropagation without storing activations. In Advances in Neural Information Processing Systems, pp. 2211-2221, 2017.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, pp. 2672-2680, 2014.
Alex Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv: 3308.0850 2013.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arXiv preprint 15Xiv:1512.03338 2015.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in neural information processing systems, pp. 6626-6637, 2017.
Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. Neural computation, 9(8): 1735-1780, 1997.
10

---

Published as a conference paper at ICLR 2020
Catalin lonescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural enviromments. IEEE transactions pattern analysis and machine intelligence, 36(7): 1020::13251130 1339, 2014.
Justin Johnson, Alexandre Alahi, and Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European Conference Computer Vision, pp. 694-711. Springer, 2016.
Nal Kalchbrenner, Aäron van den Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Video pixel networks. International Conference on Machine Learning (ICML), 2017.
Diederik P Kingma and Max Welling. Auto-encoding variational Bayes. Proceedings of the 2nd International Conference on Learning Representations, 2013.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ixl convolutions. In Advances in Neural Information Processing Systems, pp. 10236-10245, 2018.
Alex Krizhevsky, Ilya Sutskever, and Geoff Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25, pp. 1106 1114, 2012.
Alex X Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint arXiv:/804.0055 2018.
Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming-Hsuan Yang. Flow-grounded spatial-temporal video prediction from still images. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 600-615, 2018
Ziwei Liu, Raymond Yeh, Xiaoou Tang, Yiming Liu, and Aseem Agarwala. Video frame synthesis using deep voxel flow. International Conference Computer Vision (ICCV), 2017.
William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video prediction and unsupervised learning. International Conference on Learning Representations
(ICLR), 2017.
Michael Mathicu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. International Conference Learning Representations (ICLR), 2016.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing Atari with deep reinforcement learning. arXiv preprint 1312.5 2013.
Ryan Prenger, Rafael Valle, and Bryan Catanzaro. Waveglow: A flow-based generative network for speech synthesis. CoRR, 0bs//811100000 2018. URL 00002.
Prajit Ramachandran, Tom Le Paine, Pooya Khorrami, Mohammad Babacizadeh, Shiyu Chang, Yang Zhang, Mark A Hasegawa-lohnson, Roy H Campbell, and Thomas S Huang. Fast generation for convolutional autoregressive models. arXiv preprint arXiv: 7704.0600) 2017.
MarcAurelio Ranzato, Arthur Szlam, Joan Bruna, Michael Mathieu, Ronan Collobert, and Sumit Chopra. Video (language) modeling: bascline for generative models of natural videos. arXiv preprint www.iv::112.666 2014.
Scott Reed, Aäron van den Oord, Nal Kalchbrenner. Sergio Gómez Colmenarejo, Ziyu Wang, Dan Belov, and Nando de Freitas. Parallel multiscale autoregressive density estimation. arXiv preprint arXiv: /703.03664 2017.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Proceedings of The 32nd International Conference on Machine Learning, pp. 1530-1538, 2015.
Danilo J Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31 International Conference on Machine Learning (ICML-14), pp. 1278-1286, 2014.

---

Published as a conference paper at ICLR 2020
David Silver, Julian Schrittwieser, Karen Simonyan, loannis Antonoglou. Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676)355,, 2017.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representations using Istms. In International Conference on Machine Learning, 2015.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marein Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges, 2018.
Aaron Van Den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv: (000::/60000000 (609.03499, 2016.
Aaron van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with PixeICNN decoders. In Advances in Neural Information Processing Systems, pp. 4790-4798. 4798, 2016a.
Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. arXiv preprint arXiv:1601. 1.6059, 2016b.
Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Ko- ray Kavukcuoglu. Conditional image generation with PixeICNN decoders. arXiv preprint arXiv: 05328, 2016c.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pp. 5998-6008, 2017.
Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N Gomez, Stephan Gouws, Llion Jones, Lukasz Kaiser, Nal Kalchbrenner, Niki Parmar, et al. Tensor2tensor for neural machine translation. arXiv preprint arXiv: 1803.07/0 2018.
Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lee. Decomposing motion and content for natural video sequence prediction. arXiv preprint arXiv:1706.0n 08033, 2017a.
Ruben Villegas, Jimei Yang, Yuliang Zou, Sungryull Sohn, Xunyu Lin, and Honglak Lee. Learning to generate long-term future via hierarchical prediction. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 3560-3569. JMLR. org, 2017b.
Carl Vondrick and Antonio Torralba. Generating the future with adversarial transformers. In Computer Vision and Pattern Recognition (CVPR), 2017.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Anticipating the future by watching unlabeled video. arXiv preprint / 2015.
Jacob Walker, Abhinav Gupta, and Martial Hebert. Dense optical flow prediction from static image. In International Conference on Computer Vision (ICCV), 2015.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 2004.
Jianwen Xie, Song-Chur Chun Zhu, and Ying Nian Wu. Synthesizing dynamic patterns by spatial-temporal generative convnet. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Jianwen Xie, Ruiqi Gao, Zilong Zheng, Song- Chun Zhu, and Ying Nian Wu. Learning dynamic gen- erator model by alternating back proppaatpagttoon through time. Proceedings of the AAAI Conference on Artificial Intelligence, 33:5498 -3:5498-5507, Jul 2019. ISSN 2159-5399. doi: 10.1609/aaai. v33i01. 33015498, URL http:/tt/// dxx doi. www.:oooo..oooo.....oe.o 0. 609/ aaa 1 v3310 01 33015498.
Jianwen Xie, Yang Lu, Ruiqi Gao, Song-Chus Zhu, and Ying Nian Wu. Cooperative training of descriptor and generator networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(1) Jan 2020. ISSN 1939-3539, doi: 10. http://dx. doi org/ 10. 11199//PPPMT 201 8. 2879081.
URL
12

---

Published as a conference paper at ICLR 2020
SHI Xingjian, Zhourong Chen, Hao Wang, Dit- Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional Istm network: A machine learning approach for precipitation nowcasting. In Advances in Neural Information Processing Systems, 2015.
Tianfan Xue, Jiajun Wu, Katherine Bouman, and Bill Freeman. Visual dynamics: Probabilistic future frame synthesis via cross convolutional networks. In Advances in Neural Information Processing Systems, 2016.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. arXiv preprint, 2018.
A
MOVING MNIST
QUALITATIVE EXPERIMENTS
6
6
6
6
6
6
6
6
6
6
9
9
9
9
9
9
9
9
9
9
9
&
&
Y
A
6
6
6
6
6
2
3
3
3
3
0
0
0
0
Figure 8: We display ten frame rollouts conditioned on a single frame on the Moving MNIST dataset.
Similar to the Stochastic Movement Dataset described in Section 5.1, we extract random temporal patches of 2 frames on the Moving MNIST dataset (Srivastava et al., 2015). We train our VideoFlow model to maximize the log-likelihood of the second frame, given the first. Our rollouts over 10 frames capture realistic digit movement.
B HUMAN3, MMM-- QUALITATIVE EXPERIMENTS
We model the Human3.6N, 3.6 dataset (lonescu et al., 2014), by maximizing the log- log-likelihood of the 4th frame given the first three frames, in random temporal patch of 4 frames. We observe that on this dataset, our model fails to capture reasonable human motion. We hope that by increasing model capacity and using more expressive priors, we can acheive better performance on this dataset in the future.
C DISCRETIZATION AND UNIFORM QUANTIZATION
Let D== (x(0) ii be our dataset of i.i.d. observations of random variable with an unknown true distribution p" (x). Our data consist of 8-bit videos, with each dimension rescaled to the domain [0, 255/2566. We add small amount of uniform noise to the data, u ~ (O(0, 1/256.), ), matching its discretization level (Dinh et al., 2016; Kingma & Dhariwal, 2018). Let q(x) be the resulting empirical distribution corresponding to this scaling and addition noise. Note that additive noise is required to prevent qxx) from having infinite densities at the datapoints, which can result in ill-behaved optimization of the log- likeliho;hoo;; it also allows us to recast maximization the log- likellhoooo as minimization of KI. divergence.
13

---

(>1) Published as a conference paper at ICLR 2020
Figure 9: We display ten frame rollouts conditioned 3 frames the Human3.6M M.6M dataset.
D RESIDUAL NETWORK ARCHITECTURE
Here we'll describe the architecture for the residual network No that maps zet 2z to ,logo(") (Left: Figure 10). As shown in the left of Figure 10, let h (>!) be the tensor representing z, (>1) after the split operation between levels in the multi-scale architec- ture. We apply a 1 x 1 convolution over h(²l) and concatenate this across channels to each latent from the previous time-step and the same-level independently. In this way, we obtain ((Wh, (>1) (Wh, (>1), 2(_2)...
... (Wh,"") (>1), ? ()"))) We transform these values into (14)) log ( via a stack of residual blocks. We obtain a reduction in parameter count by sharing every 2 time-steps via 3-D convolutions in residual blocks.
parameters
As shown in the right Figure 10, each 3-D residual block consists of three layers. The first layer has filter size of 2x3x3 with 512 output channels followed by ReLU activation. The second layer has two 1 x x 1 convolutions via the Gated Activation Unit Van Den Oord et al. (2016); van den Oord et al. (2016a). The third layer has a filter size of 3 with the number of output channels determined by the level. This block is replicated three times in parallel, with dilation rates 1, 2 and 4, after which the results of each block, in addition to the input of the residual block, are summed.
The first two layers are initialized using Gaussian distribution and the last layer is initialized to zeroes. In that way, the residual network behaves as an identity network during initialization allowing stable optimization. After applying a sequence of residual blocks, we use the last temporal activation that should capture all context. We apply a final x I convolution to this activation to (9) This way, the network learns to predict the change latent variables for a given level. We have provided visualizations of the network architecture in this website
obtain (Az,) (!) log ( We then add Az;" to 2 to temporal skip connection to output 14
E ABLATION STUDIES
Through ablation study, experimentally evaluate the importance of the following components of our VideoFlow model: (1) the use of temporal skip connections, (2) the use Gated Activation Unit (GATU) instead of ReLUs in the residual network and (3) the use of dilations in N Noo() in Section D
We start with a VideoFlow model with 256 channels in the coupling layer, 16 steps of flow and remove the components mentioned above create baseline. We use four different combinations of our components (described in Fig. 1) and keep the rest of the hyperparameters fixed across those combinations. For each combination plot the mean bits-per-pixxr on the holdout BAIR-action
14

---

Published as a conference paper at ICLR 2020
AC
LEWARE ECTAO
1X1 conv
3-D residual network
201a diaz
203. diac
died
2G03
AP
Figure 10: Left: We predict gaussian distribution over 2 via a 3-D Residual network conditioned on 2 and vf (2+) Right: Our 3-D residual network architecture is augmented with dilations and gated activation units improving performance.
free dataset over 300K training steps for both affine and additive coupling in Figure 11. For both the coupling layers, observe that the VideoFlow model with all the components provide significant boost in bits-per-pixel over our baseline.
Affinc
Additive
277
2.
23
70-
24-
53300
100000 Training steps
150000 200000 250000 X00000
50000
300000
150000
200003
25009C 308200
Training steps
Figure 11: B: baseline, A: Temporal Skip Connection, C: Dilated Convolutions GATU, D: Dilation Convolutions Temporal Skip Connection, E: Dilation Convolutions + Temporal Skip Connection + GATU, We plot the holdout bits-per-pixel on the BAIR action-free dataset for different ablations of our VideoFlow model.
We also note that other combinations- the temporal skip connection -improve over the baseline. Finally, we experienced that increasing the receptive field in Noo () using dilated convolutions alone the absence of the temporal skip connection the GATU makes training highly unstable.
convolutions + GATU (C) and dilated convolutions +
F EFFECT TEMPERATURE ON SAVP- VAE AND SV2P
We repeat evaluations described in Figure 4 applying low temperature the latent gaussian priors SV2P and SAVP-VAE. We empirically find that decreasing temperature from 1.0 to 0.0 monotonically decreases the performance of the VAE models. Our insight is that the VideoFlow
15

---

Published as a conference paper at ICLR 2020
lerpored Sate visdeceiesddderryy
lanperd sARPad macion
Terperd SAkPaa,, Foot
013
014
Temperat sea. Plak VTO enattrty
Tenpers SIOP 2012
Terpered SIIP PoRe
1-0.2
T+E
044
012
Figure 12: We repeat evaluations described the SV2P and SAVP-VAL model Figure 4 using tempera- tures from 0.0 1.0 while sampling from the latent gaussian prior.
model gains by low-te www-temperturre sampling due to the following reason. At lower T, we obtain tradeoff between a performance gain by noise removal from the background and a performance hit due to reduced stochasticity of the robot arm. On the other hand, the VAE models have clear but slightly blurry background throughout from T = 1.0 to 0.0. Reducing T this case, solely reduces the stochasticity of the arm motion thus hurting performance.
G LIKELIHOOD VS QUALITY
bpp=1.9
bpp=2.2
bpp=2.7
T=4
T=7
T=10
Figure 13: We provide comparison between training progression (measured in the bits-per-pixel objective on the test-set) and the quality generated videos.
We show correlation between training progression (measured in bits pixel) and quality of the generated videos in Figure 13. We display the videos generated by conditioning on frames from the test set for three different values of our VideoFlow model learns to model the structure the arm with high quality as well as its motion resulting in high quality video.
the test-set. As approach lower bits- per-pixel,
16

---

Published as a conference paper at ICLR 2020
H VIDEOFLOW - BAIR HYPERPARAMETERS
H.1 QUANTITATIVE BITS-PER BITS-PER-PIXEL
To report bits-per pixel we use the following set of hyperparameters. We use a learning rate schedule of linear warmup for the first 10000 steps and apply linear-decay schedule for the last 150000 steps.
Hyperparameter
Value
Flow levels
3
Flow steps per level
24
Coupling
Affine
Number of coupling layer channels
512
Optimier
Adam
Batch size
40
Learning
3e-4
Number of 3-D residual blocks
5
Number of 3-D residual channels
256
Training steps
600K
H.2 QUALITATIVE EXPERIMENTS
For all qualitative experiments and quantitative comparisons with the baselines, used the following sets hyperparameters.
Hy Hyperparameter
Value
Flow levels
3
Flow steps per level
24
Coupling
Additive
Number of coupling layer channels
392
Optimier
Adam
Batch size
40
Learning rate
3e-4
Number of 3-D residual blocks
5
Number of 3-D residual channels
256
Training steps
500K
I HYPERPARAMETER GRID FOR THE BASELINE VIDEO MODELS.
train all our baseline models for 300K steps using the Adam optimizer. Our models were tuned using the maximum VGG cosine similarity metric with the ground-truth across 100 decodes.
We use three values of latent loss multiplier 1e-3, lc-4 and 1e-5. For the SAVP-VAE model, we additionally apply linear decay the learning rate for the last 100K steps. SAVP-GAN: We tune the gan loss multiplier and the learning rate on a logscale from 1e-2 Ic-4 and 1e-3 to le-5 respectively.
SAVP-VAE and SV2P:
J CORRELATION BETWEEN VGG PERCEPTUAL SIMILARITY AND PERS-PER-PIXEL
We plot correlation between cosine similarity using pretrained VGG network and bits-per-pixel using our trained VideoFlow model. We compare P(X, = Xx)) as done in Section 5.5 and the VGG cosine similarity between X, and X, for t = 4... 13. We report our results for every video in the test set Figure 15. We notice a weak correlation between VGG perceptual metrics and bits-per-pixel with correlation factor of -0.51.
17

---

Published as a conference paper at ICLR 2020
Bits-pen-poel VGG similancy
43
XxXx.s)
30-
145
6.70
1..5
Dall
4.55
1201
C Xx, Xa)
Figure 14: We compare P(X, xa) and VGG cosine similarity between and X for 4...
K VIDEOFLOWW: Low PARAMETER REGIME
repeated our evaluations described in Figure 4, with a smaller version of our VideoFlow model with 4x parameter reduction. Our model remains competetive with SVG-LP on the VGG perceptual metrics.
VGG_SIM SIM
0.8
VideoFlow small:
VideoFlow large:
0.96
SVG-LP: 22
0.9
0.92
0.0
088
0.86
0.84
10
15
20
25
Figure 15: We repeat our evaluations described in Figure 4 with a smaller version of our VideoFlow model.
18


Reviews:
[{"review_id": "SygDedWRtH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper \"VideoFlow: A Conditional Flow-Based Model for Stochastic Video \" proposes a new model for video prediction from a starting sequence of conditionning frames. It is based on a state-space model that encodes successive frames in a continuous hierarchical state, with contraints on trajectories of the codes in this state. \n\nI like the invertible NN framework the model relies on. It allows to avoid variational autoencoding of frames via invertible deterministic transforms. Learning the dynamics of the video is therefore easier, since there is no need of any stochastic inference process.    However, is there no risk of high latent vacancy in the representation space? Uncertainty of stochastic inference usually helps filling the space by considering larger areas of codes than deterministic process. Also, since at each step, the next code is conditionned by the whole past sequence of codes, besides the increasing complexity induced, I am wondering if such a model is able to efficiently encode the dynamics and the stochasticity of the video. In fact, a given z_t does not encode any dynamics nor uncertainty at that point, only the image (it cannot since it is fully determined via the invertible function from the image). Imagine that at a given point, two very different scenarios can follow, with very different following frames. In that case, how could the next state could encode these two different futures with a simple gaussian in the space ? Also,  it would be useful to compare the model with a version where the invertible frame encoder and the sequential model would be learned separately, to better understand what the model really does during training. A study of the impact of the hierarchy depth would also be useful.  \n\nAlso, an additional real-world dataset would be useful for really assessing the performance of the model, since BAIR is known to be fully random and the past does not highly impact the future. A possible dataset would be KTH. Other baselines could also be considered, notably the famous approach from  [Denton et al., 2017]. \n\nAt last, the clarity of some parts could be improved. Notably the description of the sequential model in the space, whih is succintly given in the appendix.   \n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJetRuM0KH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper extended the flow-based generative model for stochastic video prediction. The proposed model takes an advantage of the flow-based models which provide exact latent-variable inference, exact log-likelihood evaluation, and efficiency. The paper used the autoregressive model and the multi-scale Glow architecture. The experiments on the stochastic movement dataset (synthetic) and the BAIR Robot push dataset show the performance improvement against other state-of-the-art stochastic video generation models (SV2P and SAVP-VAE). \n\nThe main contribution in this paper is the use of flow-based models for video prediction, and it is the first work in this direction. The major idea sounds and the paper is clearly written. \n\nBelow is my concerns and the feedback. \n\nIt looks like the low-temperature sampling is important to achieve the better scores for prediction. Can the low-temperature sampling trick be applied for SV2P and SAVP-VAE as well? If then, how is the performance difference compare to the proposed model?\n\nThe authors reported the best possible values of PSNR, SSIM and VGG perceptual metrics by choosing the video closest to the ground-truth. However, I believe this evaluation does not present the benefit of the stochastic models. The better comparison I believe is to report the median/mean with the range between best and worst values. \n\nThe BAIR robot push dataset is with a pretty limited setting: a small robot and/or object motion between frames and a small variation of the background between videos. It would be interesting to see more dynamic scenarios such as driving or human motion scenes. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJe8zpxjYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a stochastic model based on Glow for conditional video generation. The major novelty of this work is to introduce the flow-based models to video modeling and learn the video dynamics via the dependencies of the latent variables. The general idea is reasonable and the proposed model is technically correct, but I have the following concerns mainly about the originality and the experiments.  \n\n**Above all, most of the text in Section 3 is very similar (or exactly the same) to the text of the Glow paper (the background section).**\n\n— Significance and originality —\na.1) In Section 1, the authors discussed some possible application scenarios of video prediction models, e.g. learning from unlabeled data and being used for downstream tasks. However, all models mentioned here, including [Mathieu et al. 2016] and [Finn et al. 2016], are deterministic models. Thus, in what way can the stochastic model proposed in this paper be used in real applications?\n\na.2) VideoFlow can be viewed as an extension of the Glow model. There are two problems. First, the originality is limited. I don’t think modeling the temporal dependencies of the latent variables with a convolutional network is a significant contribution to the conditional flow-based methods. Second, this paper is not self-contained. After reading Section 4.2, I have to check the previous literature to find the objective function, the network details, or the training procedure.\n\n— Experiments —\nb.1) Throughout the experiments, the VideoFlow model is mainly compared with two stochastic video prediction models that were probably proposed by the same research group. If it is possible, the authors might include other stochastic models such as the SVG-LP [Denton & Fergus 2018], and at least one deterministic model such as the E3D-LSTM [Wang et al. 2019] as well.\n[Wang et al. 2019] E3D-LSTM: A Model for Video Prediction and Beyond.\n\nb.2) The evaluation metric bits-per-pixel was not directly optimized by the previous video generation/prediction models. Thus, the comparisons in Table 2 might be unfair.\n\nb.3) Since training the Glow model requires a huge computational cost, how is the training efficiency of the VideoFlow model compared with other stochastic video generation models?\n\n— Other —\nc.1) In Section 4, it is not clear what the temporal border effect means?\n\nAFTER REBUTTAL:\nThough the overall novelty is still not fully convincing, this paper may shed some insights into video generation by introducing flow-based models to this topic. I have increased my score from 3 to 6.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors explore the use of flow-based models for video prediction. The idea is interesting. The paper is well-written. It is a good paper worthwhile presenting in ICLR.

For final version, we suggest that the authors can significantly improve the experiments: (1) report results on human motion datasets; (2) include the results by the FVD metric. 

Decision: Accept (Poster)