Title: PERCEPTION UPDATING NETWORKS: ON ARCHITEC-TURAL CONSTRAINTS FOR INTERPRETABLE VIDEO GENERATIVE MODELS

Abstract: We investigate a neural network architecture and statistical framework that models frames in videos using principles inspired by computer graphics pipelines. The proposed model explicitly represents "sprites" or its percepts inferred from maximum likelihood of the scene and infers its movement independently of its content. We impose architectural constraints that forces resulting architecture to behave as a recurrent what-where prediction network. * Companion code repo coming soon.

1. INTRODUCTION:

The current computer graphics pipelines are the result of efficient implementations required by limited hardware and high frequency output requirements. These requirements were also achieved with the use of explicit physics and optic constraints and modeling with constantly improving data structures (Shirley et al., 2015) .

In machine learning on the other hand, for a long time image (Olshausen et al., 1996) and video (Hurri & Hyvärinen, 2003) generative models had been investigated with statistical approaches that model images down to the pixel level (Simoncelli & Olshausen, 2001) , sometimes assuming neighborhood statistical dependencies (Osindero & Hinton, 2008) . In video prediction, the current state of the art uses variations of deep convolutional recurrent neural networks (Kalchbrenner et al., 2016 ) (Lotter et al., 2016) (Finn et al., 2016) .

As a parallel to the classic machine learning approach to image and video interpretation and prediction is a growing trend in the deep learning literature for modeling vision as inverse graphics (Kulkarni et al., 2015) (Rezende et al., 2016) (Eslami et al., 2016) . These approaches can be interpreted into two groups: supervised and unsupervised vision as inverse graphics. The supervised approach assumes that during training an image is provided with extra information about its rotation, translation, illumination, etc. The goal of the supervised model is to learn an auto-encoder that explicitly factors out the content of the image and its physical properties. The supervised approach is illustrated by Kulkarni et al. (2015) .

The unsupervised approach requires extra architectural constraints, similar to those assumed in computer graphics. For example, Reed et al. (2016) modeled the content of a scene with a Generative Adversarial Network (Goodfellow et al., 2014) and its location with Spatial Transformer Networks (Jaderberg et al., 2015) . The full model is adapted end-to-end to generate images whose appearance can be changed by independently modifying the "what" and/or "where" variables. A similar approach was applied to video generation with volumetric convolutional neural networks (Vondrick et al., 2016) .In two papers by Google DeepMind (Rezende et al., 2016 ) (Eslami et al., 2016 ) they improved the "where" representations of the unsupervised approach and modeled the 3D geometry of the scene. This way they explicitly represented object rotation, translation, camera pose, etc. Their approaches were also trained end-to-end with REINFORCE-like stochastic gradients to backpropagate through non-differentiable parts of the graphics pipeline (Rezende et al., 2016) or to count Input sprite is 8 × 8 pixels and the outputs are 64 × 64 pixels. Note that in the convolution the result shape is rotated 180 degrees and its center is where the delta equals to one at pixel (x = 16, y = 16). Note also that the edges of the spatial transformer results are blurred due to bilinear interpolation. A matrix can be read as "zoom-out" 8 times and translate up and left in a quarter of the resulting size.

the number of objects in the scene (Eslami et al., 2016) . Those papers also used Spatial Transformer Networks to model the position of the objects in the scene, but they extended it to 3D geometry so it could also model rotation and translation in a volumetric space.

Other approaches inspired by the graphics pipeline and computer vision geometry in machine learning uses the physics constraints to estimate the depth of each pixel in the scene and camera pose movements to predict frames in video (Mahjourian et al., 2016) (Godard et al., 2016) .

The present paper is closer to the unsupervised approach of vision as inverse graphics. More precisely, here we investigate frame prediction in video. Contrary to the work by Reed et al. (2016) here we first limit ourselves to simple synthetic 2D datasets and learning models whose representations can be visually interpreted. This way we can investigate exactly what the neural network is learning and validate our statistical assumptions. Also, we investigate the behavior of Spatial Transformer Networks and question it as the default choice when limited compute resources are available and no scale invariance is required.

First in the next Section we will pose a statistical model that is appropriate for machine learning but inspired by the graphics pipeline.

2. A 2D STATISTICAL GRAPHICS PIPELINE:

This section starts with a high level description of the 2D graphics pipeline, followed by a discussion of how to implement it with neural network modules, and finally we define a formal statistical model.

The 2D graphics pipeline starts from geometric primitives and follows with modeling transformations, clipping, viewing transformations and finally scan conversion for generating an image. Here, we will deal with previously rasterized bitmaps, i.e. sprites, and will model the translation transformations, rotation and clipping with differential operations. This way, the steps in the pipeline can be defined as layers of a neural network and the free parameters can be optimized with backpropagation.

For our neural network implementation, we assume a finite set of sprites (later we generalize it to infinite sprites) that will be part of the frames in the video. The image generation network selects a sprite, s, from a memorized sprite database S i∈{1,...,K} using an addressing signal c: s = j c j S j , where j c j = 1.

(1)

For interpretable results it would be optimal to do one-hot memory addressing where c j = 1 for S j = S and c j = 0 otherwise. Note that (1) is differentiable w.r.t to both c j and S j so we can learn the individual sprites from data. We can for c j to sum to 1 using the softmax nonlinearity. This approach was inspired by the recent deep learning literature on attention modules (Bahdanau et al., 2014) (Graves et al., 2014) .

When the number of possible sprites is too large it is more efficient to do a compressed representation. Instead of using an address value c we use a content addressable memory where the image generator estimates a code z that is then decoded to the desired sprite with a (possibly nonlinear) function d(z). If we interpret the addressing value z as a latent representation and the content addressable memory d(z) as a decoder, we can use the recent advances in neural networks for generative models to setup our statistical model. We will revisit this later in this section.

The translation transformation can be modeled with a convolution with a Delta function or using spatial transformers. Note that the translation of an image I(x, y) can be defined as

EQUATION

where denotes the image convolution operation. Clipping is naturally handled in such a case. If the output images have finite dimensions and δ(x -τ x , y -τ y ) is non-zero near its border, the translated image I(x -τ x , y -τ y ) will be clipped. Another way of implementing the translation operation is using Spatial Transformer Networks (STN) (Jaderberg et al., 2015) . An implementation of STN can be defined in two steps: resampling and bilinear interpolation. Resampling is defined by moving the position of the pixels (x, y) in the original image using a linear transform to new positions (x, ỹ) as

EQUATION

, where

A = A 11 A 12 A 13 A 21 A 22 A 23 .

(3)

We assume the coordinates in the original image are integers 0 ≤ x < M and 0 ≤ y < N , where M × N is the size of the image I. Once the new coordinates are defined, we can calculate the values of the pixels in the new image Ĩ using bilinear interpolation:

Ĩ(x, ỹ) = w x1,y1 I(x 1 , y 1 ) + w x1,y2 I(x 1 , y 2 )+ w x2,y1 I(x 2 , y 1 ) + w x2,y2 I(x 2 , y 2 ) (4)

where

EQUATION

To avoid sampling from outside the image we clip the values x and x + 1 between 0 and M and the values ỹ and ỹ + 1 between 0 and N . We omitted that in (5) for conciseness. Note that ( 4) is piecewise differentiable w.r.t I.

We can define translation through operations with

EQUATION

Also, we can rotate the image ρ radians counter clockwise with

EQUATION

)

Image rescaling is achieved on that framework by rescaling in the right square submatrix A 1:2,1:2 . We illustrate in Fig. 1 how to get similar results using convolutions with a delta-function and spatial transformers.

Considering the tools defined above, we can define a statistical model of 2D images the explicitly represents sprites and their positions in the scene. We can use the free energy of this statistical model to optimize a neural network. Let us start with a static single frame model and later generalize it to video.

Let an image I ∼ p θ (I) be composed of sprite s ∼ p θ (s) centered in the (x, y) coordinates in the larger image I. Denote these coordinates as a random variable δ xy ∼ p θ , where θ are the model parameters. p θ (δ xy ) can be factored in two marginal categorical distributions Cat(δ x ) and Cat(δ y ) that models the probability of each coordinate of the sprite independently. For the finite sprite dataset, p θ (s) is also a categorical distribution conditioned on the true sprites. For this finite case the generative model can be factored as

EQUATION

assuming that "what", s, and "where", δ xy , are statistically independent. Also, in such case the posterior p θ (s, δ|I) = p θ (s|I)p(δ xy |I) (9) is tractable. One could use for instance Expectation-Maximization or greedy approaches like Matching Pursuit to alternate between the search for the position and fitting the best matching shape. For the infinite number of sprites case, we assume that there is a hidden variable z from which the sprites are generated as p(s, z) = p θ (z)pθ(s|z). In such case our full posterior becomes

EQUATION

We can simplify (10) assuming p θ (z|s) = p θ (z|I) for simple images without ambiguity and no sprite occlusion. For a scalable inference in the case of unknown θ and z and intractable p θ (z|s) we can use the auto-encoding variational Bayes (VAE) approach proposed by Kingma & Welling (2013) . Using VAE we define an approximate recognition model q φ (z|s). In such case, the loglikelihood of the i.i.

EQUATION

Again, assume that the approximation p θ (z|s) = p θ (z|I) we have D KL (p θ (z|s i )||p θ (z|I i )) = 0 and the free energy (or variational lower bound) term equal to where we dropped the subindices xy and i to avoid clutter. Here we would like to train our model by maximizing the lower bound ( 12), again inspired by VAE. We can do so using the reparametrization trick assuming q φ (z|s) and the prior p θ (z) to be Gaussian and sampling

EQUATION

EQUATION

where ξ ∼ N (0, σI), I is the identity matrix, the functions m(I) and v(I) are deep neural networks learned from data.

One can argue that given z and a good approximation to the posterior q φ , estimating δ is still tractable. Nevertheless, we preemptively avoid Expectation-Maximization or other search approaches and use instead neural network layers l x and l y :

EQUATION

with ⊗ denoting the outer product of marginals. We also experiment using STNs. Such amortized inference is also faster in training and test time than EM and will also cover the case where I is itself a learned low dimensional or latent representation instead of an observable image. Bear this in mind while we use this approach even in simple experiments such as those with moving shapes in the Experiments Section. This will help us to understand what can be learned from this model.

We extend the model above to videos, i.e. sequences of images I(t) = {I(0), I(1), . . .}, assuming that the conditional log-likelihood log p θ (I t |H It ) = logp θ (I t |H δt , H zt ) follows ( 11), where H It is the history of video frames prior to time point t. Also H δt and H zt are the history of position coordinates and the history of latent variables of the sprites respectively. We should observe that one can make the assumption that the sprites don't change for a given video I(t) and only estimate one sprite s t=0 or hidden variable z t=0 . This assumption can be useful for long term predictions, but requires that the main object moving in the scene doesn't change.

In the next section, we propose a neural network architecture for maximizing our approximate variational lower bound 2D videos.

3. PERCEPTION UPDATING NETWORKS:

This Section proposes a family of neural architectures for optimizing the lower bound (12). A schematic diagram is represented in Fig.

(2). The core of our method is a Recurrent Neural Network (RNN) augmented with task specific modules, namely a sprite addressable memory and modeling transformations layers. RNNs augmented with task specific units were popularized by Graves et al. (2014) in the context of learning simple differentiable algorithms and served as inspiration for us as well.

Here since we explicitly model the perceived sprites as s or z and update it and its location and/or rotation though time we decided to call our method simply Perception Updating Networks.

Here an input frame at time t, I t , is fed to the RNN that emits 2 signals: a memory address that selects a relevant sprite and transformation parameters. If we are doing the translation transformation using convolutions and delta functions this output is equal to (14). If using STN, the translation operation returns the matrix A used in (3). Note that we could use both, letting convolutions with δ to the translation is constraining A as in (7) to do rotation transformations only. We describe the general case where both δ xy and STNs are used in Algorithm 1.

Beyond deciding between STNs vs δ xy , a few other free parameters of our method are the type of RNN (e.g. vanilla RNN, LSTM, GRU, ConvRNN, etc), the number of neurons in the hidden state of the RNN and neural network architectures that infer the correct sprite and modeling transformation parameters. Our hyperparameter choices are investigated separately in each experiment in the next Section. 

Data:

δ xy = softmax(l x (h t )) ⊗ softmax(l y (h t )) ρ = f (h t ) A = cos ρ sin ρ 0 -sin ρ cos ρ 0 ξ ∼ p θ (z) z t = m φ (h t ) + v φ (h t ) • ξ s t = d(z t ) a t = ST N (s t , A) Ĩt+1 = a t δ xy I t+1 = µ Ĩt+1 + (1 -µ)B

end Algorithm 1: Perception Updating Networks. STN denotes spatial transformer operator (3)-( 4) and denotes convolution. We experimented with several variations of this algorithm, mainly changing if and how the "where" modules δ xy and STN are used. Also changing how the sprite s t is calculated and not using a background B when not necessary.

In the next section we present experiments with the proposed architecture on synthetic datasets.

4. EXPERIMENTS:

In this section we experiment with several implementations of the proposed Perception Updating Networks. We start with a simple synthetic dataset made of videos where one of 3 moving shapes moves with constant speed bouncing in the edges of an image. This illustrates the working of the finite memory and the addressing scheme in (1). Afterwards we show results on the moving MNIST dataset (Srivastava et al., 2015) commonly used in the literature of generative neural network models of videos.

4.1. BOUNCING SHAPES:

In this first experiment we generate videos of one of three shapes moving on a non-zero background. The shapes are a square, triangle and cross. The image size is 20 × 20 pixels and the shapes are 8 × 8 pixels. The pixel values are between 0 and 1. The shapes are picked with equal probability and they move at constant speed of 1 pixel per frame. The shapes start from random initial positions with and start moving in random directions as well.

We tested two implementations of the proposed architecture: one using only convolutions, referred to as convolutional PUN in the figures, and another using using spatial transformers, called spatial transformer PUN. For the parameters of the convolutional PUN the RNN used was a Long Short Term Memory (LSTM) with 100 cells. The RNN in the Spatial Transformer PUN had 256 cells. In the convolutional PUN, the location layers used to calculate δ xy , l x and l y , output vectors of size 20 pixels and we used the finite addressable memory described in (1). The background is also learned from data as weights of neural network. This background served to make the task more difficult and force the network to avoid just exploiting any non-zero value. After the convolutional composition I t = s t δ xy , we added the background to form a new image using Ĩt = µ • I t + (1 -µ)B, where µ is a differentiable mask that accounts for the "transparency" of the image I t . B is the learned 20 × 20 pixels background image. For complex shapes this mask shape could be calculated as another module in the network, similarly to the approach in Vondrick et al. (2016) . We also show the learned sprites for the convolutional implementation of the proposed Perception Updating Networks when we over-and under-estimate the size of the desired sprites.

In the following experiments, the training videos were 10 frames long. At test time the network is fed the first 10 frames of a video and asked to predict the next 10. Results for the compared methods are shown in Fig. ??. For the baseline method, we did a hyperparameter search on conventional LSTMs with a single linear output layer until we found one that had comparable results at test time. That network had 256 hidden cells. Also, note that although the scale of the mean square error is the same, the results from our proposed architecture look smoother than those learned by the LSTM as shown in Fig. 3 .

Given such a simple experiment, it is elucidating to visualize values learned by each piece of the network. As expected the sprite memory learned the 3 investigated shapes in transposed order since they are reverted by the convolution operation to compose the frame. We also experimented with choosing the size of the learned sprites s t smaller and larger than the true shapes. We observed that for larger shapes such as 10 × 10 the sprites converge to the correct shapes but just using part of the pixels. For smaller shapes such as 6 × 6 pixels, instead of learning a part of the correct shape, the convolutional Perception Updating Network learned to compensate for the lack of enough pixels with more than one non-zero value in the location operation δ xy (see Fig. 3 ). This allow us to suggest to the interested practitioner that in order to get interpretable results it is better to use sprites larger than the expected size than smaller.

For the spatial transformer PUN the image is calculated as (see Algorithm 1 for context):

EQUATION

)

We noticed that the spatial transformer PUN was not able to learn the training videos using an equivalent architecture to the convolutional PUN one. We had to use multiple layers to define the function f (h t ). In other words, in the convolution based method δ xy can be estimated by a single affine transformation of the state h t but A cannot. We also had to use smaller learning rates to As a note on rotation, we ran experiments where the sprite are rotated by a random angle before being placed in the image. This new type of videos cannot be learned using only convolutional based Perception Updating Networks unless we increase the number of sprites proportionally to the number of possible angles. Spatial transformer based Perception Updating Networks can handle this new type of video naturally. Nevertheless, if the number of rotation angles is finite or can be discretized we found that we could learn to generate the videos faster if we combined the convolutional approach with a mechanism to select the appropriate angle from a set of possibilities. Results on this experiment are not shown in this paper due to space constraints but they can be reproduced with the companion code.

4.2. MOVING MNIST:

The Moving MNIST benchmark uses videos generated by moving 28×28 pixel images of hand written digits in a 64 × 64 pixels canvas. Just like in the Bouncing Shapes dataset, the digits move with different different speeds in different directions and can bounce in the walls. Unlike the Bouncing Shapes dataset, there are 60000 different sprites for training and 10000 for test, making it impractical to use a discrete memory module. Instead, we use the memory representation denoted by (13) followed by s t = d(z t ) as written in Algorithm 1.

We trained a convolutional Perception Updating Network using 2 layer LSTMs each one with 1024 cells for 200 epochs, with 10000 gradient updates per epoch. The latent variable z had 100 dimensions and the decoder d(•) was a single hidden layer MLP with 1000 hidden neurons and softplus activation function. The output layer of this MLP has 784 neurons, which is the size of an MNIST image, and sigmoid activation function. In the test set we obtained a negative log-likelihood of 239 nats with the proposed architecture, while a 2 layer LSTM baseline had 250 nats. Note that the our method was optimized to minimize the lower bound (12), not only the negative likelihood. These results are not as good as those obtained by the Video Pixel Networks (Kalchbrenner et al., 2016) that obtained 87 nats on the test set. Nevertheless, both approaches are not mutually exclusive and instead of a fully connected decoder we could use a similar PixelCNN decoder to generate sprites with higher likelihood. In this first paper we decided instead to focus in defining the statistical framework and interpretable "what" and "where" decoupling.

When running the proposed method in rollout mode, feeding the outputs back as next time step inputs, we were able to generate high likelihood frames for more time steps than with a baseline LSTM. Also, since the sprite to be generated and its position in the frame are decoupled, in rollout mode we can fix the sprite and only use the δ xy coming from the network. This way we can generate realistic looking frames for even longer, but after a few frames we observed the digits stopped moving or moved in the wrong direction (see video in the companion code repository). This means that the LSTM RNN was not able to maintain its internal dynamics for too long, thus, there is still room for improvement in the proposed architecture.

In Fig. 5 we show sample rollout videos. The network was fed with 10 frames and asked to generate 10 more getting its own outputs back as inputs and the companion code repository for an animated version of this figure.

This experiment also suggests several improvements in the proposed architecture. For example, we assumed that the internal RNN has to calculate a sprite at every time step, which is inefficient when the sprites don't change in the video. We should improve the architecture with an extra memory unity that snapshots the sprites and avoid the burden of recalculating the sprites at every step. We believe this would a possible way to free representation power that the internal RNN could use to model the movement dynamics for even more time steps.

5. CONCLUSIONS:

This paper introduced a statistical framework for modeling video of 2D scenes inspired by graphics pipelines and variational auto-encoding Bayes. From this statistical framework we derived a variational lower bound that decouples sprites and their dynamics in a video. To optimize this lower bound, we suggested a family of architectures called Perception Updating Networks that can take advantage of this decoupled representation by memorizing sprites or their percepts and updating in location in a scene independently. We showed that this architecture could generate videos that are interpretable and are better suited than baseline RNNs for long video generation.

:



Figures and Tables:

Figure 1: How to get similar results using convolutions with delta-functions and spatial transformers.Input sprite is 8 × 8 pixels and the outputs are 64 × 64 pixels. Note that in the convolution the result shape is rotated 180 degrees and its center is where the delta equals to one at pixel (x = 16, y = 16). Note also that the edges of the spatial transformer results are blurred due to bilinear interpolation. A matrix can be read as "zoom-out" 8 times and translate up and left in a quarter of the resulting size.

Figure 2: A schematic block diagram for a Perception Updating Network. This configuration uses both convolutions with delta functions for translation and spatial transformers for rotation. It also shows the optional background underlay.

Figure 3: Results on the Bouncing Shapes dataset. Three 8x8 sprites (a square, a cross and a triangle) were used to generate videos. The shapes move in a 20x20 pixels canvas with a Toeplitz background and bounce on the corners. a) We show one step ahead predictions with the compared methods. b)We also show the learned sprites for the convolutional implementation of the proposed Perception Updating Networks when we over-and under-estimate the size of the desired sprites.

Figure 4: Learning curves in the test task of two implementations of the proposed architecture (conv PUN and STN PUN) and an equivalent LSTM baseline. Note that the spatial transformer based PUN was not able to generalize to the test set, i.e. they did not work well for generating videos when getting its own previous outputs as next step inputs.

Figure 5: Sample rollouts of a 2 layer LSTM convolutional Perception Updating Network with .

: input videos I t , t ∈ {0, 1, 2, . . .}, initial RNN state h 0 , neural network layers m φ , v φ , d, l, f Result: video predictions I t , t ∈ {1, 2, 3, . . .} for t ∈ {0, 1, . . .} do h t ← RN N (I t , h t-1 )

Table content: <table/>

Peer review (iclr2017):
This paper proposes a generative model of videos composed of a background and a set of 2D objects (sprites). Optimization is performed under a VAE framework. The authors* proposal of an outer product of softmaxed vectors (resulting in a 2D map that is delta-like), composed with a convolution, is a very interesting way to achieve translation of an image with differentiable parameters. It seems to be an attractive alternative to more complicated differentiable resamplers (such as those used by STNs) when only translation is needed. Below I have made some comments regarding parts of the text, especially the experiments, that are not clear. The experimental section in particular seems rushed, with some results only alluded to but not given, not even in the appendix. For an extremely novel and exotic proposal, showing only synthetic experiments could be excused. However, though there is some novelty in the method, it is disappointing that there isn*t even an attempt at trying to tackle a problem with real data. I suggest as an example aerial videos (such as those taken from drone platforms), since the planar assumption that the authors make would most probably hold in that case. I also suggest that the authors do another pass at proof-reading the paper. There are missing references (*Fig. ??*), unfinished sentences (caption of Fig. 5), and the aforementioned issues with the experimental exposition.