Title: UNSUPERVISED-LEARNING OF TIME-VARYING FEATURES

Abstract: We present an architecture based on the conditional Variational Autoencoder to learn a representation of transformations in time-sequence data. The model is constructed in a way that allows to identify sub-spaces of features indicating changes between frames without learning features that are constant within a time-sequence. Therefore, the approach disentangles content from transformations. Different model-architectures are applied to affine image-transformations on MNIST as well as a car-racing video-game task. Results show that the model discovers relevant parameterizations, however, model architecture has a major impact on the feature-space. It turns out, that there is an advantage of only learning features describing change of state between images, over learning the states of the images at each frame. In this case, we do not only achieve higher accuracy but also more interpretable linear features. Our results also uncover the need for model architectures that combine global transformations with convolutional architectures.

1. INTRODUCTION:

This paper is concerned with the unsupervised learning of features that predict temporal changes in sequence data. Given a time-varying function x(t), represented by a sequence of data points x t , t = 1, . . . , N , our goal is to find a function ż(t) ∝ ∂ ∂t z(t) that encodes a representation of the local changes ẋ(t) ∝ ∂ ∂t x(t). In this context, we understand the time-discretized żt as parameterisation of an (unknown) transformation function f such that x t+1 = f (x t , żt ) = f (x t , z t+1 -z t ), where z t are unobserved feature-vectors. Recent developments for time-series data focus on finding a representation of the state z t and model the time-behaviour żt implicitely. For example Temporal-Difference Variational Auto-Encoders (Gregor et al., 2019) or Kalman Variational Auto-Encoders (Fraccaro et al., 2017) model the time-behaviour via a transition kernel p t (z t+1 |z t ). In these models, the prediction of x t+1 only depends on z t+1 akin to standard Variational Autoencoder (VAE, Kingma & Welling, 2014) . In contrast, an explicit representation of żt offers several advantages:

• Features that are constant within a time-series do not need to be encoded, as żt = 0.

Constant features are instead modeled within the transformation function f (x t , żt ), which leads to a disentanglement of transformation and content. Moreover, f can be chosen to retain fine-grained structure of x t , for example by modeling it through diffeomorphic warps of x t , as is done in deep registration approaches (Yang et al., 2017; Dalca et al., 2018) . • If the feature-space lies on a curved manifold, modeling the tangent-space might be easier than modeling z t . For example, if the observed transformations are image rotations, the transformation can be represented by elements on the unit circle S 1 . While this group can be parameterized with a single variable, the wrap-around occuring between zero and 360 degrees makes it difficult to model it directly. However, if the individual rotations between images are small, żt can be chosen to encode the tangent-space at z t with a single variable. • Calculating żt = g(x t+1 , x t ) can be more precise than modeling it via żt = z t+1 -z t .

As ż(t) is a time-derivative, its precision is highly dependent on noise on z(t). Thus, computing z t and z t+1 independently and computing żt from those point-estimates will lead to a potentially large error in calculation.

Our approach is similar in spirit to Slow Feature Analysis (SFA, Wiskott & Sejnowski, 2002) , which finds signals that vary slowly in time, but does not provide a generative model for x t+1 . Further, SFA is known to learn representations for features that are constant over a time-sequence. Because of this, we replace the principle of slowness with the information-bottleneck of the VAE and condition the reconstruction term on previous observations using a conditional VAE (CVAE, Sohn et al., 2015) to remove constant features. This way, our model learns a low-dimensional parameterization of a transformation. This allows us to analyze how commonly used neural network architectures encode geometric information of objects. To our knowledge, our model is the first that learns directly visualizable parameterisations of such geometric features. The focus of the remainder of the paper is to introduce the model and the challenges that it poses for current network architectures. In the experiments, we will train our model on datasets with geometric transformations and analyse how different design choices influence the learned representations.

2. TRANSFORMATION ENCODING VARIATIONAL AUTOENCODERS:

In this section, we describe the Transformation Encoding Variational Autoencoder (TEVAE). We assume that the generative process underlying the time-series has the form

p(x 1 , . . . , x N , z 1 , . . . , z N , c) = p(z 1 )p(c) N t=1 p(x t |c, z t )p(z t+1 |z t ) ,

where c represents a set of hidden features which are constant over the complete time-series and z t represent time-varying features. We assume that p(z 0 ) = p(z t ), t = 1, . . . , N , i.e., the process is stationary and p(z t+1 |z t ) leaves the distribution invariant. We will simplify this model in two ways. First, we will assume that each x t carries enough information to identify z t and c. Second, we replace p(z t+1 |z t ) by a random variable that models the difference żt = z t+1 -z t . With this, we approximate the generative process as

EQUATION

This model has the property that the observations x t+1 and x t-1 are conditionally independent given x t . Thus, instead of modeling complete time-series, we can restrict ourselves to pairs of observations (x t+1 , x t ). For these pairs, the initial assumption that the time-series must be stationary can be fulfilled simply by adding the time-mirrored pair (x t , x t+1 ) to the dataset. We use this to prevent that a sufficiently powerful p(x t+1 |x t , żt ) learns almost deterministic time-transitions without using żt . As p(x t+1 |x t , żt ) is typically defined via a deep-neural network, exact inference is not possible and we will use the ELBO instead:

EQUATION

Here, we use log p(x t+1 |x t , żt ) = -L(x t+1 , f (x t , żt )) + const ,where L is an error function, for example mean-squared-error, and f is the decoder model. We introduced the regularization parameter β for notational simplicity and it can be removed by re-scaling L. As a prior, we use p( żt ) = N (0, I) and we set as encoder q( żt |x t+1 , x t ) = N (µ(x t+1 , x t ), Σ(x t+1 , x t )), where Σ(x t+1 , x t ) is a diagonal covariance-matrix. To make sure that the latent space can be interpreted, we add a set of constraints on q. First, we enforce that żt = 0 represents the identity. Moreover, if we consider pairs (x t+1 , x t ) and (x t , x t+1 ), we require that µ(x t+1 , x t ) = -µ(x t , x t+1 ) and Σ(x t+1 , x t ) = Σ(x t , x t+1 ). We propose two different models that fulfill these constraints:

• We set µ(x t+1 , x t ) = µ z (x t+1 ) -µ z (x t ) and Σ(x t+1 , x t ) = Σ z (x t+1 ) + Σ z (x t ). This is equivalent to modeling żt = z t+1 -z t with z t = N (µ z (x t ), Σ z (x t )).

It is easy to see that this model fulfills the constraints we impose on the structure through symmetry.

• A relaxation of the first model leads to µ(x t , x t+1 ) = μ(x t+1 , x t ) -μ(x t , x t+1 ) and Σ(x t+1 , x t ) = Σ(x t+1 , x t ) + Σ(x t , x t+1 ).

In the first model, we obtain an explicit representation of z t , however, unlike in standard VAEapproaches, there is no structural constraint on it, i.e., we can not assume that z is normally distributed or lies with high probability in a certain range. For the second model, we do not obtain an explicit representation of z, but allow for non-linear interactions between x t and x t+1 .

3. DECODER ARCHITECTURES FOR THE TEVAE:

The fact that the decoder f (x, ż) uses a pair of a data-point x and a feature-vector ż makes it difficult to apply standard convolutional architectures on image-data. This is because ż might encode a small set of parameters of global transformations, for example an image-rotation, while convolutions apply local transformations to their input. Even though some transformations like image-translations can be reasonably encoded this way -a global translation acts the same on each pixel of the imagemost transformations, for example rotations, look very different when applied to different parts of the image. In this case, the global parameter ż must be decoded to a local transformation before a convolution is applied. At the same time, the decoder must be powerful enough to ensure that the loss of information in x is minimized so that a truthful reconstruction of the identity is possible.

To our knowledge, no suitable architecture exists that allows for this flexibility on large images. Therefore, for the remainder of this paper we will consider two simple architectures for the decoder of the TEVAE:

Densely connected layers Here, ż and x are concatenated as a single input vector to a densely connected feed-forward neural network f . This architecture ensures maximal flexibility of the network. However, each layer must have O(dim(x)) neurons to ensure that the network can learn the identity function for ż = 0. Therefore, the number of parameters and computation time grows quickly with the size of the input. Image-Registration If our goal is to model the transformation between images, training a neural network to predict the next image is inefficient. Instead, we can use a neural network to predict an image-transformation, which we can then apply to the image. To do this, let x ∈ R N ×M ×C be an image of size

N × M with C channels, let p ij ∈ [0, 1] 2 be the position of pixel x ij , I x (p) ∈ R C be the interpolated pixel-value at position p in image x and v ij ∈ R 2 a translation at p ij . With this, we define Warp ij (x, v) = I x (p ij + v ij )

. Thus, Warp returns an image where the pixel-values are the interpolated values at the translated image-positions. Finally, we arrive at the definition of the decoder f (x, ż) = Warp(x, g( ż)), where g : ż → v is a deep neural-network.

4. EXPERIMENTS:

We conduct two experiments, one on an artifical toy-dataset based on MNIST, the other is based on the CarRacing-v0 task. Note that in the following description, the number of hidden layers and the number of hidden neurons are not tuned. However, we saw qualitatively the same results for different network topologies. For the first experiment, we take images from MNIST and apply image rotations and scalings. In this experiment, the goal is to learn a parametrisation of the applied image transformations, independent of the image content. We create the training dataset as follows: Let Φ(x, θ, r) be an affine transformation of the image x, where r is a scaling parameter and θ an 

p ij → m + rR(θ)(p ij -m)

, where R(θ) is a rotation by angle θ and m is the center of the image. The identity transformation is achieved with r = 1 and θ = 0. Given an image x from the MNIST dataset, we create a pair (x 1 , x 0 ) using x 0 = Φ(x, θ 0 , r 0 ) and x 1 = Φ(x, θ 1 , r 1 ). We sample r 0 and θ 0 from distributions r 0 ∼ U(0.85, 1.15) and θ 0 ∼ U(-π, π). The first image has therefore a random orientation and a variation in scale. The second image is rotated and scaled relatively to the first by picking r 1 ∼ r 0 + U(-0.15, 0.15). For θ 1 we apply two different ranges: either, we choose θ 1 ∼ U(-π, π) and rotate the image arbitrarily relatively to the first, or we pick θ 1 ∼ θ 0 + U(-π/4, π/4), a range between -45 and 45 degrees relatively to the first image. To solve this task, the model has to find a representation of ż that encodes the differential transformation

x 1 = Φ(x 0 , θ 1 -θ 0 , r 1 /r 0 ).

As model architectures, we combine both encoder approaches with both decoder approaches. We use a 2-dimensional ż, ELU activations (Clevert et al., 2016) for hidden layers and, if not mentioned otherwise, we use linear activations for output layers. For the encoder architectures, we use a neural network with three dense hidden layers with 98, 12 and 12 hidden-neurons. The output layer encodes mean and log-variance separately for each dimension of ż. We use this model directly to encode µ z (x) and log Σ z (x), which we refer to as z-encoder. For the second encoder approach,which we refer to as ż-encoder, that uses μ(x t+1 , x t ) and log Σ(x t+1 , x t ), we just change the input by stacking both input images on top of each other. For the dense-network based decoder (FFNet), we use five hidden layers with 2 • 784 dimensions each and an output layer with 784 neurons. For the registration-based decoder (Registration), we use five hidden layers in g with 128,128, 128, 1568 and 1568 neurons. We re-interpret the output of the last hidden layer as 7×7 image with 32 channels and use a transposed convolution with stride 2, two 3 × 3 filters and tanh-activation followed by an image scaling to obtain the final v of size 28 × 28 × 2. We use border-replication in I x to extend the image outside the [0, 1] 2 area. For training, we use the mean-squared-error (mean over number of pixels and batch-size) with the Adam optimizer (Kingma & Ba, 2015) . We choose β = 10 -4 , a learning-rate schedule of α t = 10 -3 10 4 10 4 +t and a batch-size of 100. We optimize for T = 10 5 iterations. In total, the MNIST experiment has 8 combinations of encoder, decoder and dataset. For brevity, we encode this as decoder/encoder/degree, e.g. FFNet/z/45 • uses a dense-feed-forward decoder, a z-encoder and the dataset with up to 45 degree rotations between z t and z t+1 .

For the second experiment, we use the CarRacing-v0 environment of OpenAI-Gym (Brockman et al., 2016) and use the methodology by Ha & Schmidhuber (2018) to obtain a set of 10000 training sequences from random policies of length up to 1000 time-steps. We pre-process the images by applying a 3 × 3 Gaussian smoothing kernel and then cutting out an 80 × 80 image-region with 3 color-channels excluding the black information-bar of the image. The smoothing removes a few rendering artifacts around edges and corners and allows for the computation of image-gradients as the original dataset only uses flat color-areas. To make the results easier to visualize, we use pairs (x t+2 , x t ) with a time-stride of two. As in this dataset the car only moves forward between frames, the resulting sequence is non-stationary. We therefore conducted a second experiment where we also add the pairs (x t , x t+2 ).

This task is different from the MNIST task in the way that only parts of the track are visible at any given time. Given the information of the current image, reconstructing the next image in the -0.8 -0.6 -0.4 -0.2 0.0 0.2 0.4 0.6 0.8 θ 1θ 0 sequence is impossible without additional information. To solve this issue, we use an attention concept to weight the reconstruction of each pixel in the image using the reconstruction error

L(x, x) = 1 3 3 c=1 80 i=1 80 j=1 w ij |x ijc -xijc | ,

where w ij = 1 Z exp((i -30) 2 /20 2 + (j -40) 2 /20 2 ) and Z is chosen such that i,j w ij = 1. With this the center of attention is in front of the car, slightly above the middle of the image, while the borders of the image are less important. We show an example of a generated sample-pair and the pixel-wise error contribution to L in Figure 2 .

As models, we only use the registration based approach with both types of encoder, as a dense feedforward decoder would be too large for our GPUs to handle. As before, we use a similar encoder structure for both types of encoders and dim( ż) = 10. We use three convolutional layers with 32, 64 and 64 filters of size 3 × 3 using a stride of two. Afterwards, we use three hidden layers with 512, 128 and 128 neurons. The output of the last hidden layer is the input of the output layers for mean and log-variance. As before, we use ELU-activations in the hidden layer. The decoder uses the same registration based architecture as in the MNIST experiments with a different number of hidden neurons in the dense layers of g. Here we use 4 hidden layers with 10, 10, 800 and 1600 hidden neurons and afterwards interpret the output as image of size 10 × 10 × 16. As before, we use a convolution and image-rescaling to obtain a v of size 80 × 80 × 2. For training, we use Adam with batch-size of 5, β = 5 • 10 -5 and learning-rate schedule α t = 5 • 10 -5 10 4 10 4 +t and T = 5 • 10 4 training iterations.

5. RESULTS:

The results of the first experiment can be seen in Figure 5 . For each of the eight settings, we overlay a grid of reconstructions of x t for varying choices of żt with a scatter plot showing the means of q( żt | x t , x t+1 ) for varying transformations x t+1 = Φ(x t , θ, r). For the full image rotation dataset, we show the results of transforming one symmetric and asymmetric number to visualize the differences in encodings. We can see that the FFNet approach learned encodings that form circular shapes, while the registration based approaches learned linear encodings. Especially in Figure 5c we saw a very clear encoding of the first variable as scale, and the second variable as rotation of the image. For the FFNet approaches, we saw a number of artifacts. Even though the models in Figures 5a&b were trained with rotations of 90 degrees, the encoding seems to be placed on a halfcircle. To analyse this, we took the model from 5b and plotted µ(x t+1 , x t ) with r = 1 and rotation angles θ 0 ∈ {2π • i/10 | i = 1, . . . , 10} and θ 1 -θ 0 ∈ {-π/4 + π/2 • i/20 | i = 0, . . . , 20}.

The result can be seen in Figure 3b . We can see that for each choice of x t , the sampled żt lie on a half-circle. Moreover, when varying the orientation of x t , the circle rotates and when we compare the features obtained from x t with θ 0 and θ 0 + π we observe that the representations almost overlap. Therefore, by construction of the model, we can conclude that z lie on a double circle where one rotation is 180 degrees. When training on full rotations, Figures 5e, f ,i&j, we see that the rotations of the symmetric number one lie on a full circle, while the asymmetric number six has a double-circle shape, where both circles are connected by a twist, forming a prezel-shape. Closer inspection reveals that each circle maps a range of 180 degree rotations. When we performed similar reconstructions of other numbers (not shown), we saw that the inner circle growed the more symmetric the numbers were until they overlap for fully symmetric numbers. There is no visible difference between the z and ż encodings. For the registration approach, we saw a different set of artifacts. In Figure 5d only a subset of sampled points are visible. A closer inspection of the range of ż values, Figure 3b , reveals that the learned encoding of ż (and z by construction) is a piecewise-linear function with large jumps. When plotting a larger area of the space (not shown), we found that the decoder had matching discontinuities that allow for correct reconstruction. Thus, the encoding developed multiple ż-values that parameterize the same rotation. However, when using the full dataset, the model encoding z, Figures 5h&l only managed to reconstruct one half of the transformations correctly. We did not see any relevant artifacts for the registration model that encoded ż.

Looking at the second experiment, Figure 4 , we observe that encoding ż lead to superior performance (Figure 4d ). Using PCA, we found that only a 3 dimensional subspace was used, encoding image scaling, rotation and movement. However, the subspace was not axis-aligned with any three żi . In contrast, the model encoding z failed to model the movement feature. When comparing the stationary with the non-stationary dataset, the models obtained were similar, however with different distribution of features, as can be seen in Figures 4e-h . While all features were non-normal, we see that between the stationary and non-stationary variant the dimension encoding scaling differed a lot. This is because in the task the first few frames are a zoom-in towards the car. We saw a similar effect for the movement direction(not shown), but no effect on rotation, as expected.

6. DISCUSSION & CONCLUSION:

In this paper we introduced a novel approach to learning transformations in time-sequences. By learning a representation of transformations between images, we show that the VAE can be used to reconstruct and predict image-sequences with high accuracy and find minimal, disentangled parameterisations of the modeled transformations.

Our results allow us to get a glimpse into how neural networks encode geometric features. We show that deep dense neural networks encode image-orientation on a circular structure. In all experiments we obtained two rotations around the origin for encoding the full 360 degrees, with a shape depending on magnitude of transformations in the dataset and symmetry of the image. This circular shape makes it difficult to interpret differences in orientation between images as the direction of the difference vector depends on the position of the orientations on the circle and not only their relative position. We enforced the difference vectors to lie on a flat manifold using a registration based approach where the transformation from difference-vector to warp is independent of the input images. This lead to discontinuous and ambigous encodings as the model tried to flatten the circular structure into a shape where difference vectors form a line. However, when we used a more powerful encoder that did not compute the difference-vector from individual states, but instead from the pair of images directly, we obtained perfectly disentangled, linear features. We hypothesize, this is because we provided the encoder with enough information to locally linearize the features and transform them in a single coordinate-system for the decoder. Our registration based architectures are most similar to Yang et al. (2017) and Dalca et al. (2018) in the deep-registration literature, which can be understood in our framework as using a different type of encoder model. We see the applicability of our approach in the medical domain as well as reinforcement learning to discover and parameterize the space of dynamic image-features without having to learn irrelevant features that do not vary over time. Our results also highlight the need for new architectures that merge global parameterisations with convolutional architectures.

:



Figures and Tables:

Figure 1: We assume the underlying sequence to follow a graphical model in the form of (a), where c encodes constant and z t encode dynamic features. Observed variables are shaded. Our model uses the graphical model shown in (b) to approximate this sequence without a representation of c.

Figure2: A pair of samples (x 0 , x 1 ) from the CarRacing-v0 experiment. Shown are x 0 , x 1 and the weighted pixel-wise error contribution to L(x 0 , x 1 ), brighter indicates larger error-contribution.

Figure 3: Visualisation of µ(x t+1 , x t ) for two of the models depicted in Figure 5 obtained by choosing the same input image and transforming it with Φ, varying θ 0 and θ 1 for fixed r = 1. (a): For the model in Figure 5d, fix θ 0 and vary θ 1 -θ 0 ∈ [-π/4, π/4]. It is clear to see that the features approximate a piece-wise linear function with discontinuity. Note that features żi < 2 are outside the visible range of Figure 5d. (b): For the model in Figure 5b, vary θ 0 ∈ {2π • i/10 | i = 1, . . . , 10} and θ 1 -θ 0 ∈ [-π/4, π/4]. Colors encode θ 1 -θ 0 . The black line is an example track of the features for fixed θ 0 . Note that θ 0 and θ 0 + π lead to almost the same encoding.

Figure 4: Visualization of the three features with largest variance of µ(x t+1 , x t ). (a) -(c): Reconstruction of features with ż chosen along the direction of the eigenvectors with largest variance using the VAE-ż trained on the stationary dataset. (d): Histograms of reconstruction errors of the the models trained on the stationary dataset. The non-stationary results are similar. (e) -(h): Histogram and scatter-plot of the two features with largest variance in the VAE-ż-model trained on both datasets. Best viewed on screen.

Figure 5: Visualization of the 2D latent space of ż learned by different architectures on MNIST. Shown are models where the decoder has either an FFNet or registration architecture and the encoder either models z or ż. Models are trained with transformed pairs (x t , x t+1 ) with relative rotations with an angle in either

Peer review (iclr2020):
Review of “Unsupervised-Learning of time-varying features” This work looks at using a conditional VAE-based approach to model transformations of sequential data (transformations can be spatial, such as rotation of MNIST, or temporal, i.e. screenshots of a car racing game). Like how our own visual system encodes differences in time and space [1], they show that in generative modelling with VAEs, “there is an advantage of only learning features describing change of state between images, over learning the states of the images at each frame.” Such an encoding allows the model to ignore features that are constant over time, and also makes it easier to represent data in a rotating curved manifold. They demonstrate this using data collected from CarRacing-v0 task (a task where agents have to learn to drive from pixel observation of a top-down track), and also on MNIST where the digits are rotated around the center of an image. They provide interesting analysis of the latent space learned and show that indeed this approach can handle both stationary and non-stationary features well (in CarRacing-v0). For MNIST, they compare the latent space learned from transformations (z_dot) and show that this approach can encode image geometric transformations quite well. While this paper is interesting and highlights advantages of modeling transformations of sequential data, I don*t think the contributions are currently sufficient for ICLR conference (right now it is a good workshop paper IMHO). For it to be at the conference level, I can make a few suggestions of things that will bring it there, hopefully the authors can take these points as feedback to help improve the work: 1) Would be great to see how this approach can compare to existing proposed algorithms (i.e. TD-VAE as cited)? Are there problems where this approach will perform really well that current methods are inadequate? 2) As the method is based on an RL-task, would the latent representation learned be useful for an RL agent that relies on the latent code across several representative RL tasks (in both sample efficiency, and/or terminal performance)? I don*t mean to discourage the authors (esp as an Anon Reviewer #2...), as I like the direction of the work, and also appreciate that a lot of effort has gone into this work. I hope to see the authors take the criticism to make their work better. Good luck! [1] Concetta1988, Feature detection in human vision: A phase-dependent energy model