Unsupervised-Learning of time-varying features
Authors: Henrik Høeg
Matthias Brix
Oswin Krause
Venue:  (2020)
Keywords: Representation Learning
Variational Autoencoder
Unsupervised Learning
Deep-Learning
Registration

Abstract: We present an architecture based on the conditional Variational Autoencoder to learn a representation
of transformations in time-sequence data. The model is constructed in a way that allows to identify sub-spaces of features indicating changes between frames without learning features that are constant within a time-sequence. Therefore, the approach disentangles content from transformations. Different model-architectures are applied to affine image-transformations on MNIST as well as a car-racing video-game task.
Results show that the model discovers relevant parameterizations, however, model architecture has a major impact on the feature-space. It turns out, that there is an advantage of only learning features describing change of state between images, over learning the states of the images at each frame. In this case, we do not only achieve higher accuracy but also more interpretable linear features. Our results also uncover the need for model architectures that combine global transformations with convolutional architectures.

Under review as a conference paper at ICLR 2020
UNSUPERVISED-LEARNNIG OF TIME-VARYING FEATURES
Anonymous authors Paper under double-blind review
ABSTRACT
We present an architecture based on the conditional Variational Autoencoder learn a representation of transformations in time-seuuence data. The model is con- structed in a way that allows to identify sub-spaces of features indicating changes between frames without learning features that are constant within a time-sequence. Therefore, the approach disentangles content from tramsformations. Different model-architectures are applied to affine image-transformaaions on MNIST as well as a car-racing video-game task, Results show that the model discovers rel- evant parameterizations, however, model architecture has major impact on the feature-space. It turns out, that there an advantage only learning features describing change of state between images, at each frame. this case, we do not only achieve higher accuracy but also more interpretable linear features. Our results also uncover the need for model architec- tures that combine global transformations with convolutional architectures.
learning the states of the images
INTRODUCTION
This paper concerned with the unsupervised learning of features that predict temporal changes in sequence data. Given time-varying function x(t). represented by sequence of data points X,, our goal is to find function z(t) z() that encodes representation of the local changes x(t) 3 x(t). In this context, we understand the time-discretized é, parameteri- sation of an (unknown) transformation function f such that X = fxx, 2.) f(x,, Zt+1 2t), where 2, are unobserved Peature-vectors, Recent developments for time-series data focus on finding representation of the state and model the time-behaviour 2, implicitely. For example Temporal- Difference Variational Auto-Encoders (Gregor et al. 2019) or Kalman Variational Auto-Encoderr (Facccaro et al. 2017) model the time-behaviour via a transition kernel Pa Pr(2+41/2;) In these mod- cls. the prediction of X only depends on Z;+1 akin to standard Variational Autoencoder (VAE, Kingma & Welling 2014). In contrast, explicit representation of 2, offers several advantages:
t ,.,..,N, .,,
Features that are constant within a time-series do not need Constant features are instead modeled within the transformation function f f(x,, 2t), which leads to a disentanglement of transformation and content. Moreover, f can be chosen to retain fine- -iniigr structure of x,, for example by modeling it through diffeomorphic warps of X4, as is done in deep registration approaches (Yang et al.) 2017 Dalca et al. (0018).
encoded, as 2; 0.
If the feature-space lies on a curved manifold, modeling the tangent-space might easier than modeling Z₁. For example, if the observed transformations are image rotations, the transformation can be represented by elements on the unit circle Sy. While this group can be parameterized with a single variable, the wrap-aroum occuring between zero and 360 degrees makes difficult to model it directly. However, the individual rotations between images are small, 2, can be chosen encode the tangent-space at 2, with single variable.
Calculating d, = gxx++1, x.) can be more precise than modeling it via 2, = Z141 - Z₁- As &(l) time-derivative, its precision highly dependent noise on 2(). Thus, computing 2, and 2,+1 independently and computing 2; from those point-estimates will lead potentially large emror in calculation.
Our approach is similar in spirit to Slow Feature Analysis (SFA, Wiskott & Sejnowski 2002). which finds signals that vary slowly in time, but does not provide generative model for X++1. Further,

---

Under review as a conference paper at ICLR 2020
Zt-1
Z
Zt+1
Z
Z
Xt-1
Xt
X++1
1
X
X++1
C
(a)
(b)
Figure I: We assume the underlying sequence to follow graphical model in the form of (a) where encodes constant and 2, encode dynamic features. Observed variables are shaded. Our model uses the graphical model shown in (b) to approximate this sequence without representation of
SFA is known to learn representations for features that are constant over time- sequence. Because of this, we replace the principle of slowness with the information bottllenek of the VAE and condition the reconstruction term on previous observations using conditional VAE (CVAE. Sohn to remove constant features. This way, our model learns low-dimensional parameterization of transformation. This allows us to analyze how commonly used neural network architectures encode geometric information of objects. To our knowledge, our model is the first that learns directly visualizable parameterisations of such geometric features. The focus of the remainder of the paper is to introduce the model and the challenges that it poses for current network architectures. In the experiments, we will train our model on datasets with geometric transformations and analyse how different design choices influence the learned representations.
2015)
TRANSFORMATION ENCODING VARIATIONAL AUTOENCODERS
In this section, we describe the Transformation Encoding Variational Autoencoder (TEVAE). We assume that the generative process underlying the time-series has the form
N
p(x XN. 21 ZN. = p(z))p(c) II (x,le, 2. 2))p(((+12)))
t=1
where represents of hidden features which constant the complete time-series and Z4 represent time-varying features. We assume that p(zo) = p(z),, L 1, 1,.... .N, i.e., the process stationary and p( 2 [z,) leaves the distribution invariant. We will simplify this model in two ways. First, we will assume that each X, carries enough information to identify z, and c. Second, replace p(Z++1/2 |2e) by a random variable that models the difference we approximate the generative process
Ze-1 Zz. With this,
N 2N) = p(x)) II p pxx+1 X,, 2x)p(())
p(x...... XN, 21
This model has the property that the observations X++1 and x are conditionally independent given X,. Thus, instead of modeling complete time-series, we can restrict ourselves to pairs of observations (X++1, x,). For these pairs, the initial assumption that the time- time-series must be stationary can be fulfilled simply by adding the time-mirrored pair (X,, X+++) 1) to the dataset. We use this to prevent that a sufficiently powerful p(X++ [Xe, 2t) learns almost deterministic time-transitions without using z, As p(x++1 [x,, 2,) is typically defined via deep-neural network, exact inference is not possible and we will use the ELBO instead:
logp(x++1 (x,) & (x (log p( (X++1 (x,, 3KL (1(4 ())p(((.)
Here, use log p(x (,,,4111,,,,00 for example weean-qquaredeene error, and f is the decoder model. We introduced the regularization parameter for notational simplicity and it can be removed by re-scaling I.. As prior, we
-L(x f(x, 2 const where is enror function,
2

---

Under review as a conference paper at ICLR 2020
((X(+1,x1)), where maariance-matria To make sure that the latent space can be interpreted, add set of constraints on q. First, enforce that 2, 0 represents the identity, Moreover, we consider pairs (x++1, x,) and (x,, Xx+1), we require that ju (x(+++1 Xt) = - [i(x,; X(+1) and E(x++1,xx -(xx,x++11 We propose two different models that fulfill these constraints:
p(2,) (Q, I) and set encoder q Xx+1, xx)
xx), (((xx++1
[( ) is a diagonal
We set ,(xx+1,,Xi xz) (2(x)+1) - (x{x)) and (x (x((++1) + (x()). This is equivalent modeling 2, Zc+1 Z, with Z N (Uz((e)), (x:)). is easy to see that this model fulfills the constraints we impose on the structure through symmetry.
A relaxation of the first model leads to u(xx,x+++) = f((x+11.x)) - [ixx xx+1) and E(x+11.X) X+) = E(x++1.x x4) + E(x,, X(+1).
In the first model, we obtain explicit representation of Zr, however, unlike in standard VAE- approaches, there is no structural constraint on it, i.e., we can not assume that z is normally dis- tributed lies with high probability in certain range. For the second model, do not obtain an explicit representation of z, but allow for non- non-linear interactions between X₁ and X
3 DECODER ARCHITECTURES FOR THE TEVAE
The fact that the decoder fx, 2) uses a pair of a data-point x and a feature- vector makes it difficult apply standard convolutional architectures image-data. This is because à might encode small set of parameters of global transformations, for example an image-rotation, while convolutions apply local transformations to their input. Even though some transformations like image-translations can be reasonably encoded this way - a global translation acts the same on each pixel of the image most transformations, for example rotations, look very different when applied different parts of the image. In this case, the global parameter à must be decoded to a local transformation before convolution applied. At the same time, the decoder must be powerful enough to ensure that the loss of information in x is minimized so that a truthful reconstruction of the identity is possible. To our knowledge, no suitable architecture exists that allows for this flexibility on large images. Therefore, for the remainder of this paper will consider two simple architectures for the decoder of the TEVAE:
Densely connected layers Here, and concatenated as single input vector densely connected feed-forward neural network f. This architecture ensures maximal flexibility of the network. However, each layer must have (dim(x)) neurons to ensure that the network can learn the identity function for à = 0. Therefore, the number of parameters and computation time grows quickly with the size of the input.
Regge-Regstttttinn If our goal is to model the transformation between images, training neural network to predict the next image is inefficient. Instead, we can use a neural network to predict an amage-transformart
then apply to the image. To do this, let R Nx MxC be an image of size N x M with C channels, let Pay E [0, 1]2 be the position of pixel Xij, Ix()) E RC be the interpolated pixel-value at position p in image x and Vi E IR2 translation Pij- With this, we define
which
Warp,, (x, v) (P + v.z)
Thus, Warp returns an image where the pixel-values the interpolated values at the translated image-positions Finally, we arrive at the definition of the decoder f(x, 2) = Warp(x, g()),, where g is deep neural-network.
4 EXPERIMENTS
We conduct two experiments, one on an artifical toy-dataset based on MNIST, the other is based on the CarRacing-v0 task. Note that in the following description, the number of hidden layers and the number of hidden neurons are not tuned. However, saw qualitatively the same results for different network topologies. For the first experiment, we take images from MNIST and apply image rotations and scalings. In this experiment, the goal is to learn parametrisation of the applied image transformations, independent of the image content. We create the training dataset as follows: Let Φ(x,0, 0,r) be an affine transformation of the image x, where is scaling parameter and an
3

---

Under review as a conference paper at ICLR 2020
Sample xo
(b) Sample x1
(c) Error contribution
Figure 2: A pair of samples (xo, x1) from the CarRacing-vO experiment. Shown are Xo, X₁ and the
weighted pixe-wwie error contribution to L(xo; x1), brighter indicates larger error-contribution.
angle of rotation around the center the image. The transformation warps the pixel positions Piy + (R) m), where R(0) is rotation by angle 9 and is the center of the image. The identity transformation is achieved with r = and = 0. Given an image x from the MNIST dataset, we create a pair (x1,X) using X (xx, lo rn) and X1 0(x,01, sample ro and fg from distributions ro ~ 2/(.855 1.15) and 9g ~ 22(1(17, T). The first image has therefore a random orientation and variation in scale. The second image is rotated and scaled relatively to the first by picking either, we choose 01 ((- TT) and rotate the image arbitrarily relatively to the first, or we pick 81 ~ 6g +0+ /4, TT /4,, range between -45 and 45 degrees relatively to the first image. To solve this task, the model has to find a representation of that encodes the differential transformation X1 = ((x0. 0, 01.r./.rm /r/o).
We
+ ((00115 0.1 For 0 we apply two different ranges:
As model architectures, we combine both encoder approaches with both decoder approaches. We 2-dimensional à. ELU activations (Clevert et al. (016) for hidden layers and, if not mentioned otherwise, we use linear activations for output layers. For the encoder architectures, we use a neural network with three dense hidden layers with 98, 12 and 12 hidden-neurons. The output layer encodes mean and log-vat Var-variance separately for each dimension of We use this model directly to encode (x(x) and log (x), which refer to as z-encoder. For the second encoder approach, which refer to ž-encoder, that uses (( xx) and log Σ(x(+1, Xt), we just change the input by stacking both input images on top of each other. For the dense-network based decoder (FFNet), we use five hidden layers with 2 784 dimensions each and an output layer with 784 neurons. For the registration-baa based decoder (Registration), we use five hidden layers in with 128, 188, 128, 1568 and 1568 neurons. We re-interpret the output of the last hidden layer as 7 image with 32 channels and use transposed convolution with stride 2, two 3 3 filters and tanh-activation followed by an image scaling to obtain the final v size 28 x
We use border-replicationn in Ix to extend the image outside the [0, 1 area. For training. we use the mean- squared-rrr- (mean over number of pixels and batch-size) with the Adam optimizer (Kingma & Ba 2015). We choose 3 = 10 a learning- rate schedule of a 10-3 10 10 and batch-size of 100. We optimize for T = 10 iterations. In total, the MNIST experiment has 8 combinations of encoder, decoder and dataset. For brevity, we encode this decoder//nncoder/degrre e.g. FFNet/z/455 uses dense-ffed--orwaat decoder, a z-encoder and the dataset with up to 45 degree rotations between Z, and Z,+1-
For the second experiment, we use the CarRacing-vO environment of OpenAI-Gym (Brockman et al., (016) and use the methodology by Ha & Schmidhuber (2018) to obtain set of 10000 training sequences from random policies length up 1000 time-steps. We pre-process the images by applying 3 x 3 Gaussian smoothing kernel and then cutting out an 80 x 80 image-region with of the image. The smoothing removes a few
3 color-channels excluding the black information- rendering artifacts around edges and corners and allows for the computation of matage-gradient, as the original dataset only uses flat color-areas. To make the results easier to visualize, we use pairs (x++2, x,) with time-stride of two. As in this dataset the car only moves forward between frames, the resulting sequence is non-stationary. We therefore conducted a second experiment where we also add the pairs (x, xx+2)
This task is different from the MNIST task in the way that only parts of the track are visible at any given time. Given the information of the current image, reconstructing the next image in the
4

---

Under review as a conference paper at ICLR 2020
0.
Feature Value
05
-15
0.
-24
-05
-0.8
-06
-0.4
-02
0.0
0.4
6:
-
(a) FFNet/2/15
(b) Registration///455
Figure 3: Visualisation of (1 Xe) for of the models depicted in Figure obtained choos- ing the same input image and transforming it with ④ varying Ou and 02 for fixed = 1. (a) For the model in Figure 5d fix du and vary 01 Bu E [-T/4, 4].
clear to see that the features
approximate piece-wiss wise linear function with discontinuity. Note that features 2; < 2 are outside the visible range of Figure 5d (b)) For the model in Figure vary 0g E i/10 i 1..... 10)
and 01 - E - /4, т /4] Colors encode 01 - 0g. The black line is an example track of the features for fixed 0g. Note that Og and do + π lead to almost the same encoding.
sequence impossible without additional information. To solve this issue, concept to weight the reconstruction of each pixel in the image using the reconstruction error
attention
3
80
80
I(x, x)
wigl Pijel
3
and Z is chosen such that Ei, Waj 1. With this the center of attention is in front of the car, slightly above the middle of the image, while the borders of the image less important. We show example of generated sample-pair and the pixel-wise error contribution to L in Figure 2
where Why = exp((i -0003
As models, we only use the registration based approach with both types of encoder, as a dense feed- forward decoder would be too large for our GPUs to handle. As before, we use similar encoder structure for both types of encoders and dim(z) = We use three convolutional layers with 32, 64 and 64 filters of size 3 x 3 using stride of two. Afterwards, we use three hidden layers with 512, 128 and 128 neurons. The output of the last hidden layer is the input of the output layers for mean and log-variance. As before, we use ELU-activations in the hidden layer. The decoder uses the same registration based architecture as the MNIST experiments with a different number of hidden neurons in the dense layers of g. Here use 4 hidden layers with 10, 10, 800 and 1600 hidden neurons and afterwards interpret the output as image size 10 x 10 16. As before, we Adam
convolution and image-res obtain of size 80 x 80 2. For training,
with batch-size of 5, B = 10-5 and learning-rate schedule at = 10- 10" and T 10a training iterations.
10
RESULTS
The results of the first experiment can be seen in Figure For each of the eight settings, we overlay grid of reconstructions of x, for varying choices of 2; with a scatter plot showing the means of q(ée xr, Xt+1) for varying transformations X 4(x,, 0, r). For the full image rotation dataset, we show the results of transforming one symmetric and asymmetric number to visualize the differences in encodings. We can see that the FFNet approach learned encodings that form circular shapes, while the registration based approaches leamndd linear encodings. Especially in Figure 5c we saw a very clear encoding of the first variable as scale, and the second variable as rotation of the image. For the FFNet approaches,
number of artifacts. Even though the models in

---

Under review as a conference paper at ICLR 2020
(a) Feature-1 (scaling)
(b) Feature-2 (rotation)
(c) Feature-3 (movement)
Error VAE-ž Error VAE-2
SUNT 2
Foature 2
Count
0.00
0:21
3.00
5.03
0.04
6.55
11.0
-75
-10
-25
00
30
75
163
Reconstruuction error
Feature
Feature
(d) Error-histogram
(e) Feature-scatter, stationary
(i) Feature-scatter, non-stationary
1.9
Feature
Feature Feature
153
Feature
57
53
Feature value
Feature value
(g) Feature-histogram, stationary (h) Feature-histogram,
non-
stationary
Figure 4: Visualization of the three features with largest variance of 4(x++1, xx).
Recon-
struction of features with & chosen along the direction of the eigenvectors with largest variance using the VAE-2 trained on the stationary dataset. [d) Histograms reconstruction errors of the the mod-
els trained on the stationary dataset. The non-stationary results are similar, (e) (h) Histogram and
scatter-plot of the two features with largest variance the VAF-2-model trained on both datasets. Best viewed on screen.

---

Under review as a conference paper at ICLR 2020
Figure 5: Visualization of the 2D latent space of à learned by different architectures on MNIST. Shown are models where the decoder has either an FFNet or registration architecture and the encoder either models z or 2. Models are trained with transformed, pairs (xt, x with relative rotations with an angle in either [-45, 45] (top row) or |- 180, 180] (bottom rows) degrees. Reconstructions are sampled on a à E [-1.5, 1.5) x [-2, 2) grid and the reconstruction of the original image is in the center. The overlayed scatter-ple reatter--pot plot represents the mean ž- values computed by p(x(+1, x,) for different transformed versions of X(++1 Colors/marker type symbolize different relative scalings: orange aquares=85%% blue cirrle== 1==100% and green- crosses=1 11111155 15% Each color has 30 points with angles between the min-and max trained angle range. Note that in d), only a fraction of points are visible.
(1) Registration// 1180
(h) Regisstaation////10
Registration/zZ
(g) Registration/ 1800
(k) Registraation/2/1800
Registration
(b) PFNet/Z
( FFNeU 180
( HFNeti2/1800
FFNeVE//88 180%
FFNet/2/115
( FFNet/2/1800

---

Under review as a conference paper at ICLR 2020
Figures 5a were trained with rotations of 90 degrees, the encoding seems to be placed half- circle. To analyse this, we took the model from 5b and plotted ((xx+1,,x) Xx) with r = 1 and rotation angles 8 E {2TT i/10
L..... 10} and 0 0n E (- + x/2 i/20 | i 0, ..... 20)- The result can be seen in Figure We can see that for each choice of xt, the sampled &, lie on a half-circle. Moreover, when varying the orientation of x,, the circle rotates and when we compare the features obtained from X₁ with 0g and 0 + we observe that the representations almost overlap. Therefore, by construction of the model, we can conclude that z lie on a double circle where one rotation is 180 degrees. When training on full rotations, Figures 5e T we see that the rotations of the symmetric number one lie on full circle, while the asymmetric number six has a double-circle shape, where both circles are connected by a twist, forming a prezel-shape. Closer inspection reveals that each circle maps range of 180 degree rotations. When we performed similar reconstructions of other numbers (not shown). we saw that the inner circle growed the more symmetric the numbers were until they overlap for fully symmetric numbers. There is no visible difference between the z and à encodings. For the registration approach, we saw different set of artifacts. In Figure LeguureSdoon oonly subset of sampled points are visible. A closer inspection of the range of 1 values, Figure [b] reveals that the learned encoding of & (and z by construction) is a piecewise- linear function with large jumps. When plotting a larger area of the space (not shown), we found that the decoder had matching discontinuities that allow for correct reconstruction. Thus, the encoding developed multiple à-values that parameterize the same rotation. However, when using the full dataset, the model encoding z, Figures [ only managed reconstruct one half of the transformations correctly. We did not see any relevant artifacts for the registration model that encoded 2.
Looking at the second experiment Figure we observe that encoding à lead to superior perfor- mance (Figure 4d). Using PCA, we found that only a 3 dimensional subspace was used, encoding image scaling, rotation and movement. However, the subspace was not axis-aligned with any three 2,. In contrast, the model encoding failed to model the movement feature. When comparing the stationary with the non-stationary dataset, the models obtained were similar, however with different distribution of features, as can be seen in Figures 4e h
While all features were non -ormall that between the stationary and non-stationary variant the dimension encoding scaling differed a lot. This is because in the task the first few frames a zoom-in towards the car. We saw a similar effect for the movement direction(not (not shown), but no effect on rotation, as expected.
DISCUSSION & CONCLUSION
In this paper we introduced a novel approach to learning transformations in time-sequences. By learning a representation of transformations between images, we show that the VAE can be used to reconstruct and predict image-sequence, with high accuracy and find minimal, disentangled param- eterisations of the modeled transformations.
Our results allow us to get glimpse into how neural networks encode geometric features. We show that deep dense neural networks encode image-orientation on a circular structure. In all experiments we obtained two rotations around the origin for encoding the full 360 degrees, with a shape de- pending on magnitude of transformations in the dataset and symmetry of the image. This circular shape makes it difficult to interpret differences in orientation between images as the direction of the difference vector depends on the position of the orientations on the circle and not only their relative position. We enforeed the difference vectors to lie on flat manifold using a registration based ap- proach where the transformation from difference-vectoo to warp is independent of the input images. This lead to discontinuous and ambigous encodings as the model tried flatten the circular structure into a shape where difference vectors form line. However, when we used a more powerful encoder that did not compute the difference-vector from individual states, but instead from the pair of im- ages directly, we obtained perfectly disentangled, linear features. We hypothesize, this is because we provided the encoder with enough information to locally linearize the features and transform for the decoder. Our registration based architectures are most
them in a single coordinate-syste similar to Yang et al. (2017) and Dalca et al. (2018) 8) in the deep-regitttation literature, which can be understood in our framework as using different type of encoder model. We see the applicability of our approach in the medical domain as well as reinforcement learning to discover and parameterize the space of dynamic image- features without having to learn irrelevant features that do not vary over time. Our results also highlight the need for new architectures that merge global parameterisations with convolutional architectures.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI Gym, 2016.
Djórk-Amé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (ELJUs). In 4th International Conference on Learning Rep- resentations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
Adrian V. Dalea, Guha Balakrishnan, John Guttag, and Mert R. Sabuneu. Unsupervised learning for fast probabilistic diffeomorphic registration. Medical Image Computing and Computer Assisted Intervention, pp- 729-738. Springer, 2018.
Marco Fraccaro, Simon Kamronn, Ulrich Paquet, and Ole Winther. A disentangled recognition and nonlinear dynamics model for unsupervised learning. In Advances in Neural Information Processing Systems 30, pp. 3601-3610. Curran Associates, Inc., 2017.
Karol Gregor, George Papamakarios, Frederic Besse, Lars Buesing, and Theophane Weber. Tempo- ral difference variational auto-encoder. In International Conference Learning Representattons, 2019.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution, In Advances in Neural Information Processing Systems 31, pp. 2451-2463. Curran Associates, Inc., 2018
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Truck Proceedings, 2015.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. 2nd International Conference Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In Advances in Neural Information Processing Systems 28, pp. 3491.. Curran Associates, Inc., 2015.
Laurenz Wiskott and Terrence J Sejnowski. Slow feature analysis: Unsupervised learning of invari- ances. Neural computation, 14(4):715-770, 2002.
Xiao Yang, Roland Kwitt, Martin Styner, and Mare Niethammer. Quicksilver: Fast predictive image registration-A deep learning approach. Neurolmage, 158:378-396, 2017.


Reviews:
[{"review_id": "ryxFa1xkqr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Review of “Unsupervised-Learning of time-varying features”\n\nThis work looks at using a conditional VAE-based approach to model transformations of sequential data (transformations can be spatial, such as rotation of MNIST, or temporal, i.e. screenshots of a car racing game). Like how our own visual system encodes differences in time and space [1], they show that in generative modelling with VAEs, “there is an advantage of only learning features describing change of state between images, over learning the states of the images at each frame.”\n\nSuch an encoding allows the model to ignore features that are constant over time, and also makes it easier to represent data in a rotating curved manifold. They demonstrate this using data collected from CarRacing-v0 task (a task where agents have to learn to drive from pixel observation of a top-down track), and also on MNIST where the digits are rotated around the center of an image. They provide interesting analysis of the latent space learned and show that indeed this approach can handle both stationary and non-stationary features well (in CarRacing-v0). For MNIST, they compare the latent space learned from transformations (z_dot) and show that this approach can encode image geometric transformations quite well.\n\nWhile this paper is interesting and highlights advantages of modeling transformations of sequential data, I don't think the contributions are currently sufficient for ICLR conference (right now it is a good workshop paper IMHO). For it to be at the conference level, I can make a few suggestions of things that will bring it there, hopefully the authors can take these points as feedback to help improve the work:\n\n1) Would be great to see how this approach can compare to existing proposed algorithms (i.e. TD-VAE as cited)? Are there problems where this approach will perform really well that current methods are inadequate?\n\n2) As the method is based on an RL-task, would the latent representation learned be useful for an RL agent that relies on the latent code across several representative RL tasks (in both sample efficiency, and/or terminal performance)?\n\nI don't mean to discourage the authors (esp as an Anon Reviewer #2...), as I like the direction of the work, and also appreciate that a lot of effort has gone into this work. I hope to see the authors take the criticism to make their work better. Good luck!\n\n[1] Concetta1988, Feature detection in human vision: A phase-dependent energy model\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "SJeUvbXF_H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a VAE architecture that separates a fixed content representation and time varying features of an image, that can be used to learn a representation of image transformations in a temporal setting.\n\nThe paper is well written and the ideas presented are interesting, but in my opinion not novel enough or thoroughly demonstrated to justify acceptance:\n\n- there is a very relevant work that is not mentioned by the authors and that can be seen as a generalization of the model presented in this paper: \"Disentangled Sequential Autoencoder\" by Li and Mandt (ICML 2018), which introduces a model that is also disentangling a content and a temporal representation of sequential data. This is basically the more general model introduced by the authors of this submission in the beginning of section 2, without all the assumptions made in the rest of section 2. A comparison with this related work would help assess the differences in terms of modelling power and in performances.\n\n- The assumptions made in this work are fairly strong for most interesting applications, in particular the fact that the content cannot change across time steps.\n\n- To me, the issue with the novelty of this model would not be a big problem if the authors focused more on showing its usefulness in different applications (e.g. medical domain or RL as mentioned in the conclusions). However, the authors only demonstrate the TEVAE on relatively simple experiments that are only tailored to simple image transformations.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Byxeg0D3KH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a VAE model. The authors consider time series data and claim that in this situation it is better to model the transformations in the latents instead of each datum independently. The setup is reasonable and seems novel, but since it stops at image registration, which is a well-known existing model I cannot qualify the paper as novel. The paper is mostly clear, some claims are not backed up by experiments and the experiments are lacking. As I motivate below I find the current content more at a workshop level than a conference paper. \n\nMajor issues:\n* This paper can become a conference paper in two ways in my opinion. 1) It either needs to show that richer modeling has benefits (if anything it would seem from fig 5 that this is not the case). A way towards that would be to take data where there are no simple transformations that we can introduce and show that it discovers reasonable ones. And 2) show on some highly varying temporal domain that this is better than differences of z. \n* on page 2 \" the initial assumption that the time-series must be stationary can be fulfilled\" -- The data doesn't have to comply with our standards of stationarity. A more sensible formulation is we add these additional constraints to our model which are correct if the data is stationary. \n* On page 3 \"to make sure that the latent space can be interpreted\". This is a very strong claim, it implies that if we do this the latent space will always be interpretable, which I think is false and definitely not backed up by experiments.\n* The conditions on \\dot{z} are interesting and potentially useful and they should be explored in experiments. Putting them in or not does it really make the sense that we think it should make ? Ideally in a setup where the data is not trivial.\n* I am not sure what insight a reader can possibly get from figure 3. \n* Given the final image-registration setup I find that the following citations are necessary: \njaderberg et al. Spatial transformer networks, Shu et al. Deforming auto-encoders: unsupervised disentangling of shape and appearance. \nMinor issues:\n* the authors should number all equations. \n* In their first equation (not numbered) the indices go beyond N+1. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a VAE-based model for learning transformations of sequential data (the main here intuition is to have the model learn changes between frames without learning features that are constant within a time-sequence). All reviewers agreed that this is a very interesting submission, but have all challenged the novelty and rigor of this paper, asking for more experimental evidence supporting the strengths of the model. After having read the paper, I agree with the reviewers and I currently see this one as a weak submission without potentially comparing against other models or showing whether the representations learned from the proposed model lead in downstream improvements in a task that uses this representations.
Decision: Reject