Representing dynamically: An active process for describing sequential data

Abstract: We propose an unsupervised method for building dynamic representations of sequential data, particularly of observed interactions. The method simultaneously acquires representations of input data and its dynamics. It is based on a hierarchical generative model composed of two levels. In the first level, a model learns representations to generate observed data. In the second level, representational states encode the dynamics of the lower one. The model is designed as a Bayesian network with switching variables represented in the higher level, and which generates transition models. The method actively explores the latent space guided by its knowledge and the uncertainty about it. That is achieved by updating the latent variables from prediction error signals backpropagated to the latent space. So, no encoder or inference models are used since the generators also serve as their inverse transformations.
      The method is evaluated in two scenarios, with static images and with videos. The results show that the adaptation over time leads to better performance than with similar architectures without temporal dependencies, e.g., variational autoencoders. With videos, it is shown that the system extracts the dynamics of the data in states that highly correlate with the ground truth of the actions observed.

Reviews and author responses:

The authors propose an architecture and generative model for static images and video sequences, with the purpose of generating an image that looks as similar as possible to the one that is supplied. This is useful for for example frame prediction in video and detection of changes in video as a consequence to changes in the dynamics of objects in the scene.
The architecture minimizes the error between the generated image(s) and the supplied image(s) by refining the generated image over time when the same image is shown and by adapting when the image is changed. The model consists of three neural networks (F_Zµ, F_Zsigma, f_X|Z) and three multivariate Gaussian distributions P(S_t), P(Z_t) and P(X_t,Z_t) with diagonal covariances. The NNs do not change over time but they relate the three Gaussian distributions in different ways and these distributions are changing over time in order to minimize the error of the generated image(s).

The paper took a while to understand due to its structure and how it is written. A short overview of the different components of Figure 1 giving the general idea and explaining
* what nodes are stochastic variables and NNs
* what is trained offline/online
It would also help the structure if the links/arrows/nodes had numbers corresponding to the relevant equations defining the relations/computations. Some of these relations are defined with explicit equation numbers, others are baked into the text which makes it difficult to jump around in the paper when reading it and trying to understand the architecture.

There are also numerous language errors in the paper and many of them are grammatical. For example:
  Page 5, second paragraph: "osculations" -> "oscillations"
  Page 5, fourth paragraph: "Defining .. is defined as.."

The results seem impressive and the problem under consideration is important and have several applications. There is however not much in terms of discussion nor analysis of the two experiments.
I find the contribution fairly significant but I lack some clarity in the presentation as well as in the experiments section.
I do not find the paper clearly written. The presentation can be improved in several chapters, such as the introduction and the method section.
The paper seem to be technically correct. I did not spot any errors.

General comments:
- Why is 2 a suitable size of S?
- Why use two by two encoding instead of stride for the VAE baseline? How does this affect the experiments?
- How is S_0 and the Z_0 prior set (initialized) in the experiments?
- It would improve the readability of the paper, not least for a broader audience, if more details are added on how the VAE baseline architecture differ from the proposed architecture.

- In the first experiment: 
-- How many iterations does it take for your method to beat VAE?
-- What is the difference between the VAE basline and your approach that make VAE perform better than your approach initially (or even after a few iterations)? 
-- What affect does the momentum formulation have on the convergence rate (number of iterations necessary to reach VAE and your methods result at t=10)?  

- In the second experiment and Figure 5 in particular it was observed that some actions are clearly detected while others are not. It is mentioned that those that are not detected by the approach are more similar. In what sense are the actions more similar, which are the most prominent (from a humans perspective) such actions, what is making the model not detecting them and what can be done (within your approach) in order to improve or adjust the detection fidelity?

Please add more time labels under the time axis in Figure 4 and Figure 5. Also please annotate the figures at the time points where the action transitions are according to the ground truth.

This paper, to me, is a clear rejection from these basic observations:

A *model* is not a computation graph and should never be presented that way. 
Calling a computation graph a Bayesian network without even writing down how *inference* can ever result in such a computation graph is a basic error. 
The authors claim this: "So, no encoder or inference models are used since the generators also serve as their inverse transformations." Well, then this is not a Bayesian network. 
The authors spend a lot of time analyzing constant inputs theoretically, and I'm not sure why this is even relevant. 

Summary:

The paper proposed an Bayesian network model, realized as a neural network, that learns
1. latent representation of observed data (images in the paper).
2. dynamics of interaction in sequential data in the form of a linear dynamical system w.r.t. latent representation, controlled by secondary latent variables.

The model is evaluated in two scenarios:
1. Static images on CelebA dataset, where it shows that iterative guided updates of the latent representation improve reconstruction quality, compared to VAE.
2. Sequential data experiment, the authors show that the interaction states can be used for semantic action segmentation.

-------------------------------------------------------
Pros:
1. The proposed model is unsupervised, and it can iteratively improve the latent representation and consequently the systhesized output, given more observations at inference.

2. The paper proposes novel strategies to update the latent distributions' parameters.

-------------------------------------------------------
Cons:
1. The problem fornulation with Z, X and S are not clearly defined at the beginning, the reviewer must read further into page 4 and 5 to understand. Brief description should be provided in/around Fig 1. Furthermore, the log likelihood loss is not clearly defined before Eq (1).

2. The proposed updates to mu_z, sigma_z, mu_s and sigma_s in Eq (2,3,6,9) and their convergence properties as well as stability are not justified by actual experiments and analysis.

3. The overall training procedure is unclear: do the hidden layer weights get updated given repeated observations in the case of static model?

4. In the static image experiment, why the authors did not compare to at least GAN (or better if GLO [Bojsnowski et al. 2017] is included)?
The iterative updates clearly give the proposed model advantage over VAE. VAE also relies on reconstruction loss, hence the synthesized output are often blurry.
GAN, which can generate high quality images, should provide a better benchmark. One can use the same method as this paper, backpropagating the reconstruction error of GAN generator to the input layer to find the suitable noise vector of an image, then synthesize the image with the generator from that noise input.

5. One the two objectives of the paper, image reconstruction in sequence, is not evaluated in the dynamic experiment. The paper does not provide any analysis for A, Z and \hat{X} in the dynamic setting.
Instead, the secondary latent states S are used for sequence segmentation. Why is it not compared to HMM baseline at least, in that case?
Furthermore, the paper does not provide a standard methodology to segment actions from video, but rather, the reviewer must look at the plot in Fig 4 & 5 and read the description in page 8 to see the correspondence between the variation of S and the action switch in the video.
In addition, the segmentation is only carried out on one video in the paper.
Moreover, the paper only experiments with S of length 2. Given the limited evaluation, the authors can try and report the results with different lengths of S (e.g. 1 or 3, 4).

The paper proposes a hierarchical probabilistic model that learns both static representations and the dynamics of the data. The model iteratively updates its latent representations of the data in order to improve its generative power. The model is applicable to both static images (iterative improvements of the samples) and videos (predictive coding like repesentations).

Pros:

-- the motivation for the work and its connections to the cognitive/philosophical models of concepts and predictive coding is very interesting
-- the iterative improvements in the celebA samples in Fig. 3 and the corresponding improvements in the log-likelihood in Tbl. 1 vs the vanilla VAE baseline are promising and suggest that the approach has potential

Cons:

-- The major problem with this paper in my opinion is that the methods section is very confusing:
    1) The section is too brief and there is absolutely no description of the model until page 4
    2) The figures are hard to understand and the annotations are not informative (e.g. what is the difference between Fig.1 and Fig.2?) 
    3) The notation is unconventional and keeps changing (e.g. the generator is referred to as either \varphi_A, \varphi_X,  \varphi_X(Z_t), X_t|Z_t, or \mu_X; \sigma_X... the dimensionality of the image is denoted as i, N * c or Nc... I can go on).  
    4) The rescaling of the latent parameters seems engineered and arbitrary (e.g. \beta scaling factor in Eq. 8 is chosen so that the sigmoid reaches 0.75 when the value is 0.5\sigma of the threshold).

Due to the points above I failed to fully understand the model despite trying hard to do so. In particular, I did not understand the most important part of the paper addressing the iterative update of the latents vs backprop update of the generative weights. 

Minor points:
-- The introduction is too long and repetitive. The space saved should be used to describe the model more precisely.
-- The parametrisation of S_t should be described when it is first introduced, not 2 paragraphs later.
-- How does an inner product of two vectors result in a matrix (Sec. 3.3)?
-- GANs also do not have an encoder network (despite what the authors claim in Sec. 4.1) and should be used as a baseline
-- Why does the VAE baseline have a different decoder architecture than the proposed model?
-- What is the pre-processing done for CelebA?
-- What is the ground truth that was supposed to be matched by \mu_S_t in the dynamic dataset?
-- Figs. 4-5 are hard to understand. What do the different colours of the lines mean? The time stamps where the behaviours are changing should be marked in the plot (not just described in the text).

To conclude, the authors are advised to shorten the introduction and literature review sections and use the extra space to re-write and expand the methods section to make it very clear how their model works using the standard notation used in the literature. The results section detailing the dynamic setup of their approach needs to be made more clear as well. In the current form the paper is not ready for publication.








Meta-review: This paper proposes a model which learns simultaneously the dynamics of sequential data, together with a static latent representation. The idea and motivation is interesting and the results are promising.

However, all reviewers agree that the presentation needs much more work to convey the messages correctly and convincingly. Moreover, the reviewers question some design choices and lack of discussion of the results. No rebuttal has been provided.



Decision: rejected-papers