Learning to encode motion using spatio-temporal synchrony
Authors: Kishore Reddy Konda, Roland Memisevic, Vincent Michalski
ICLR 2014

Abstract: We consider the task of learning to extract motion from videos. To this end, we show that the detection of spatial transformations can be viewed as the detection of synchrony between the image sequence and a sequence of features undergoing the motion we wish to detect. We show that learning about synchrony is possible using very fast, local learning rules, by introducing multiplicative 'gating' interactions between hidden units across frames. This makes it possible to achieve competitive performance in a wide variety of motion estimation tasks, using a small fraction of the time required to learn features, and to outperform hand-crafted spatio-temporal features by a large margin. We also show how learning about synchrony can be viewed as performing greedy parameter estimation in the well-known motion energy model.


Peer review:
[Review]:
The paper introduces a variation of common mathematical forms for encoding motion.  The basic approach is to encode first-order motion information through a multiplication of two filter outputs.  This approach is closely related to the motion-energy model and the cross-correlation model.

There is a lot of math leading up to a rather simple transform (after learning).  I would explain the resulting model as simply 'squared spatio-temporal Gabor filters followed by k-means pooling.'  The squared outputs are the components of the energy-based model (quadrature pair squared spatio-temporal Gabor filters are added in the energy based model) and k-means is used to pool first-order motion selective responses.

The more general problem of motion encoding needs to address the goal of motion selectivity and form or pattern invariance/tolerance.  As the authors point out, their squared outputs do not solve the motion encoding problem and the following pooling layer is intended to provide the solution.  The simplest next step would be to combine (add) two outputs to increase form invariance (this is the motion energy model).  Slightly more complex would be to group multiple squared outputs (these are the ISA models applied to spatio-temporal input).  The authors propose to use k-means for the pooling operation.

The results are validated on common action recognition computer vision databases.  I do not find the results surprising.  Given the very close similarities of the proposed algorithm to the Le et al. ISA algorithm it is not at all surprising that the results are nearly identical.  The training time improvements are also not surprising given the results from Coates et al. 2011.

This paper should probably be cited in regards to the learned filters:
Olshausen 2003, Learning Sparse, Overcomplete Representations of Time-Varying Natural Images
There are other results in the ICA community that should be cited, as they give similar results.

I am confused about some details on the model architecture used for SAE and SK-means.  The beginning of section 4 appears to only describe the SAE architecture.  What about the Sk-means architecture?  and how does the k-means in section 3.4 relate to the models evaluated in section 4?  My apologies if this is discussed somewhere in the text, I just can't find it in the places I expect.

Here are some suggestions/comments:
The exposition is useful for bringing together many of motion models in the literature.

You argue for even-symmetric non-linearities, but couldn't a rectified linear unit also produce the desired effect? ; however it would not associate the contrast reversal, which may be the better inductive bias.

In the current form I remain uncertain what the benefit is of the proposed approach over the motion energy approach or ISA.  K-means does pool over more than 2 features, but this is to be expected given the architecture.  Can you show that 2-d subspaces are sub-optimal in terms of performance?  You do show that the covAE underperforms, why?  Is the Le et al. 2011 pipeline giving the major benefit?

In my opinion, the most novel contribution of the paper is the 'synchrony k-means' algorithm.  As far as I am aware this is a nice extension of the fast convergence results shown by Coates et al..  This general framework appears promising for the related problems of 'relating images.'  Therefore it could be the focus of the paper, with the SAE algorithm used merely as a control or motivation.

I am also skeptical about performance measurements of activity recognition as proxy for motion encoding.  These datasets likely suffer from various confounds not related to motion encoding.  Maybe a simpler test would be beneficial, scientifically?