Learning to encode motion using spatio-temporal synchrony
Authors: Kishore Reddy Konda, Roland Memisevic, Vincent Michalski
ICLR 2014

Abstract: We consider the task of learning to extract motion from videos. To this end, we show that the detection of spatial transformations can be viewed as the detection of synchrony between the image sequence and a sequence of features undergoing the motion we wish to detect. We show that learning about synchrony is possible using very fast, local learning rules, by introducing multiplicative 'gating' interactions between hidden units across frames. This makes it possible to achieve competitive performance in a wide variety of motion estimation tasks, using a small fraction of the time required to learn features, and to outperform hand-crafted spatio-temporal features by a large margin. We also show how learning about synchrony can be viewed as performing greedy parameter estimation in the well-known motion energy model.


Peer review:
[Review]:
The paper introduces an algorithm to learn to detect motion from video data using coincidence of features between consecutive frames. While there are novel elements, the basic ideas are similar to papers of the same authors and the algorithm by other authors referenced in this paper. 

Details:
- The formula (20) looks remarkably similar the paper you reference (and that beats your performance) - sec 3.1 of 'Learning hierarchical spatio-temporal features for action recognition with independent subspace analysis'. The only difference is mainly that you use sigmoid nonlinearity and they use square root. 

- 4.2 - which auto encoder you are comparing to - there is a large number of them. In particular what if you just used k-means on concatenated frames? That would also detect 'coincidence' between frames. It would also train extremely quickly (and then use smoother version during inference as in Coates et al).

- In table 1 - is this on sequence of frames or pairs? (3.1,3.2 vs 3.3?)