Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper presents a method that given a sequence of frames, estimates a corresponding motion embedding to be the hidden state of an RNN (over convolutional features) at the last frame of the sequence. The parameters of the motion embedding are trained to preserve properties of associativity  and invertibility of motion, where the frame sequences have been recomposed (from video frames) in various way to create pairs of frame sequences with those -automatically obtained- properties. This means, the motion embedding is essentially trained without any human annotations.
Experimentally, the paper shows that in synthetic moving MNIST frame sequences motion embedding discovers different patterns of motion, while it ignores image appearance (i.e., the digit label). The paper also shows that linear regressor trained in KITTI on top of the unsupervised motion embedding to estimate camera motion performs better than chance. 
Q to the authors: what labelled data were used to train the linear regressor in the KITTI experiment? 
Empirically, it appears that supervision by preserving group transformations may not  be immensely valuable for learning motion representations. 



Pros
1)The neural architecture for motion embedding computation appears reasonable
2)The paper tackles an interesting problem

Cons
1)For a big part of the introduction the paper refers to the problem of `` ````"learning motion” or `''understanding motion”  without being specific what it means by that. 
2)The empirical results are not convincing of the strength of imposing group transformations for self-supervised learning of motion embeddings.
3)The KITTI experiment is not well explained as it is not clear how this regressor was trained to predict egomotion out of the motion embedding.

review_title: The proposed unsupervised losses appear weak

final_score: rating: 5: Marginally below acceptance threshold
review_content: Paper proposes an approach for learning video motion features in an unsupervised manner. A number of constraints are used to optimize the neural network that consists of CNN + RNN (LSTM). Constraints stem from group structure of sequences and include associativity and inevitability. For example, forward-backward motions should cancel each other out and motions should be additive. Optimized network is illustrated to produce features that can be used to regress odometry. 

Overall the approach is interesting from the conceptual point of view, however, experimental validation is very preliminary. This makes it difficult to asses the significance and viability of the approach. In particular, the lack of direct comparison, makes it difficult to asses whether the proposed group constraints are competitive with brightness constancy (or similar) constraints used to learn motion in an unsupervised manner in other papers. 

It is true that proposed model may be able to learn less local motion information, but it is not clear if this is what happens in practice. In order to put the findings in perspective authors should compare to unsupervised optical flow approach (e.g., unsupervised optical flow produced by one of the proposed CNN networks and used to predict odometer on KITTI for fair comparison). Without a comparison of this form the paper is incomplete and the findings are difficult to put in the context of state-of-the-art.

Also, saying that learned features can predict odometry “better than chance” (Section 4.2 and Table 2) seems like a pretty low bar for a generic feature representation. 
review_title: Interesting idea, but more detailed experimental validation is needed. 

final_score: rating: 7: Good paper, accept
review_content: The authors propose to learn the rigid motion group (translation and rotation) from a latent representation of image sequences without the need for explicit labels.
Within their data driven approach they pose minimal assumptions on the model, requiring the group properties (associativity, invertibility, identity) to be fulfilled.
Their model comprises CNN elements to generate a latent representation in motion space and LSTM elements to compose these representations through time.
They experimentally demonstrate their method on sequences of MINST digits and the KITTI dataset.

Pros:
- interesting concept of combining algebraic structure with a data driven method
- clear idea development and well written
- transparent model with enough information for re-implementation
- honest pointers to scenarios where the method might not work well

Cons:
- the method is only intrinsically evaluated (Tables 2 and 3), but not compared with results from other motion estimation methods
review_title: Interesting approach to estimate motion

Meta-review: An interesting model, for an interesting problem but perhaps of limited applicability - doesn't achieve state of the art results on practical tasks.
Paper has other limitations, though the authors have addressed some in rebuttals.
Decision: Accept (Poster)