Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This work presents a Motion Capsule Autoencoder (MCAE) which learns to represent motions. The two-staged architecture first learns a set of snippet capsules, which learn motions for short time-spans, and then learns segment capsules that learn the long-term motions from all the snippet capsules. These capsules are learned end-to-end in an unsupervised manner with reconstruction losses and a contrastive loss. The approach is evaluated on a synthetic motion dataset proposed in this work, Trajectory20, as well as on three real-world skeleton-based human action recognition datasets, where it is shown to achieve strong performance when compared to previous state-of-the-art methods.
main_review: Overall, this is an interesting work which presents a novel capsule architecture (MCAE) to learn motion patterns in an unsupervised manner. Their experimental evaluations on both synthetic and real data, comparisons with previous approaches, and analysis all support the strengths of this work. I suggest that is paper should be accepted, but the following questions/comments should be addressed to improve the clarity of the work.

1) How are the templates learned/obtained? They are defined as $ \mathcal{T} = t_i | t_i \in \mathbb{R}^d, i=1,..., l $. Are they a set of learned parameters, which are trained with the full system in an end-to-end manner? Also, $d$ is not defined - since the network works on 2d points, and $A$ is a $3\times3$ matrix, I assume $d=2$, but it should be stated explicitly.

2) In the ablation study, how is the accuracy obtained? Later in lines 227-228, it states that an MLP is learned on top of the model for the motion classification experiments. Is it the same for the ablations? If so, this information should be included in the, or prior to the ablations section.

3) How is the LSTM used in Eq. 3 to encode the SniCaps? Since $C_i^\text{Sni}$ is a set of tuples $(A_i,\mu_i)$, what operation is used to combine each $C_i^\text{Sni}$ into a single representation/vector?

4) In Section 4.3, the authors state that Table 4 shows that the top-5 segment templates for the hexagon remain consistent across all listed rotations, but changes for abs_sine (e.g. IDs 33 and 32 are present in $\theta=-10$ and $\theta=10$ respectively). Is this due to the nature of the motion pattern (i.e. the abs_sine's open trajectory rather than the hexagon's closed trajectory), or is such behavior common for many different motion classes?

5) Is $\mu$ the result of a sigmoid or softmax activation?
limitations_and_societal_impact: The authors address potential negative societal impacts, but could include more discussion about the limitations. For example, what type of motion patterns does the MCAE not perform well on? Can the method generalize to longer sequences (i.e. $L>32$)?
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work proposes a motion representation learning method based on capsule autoencoders. It decomposes an input 2D keypoint trajectory into snippets and segments, and form a hierarchical representation that is also transformation invariant. The proposed motion representation achieves state-of-the-art motion classification results on a self-proposed synthetic 2D motion dataset as well as skeleton-based human action recognition.
main_review: Strength: 

- The proposed method is based on the important insight that movement patterns should be transformation invariant and also should naturally consist of a hierarchy of patterns. Thus, the proposed method is intuitive and easy to understand. To the best of my knowledge, using the formulation of capsule autoencoders to explicitly learn transformable trajectory templates through self-supervised learning is novel and elegant. Such formulation can also provide an interpretable representation for future motion analysis tasks. 
- The paper conducts extensive experiments and studies that backed up its claims:
    - Experiments on the synthetic T20 dataset suggest that the proposed method is an effectively self-supervised motion representation learning method.
    - Real-world applications on self-supervised action recognition also suggest the strength of the proposed method.
    - Ablation studies that specify the interpretability of the proposed method: I find section 4.3 especially helpful in showcasing the benefit of the proposed method, where the transformation invariance is well-documented.
- The proposed T20 dataset would be beneficial as a standard test ground for future 2D motion analysis methods.

Issues:

- In section 4.3, it seems old that theta is constrained to [-10, 10] degrees, where a hexagon should be expected to activate similar segments for large rotation perturbation?
- To what extent can a motion trajectory be transformed but still be classified as a class? The proposed method seems to be confined to 2D similarity transformation instead of the full affine/projective transformation. 
- (Minor) It would be beneficial to include state-of-the-art results on supervised skeleton-based action classification, to give readers an idea of how far unsupervised methods are away from the supervised ones.
- (Minor) It would be beneficial if additional ablation on the effect of the contrastive learning loss could be provided. While conceptually the contrastive learning loss would be beneficial in learning the segments, I am curious to see its effects compared to only using reconstruction loss. 

====================================================================================================

After the author response:

The answers from the authors are largely satisfactory to me (concerning robustness with rotation, contrastive loss). I think the proposed MCAE would be a worthy contribution to the current capsule network literature. I would like to keep my original score.
limitations_and_societal_impact: Yes, the authors have adequately addressed the limitations and potential negative societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents a method for incorporating transformation invariance using capsule networks in the unsupervised learning of motion representations. The proposed method solves the problem in two steps, snippet and segment learning, which correspond to a lower and higher level of motion signals (basic trajectories -> reconstruction/time -> Contrastive learning and reconstruction). The method was evaluated on a synthetic dataset and three skeleton datasets for action recognition achieving convincing results.
main_review: Strengths: 
-The paper is well-structured
-Experimental results are promising 

Limitations:
- I found the description of the model very hard to follow (both the snipped and segment learning). For someone that is not familiar to CapsNet, it basically is impossible to understand what is happening. Basics of CapsNet should be recalled to build a parallel with the proposal. 
-The authors noted a few works that employ capsule networks in motion representation in Related Works [3,30, 47] but did not attempt to compare against them in the experiments. 
-2D projected inputs were used instead of 3D pose sequence, did the authors try to experiment with the original 3D sequences?
-It would be interesting to see what has the model learnt on the skeleton dataset.

Originality:
- The work is original. Capsule Nets have been designed especially to do this. However, few works that were cited introduced ways of representing "images". A comparison should have been made to see if the introduction of "motion" really is bringing something to the Table. 

Clarity:
- The model Section should be clarified, making references to what a CapsNet is (formal definition).

Significance:
-The proposed method could have an impact on motion representation.
limitations_and_societal_impact: n/a
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper presents an unsupervised motion representation learning method with capsule autoencoders. Its framework is composed of two-level motion modeling: snippet-level for short and local motion capturing, and segment-level for full-length semantic-aware motion capturing. It presents a new dataset called as Trajectory 20 and performs experiments on two datasets.
main_review: 1. The proposed autoencoder is a feed-forward network of temporal convolutions to predict the transformation parameter and activations. I cannot figure out the relation between the proposed autoencoder and the traditional capsule network. This paper needs to make a clear description on its work with respect to the capsule network.

2. This paper fails to present experiments on more common motion datasets, like movingMNIST, and thus I cannot clearly tell the advantages of the proposed method over previous ones.

3. In the ablation studies, the authors fail to provide a study on the importance of two-level motion modeling. For example, it should compare with a simple baseline of one-level modeling with the same number of snippets.

4. The evaluation datasets are quite clear and like toy examples. I would like to see more experiments on motion representation learning on more realistic datasets, like cloud points for scene flow estimation, RGB videos for optical flow estimation.

5. The idea of two-level motion modeling has appeared in many video action recognition works. For example: V4D:4D Convolutional Neural Networks for Video-level Representation Learning [ICLR 2020]. It should discuss these works in the related work section.

limitations_and_societal_impact: N/A.
review_title: null

Meta-review: This work introduces a novel capsule architecture that learns motion patterns in an unsupervised manner. Evaluation on synthetic and real-work skeleton-based human action datasets demonstrate the model good performance when compared to previous state-of-art methods. 

Using the formulation of capsule autoencoders to explicitly learn transformable trajectory templates through self-supervised learning is novel and an interesting addition to the capsule network literature.

Decision: Accept (Poster)