Unsupervised Motion Representation Learning with Capsule Autoencoders
Year: 2021

Abstract:
We propose the Motion Capsule Autoencoder (MCAE), which addresses a key challenge in the unsupervised learning of motion representations: transformation invariance. MCAE models motion in a two-level hierarchy. In the lower level, a spatio-temporal motion signal is divided into short, local, and semanticagnostic snippets. In the higher level, the snippets are aggregated to form fulllength semantic-aware segments. For both levels, we represent motion with a set of learned transformation invariant templates and the corresponding geometric transformations by using capsule autoencoders of a novel design. This leads to a robust and efficient encoding of viewpoint changes. MCAE is evaluated on a novel Trajectory20 motion dataset and various real-world skeleton-based human action datasets. Notably, it achieves better results than baselines on Trajectory20 with considerably fewer parameters and state-of-the-art performance on the unsupervised skeleton-based action recognition task.

heading: 1 Introduction
text: Real-world movements contain a plethora of information beyond the literal sense of moving. For example, honeybees “dance” to communicate the location of a foraging site and human gait alone can reveal activities and identities [10]. Understanding these movements is vital for an artificial intelligent agent to comprehend and interact with the ever-changing world. Studies on social behavior analysis [8, 9], action recognition [59, 64], and video summarizing [60] have also acknowledged the importance of movement.
A key step towards understanding movements is to analyze their patterns. However, learning motion pattern representations is non-trivial due to (1) the curse of dimensionality from input data, (2) difficulties in modeling long-term dependencies in motion sequences, (3) high intra-class variation as a result of subject or viewpoint change, and (4) insufficient data annotation. The first two challenges have been ameliorated by the advances in keypoint detection [57], spatial-temporal feature extractors [38, 43, 50], and hierarchical temporal models [13, 49, 56]. The third and the fourth nonetheless remain hurdles and call for unsupervised transformation-invariant motion models.
Inspired by the viewpoint-invariant capsule-based representation for images [11, 17], we exploit capsule networks and introduce the Motion Capsule Autoencoder (MCAE), an unsupervised capsule framework that learns the transformation-invariant motion representation for keypoints. MCAE models motion signals in a two-level snippet-segment hierarchy. A snippet is a movement of a narrow time span, while a segment consists of multiple temporally-ordered snippets, representing a longer-time motion. In both the lower and the higher levels, the snippet capsules (SniCap) and the segment capsules (SegCap) maintain a set of templates as their identities—snippet templates and
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
segment templates, respectively—and transform them to reconstruct the input motion signal. While the snippet templates are explicitly modeled as motion sequences, the SegCaps are built upon the SniCaps and parameterize the segment templates in terms of the snippet templates, resulting in fewer parameters compared with single-layer modeling. The SniCaps and SegCaps learn transformationinvariant motion representation in their own time spans. The activations of the SegCaps serve as a high-level abstraction of the input motion signal.
The contributions of this work are as follows:
• We propose MCAE, an unsupervised capsule framework that learns a transformationinvariant, discriminative, and compact representation of motion signals. Two motion capsules are designed to generate representation at different abstraction levels. The lower-level representation captures the local short-time movements, which are then aggregated into higher-level representation that is discriminative for motion of wider time spans.
• We propose Trajectory20, a novel and challenging synthetic dataset with a wide class of motion patterns and controllable intra-class variations.
• Extensive experiments on both Trajectory20 and real-world skeleton human action datasets show the efficacy of MCAE. In addition, we perform ablation studies to examine the effect of different regularizers and some key hyperparameters of the proposed MCAE.

heading: 2 Related Works
text: Motion Representation A variety of methods have been proposed to learn (mostly human) motion representation from video frames [3, 23, 28, 53], depth maps [14, 22, 27, 37, 47], keypoints/skeletons [4, 21, 24, 26, 29, 33, 41, 52, 55, 58], or point clouds [6, 7]. Earlier works use handcrafted features like Fourier coefficients [47], dense trajectory features [46, 30], and Lie group representations [44]. Some works use canonical human pose [32] or view-invariant short tracklets to learn robust feature for recognition [19]. The development of deep learning brings the usage of convolution networks (ConvNet) and recurrent networks for motion representation. Simonyan et al. [39] proposes a two-stream ConvNet which combines video frame with optical flow. C3D [43] proposes to use 3D convolution on the spatial-temporal cubes. Srivastava et al. [40] uses an Long Short-Term Memory (LSTM)-based encoder to map input frames to a fixed-length vector and apply task-dependent decoders for applications such as frame reconstruction and frame prediction. The combined use of convolution module and LSTM has also been proved effective in [3, 38, 51].
A series of works [15, 18, 20, 45, 48] have been proposed to address the problem of learning viewpoint-invariant motion representation from videos or keypoint sequences. MST-AOG [48] uses an AND-OR graph structure to separate appearance of mined parts from their geometry information. Li et al. [20] learn view-invariant representation by extrapolating cross-view motions. View-LSTM [18] defines a view decomposition, where the view-invariant component is learned by a Siamese architecture. While most of these works exploits multi-modal input of RGB frames, depth maps or keypoint trajectories, MCAE focuses on the pure keypoint motion.
Capsule Network MCAE is closely related to the Capsule Network [11], which is designed to represent objects in images using automatically discovered constituent parts and their poses. A capsule typically consists of a part identity, a set of transformation parameters (i.e. pose), and an activation. The explicit modeling of poses helps learning viewpoint-invariant part features that are more compact, flexible, and discriminative than traditional ConvNets. Capsules can be obtained via agreement-based routing mechanisms [12, 34]. More recently, Kosiorek et al. [17] proposed the unsupervised stacked capsule autoencoder (SCAE), which uses feed-forward encoders and decoders to learn capsule representations for images.
Apart from images, capsule network has been studied in other vision tasks. In [61, 62], capsule network is used for point cloud processing for 3D object classification and reconstruction. VideoCapsuleNet [5] proposes to generalize capsule networks from 2D to 3D for action detection in videos. Yu et al. [54] proposed a limited study on supervised skeleton-based action recognition using Capsule Network. Sankisa et al. [36] proposed to use Capsule Network for error concealment in videos.
Despite the success of capsule networks in various vision tasks, the study of capsule networks on motion representation is scarce. In this work, MCAE performs unsupervised learning of motion rep-
resented as coordinates rather than pixels. It aims at learning an appearance-agnostic transformationinvariant motion representation. We believe that introducing motion to the capsule network, or the other way round, provides (1) A new, robust, and efficient view into motion signals in any dimension space under the transformation-invariance assumption (while the motion and transformation in dimension space could have semantics different from their 2D/3D counterparts), and (2) proof that disentangling identity from transformation variance works not only for vision problems but a possibly larger family of time series analysis problems.

heading: 3 Methodology
text: We consider a single point1 in d-dimension space. The motion of the point, i.e. a trajectory, is described by X = {xi|i = 1, . . . , L}, where xi ∈ Rd is the coordinates at time i in a d-dimension space. Semantically, X belongs to a motion pattern, subject to an arbitrary and unknown geometric transformation. Given sufficient samples of X, we aim to learn a discriminative (in particular, transformation-invariant) representation for those motion samples without supervision.

heading: 3.1 Framework Overview
text: We solve this problem in two steps, namely snippet learning and segment learning. Snippets and segments correspond to the lower and higher levels of how MCAE views the motion signal. Both snippets and segments are temporally consecutive subsets ofX , but snippets have a shorter time span than segments. In the snippet learning step, the input X is first divided into S = L/l temporally non-overlapping snippets, where l is the length of snippets. Each of these snippets will be mapped into a semantic-agnostic representation by the Snippet Autoencoder. In the segment learning step, the snippet representations are combined and fed into the Segment Autoencoder, where the full motion is represented as a weighted mixture of the transformed canonical representations. The segment activations are used as the motion representation for downstream tasks. An overview of the framework is shown in Fig. 1. In the following section, we delineate the details for each module and explain the training procedure.

heading: 3.2 Snippet Autoencoder
text: To encode the snippets’ motion variation, we propose the Snippet Capsule (SniCap), which we denote as CSni. SniCap is parameterized as CSni = {T ,A, µ}, where T , A, and µ are the snippet template, snippet transformation parameter, and snippet activation, respectively. The snippet
1We show a way to generalize MCAE to multi-point systems in Section 4.2
template T = { ti|ti ∈ Rd, i = 1, ..., l } describes a motion template of length l and is the identity information of a SniCap. A and µ depend on the input snippet. The transformation parameter A ∈ R(d+1)×(d+1) descries the geometric relation between the input snippet and the snippet template. The snippet activation µ ∈ [0, 1] denotes whether the snippet template is activated to represent the input snippet.
Snippet Encoding/Decoding For a given snippet xi:i+l, the snippet module performs the following steps: (1) encode motion properties with Snippet Encoder into SniCaps, and (2) decode SniCaps to reconstruct the original xi:i+l. For the encoding step, a 1D-ConvNet fCONV is used to extract the motion information from xi:i+l and predict SniCap parameters, i.e. {(Aj , µj)|j = 1, . . . , N} = fCONV(xi:i+l) where N is the number of SniCaps. The range of µ is confined by a sigmoid activation function. For the decoding step, we first apply the transformation A to the snippet templates as(
t̂ij
1
) = Ai ( tj
1
) , i = 1, . . . , N, j = 1, . . . , l. (1)
Then, the transformed templates from different SniCaps are mixed, according to their activations, and the corresponding reconstructed input is
x̂j = N∑ i=1 µit̂ij , j = 1, . . . , l, (2)
where t̂ij indicates the transformed coordinate of the ith SniCap at jth time step.

heading: 3.3 Segment Autoencoder
text: The motion information encoded in SniCaps is agnostic to the segment level motion patterns. This makes it less biased towards the training data domain. However, its utility on high-level applications, such as activity analysis or motion classification, is greatly undermined. For example, consider Fig. 2(a) as a reference “triangle” trajectory. Fig. 2(b) illustrates a possible intra-class variation. Since the two trajectories differ greatly in their local movement, they could be considered as different classes without transformation-invariant information from the full trajectory.
Hence, we introduce a segment encoder to gain a holistic understanding of motion and encapsulate such information in the segment capsules (SegCap). A segment is a motion of length L (generally the segment length do not have to be the same as the signal length) and can be interpreted as S = L/l consecutive non-overlapping snippets. A SegCap is parameterized as CSeg = {P,B, ν}, where P , B, and ν are the segment template, segment transformation parameter, and segment activation, respectively.
Different from the SniCap, whose template is explicitly a motion sequence, the SegCap parameterizes the segment template P in terms of the N snippet templates. Specifically, P = {(P i,αi) | i = 1, . . . , S}, where P i ∈ RN×(d+1)×(d+1) and αi ∈ RN . Each P ij ∈ R(d+1)×(d+1) (j ∈ [N ] additionally
indexes the first dimension of P i) describes how the jth snippet template is aligned to form the ith snippet of the segment template. The weight αij (j ∈ [N ] additionally indexes the elements of αi) controls the importance of jth snippet template for the ith snippet of the segment template. In other words, (P i,αi) describes how the N snippet templates are used to construct an l-long snippet and a SegCap requires S such parameters to describe an L-long segment template. Fig. 2(c) illustrates the interpretation of P . B and ν are dependent on the input. B ∈ R(d+1)×(d+1) is a transformation on P , and ν ∈ [0, 1] is the activation of the SegCap. The segment template P is fixed for a SegCap w.r.t the training domain.
Segment Encoding/Decoding Assume we have M SegCaps with which we hope to reconstruct the low-level motion encoded in the SniCap parameters. This is equivalent to reconstructing all the data-dependent SniCap parameters [CSni1 , . . . , CSniS ], where CSnii = {(Aij , µij) | j = 1, . . . , N} is the set of SniCap parameters for the ith snippet. To obtain the SegCap parameters, we first flatten each SniCap’s A into a vector and concatenate it with its corresponding µ. Then we encode the S-long sequence of flattened SniCap parameters with an LSTM model fLSTM shared by all SegCaps, and M fully-connected layers (one for each SegCap) to produce {B, ν}. Formally,
h = fLSTM ([ CSni1 , . . . , CSniS ]) ,
{B(k), ν(k)} = f (k)FC (T ,h), k = 1, . . . ,M, (3)
where T = {Ti|i = 1, . . . , N}, and superscript (k) refers to the kth SegCap. The transformation and activation parameters are then applied to P to reconstruct snippet parameters
P̂ (k) ij = B (k) × P (k)ij , i = 1, . . . , S, j = 1, . . . , N, k = 1, . . . ,M,
ĈSnii = (Âi, µ̂i) = ( M∑
k=1
ν(k)P̂ (k)
i , M∑ k=1 ν(k)α (k) i ) , i = 1, . . . , S,
(4)
where Âi ∈ RN×(d+1)×(d+1) and µ̂i ∈ RN are the reconstructed snippet transformation and activation of the snippet templates for the ith snippet. Note that S = L/l, which means fLSTM can have a much smaller footprint than a recurrent network that handles the whole L-long sequence.
The above formulation enables SegCap to learn a transformation-invariant representation of motion. Intuitively, P describes snippet-segment relation, and B can be regarded as the spatial relation between a segment template pattern and the observed trajectory. The segment activation ν ∈ RM reveals the semantics of the input trajectory and can be used for self-supervised training.

heading: 3.4 Training
text: As delineated in Section 3.2 and 3.3, SniCap and SegCap play different roles by capturing information at two different abstraction levels. SniCap focuses on short-time motion while SegCap is defined upon SniCap to model long-time semantic information. Hence, the two autoencoders are trained using different objective functions.
The only objective of the snippet autoencoder is to faithfully reconstruct the original input. Therefore, for a training sample X = {xi|i = 1, . . . , L}, we use a self-supervised reconstruction loss:
LSniRec = L∑
i=1
||(x̂i − xi)||22, (5)
where x̂i denotes the reconstructed coordinate following Equation (2).
The segment autoencoder’s primary goal is to reconstruct the input SniCap parameters, hence the reconstruction loss
LSegRec = S∑
i=1
||(Âi −Ai)||22 + ||(µ̂i − µi)||22. (6)
Furthermore, we use unsupervised contrastive training to learn semantic meaningful activations ν. For a batch of B samples, the contrastive loss is
LSegCon = − 1
B B∑ i=1 log exp
( cossim(ν′i,ν ′′ i )/τ )∑B j=1 exp ( cossim(ν′i,ν ′′ j )/τ ) , (7)
where τ = 0.1 is the temperature used for all experiments, ν′i and ν′′i is the segment activation of sample X ′i and X ′′i , respectively. Here, X ′i and X ′′i are the spatial-temporally disturbed versions of Xi. The disturbance is dataset-dependent and will be discussed in the supplementary material.
In additional to the above loss terms, we impose two regularizers: a smoothness constraint on reconstructed sequence, and a sparsity regularization on the segment activations
LRegSmt = L∑
i=2
||x̂i − x̂i−1||22, L Reg Sps = ||ν||22. (8)
The final training objective is:
L = λSniLSniRec + λSegLSegRec + L Seg Con + 0.5L Reg Smt + 0.05L Reg Sps, (9)
where the weights are empirically determined. λSni and λSeg are dependent on the target dataset and the remaining weights are consistent for all datasets in our experiment.

heading: 4 Experiments
text: In this section, we first assess the proposed MCAE on a synthetic motion dataset to show its ability in learning transformation-invariant robust representations. Then, we generalize MCAE to multi-point systems and show its efficacy on real-world skeleton-based human action datasets. All unsupervised accuracies are produced by an auxiliary linear classifier that is trained on the motion representation learned by MCAE or the baselines, but whose gradient is blocked from back-propagating to the model. We report the mean accuracy and standard error based on three runs with random initialization. The experiments are run on an NVIDIA Titan V GPU, where we use a batch size of 64, and the Adam [16] optimizer with a learning rate of 10−3. Please refer to the supplementary material for details.

heading: 4.1 Learning from Synthesized Motion
text: The Trajectory20 Dataset Although commonly used in the motion representation learning literature, datasets like movingMNIST [40] are innately linear and have limited motion variations. Moreover, the prediction-oriented setting makes it difficult to examine the motion category of each trajectory. In this paper, we introduce the Trajectory20 (T20), a synthetic trajectory dataset based on 20 distinct motion patterns (as shown in Fig. 3). Each sample in T20 is a 32-step-long sequence of coordinates in [−1, 1]2. In the data generating process, a motion template is randomly picked, randomly rotated and scaled, and translated to a random position to produce a trajectory. A closed trajectory (marked blue in Fig. 3) starts at a random point on the trajectory and end at the same point, whereas an open trajectory (marked yellow in Fig. 3) starts at an either end’s vicinity. The randomized generating process ensures the trajectories are controllably diverse in scale, rotation, and position. The training data is generated on-the-fly and a fixed test set of 10,000 samples is used for evaluation. Examples of T20 are shown in the supplementary material.
Ablation Study We perform an ablation study of MCAE on T20 to examine the effect of different regularizers and three key hyperparameters: snippet length l, the numbers of SniCap
(#Sni) and SegCap (#Seg). The result is shown in Table 1. The length of snippet l plays a vital role in learning a useful representation. A very small l results in a narrow receptive field for snippet capsules, which makes it less useful for inferring semantics of the whole sequence.
At the other end, a large l makes snippets challenging to reconstruct. The numbers of SniCap and SegCap also have major effect on the outcome. Too few SniCaps makes it difficult to reconstruct the input motion signal. Too few SegCaps undermines the expressiveness of the segment autoencoder. Too many SniCaps could cause difficulty in learning proper alignments between SegCaps and SniCaps. Both degrade the quality of the learned features. Moreover, increasing #Seg from 80 to 128 does not bring further improvements. As the result shows, (l, #Sni, #Seg) = (8, 8, 80) performs well and we will use it in all experiments below. As for the regularizers, while both regularizers improve the performance, the sparsity regulation (LRegSps) on segment activation is more helpful for learning discriminative features.
Motion Classification We compare MCAE with the following baseline models, namely KMeans, DTW-KMeans, k-Shape [31], LSTM and 1D-Conv2. KMeans, DTW-KMeans, and k-Shape are parameter-free time series clustering algorithms. Briefly, KMeans uses Euclidean distance to measure the similarity between signals. DTW-KMeans normalizes input signals using dynamic time warping [35], and performs KMeans on the normalized signals. k-Shape uses cross-correlation based distance measure to cluster time series. We use the implementation by tslearn [42] for the three clustering methods. LSTM, 1D-Conv, and MCAE are used as backbone networks, which take the raw coordinate sequence as input and output a feature vector of a pre-defined dimension. The feature vector is used for contrastive learning following Equation (7). The corresponding accuracy reflects the quality of the learned representation.
For LSTM and 1D-Conv backbone, different numbers of hidden units/channels have been explored (shown as Hidden Param. in Table 2), which has resulted in different model sizes (measured by #Param. in Table 2).
As shown in Table 2, since the spatial variance (e.g. viewpoint changes) within motion signal cannot be directly captured by temporal warping/correlation, all the three parameter-free clustering methods perform poorly on T20. On the other hand, with considerably fewer parameters, MCAE outperforms LSTM and 1D-CNN by a large margin. This provides quantitative evidence that MCAE can capture the transformationinvariant semantic information more efficiently than the compared baselines.

heading: 4.2 Generalizing to Multiple Points
text: The MCAE running on T20 dataset handles a single moving point while most real-world problems involve multiple points. This section presents MCAE-MP, a naive but effective extension of
2Architectures of LSTM and 1D-Conv are detailed in the supplementary material.
MCAE, to enable processing motion for multi-point systems. Such motion can be described as X = {Xi|i = 1, . . . ,K}, where K is the number of moving points. The extension works as follows:
1. The K moving points are processed separately by an MCAE. This results in K segment activation vectors {νi, |i = 1, . . . ,K}.
2. The K activation vectors are concatenated into a single representation ν ∈ RKM , which is used for unsupervised learning following Equation (9).
Skeleton-based Human Action Recognition We apply MCAE-MP to solve the unsupervised skeleton-based action recognition problem, where a human skeleton is a system consisting of multiple moving joints (points). Three widely-used datasets are used for evaluation: NW-UCLA [48], NTU-RGBD60 (NTU60) [37], and NTU-RGBD120 (NTU120) [25]. The three datasets consist of sequences with 1 or 2 subjects whose movement is measured in 3D space. For NW-UCLA, we follow previous works [41] to train the model on view 1 and view 2, and test the model on view 3. For NTU60, we follow the official data split for the cross-subject (XSUB) and cross-view (XVIEW) protocols. The similar is implemented on NTU120 for the cross-subject (XSUB) and cross-setting (XSET) protocol. For ease of implementation, we project the 3D sequence into three orthonormal 2D spaces and use an MCAE defined on the 2D space to process the three views of the sequences. Then the segment activations from the three views are concatenated to form the representation. Four types of disturbance are introduced for contrastive learning, namely jittering, spatial rotation, masking, and temporal smoothing. The readers are referred to the supplementary material for details.
The classification accuracy is put into three groups in Table 3. In the first group are the prior works that are not directly comparable as they use depth map [20, 27] or stronger auxiliary classifiers for supervised training [29]. In the second group, where our model is marked as MCAE-MP (SLP), a single layer perceptron (SLP) is trained as the auxiliary classifier with backbone parameters frozen. In the third group, where our model is marked as MCAE-MP (1NN), a 1-nearest-neighbor classifier is used instead of an SLP. For completeness, the fourth group shows state-of-the-art results from supervised methods. Although MCAE-MP is a naive extension as it encodes joints separately and largely ignores their interactions, it achieves better or competitive performance compared with the baselines. Notably, on NTU60-XVIEW and NTU120-XSET where the training set and test set have different viewpoints, our model outperforms baselines by a clear margin thanks to the capsule-based representation which effectively captures viewpoint changes as transformations on input.

heading: 4.3 What does MCAE Learn?
text: To better understand what is encoded, we plot the learned snippet templates T and segment templates P in Fig. 4. Note that T are initialized as random straight lines, and P are initialized as arbitrary patterns composed randomly of T . As shown in Fig. 4a, the snippets are mainly simple lines and hook-like curves that does not carry semantic information. Segment templates in Fig. 4b, how-
ever, bear some resemblance to the patterns shown in Fig. 3. This suggests that semantic-agnostic snippets are being aggregated into semantic-aware segments.
We proceed to explore the information in SegCaps. In particular, we would like to see if SegCaps have learned transformation-invariant information. To this purpose, we randomly sample a trajectory from T20 dataset. The trajectory is first normalized so that its centroid is at (0, 0), then rotated clockwise by an angle θ, and finally fed into the model. We examine the segment templates with the highest activation values (which reflects the trajectory’s semantics) and calculate the rotation angle φ from those templates’ parameterB. As shown in Table 4, the calculated φ reveals two types of segments templates as we rotate the input. One type yields constant φ (e.g. segment ID 2 for sample “absolute sine”), which indicates its rotation-invariance, the other has φ that changes monotonically with θ (e.g. segment ID 8 for sample “hexagon”), which shows its rotation-awareness. As for the activation values, samples from different categories activate different sets of segment templates. Meanwhile, the same sample under different rotation angle θ gives stable segment template activations, despite some changes which are found to have no effect on the classification result.
We do a similar study on the translation component (x, y), where we translate the input by (∆x,∆y). As shown in Table 5, (x, y) changes monotonically with (∆x,∆y) while the activated segment templates remain stable. These results prove that the semantics and transformation information has been
encoded separately in the segment activation ν and transformation parameters B. In other words, the encoded semantic information is robust against geometric transformations.

heading: 5 Conclusion
text: In this paper, we introduce MCAE, a framework that learns robust and discriminative representation for keypoint motion. To resolve the intra-class variation of motion, we propose to learn a compact and transformation-invariant motion representation using a two-level capsule-based representation hierarchy. The efficacy of the learned representation is shown through an experimental study on synthetic and real-world datasets. The output of MCAE could serve as mid-level representation in other frameworks, e.g. Graph Convolution Network, for tasks that involve more context than classification. We anticipate this work to inspire further studies that apply capsule-based models to other time series processing tasks, such as joint modeling of visual appearance and motion in video. The source code and the T20 dataset of our research are accessible at https://github.com/ ZiweiXU/CapsuleMotion.
Motion analysis techniques are in the foreground of the misuse of machine learning methods, among which adverse societal impacts and privacy breach are two major concerns. Regarding the societal impacts, admittedly, our method has both upside and downside. On one hand, a transformationinvariant motion representation enables us better decode the information implicit in the trajectory, which has applications for example in ethology. On the other hand, it could also be misused in mass surveillance. Appropriate boundaries of use and ethical review are required to prevent potential malicious applications. Regarding the privacy concerns, our method isolates the subjects’ motion from their sensitive information, such as gender and race.
Acknowledgments and Disclosure of Funding
This research/project is supported by the National Research Foundation, Singapore under its Strategic Capability Research Centres Funding Initiative. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore. The computational work for this article was partially performed on resources of the National Supercomputing Centre, Singapore (https://www.nscc.sg).

Review summaries:
This work proposes a motion representation learning method based on capsule autoencoders. It decomposes an input 2D keypoint trajectory into snippets and segments, and form a hierarchical representation that is also transformation invariant. The proposed motion representation achieves state-of-the-art motion classification results on a self-proposed synthetic 2D motion dataset as well as skeleton-based human action recognition.

This paper presents an unsupervised motion representation learning method with capsule autoencoders. Its framework is composed of two-level motion modeling: snippet-level for short and local motion capturing, and segment-level for full-length semantic-aware motion capturing. It presents a new dataset called as Trajectory 20 and performs experiments on two datasets.

This work presents a Motion Capsule Autoencoder (MCAE) which learns to represent motions. The two-staged architecture first learns a set of snippet capsules, which learn motions for short time-spans, and then learns segment capsules that learn the long-term motions from all the snippet capsules. These capsules are learned end-to-end in an unsupervised manner with reconstruction losses and a contrastive loss. The approach is evaluated on a synthetic motion dataset proposed in this work, Trajectory20, as well as on three real-world skeleton-based human action recognition datasets, where it is shown to achieve strong performance when compared to previous state-of-the-art methods.

The paper presents a method for incorporating transformation invariance using capsule networks in the unsupervised learning of motion representations. The proposed method solves the problem in two steps, snippet and segment learning, which correspond to a lower and higher level of motion signals (basic trajectories -> reconstruction/time -> Contrastive learning and reconstruction). The method was evaluated on a synthetic dataset and three skeleton datasets for action recognition achieving convincing results.