Unsupervised Motion Representation Learning with Capsule Autoencoders
Authors: Ziwei Xu
Xudong Shen
Yongkang Wong
Mohan Kankanhalli
Venue: NeurIPS 2021 Poster (2021)
Keywords: Capsule Network
motion representation
skeleton-based action recognition

Abstract: We propose the Motion Capsule Autoencoder (MCAE), which addresses a key challenge in the unsupervised learning of motion representations: transformation invariance. MCAE models motion in a two-level hierarchy. In the lower level, a spatio-temporal motion signal is divided into short, local, and semantic-agnostic snippets. In the higher level, the snippets are aggregated to form full-length semantic-aware segments. For both levels, we represent motion with a set of learned transformation invariant templates and the corresponding geometric transformations by using capsule autoencoders of a novel design. This leads to a robust and efficient encoding of viewpoint changes. MCAE is evaluated on a novel Trajectory20 motion dataset and various real-world skeleton-based human action datasets. Notably, it achieves better results than baselines on Trajectory20 with considerably fewer parameters and state-of-the-art performance on the unsupervised skeleton-based action recognition task.

Unsupervised Motion Representation Learning with Capsule Autoencoders
Ziwei Xut, Xudong Shen', Yongkang Wong', Mohan S Kankanhalli i School of Computing, National University of Singapore NUS Graduate School, National University of Singapore
(ziwei-xu, mohan @ccmpc nus. nus.edu... edu. xudong. nus.eeeeuuuuuueeem nus. edu, yongkang
waiikangg.wngeene wongê edu.sg
Abstract
We propose the Motion Capsule Autoencoder (MCAE), which addresses key challenge in the unsupervised learning of motion representations: transforma- tion invariance. MCAE models motion in two-level hierarchy. In the lower level, a spatio-temporal motion signal divided into short, local, and semantic- agnostic snippets. In the higher level, the snippets are aggregated to form full- length semantic-awarc segments. For both levels, we represent motion with a set of learned transformation invariant templates and the corresponding geometric transformations by using capsule autoencoders of a novel design. This leads to robust and efficient encoding of viewpoint changes. MCAE is evaluated novel Trajectory20 motion dataset and various real-world skeleton- based human action datasets. Notably, it achieves better results than baselines Trajectory20 with considerably fewer parameters and state-of-the-an performance on the unsu- pervised skeleton-based action recognition task.
Introduction
Real-world movements contain a plethora of information beyond the literal sense moving. For example, honeybees "dance" to communicate the location of foraging site and human gait alone can reveal activities and identities [10]. Understanding these movements is vital for an artificial in- telligent agent to comprehend and interact with the ever-changing world. Studies social behavior analysis [ [9] action recognition [59] 64] and video summarizing [60] have also acknowledged the
importance of movement.
key step towards understanding movements is to analyze their patterns. However, learning motion pattern representations is non-trivial due to (1) the curse of dimensionality from input data, (2) diffi- culties in modeling long-term dependencies in motion sequences, (3) high intra-class variation as result of subject or viewpoint change, and (4) insufficient data annotation. The first two challenges have been ameliorated by the advances keypoint detection
[57] spatial-temporal feature extrac- tors [38] 43 [0], and hierarchical temporal models [13 49 [66] The third and the fourth nonetheless remain hurdles and call for unsupervised transformation-inaatiin motion models.
Inspired by the viewpoint-invariant capsule-based representation for images capsule networks and introduce the Motion Capsule Autoencoder (MCAE), an unsupervised capsule framework that learns the transformationinnaae invariant motion representation for keypoints. MCAE models motion signals in a two-level snippet-segment hierarchy. snippet movement of narrow time span, while segment consists of multiple temporally-ordere snippets, representing longer-time motion. In both the lower and the higher levels, the snippet capsules (SniCap) and the segment capsules (SegCap) maintain set of templates their identities seaittees--niipeet templates and
we exploit
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

segment templates, respectively-aant and the snippet templates are explicitly modeled motion sequences, the SegCaps are built upon the SniCaps and parameterize the segment templates in terms of the snippel templates, resulting in fewer parameters compared with single-layer modeling. The SniCaps and SegCaps learn transformation- invariant motion representation in their own time spans. The activations of the SegCaps serve as high-level abstraction of the input motion signal.
and transform them to reconstruct the input motion signal. While
The contributions of this work are as follows:
We propose MCAE, an unsupervised capsule framework that learns transformation- invariant, discriminative, and compact representation of motion signals. Two motion cap- sules are designed to generate representation at different abstraction levels. The lower-level representation captures the local short-time movements, which are then aggregated into higher-level representation that is discriminative for motion of wider time spans.
We propose Trajectory20, novel and challenging synthetic dataset with wide class of motion patterns and controllable intra-class variations.
Extensive experiments on both Trajectory20 and real-world skeleton human action datasets show the efficacy of MCAE. In addition, we perform ablation studies examine the effect of different regularizers and some key hyperparameters of the proposed MCAE.
2 Related Works
Motion Representation variety of methods have been proposed to learn (mostly human) mo-
key- 7 Farlier works use and Lie group representations [44]. Some works use canonical human pose [32] or view-invariant short tracklets to learn robust feature for recognition [19]. The development of deep learning brings the usage of con- volution networks (ConwNet) and recurrent networks for motion representation. Simonyan et al. [39 proposes two-stream ConvNet which combines video frame with optical flow. C3D [43] pro- poses to use 3D convolution on the spatial-temports cubes. Srivastava et al. [40] uses an Long Short-Term Memory (LSTM)- encoder to map input frames fixed-length vector and apply task-dependent decoders for applications such as frame reconstruction and frame prediction. The combined use of convolution module and L.STM has also been proved effective in 38, 51].
tion representation from video frames 3
23
28
53
depth maps [14
22
27
37
47
points/skeletons 4 21
24 26
52
58
or point clouds 6
handcrafted features like Fourier coefficients 47
dense trajectory features [46 30
45 48) have been proposed to address the problem of learning viewpoint-invariant motion representation from videos or keypoint sequences. MST-AOG [48] uses an AND-OR graph structure to separate appearance of mined parts from their geometry in- formation. Li et al. 200 learn view-invariant representation by extrapolating cross-view motions. View-LSTM [88 defines a view decomposition, where the view-invariant component is learned by Siamese architecture. While most of these works exploits multi-moda input of RGB frames, depth maps or keypoint trajectories, MCAE focuses on the pure keypoint motion.
A series of works [ 18
20
Capsule Network MCAE is closely related to the Capsule Network [TT, which is designed to represent objects in images using automatically discovered constituent parts and their poses. A capsule typically consists of part identity, set of transformation parameters (i.e. pose), and an activation. The explicit modeling of poses helps learning viewpoint-invarian part features that are more compact, flexible, and discriminative than traditional ConvNets. Capsules can be obtained via agreement-based routing mechanisms 12 34 More recently, Kosiorek el al. [17 proposed the unsupervised stacked capsule autoencoder (SCAE), which uses feed-forward encoders and decoders to learn capsule representations for images.
Apart from images, capsule network has been studied in other vision tasks. In 61 capsule net- work is used for point cloud processing for 3D object classification and reconstruction. VideoCap- suleNet [3] proposes generalize capsule networks from 2D 3D for action detection in videos. Yu et al. [54) proposed a limited study on supervised dceleton-basedd action recognition using Cap- sule Network. Sankisa al. (36] proposed to Capsule Network for error concealment in videos.
Despite the success of capsule networks in various vision tasks, the study capsule networks on motion representation is scarce. In this work, MCAE performs unsupervised learning of motion rep-

---

Reconstruction
Tg
Ts
Snippet Decoder
Reconstruction
Sagment Decoder
Leo Sara
Parameter free Procedure
Snippet
A12
Segment Capsules Cffee
piu
p(3)
Training Objective
Segment Encoder
B
B
Capsule
Ts
T/PA/B p/v
Snippet Encoder
Template
Activation
(123
M23
Transformation
Snippel Capules C
Contrastive Training
(c)
Figure 1: Overview of MCAE (best viewed in color). (a) The Snippet Autoencoder, which learns the semantic-agnostic short-time representation (snippet capsules) by reconstructing the input signal
X. (b) The Segment Autoencoder, which learns the semantic-aware long-time representation (seg- ment capsules) by aggregating and reconstructing snippet capsule parameters. The activation values in segment capsules are used as semantic information for self-supervised contrastive training. (c)
Meanings for different shapes and variables.
resented as coordinates rather than pixels. It aims at learning appearance-agnostir transformation- invariant motion representation. We believe that introducing motion to the capsule network, or other way round, provides (1) A new, robust, and efficient view into motion signals in any dimen- sion space under the ienfformmtiionnnvvaiaane
assumption (while the motion and transformation in dimension space could have semanties different from their 2D/3D counterparts), and (2) proof that disentangling identity from transformation variance works not only for vision problems but possibly larger family of time series analysis problems.
Methodology
We consider single point in d-dimension space. The motion of the point, i.e. a trajectory, is described by X = fx
LT. where E R is the coordinates at time i in d-dimension space. Semantically. X belongs motion pattern, subject an arbitrary and unknown geometric transformation. Given sufficient samples of X, we aim to learn a discriminative (in particular, cransformation-invarian) representation for those motion samples without supervision.
3.1 Framework Overview
We solve this problem in steps, namely snippel learning and segment learning. Snippets and segments correspond to the lower and higher levels of how MCAE views the motion signal. Both snippets and segments temporally consecutive subsets of X, but snippets have shorter time span than segments. In the snippet learning step, the input X is first divided into S = L/l temporally non-overlapping snippets, where | the length of snippets. Each of these snippets will be mapped into a semantic-agnostic representation by the Snippet Autoencoder. In the segment learning step, the snippet representations are combined and fed into the Segment Autoencoder, where the full motion is represented as a weighted mixture of the transformed canonical representations. The segment activations are used as the motion representation for downstream tasks. An overview the framework is shown in Fig. [ In the following section, we delineate the details for each module and explain the training procedure.
3.2 Snippet Autoencoder
To encode the snippets' motion variation, propose the Snippet Capsule (SniCap), which denote as Car1 SniCap is parameterized as Sh = (T. A, μ.), where T, A, and are the snippet template, snippet transformation parameter, and snippet activation, respectively. The snippet
We show way generalize MCAE multi-point systems in Section

---

template T (4.l4 E R I.....} describes a motion template of length 1 and is the identity information of SniCap. and μ depend on the input snippet. The transformation parameter A 1(4+1)x(++1) descries the geometric relation between the input snippet and the snipper template. The snippet activation js E [0, 1] denotes whether the snippet template activated represent the input snippet.
Snippet Encoding/Decoding For a given snippet æor-t, the snippet module performs the follow- ing steps: (1) encode motion properties with Snippet Encoder into SniCaps, and (2) decode SniCaps to reconstruct the original Tiitt- For the encoding step, a ID-ConvNet fenw is used to extract the motion information from #sst and predict SniCap parameters, i.e. {(AA,144) feow((aast where N is the number of SniCaps. The range of ş is confined by a sigmoid activation function. For the decoding step, first apply the transformation to the snippet templates
l..... N)
Ees 1
t
A.
i= N, j -= 1, 1.,,,,,,
Then, the transformed templates from different SniCaps mixed, according their activations, and the corresponding reconstructed input
N
Σ
pelign J I.....4
where łig indicates the transformed coordinate of the 8th SniCap at jth time step.
3.3 Segment Autoencoder
The motion information encoded in SniCaps is agnostic to the segment level motion patterns. This makes it less biased towards the training data domain. However, its utility on high-level applications, such as activity analysis or motion classification, is greatl undermined. For example, consider Fig. aaa) as a reference "triangle" trajectory. Fig. (b)) illustrates a possible intra-class variation. Since the two trajectories differ greatly in their local movement, they could be considered different classes without transformation-ing intermattiooiiiiiiantt information from the full trajectory.
Hence, introduce segment encoder to gain holistic understanding of motion and encapsulate such information in the segment capsules (SegCap). segment is a motion of length (generally the segment length do not have to be the same the signal length) and can be interpreted as S L/l consecutive non-overlapping snippets. SegCap is parameterized as Seg = [P. B, v). where P, B, and v the segment template, segment transtormation parameter, and segment activation, respectively.
Different from the SniCap, whose template is explicitly motion sequence, the SegCap parameter- izes the segment template PP in terms of the N snippet templates. Specifically, P = {(P., a.))
I...., S), where P: R (d++))((++) and IR Each P.j B [[(++))((+11) (j E [N] additionally
A
Srippet Decoder
P
Ti
Snippet Decoder
(a) A reference pattern.
(b) A variant of (a).
(c)
Figure 2: (a) and (b) show reference motion pattern and variant of it. The circle and the shows the start and the direction of motion respectively. (c) Interpretation of a segment template P. is functionally the same as snippet parameters (A, Ju). When combined with T. it can be decoded into an L-long sequence. The segment autoencoder maintains multiple segment templates, which can be transformed and mixed to reconstruct the input snippet parameters.

---

indexes the first dimension of P:) describes how the 1th snippet template is aligned to form the 1t snippet of the segment template. The weight az (j E [N] additionally indexes the elements of a.) controls the importance of 5dh snippet template for the tth snippet of the segment template. In other words, (P,, ax)) descrihes how the snippet templates are used construct I-long snippet and SegCap requires 5 such parameters to describe an I-long segment template. Fig. (] illustrates the interpretation of P. and v are dependent on the input. B (R(--))(d+1)) is transformation on P. and (0, is the activation of the SegCap. The segment template P is fixed for SegCap w.r.t the training domain.
Segment Encoding/ Deeiiiing/ Assume we have M SegCaps with which we hope to reconstruct the low-level motion encoded the SniCap parameters. This is equivalent to reconstructing all the data-dependent SniCap parameters [C ..... C. where C, {(Ag,, (ig) j ,....,N .N is the set of SniCap parameters for the jth snippet. SniCap's A into vector and concatenate it with its corresponding sequence of flattened SniCap parameters with an LSTM model fISIN shared by all SegCaps, and M cullecconeected layers (one for each SegCap) to produce VB. Formally,
obtain the SegCap parameters, we first flatten each
Then we encode the S-long
h = fLSTY. [[c ....,,C511 [B (8)) p(A)] (T, (T,h), k 1,,,..., M,,
(3)
where = iTil = 1 I,.... N), and superscript (A)) refers to the hth SegCap. The transformation and activation parameters then applied to P reconstruct snippet parameters
p B ((k)
..... S,
j j=1, I,....,,, k ==1,.... -....,,M M,
M
M
G8t = (,,
(R)
a (K) i= 1= S
1=1
where A, E (=Mx(d+1)x((+1)) and fA; R are the reconstructed snippet transformation and acti- vation of the snippet templates for the 1th snippet. Note that S L/l, which means fLSIM can have a much smaller footprint than a recurrent network that handles the whole L-long sequence.
The above formulation enables SegCap to learn transfornation-invariann representation of motion. Intuitively, P describes snippet-segment relation, and B can be regarded as the spatial relation between a segment template pattern and the observed trajectory. The segment activation v E R reveals the semantics of the input trajectory and can be used for supervisdd training.
3.4 Training
delineated in Section 3.2 and 3.3 SniCap and SegCap play different roles by capturing infor- mation at two different abstraction levels. SniCap focuses on short-time motion while SegCap is defined upon SniCap to model long-time semantic information. Hence, the two autoencoders are trained using different objective functions.
The only objective of the snippet autoencoder is to faithfully reconstruct the original input. There- fore, for training sample X = (wxli = ...., L}, we use self-supervised reconstruction loss:
....
Sn1 Rea
Σ [1(2; x.)1
(5)
where è; denotes the reconstructed coordinate following Equation
The segment autoencoder's primary goal is to reconstruct the input SniCap parameters, hence the reconstruction loss
Soe Rec
(I - A + (1 - 14)112
(6)
1=1
Furthermore, we use unsupervised contrastive training to learn semantic meaningful activations v. For batch of samples, the contrastive loss is
1
exp (cossim(v/,
[eeg
log
B
ΣS exp (cossim(v/, u/))
5

---

where т 0.1 is the temperature used for all experiments, v and v is the segment activation of sample and X", respectively. Here, x and X are the tempoallyy disturbed versions of x The disturbance is dataset-dependent and will be discussed in the supplementary material.
In additional the above loss terms, impose regularizers: smoothness constraint constructed sequence, and sparsity regularization on the segment activations
Reg SmC
[ 1
CReg Sps
||212.
(8)
k
The final training objective is:
8n1 Shi + Seg Cee + L5eg + 0.5 0.0
where the weights are empirically determined. A Sni and A are dependent on the target dataset and the remaining weights consistent for all datasets in experiment.
4 Experiments
In this section, we first assess the proposed MCAE on a synthetic motion dataset to show its ability in learning wranformatiooniinariian robust representations. Then, we generalize MCAE to multi-point systems and show its efficacy on real-world skeleton-bar based human action datasets. All unsupervised accuracies are produced by an auxiliary linear classifier that trained on the motion representation learned by MCAE or the baselines, but whose gradient is blocked from back-propagating to the model. We report the mean accuracy and standard error based on three runs with random initializa- tion. The experiments are run on an NVIDIA Titan V GPU, where we use batch size of 64, and the Adam [16] optimizer with learning rate of 10-3 Please refer the supplementary material for details.
4.1 Learning from Synthesized Motion
The Trajectory20 Dataset Although commonly used in the motion representation learning lit- erature, datasets like movingMNIST [40] are innately linear and have limited motion variations. Moreover, the prediction-oriente setting makes difficult examine the motion category of each trajectory. In this paper, we introduce the Trajectory20 (T20), synthetic trajectory dataset based on 20 distinct motion patterns (as shown in Fig. 3 Each sample in T20 a 32-step-long sequence of coordinates in [-1, 1).. In the data generating process, motion template is randomly picked, randomly rotated and scaled, and translated to a random position to produce a trajectory. A closed trajectory (marked blue in Fig. 3 starts at random point on the trajectory and end at the same point, whereas an open trajectory (marked yellow in Fig. 3 starts at an either end's vicinity. The randomized generating process ensures the trajectories are controllably diverse in scale, rotation, and position. The training data is generated on-the-fly fly and a fixed test set 10,000 samples used for evaluation. Examples of T20 are shown in the supplementary material.
Ablation Study
We perform ablation study of MCAE on T20 to examine the effeet of different regularizers and three key hyperparameters: snippet length the numbers of SniCap
M
triangle
rectangle
pentagon
hexagon
astroid
line
parabooaa
absolute sine
dircle
heart
hippopado
lamriscato
spiral
bell
cuspidal cuble
aubic a.l. line
asymmettric cubic line
cuspidal cubio
Figure 3: The motion patterns in the Trajectory20 (T20) dataset. a.t. is short for "asymptotic to".

---

(#Sni) and SegCap (#Seg). The result is shown in Table[] The length of snippet plays a vi- tal role in learning a useful representation. A very small results a narrow receptive field for snippet capsules, which makes it less useful for inferring semantics of the whole sequence.
At the other end, large/ makes snippets challeng-
ing to reconstruct. The numbers of SniCap and Seg- Cap also have major effect on the outcome. Too few SniCaps makes it difficult to reconstruct the in- put motion signal. Too few SegCaps undermines the expressiveness of the segment autoencoder. Too many SniCaps could cause difficulty in learning proper alignments between SegCaps and SniCaps. Both degrade the quality of the learned features. Moreover, increasing #Seg from 80 to 128 does not bring further improvements. As the result shows, (1, #Sni, #Seg) (8, 8, 80) performs well and will use it in all experiments below. As for the reg- ularizers, while both regularizers improve the per- formance, the sparsity regulation (Capa) on segment activation is more helpful for learning discrimina- tive features.
Table 1: Ablation study on T20.
Reg.
I
#Sni
#Seg
Acc. (%)
8
8
80
69.30
0.76
4
8
80
41.01
8.81
16
8
80
45.83 = 8.36
8
2
80
64.02 = 2.10
Full
8
4
80
68.17 0.36
8
16
80
48.11 = 1.60
8
8
32
42.36 = 3.15
8
8
64
63.94 = 1.41
8
8
128
69.44 = 1.69
w/o CReg
8
8
80
67.60 = 1.69
w/o
8
8
80
65.92 1.63
Motion Classification We compare MCAE with the following baseline models, namely KMeans, DTW-KMeans, A-Shape B1. LSTM and ID-Cony KMeans, DTW-KMeans, and k-Shape parameter-free time series clustering algorithms. Briefly, KMeans uses Euclidean distance to mea- the similarity between signals. DTW-KMMans normalizes input signals using dynamic time warping (35), and performs KMeans on the normalized signals. k-Shape uses cross-correlation based distance measure cluster time series. We use the implementation by tslearn 42] for the three clustering methods. LSTM, ID-Conv, and MCAE are used as backbone networks, which take the coordinate sequence input and output feature vector of pre-defined dimension. The feature vector is used for contrastive learning following Equation The corresponding accuracy reflects the quality of the learned representation.
For LSTM and ID-Conv backbone, dif- ferent numbers of hidden units/channels have been explored (shown as Hidden Param. in Table which has resulted in different model sizes (measured by #Param. in Table 2).
Table 2: Unsupervised learning performance of MCAE and baselines on T20.
Hidden Param.
#Param.
Acc. (%)
K MMans OTW-KMeans k-Shape B1
8.57 0.04
9.12 + 0,20
12.94 1: 0.34
As shown in Table since the spatial variance (e.g. viewpoint changes) within motion signal cannot be directly cap- tured by temporal warping/correlation, all the three parameter-free cluster- ing methods perform poorly T20. On the other hand, with considerably fewer parameters, MCAE outperforms LSTM and ID-CNN by a large margin. This provides quantitative evidence that MCAE can capture the transformation- invariant semantic information more ef- ficiently than the compared basclines.
128
29.17 + 2.45
256
669k
40.03 1 0.57
LSTM
512
805k
45.59 + 1.37
1,024
1,078k
53.47 = 1.52
2,048
1.625k
54.32 = 0.55
128
588k
44.78 1 0.57
256
787k
53.69 + 0.53
ID-Conv
512
1,185k
57.57 1. 0.56
1,024
1,982k
57.58 + 0.08
(#Sni, #Seg)
#Param.
Acc. (%)
MCAE
(8, 80)
277k
69.30 0.76
4.2 Generalizing to Multiple Points
The MCAE running on T20 dataset handles single moving point while most real-world prob- lems involve multiple points. This section presents MCAE-MP, a naive but effective extension of
Architectures of LSTM and ID-Conv detailed in the supplementary material.
7

---

Table 3: Performance (%) for skeleton-base based action classification. Column "Mod." shows the data modality, where "S" indicates skeleton and "D indicates depth map. Column "Cls." shows the auxiliary classifier used for supervised training. We also report supervised SOTAs for completeness.
NTU60
NTU120
NW-UCLA
Model
Mod.
Cls.
XSUB
XVIEW
XSUB
XSET
V1&V2 + V3
Luo ol.
27
S+D
SLP
61.4
53.2
50.7
Li al.
20
S+D
SLP
68.1
63.9
62.5
Supv. Unsupervised
SeBiReNet
29
S
LSTM
79.7
80.3
LongT GAN
63
S
SLP
39.1
48.1
74.3
MS
24
S
SLP
52.6
76.8
CAF+ 33
S
SL.P
58.5
48.6
49.2
MCAE-MP (SLP)
S
SLP
65.6
74.7
52.8
54.7
83.6
P&C 41
S
I-NN
50.7
76.1
84.9
MCAE-MP (I-NN)
S
I-NN
51.9
82.4
42.3
46.
79.1
DropGraph
2
S
90.5
96.6
82.4
84.3
93.8
JOLO-GCN
S
93.8
98.1
87.6
89.7
MCAE, to enable processing motion for multi-point systems. Such motion can described as X X 1,. KY, where K is the number of moving points. The extension works as follows:
1. The K moving points are processed separately by an MCAE. This results in K segment activation vectors {w:, / = ...... K}.
2. The K activation vectors are concatenated into a single representation E B KM which is used for unsupervised learning following Equation
Skeleton-based Human Action Recognition We apply MCAE-MP to solve the unsupervised skeleton-based action recognition problem, where human skeleton is system consisting of mul- tiple moving joints (points). Three widely-used datasets are used for evaluation: NW-UCLA (48) NTU-RGBD60 (NTU60) (37), and NTU-RGBD120 (NTU120) [25]] The three datasets consist of sequences with or 2 subjects whose movement measured 3D space. For NW-UCLA, we follow previous works [4] to train the model on view and view 2, and test the model view 3. official data split for the cross-subject (XSUB) and cross-view (XVIEW) NTU120 for the cross-subject (XSUB) and cross-setting
For NTU60, we follow protocols. The similar is implemented (XSET) protocol. For case of implementation. we project the 3D sequence into three orthonormal 2D spaces and use MCAE defined the 2D space to process the three views of the sequences. Then the segment activations from the three views are concatenated to form the representation. Four types of disturbance are introduced for contrastive learning, namely jittering, spatial rotation, mask- ing, and temporal smoothing. The readers are referred to the supplementary material for details.
The classification accuracy is put into three groups in Table 3 that are not directly comparable as they use depth map 20 27 or stronger auxiliary classifiers for supervised training [29]. In the second group, where our model is marked as MCAE-MP (SLP), single layer perceptron (SLP) is trained as the auxiliary classifier with backbone parameters frozen. In the third group. where model is marked as MCAE-MP (INN), l-ncarest-neighbor classifier is used instead of an SLP. For completeness, the fourth group shows state-of-the-arr results from naive extension
In the first group are the prior works
supervised methods. Although MCAE-MP largely ignores their interactions, it achieves better or competitive performance compared with the baselines. Notably, NTUGO-XVIEW and NTU120 XUETTT where the training set and test set have different viewpoints, our model outperforms baselines by a clear margin thanks to the capsule- based representation which effectively captures viewpoint changes as transformations on input.
encodes joints separately and
4.3 What does MCAE Learn?
To better understand what is encoded, we plot the learned snippet templates T and segment tem- plates P in Fig. 4 Note that T are initialized random straight lines, and P are initialized as arbi- trary patterns composed randomly of T. As shown in Fig. 4a the snippets are mainly simple lines and hook-like curves that does not carry semantic information. Segment templates in Fig. 4b how-

---

{
3
(a) Snippel templates T.
(b) Samples of segment templates P.
Figure 4: Templates learned from Trajectory20 dataset. Color indicates time.
ever, bear some resemblance to the patterns shown in Fig. This suggests that semantic-agnostic snippets are being aggregated into semantic-aware segments.
proceed to explore the information in SegCaps. In particular, we would like to see SegCaps have learned transformation-inaaatt tory from T20 dataset. The trajectory first normalized so that centroid at (0, 0), then rotated clockwise by angle 0, and finally fed into the model. We examine the segment templates with the highest activation values (which reflects the trajectory's 's semantics) and calculate the rotation angle Φ from those templates' parameter B. As shown in Table the calculated á reveals two types of segments templates as we rotate the input. One type yields constant Ø (e.g. segment ID for sample "absolute sine"), which indicates its rotation-invariance the other has that changes mono- tonically with 0 (e.g. segment ID for sample "hexagon"), "), which shows its cotation-awareness As for the activation values, samples from different categories activate different plates. Meanwhile, the same sample under different rotation angle 0 gives stable segment template activations, despite some changes which are found have no effect on the classification result.
information. To this purpose, randomly sample trajec-
of segment tem-
We do a similar study on the translation component (æ, v), where we translate the input by (Ax, Ay). As shown in Table (ar, y) changes monotonically with (Az, Ay) while the activated segment tem- plates remain stable. These results prove that the semantics and transformation information has been
Table Top-5 segment templates (sorted by segment activation y then segment ID for better visu- alization), and the rotation calculated from their parameters B. Bold IDs are segments repeating across different 6.
0
10"
5
I 0"
0
5
0
10"
Input
ID
Φ
ID
6.3
6.7
2
6.8
2
7.0
2
7.1
Teosgon
8
6,9
9.0
8
11.2
8
13.9
8
16.5
12
54.9
12
55.5
12
55.8
12
56.5
12
56.8
37
-20.8
37
-19.8
37
-18.9
37
-17.9
37
-16.9
66
50.2
66
52.5
66
55.4
66
59.0
66
62.4
zho
12.1
12.3
2
12.2
2
12.1
2
11.9
8.2
-10.7
5
-10.1
5
-9,9
7
17.2
33
65.1
7
10.7
7
13.4
7
15.4
32
-9.7
37
-22.9
37
-22.3
37
-21.8
37
-21.3
37
-19.9
46
45.7
46
47.5
46
48.6
46
50.2
46
51.6
Table 5: Top-5 segment templates (sorted by segment activation v then segment ID for better visu- alization), and the translation (z. y) calculated from their parameters B.
(Ar. Ag) = (-0.2, 0)
(Ar, Ag) (-0.1.
(Ar. Ag) = (0,
(Ar, Au) = (0,0.1) 0.1))
(Ar, Ag) 0.2)
Ingut
ID
ID
ID
0.05
0.18
2
0.17
0.19
2
0.27
0.19
2
0.28
0.28
0.27
0.37
0.01
-0.07
0.09
-0.06
8
0.18
-0.04
0.19
0.04
0.19
0.12
.0.09
0.13
12
0.00
0.13
12
0.09
0.13
12
0.09
0.23
0.09
0.32
0,10
-0.11
37
0.18
-0.11
37
0.27
-0.11
37
0.27
-0.03
0.27
0.05
-0.12
0.16
-0.03
0.16
66
0.17
0.06
0.04
0.14
0.19
2
0.24
0.19
0.24
0.28
0.23
0.38
-0.01
0.30
0.07
0.29
0.16
0.29
0.16
0.38
0.15
0.46
0.20
-0.16
7
0.28
-0.16
7
0.37
-0.15
0.36
-0.06
0.36
0.04
-0.17
0.12
-0.16
46
0.02
0.01
46
0.13
0.02
46
0.23
0.04
46
0.23
0.13
46
0.22
0.23

---

encoded separately in the segment activation and transformation parameters B. In other words, the encoded semantic information is robust against geometric transformations.
5 Conclusion
In this paper, we introduce MCAE, framework that learns robust and discriminative representation for keypoint motion. To resolve the intra-class variation of motion, we propose learn a compact and transformation-inaarre
motion representation using two-level sapsule-based representation hierarchy. The efficacy of the learned representation is shown through an experimental study on synthetic and real-world datasets. The output of MCAE could serve as mid-level representation in other frameworks, e.g. Graph Convolution Network, for tasks that involve more context than classification. We anticipate this work to inspire further studies that apply capsule-based models to other time series processing tasks, such as joint modeling of visual appearance and motion video. The source code and the T20 dataset of our research are accessible at htttt::////// /ttp//gtttttttttt com/ #iweix0/Capsulete eoootiiioooonn
Motion analysis techniques are in the foreground of the misuse of machine learning methods, among which adverse societal impacts and privacy breach are two major concerns. Regarding the societal impacts, admittedly, our method has both upside and downside. On one hand, transformation- invariant motion representation enables us better decode the information implicit in the trajectory, which has applications for example in ethology. On the other hand, it could also be misused in mass surveillance. Appropriate boundaries of use and ethical review are required to prevent potential malicious applications. Regarding the privacy concerns, our method isolates the subjects' motion from their sensitive information, such as gender and race.
Acknowledgments and Disclosure of Funding
This research/project is supported by the National Research Foundation, Singapore under its Strategic Capability Research Centres Funding Initiative. Any opinions, findings and conclu- sions or recommendations expressed in this material are those of the author(s) and do not re- flect the views of National Research Foundation, Singapore. The computational work for this article was partially performed resources of the National Supercomputing Centre, Singapore (https: htttpp::/www.nnc.. nscc. sg).
References
[1] Jinmiao Cai, Nianjuan Jiang, Xiaoguang Han, Kui Jia, and Jiangbo Lu. JOLO-GCN: mining joint-centered weight information for skeleton-based action recognition. In WACV, pages 2734-2743, 2021.
[2] Ke Cheng. Yifan Zhang, Congqi Cao, Lei Shi, Jian Cheng, and Hanqing Lu. Decoupling GCN with dropgraph module for skeleton-based action recognition. In ECCV, volume 12369 of Lecture Notes in Computer Science, pages 536-553, 2020.
[3] Jeff Donahue, Lisa Anne Hendricks, Marcus Rohrbach, Subhashini Venugopalan, Sergio Guadarrama, Kate Saenko, and Trevor Darrell. Long-term recurrent convolutional networks for visual recognition and description. IEEE Transactions on Pattern Analysis and Machine Intelligence, (9(4))677-69) 2017.
[4] Yong Du, Wei Wang, and Liang Wang. Hierarchical recurrent neural network for skeleton based action recognition. In CVPR, pages 1110-1118, 2015.
[5] Kevin Duarte, Yogesh Rawat, and Mubarak Shah. VideoCapsuleNet: A simplified network for action detection. In NeurIPS, pages 7610-7619, 2018.
[6] Hehe Fan, Yi Yang, and Mohan Kankanhalli. Point 4D transformer networks for spatio- temporal modeling in point cloud vidcos. In CVPR, pages 14204-14213, 2021.
Hehe Fan, Xin Yu, Yuhang Ding, Yi Yang, and Mohan Kankanhalli. PSTNet: Point spatio- temporal convolution on point cloud sequences. In ICLR, 2021.
[8] Tian Gan, Yongkang Wong, Daqing Zhang, and Mohan Kankanhalli. Temporal encoded f-
formation system for social interaction detection. In ACM Multimedia, pages 937-946, 2013.
10

---

[9] Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social GAN: Socially acceptable trajectories with generative adversarial networks. In CVPR, pages 2255- 2264, 2018.
[10] Ahmed Refaat Hawas, Heba A. EI-Khobby, Mohammed Abd-Elnaby, and Fathi E. Abd El- Samie. Gait identification by convolutional neural networks and optical flow. Multimedia Tools and Applications, 78( (8(11))258833-5888888 2019.
[11] Geoffrey E. Hinton, Alex Krizhevsky, and Sida D. Wang. Transforming auto-encoders. In ICANN, volume 6791 of Lecture Notes in Computer Science, pages 44-51, 2011.
[12] Geoffrey E. Hinton, Sara Sabour, and Nicholas Frosst. Matrix capsules with EM routing, In ICLR, 2018.
[13] Noureldien Hussein, Efstratios Gavves, and Amold W. M. Smeulders. Timeception for com- plex action recognition. In CVPR, pages 254-263, 2019.
[14] Mariano Jaimez, Mohamed Souiai, Javier Gonzalez-Jimenez, and Daniel Cremers. A primal- dual framework for real-time dense RGB-D scene flow. In ICRA, pages 98-104, 2015.
[15] Yanli Ji, Feixiang Xu, Yang Yang, Ning Xie, Heng Tao Shen, and Tatsuya Harada. Attention transfer (ANT) network for view-invariant action recognition. In ACM Multimedia, pages 574-582, 2019.
[16] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[17] Adam Kosiorek, Sara Sabour, Yee Whye Teh, and Geoffrey E Hinton. Stacked capsule autoen- coders. In NeurlPS, pages 15512-15522, 2019.
[18] Mohamed Hyee Lakhal, Oswald Lanz, and Andrea Cavallaro. View-lstm: Novel-view video synthesis through view decomposition. In ICCV, pages 7576-7586, 2019.
[19] Binlong Li, Octavia I. Camps, and Mario Sznaier. Cross-view activity recognition using han- kelets. In CVPR, pages 1362-1369, 2012.
[20] Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S. Kankanhalli. Unsupervised learning of view-invariant action representations. In NeurIPS, pages 1262-1272, 2018.
[21] Maosen Li, Siheng Chen, Xu Chen, Ya Zhang, Yanfeng Wang, and Qi Tian. Actional-structure graph convolutional networks for skeleton- action recognition. In CVPR, pages 3595- 3603, 2019.
[22] Wanqing Li, Zhengyou Zhang, and Zicheng Liu. Action recognition based on a bag of 3D points. CVPR Workshops, pages 9-14, 2010.
[23] Yan Li, Bin Ji, Xintian Shi, Jianguo Zhang, Bin Kang, and Limin Wang. TEA: Temporal excitation and aggregation for action recognition. In CVPR, pages 906-915, 2020.
[24] Lilang Lin, Sijie Song, Wenhan Yang, and Jiaying Liu. MS2L: multi-task self-supervised learning for skeleton based action recognition. In ACM Multimedia, pages 2490-2495 2020.
[25] Jun Liu, Amir Shahroudy, Mauricio Perez, Gang Wang, Ling- Yu Duan, and Alex C. Kot. NTU RGB+D 120: A large-scale scale benchmark for 3D human activity understanding. IEEE Transaction on Pattern Analysis and Machine Intelligence, 42(10):264--770, 2020.
[26] Ziyu Liu, Hongwen Zhang, Zhenghao Chen, Zhiyong Wang, and Wanli Ouyang. Disentangling and unifying graph convolutions for skeleton-based action recognition. In CVPR, pages 143- 152, 2020.
[27] Zelun Luo, Boya Peng, De-An Huang, Alexandre Alahi, and Fei-Fei. Unsupervised learning of long-term motion dynamics for videos. In CVPR. pages 7101-7110, 2017.
[28] Joe Yue-Hei Ng, Matthew J. Hausknecht, Sudheendra Vijayanarasimhan, Oriol Vinyals, Rajat Monga, and George Toderici. Beyond short snippets: Deep networks for video classification. In CVPR, pages 4694-4702, 2015.
[29] Qiang Nie, Ziwei Liu, and Yunhui Liu. Unsupervised 3d human pose representation with viewpoint and pose disentanglement. In ECCV, volume 12364 of Lecture Notes in Computer Science, pages 102-118, 2020.
[30] Dan Oneata, Jakob Verbeek, and Cordelia Schmid. Action and event recognition with fisher vectors on compact feature set. In ICCV, pages 1817-1824, 2013.

---

[31] John Paparrizos and Luis Gravano. k-shape: Efficient and accurate clustering of time series. In SIGMOD, pages 1855-1870, 2015.
[32] Vasu Parameswaran and Rama Chellappa. View invariance for human action recognition. International Journal of Computer Vision, (6(3)183-11, 2006.
[33] Haocong Rao, Shihao Xu, Xiping Hu, Jun Cheng, and Bin Hu. Augmented skeleton based contrastive action learning with momentum LSTM for unsupervised action recognition. Infor- mation Sciences, 569:90-109, August 2021.
[34] Sara Sabour, Nicholas Frosst, and Geoffrey Hinton. Dynamic routing between capsules. In NIPS, pages 3856-3866, 2017.
[35] Hiroaki Sakoe and Seibi Chiba. Dynamic programming algorithm optimization for spoken word recognition. IEEE Transactions Acoustics, Speech, and Signal Processing, 26 ())4: 49, 1978.
[36] Arun Sankisa, Arjun Punjabi, and Aggelos K. Katsaggelos. Temporal capsule networks for video motion estimation and enror concealment. Signal Image Video Process, 14(7):1369) 1377, 2020.
[37] Amir Shahroudy, Jun Liu, Tian-Tsong Ng. and Gang Wang. NTU RGB+D: A large scale dataset for 3D human activity analysis. In CVPR, pages 1010-1019, 2016.
[38] Xingjian Shi, Zhourong Chen, Hao Wang, Dit- Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional LSTM network: A machine learning approach for precipitation nowcast- ing. In NIPS, pages 802-810, 000, 2015.
[39] Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recog- nition in videos. In NIPS, pages 568-576, 2014.
[40] Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov. Unsupervised learning of video representations using LSTMs. In ICMI., volume 37 of JMLR Workshop and Conference Proceedings, pages 843-852, 2015.
[41] Kun Su, Xiulong Liu, and Eli Shlizerman. PREDICT & CLUSTER: unsupervised skeleton
based action recognition. In CVPR, pages 9628-9637, 2020.
[42] Romain Tavenard, Johann Faouzi, Gilles Vandewiele, Felix Divo, Guillaume Androz, Chester Holtz, Marie Payne, Roman Yurchak, Mare Rußwurm, Kushal Kolar, and Eli Woods. Tslearn, a machine learning toolkit for time series data. Journal of Machine Learning Research, 21(118):1-6, 2020.
[43] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3D convolutional networks. In ICCV, pages 4489-4497, 2015.
[44] Raviteja Vemulapalli, Felipe Arrate, and Rama Chellappa. Human action recognition by rep- resenting 3D skeletons as points in Lie group. In CVPR, pages 588-595, 2014.
[45] Shruti Vyas, Yogesh Singh Rawat, and Mubarak Shah. Multi-view action recognition using cross-view video prediction. In ECCV, volume 12372 of Lecture Notes in Computer Science, pages 427-444-44444 2020.
[46] Heng Wang and Cordelia Schmid. Action recognition with improved trajectories. In ICCV, pages 3551-3555 2013.
[47] Jiang Wang, Zicheng Liu, Ying Wu, and Junsong Yuan. Mining actionlet ensemble for action recognition with depth cameras. In CVPR, pages 1290-1297, 2012.
[48] Jiang Wang. Xiaohan Nie. Yin Xia. Ying Wu, and Song-Chun Zhu. Cross-view action model- ing, learning, and recognition. In CVPR, pages 2649-2656, 2014.
[49] Limin Wang, Zhan Tong, Bin Ji, and Gangshan Wu. TDN: temporal difference networks for efficient action recognition. In CVPR, pages 1895-1904, 2021.
[50] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, pages 7794-7803, 2018.
[51] Yunbo Wang, Lu Jiang, Ming-Hsuan Yang, Li-Jia Li, Mingsheng Long, and Li Fei-Fei. Eidetic 3D LSTM: A model for video prediction and beyond. In ICIRR 2019.
[52] Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for skeleton-based action recognition. In AAAI, pages 7444-7452, 2018
12

---

[53] Ceyuan Yang, Yinghao Xu, Jianping Shi, Bo Dai, and Bolei Zhou. Temporal pyramid network for action recognition. CVPR, pages 588-597, 2020.
[54] Yue Yu, Niehao Tian, Xiangru Chen, and Ying Li. Skeleton capsule An efficient network for action recognition. In ICVRV, pages 74-77, 2018.
[55] Pengfei Zhang, Cuiling Lan, Wenjun Zeng, Junliang Xing, Jianru Xue, and Nanning Zheng. Semantics Semantics-guided guided neural networks for efficient akeleton-basee human action recognition. In CVPR. pages 11 1112-11211 2020.
[56] Shiwen Zhang, Sheng Guo, Weilin Huang, Matthew R. Scott, and Limin Wang. V4D: 4d convolutional neural networks for video-level representation learning. In ICLR, 2020.
[57] Xiheng Zhang, Yongkang Wong, Xiaofei Wu, Juwei Lu, Mohan Kankanhalli, Xiaongdong Li, and Weidong Geng. Learning causal representation for training cross-domain pose estimator via generative inventions. In ICCV, pages 1270--1220, 2021.
[58] Xikun Zhang. Chang Xu, and Dacheng Tao. Context aware graph convolution for skeleton- based action recognition. In CVPR, pages 14333-14342, 2020.
[59] Yiyi Zhang, Li Niu, Ziqi Pan, Meichao Luo, Jianfu Zhang, Dawei Cheng, and Liqing Zhang- Exploiting motion information from unlabeled videos for static image action recognition. In AAAI, pages 22918-12225, 2020.
[60] Yujia Zhang, Xiaodan Liang, Dingwen Zhang, Min Tan, and Eric P Xing. Unsupervised object-level video summarization with online motion auto-encoder. Pattern Recognition Let- ters, 130:376-385, 2020.
[61] Yongheng Zhao, Tolga Birdal, Haowen Deng, and Federico Tombari. 3D point capsule net- works. In CVPR, pages 1009-1018, 2019.
[62] Yongheng Zhao, Tolga Birdal, Jan Eric Lenssen, Emanuele Menegatti, Leonidas J. Guibas, and Federico Tombari. Quatermion equivariant capsule networks for 3D point clouds. In ECCV, volume 12346 of Lecture Notes in Computer Science, pages 1-19, 2020.
[63] Nenggan Zheng, Jun Wen, Risheng Liu, Liangqu Long, Jianhua Dai, and Zhefeng Gong, Un- supervised representation learning with long-term dynamics for skeleton based action recog- nition. In AAAI, pages 2644-2044 1144-2651, 2018.
[64] Tao Zhuo, Zhiyong Cheng, Peng Zhang. Yongkang Wong, and Mohan Kankanhalli. Explain- able video action reasoning via prior knowledge and state transitions. In ACM Multimedia, pages 521-529, 2019.
13


Reviews:
[{"review_id": "wjr18MWcg4Z", "reviewer": "Reviewer_B1sR", "summary": "The paper presents a method for incorporating transformation invariance using capsule networks in the unsupervised learning of motion representations. The proposed method solves the problem in two steps, snippet and segment learning, which correspond to a lower and higher level of motion signals (basic trajectories -> reconstruction/time -> Contrastive learning and reconstruction). The method was evaluated on a synthetic dataset and three skeleton datasets for action recognition achieving convincing results.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n-The paper is well-structured\n-Experimental results are promising \n\nLimitations:\n- I found the description of the model very hard to follow (both the snipped and segment learning). For someone that is not familiar to CapsNet, it basically is impossible to understand what is happening. Basics of CapsNet should be recalled to build a parallel with the proposal. \n-The authors noted a few works that employ capsule networks in motion representation in Related Works [3,30, 47] but did not attempt to compare against them in the experiments. \n-2D projected inputs were used instead of 3D pose sequence, did the authors try to experiment with the original 3D sequences?\n-It would be interesting to see what has the model learnt on the skeleton dataset.\n\nOriginality:\n- The work is original. Capsule Nets have been designed especially to do this. However, few works that were cited introduced ways of representing \"images\". A comparison should have been made to see if the introduction of \"motion\" really is bringing something to the Table. \n\nClarity:\n- The model Section should be clarified, making references to what a CapsNet is (formal definition).\n\nSignificance:\n-The proposed method could have an impact on motion representation.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "n/a", "review_text": "", "extra_scores": {"ethical_concerns": "n/a", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "i6FZYYp2rO4", "reviewer": "Reviewer_fGts", "summary": "This paper presents an unsupervised motion representation learning method with capsule autoencoders. Its framework is composed of two-level motion modeling: snippet-level for short and local motion capturing, and segment-level for full-length semantic-aware motion capturing. It presents a new dataset called as Trajectory 20 and performs experiments on two datasets.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The proposed autoencoder is a feed-forward network of temporal convolutions to predict the transformation parameter and activations. I cannot figure out the relation between the proposed autoencoder and the traditional capsule network. This paper needs to make a clear description on its work with respect to the capsule network.\n\n2. This paper fails to present experiments on more common motion datasets, like movingMNIST, and thus I cannot clearly tell the advantages of the proposed method over previous ones.\n\n3. In the ablation studies, the authors fail to provide a study on the importance of two-level motion modeling. For example, it should compare with a simple baseline of one-level modeling with the same number of snippets.\n\n4. The evaluation datasets are quite clear and like toy examples. I would like to see more experiments on motion representation learning on more realistic datasets, like cloud points for scene flow estimation, RGB videos for optical flow estimation.\n\n5. The idea of two-level motion modeling has appeared in many video action recognition works. For example: V4D:4D Convolutional Neural Networks for Video-level Representation Learning [ICLR 2020]. It should discuss these works in the related work section.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "DAttrJOjlld", "reviewer": "Reviewer_8Ljw", "summary": "This work presents a Motion Capsule Autoencoder (MCAE) which learns to represent motions. The two-staged architecture first learns a set of snippet capsules, which learn motions for short time-spans, and then learns segment capsules that learn the long-term motions from all the snippet capsules. These capsules are learned end-to-end in an unsupervised manner with reconstruction losses and a contrastive loss. The approach is evaluated on a synthetic motion dataset proposed in this work, Trajectory20, as well as on three real-world skeleton-based human action recognition datasets, where it is shown to achieve strong performance when compared to previous state-of-the-art methods.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, this is an interesting work which presents a novel capsule architecture (MCAE) to learn motion patterns in an unsupervised manner. Their experimental evaluations on both synthetic and real data, comparisons with previous approaches, and analysis all support the strengths of this work. I suggest that is paper should be accepted, but the following questions/comments should be addressed to improve the clarity of the work.\n\n1) How are the templates learned/obtained? They are defined as $ \\mathcal{T} = t_i | t_i \\in \\mathbb{R}^d, i=1,..., l $. Are they a set of learned parameters, which are trained with the full system in an end-to-end manner? Also, $d$ is not defined - since the network works on 2d points, and $A$ is a $3\\times3$ matrix, I assume $d=2$, but it should be stated explicitly.\n\n2) In the ablation study, how is the accuracy obtained? Later in lines 227-228, it states that an MLP is learned on top of the model for the motion classification experiments. Is it the same for the ablations? If so, this information should be included in the, or prior to the ablations section.\n\n3) How is the LSTM used in Eq. 3 to encode the SniCaps? Since $C_i^\\text{Sni}$ is a set of tuples $(A_i,\\mu_i)$, what operation is used to combine each $C_i^\\text{Sni}$ into a single representation/vector?\n\n4) In Section 4.3, the authors state that Table 4 shows that the top-5 segment templates for the hexagon remain consistent across all listed rotations, but changes for abs_sine (e.g. IDs 33 and 32 are present in $\\theta=-10$ and $\\theta=10$ respectively). Is this due to the nature of the motion pattern (i.e. the abs_sine's open trajectory rather than the hexagon's closed trajectory), or is such behavior common for many different motion classes?\n\n5) Is $\\mu$ the result of a sigmoid or softmax activation?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors address potential negative societal impacts, but could include more discussion about the limitations. For example, what type of motion patterns does the MCAE not perform well on? Can the method generalize to longer sequences (i.e. $L>32$)?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "-9iymStFdh", "reviewer": "Reviewer_Vi9D", "summary": "This work proposes a motion representation learning method based on capsule autoencoders. It decomposes an input 2D keypoint trajectory into snippets and segments, and form a hierarchical representation that is also transformation invariant. The proposed motion representation achieves state-of-the-art motion classification results on a self-proposed synthetic 2D motion dataset as well as skeleton-based human action recognition.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: \n\n- The proposed method is based on the important insight that movement patterns should be transformation invariant and also should naturally consist of a hierarchy of patterns. Thus, the proposed method is intuitive and easy to understand. To the best of my knowledge, using the formulation of capsule autoencoders to explicitly learn transformable trajectory templates through self-supervised learning is novel and elegant. Such formulation can also provide an interpretable representation for future motion analysis tasks. \n- The paper conducts extensive experiments and studies that backed up its claims:\n    - Experiments on the synthetic T20 dataset suggest that the proposed method is an effectively self-supervised motion representation learning method.\n    - Real-world applications on self-supervised action recognition also suggest the strength of the proposed method.\n    - Ablation studies that specify the interpretability of the proposed method: I find section 4.3 especially helpful in showcasing the benefit of the proposed method, where the transformation invariance is well-documented.\n- The proposed T20 dataset would be beneficial as a standard test ground for future 2D motion analysis methods.\n\nIssues:\n\n- In section 4.3, it seems old that theta is constrained to [-10, 10] degrees, where a hexagon should be expected to activate similar segments for large rotation perturbation?\n- To what extent can a motion trajectory be transformed but still be classified as a class? The proposed method seems to be confined to 2D similarity transformation instead of the full affine/projective transformation. \n- (Minor) It would be beneficial to include state-of-the-art results on supervised skeleton-based action classification, to give readers an idea of how far unsupervised methods are away from the supervised ones.\n- (Minor) It would be beneficial if additional ablation on the effect of the contrastive learning loss could be provided. While conceptually the contrastive learning loss would be beneficial in learning the segments, I am curious to see its effects compared to only using reconstruction loss. \n\n====================================================================================================\n\nAfter the author response:\n\nThe answers from the authors are largely satisfactory to me (concerning robustness with rotation, contrastive loss). I think the proposed MCAE would be a worthy contribution to the current capsule network literature. I would like to keep my original score.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have adequately addressed the limitations and potential negative societal impact. ", "review_text": "", "extra_scores": {"ethical_concerns": "The authors have adequately acknowledged the ethical concerns of their work.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work introduces a novel capsule architecture that learns motion patterns in an unsupervised manner. Evaluation on synthetic and real-work skeleton-based human action datasets demonstrate the model good performance when compared to previous state-of-art methods. 

Using the formulation of capsule autoencoders to explicitly learn transformable trajectory templates through self-supervised learning is novel and an interesting addition to the capsule network literature.

Decision: Accept (Poster)