Unsupervised Motion Representation Learning with Capsule Autoencoders

1 Introduction . Real-world movements contain a plethora of information beyond the literal sense of moving . For example , honeybees “ dance ” to communicate the location of a foraging site and human gait alone can reveal activities and identities [ 10 ] . Understanding these movements is vital for an artificial intelligent agent to comprehend and interact with the ever-changing world . Studies on social behavior analysis [ 8 , 9 ] , action recognition [ 59 , 64 ] , and video summarizing [ 60 ] have also acknowledged the importance of movement . A key step towards understanding movements is to analyze their patterns . However , learning motion pattern representations is non-trivial due to ( 1 ) the curse of dimensionality from input data , ( 2 ) difficulties in modeling long-term dependencies in motion sequences , ( 3 ) high intra-class variation as a result of subject or viewpoint change , and ( 4 ) insufficient data annotation . The first two challenges have been ameliorated by the advances in keypoint detection [ 57 ] , spatial-temporal feature extractors [ 38 , 43 , 50 ] , and hierarchical temporal models [ 13 , 49 , 56 ] . The third and the fourth nonetheless remain hurdles and call for unsupervised transformation-invariant motion models . Inspired by the viewpoint-invariant capsule-based representation for images [ 11 , 17 ] , we exploit capsule networks and introduce the Motion Capsule Autoencoder ( MCAE ) , an unsupervised capsule framework that learns the transformation-invariant motion representation for keypoints . MCAE models motion signals in a two-level snippet-segment hierarchy . A snippet is a movement of a narrow time span , while a segment consists of multiple temporally-ordered snippets , representing a longer-time motion . In both the lower and the higher levels , the snippet capsules ( SniCap ) and the segment capsules ( SegCap ) maintain a set of templates as their identities—snippet templates and 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . segment templates , respectively—and transform them to reconstruct the input motion signal . While the snippet templates are explicitly modeled as motion sequences , the SegCaps are built upon the SniCaps and parameterize the segment templates in terms of the snippet templates , resulting in fewer parameters compared with single-layer modeling . The SniCaps and SegCaps learn transformationinvariant motion representation in their own time spans . The activations of the SegCaps serve as a high-level abstraction of the input motion signal . The contributions of this work are as follows : • We propose MCAE , an unsupervised capsule framework that learns a transformationinvariant , discriminative , and compact representation of motion signals . Two motion capsules are designed to generate representation at different abstraction levels . The lower-level representation captures the local short-time movements , which are then aggregated into higher-level representation that is discriminative for motion of wider time spans . • We propose Trajectory20 , a novel and challenging synthetic dataset with a wide class of motion patterns and controllable intra-class variations . • Extensive experiments on both Trajectory20 and real-world skeleton human action datasets show the efficacy of MCAE . In addition , we perform ablation studies to examine the effect of different regularizers and some key hyperparameters of the proposed MCAE . 2 Related Works . Motion Representation A variety of methods have been proposed to learn ( mostly human ) motion representation from video frames [ 3 , 23 , 28 , 53 ] , depth maps [ 14 , 22 , 27 , 37 , 47 ] , keypoints/skeletons [ 4 , 21 , 24 , 26 , 29 , 33 , 41 , 52 , 55 , 58 ] , or point clouds [ 6 , 7 ] . Earlier works use handcrafted features like Fourier coefficients [ 47 ] , dense trajectory features [ 46 , 30 ] , and Lie group representations [ 44 ] . Some works use canonical human pose [ 32 ] or view-invariant short tracklets to learn robust feature for recognition [ 19 ] . The development of deep learning brings the usage of convolution networks ( ConvNet ) and recurrent networks for motion representation . Simonyan et al . [ 39 ] proposes a two-stream ConvNet which combines video frame with optical flow . C3D [ 43 ] proposes to use 3D convolution on the spatial-temporal cubes . Srivastava et al . [ 40 ] uses an Long Short-Term Memory ( LSTM ) -based encoder to map input frames to a fixed-length vector and apply task-dependent decoders for applications such as frame reconstruction and frame prediction . The combined use of convolution module and LSTM has also been proved effective in [ 3 , 38 , 51 ] . A series of works [ 15 , 18 , 20 , 45 , 48 ] have been proposed to address the problem of learning viewpoint-invariant motion representation from videos or keypoint sequences . MST-AOG [ 48 ] uses an AND-OR graph structure to separate appearance of mined parts from their geometry information . Li et al . [ 20 ] learn view-invariant representation by extrapolating cross-view motions . View-LSTM [ 18 ] defines a view decomposition , where the view-invariant component is learned by a Siamese architecture . While most of these works exploits multi-modal input of RGB frames , depth maps or keypoint trajectories , MCAE focuses on the pure keypoint motion . Capsule Network MCAE is closely related to the Capsule Network [ 11 ] , which is designed to represent objects in images using automatically discovered constituent parts and their poses . A capsule typically consists of a part identity , a set of transformation parameters ( i.e . pose ) , and an activation . The explicit modeling of poses helps learning viewpoint-invariant part features that are more compact , flexible , and discriminative than traditional ConvNets . Capsules can be obtained via agreement-based routing mechanisms [ 12 , 34 ] . More recently , Kosiorek et al . [ 17 ] proposed the unsupervised stacked capsule autoencoder ( SCAE ) , which uses feed-forward encoders and decoders to learn capsule representations for images . Apart from images , capsule network has been studied in other vision tasks . In [ 61 , 62 ] , capsule network is used for point cloud processing for 3D object classification and reconstruction . VideoCapsuleNet [ 5 ] proposes to generalize capsule networks from 2D to 3D for action detection in videos . Yu et al . [ 54 ] proposed a limited study on supervised skeleton-based action recognition using Capsule Network . Sankisa et al . [ 36 ] proposed to use Capsule Network for error concealment in videos . Despite the success of capsule networks in various vision tasks , the study of capsule networks on motion representation is scarce . In this work , MCAE performs unsupervised learning of motion rep- resented as coordinates rather than pixels . It aims at learning an appearance-agnostic transformationinvariant motion representation . We believe that introducing motion to the capsule network , or the other way round , provides ( 1 ) A new , robust , and efficient view into motion signals in any dimension space under the transformation-invariance assumption ( while the motion and transformation in dimension space could have semantics different from their 2D/3D counterparts ) , and ( 2 ) proof that disentangling identity from transformation variance works not only for vision problems but a possibly larger family of time series analysis problems . 3 Methodology . We consider a single point1 in d-dimension space . The motion of the point , i.e . a trajectory , is described by X = { xi|i = 1 , . . . , L } , where xi ∈ Rd is the coordinates at time i in a d-dimension space . Semantically , X belongs to a motion pattern , subject to an arbitrary and unknown geometric transformation . Given sufficient samples of X , we aim to learn a discriminative ( in particular , transformation-invariant ) representation for those motion samples without supervision . 3.1 Framework Overview . We solve this problem in two steps , namely snippet learning and segment learning . Snippets and segments correspond to the lower and higher levels of how MCAE views the motion signal . Both snippets and segments are temporally consecutive subsets ofX , but snippets have a shorter time span than segments . In the snippet learning step , the input X is first divided into S = L/l temporally non-overlapping snippets , where l is the length of snippets . Each of these snippets will be mapped into a semantic-agnostic representation by the Snippet Autoencoder . In the segment learning step , the snippet representations are combined and fed into the Segment Autoencoder , where the full motion is represented as a weighted mixture of the transformed canonical representations . The segment activations are used as the motion representation for downstream tasks . An overview of the framework is shown in Fig . 1 . In the following section , we delineate the details for each module and explain the training procedure . 3.2 Snippet Autoencoder . To encode the snippets ’ motion variation , we propose the Snippet Capsule ( SniCap ) , which we denote as CSni . SniCap is parameterized as CSni = { T , A , µ } , where T , A , and µ are the snippet template , snippet transformation parameter , and snippet activation , respectively . The snippet 1We show a way to generalize MCAE to multi-point systems in Section 4.2 template T = { ti|ti ∈ Rd , i = 1 , ... , l } describes a motion template of length l and is the identity information of a SniCap . A and µ depend on the input snippet . The transformation parameter A ∈ R ( d+1 ) × ( d+1 ) descries the geometric relation between the input snippet and the snippet template . The snippet activation µ ∈ [ 0 , 1 ] denotes whether the snippet template is activated to represent the input snippet . Snippet Encoding/Decoding For a given snippet xi : i+l , the snippet module performs the following steps : ( 1 ) encode motion properties with Snippet Encoder into SniCaps , and ( 2 ) decode SniCaps to reconstruct the original xi : i+l . For the encoding step , a 1D-ConvNet fCONV is used to extract the motion information from xi : i+l and predict SniCap parameters , i.e . { ( Aj , µj ) |j = 1 , . . . , N } = fCONV ( xi : i+l ) where N is the number of SniCaps . The range of µ is confined by a sigmoid activation function . For the decoding step , we first apply the transformation A to the snippet templates as ( t̂ij 1 ) = Ai ( tj 1 ) , i = 1 , . . . , N , j = 1 , . . . , l. ( 1 ) Then , the transformed templates from different SniCaps are mixed , according to their activations , and the corresponding reconstructed input is x̂j = N∑ i=1 µit̂ij , j = 1 , . . . , l , ( 2 ) where t̂ij indicates the transformed coordinate of the ith SniCap at jth time step . 3.3 Segment Autoencoder . The motion information encoded in SniCaps is agnostic to the segment level motion patterns . This makes it less biased towards the training data domain . However , its utility on high-level applications , such as activity analysis or motion classification , is greatly undermined . For example , consider Fig . 2 ( a ) as a reference “ triangle ” trajectory . Fig . 2 ( b ) illustrates a possible intra-class variation . Since the two trajectories differ greatly in their local movement , they could be considered as different classes without transformation-invariant information from the full trajectory . Hence , we introduce a segment encoder to gain a holistic understanding of motion and encapsulate such information in the segment capsules ( SegCap ) . A segment is a motion of length L ( generally the segment length do not have to be the same as the signal length ) and can be interpreted as S = L/l consecutive non-overlapping snippets . A SegCap is parameterized as CSeg = { P , B , ν } , where P , B , and ν are the segment template , segment transformation parameter , and segment activation , respectively . Different from the SniCap , whose template is explicitly a motion sequence , the SegCap parameterizes the segment template P in terms of the N snippet templates . Specifically , P = { ( P i , αi ) | i = 1 , . . . , S } , where P i ∈ RN× ( d+1 ) × ( d+1 ) and αi ∈ RN . Each P ij ∈ R ( d+1 ) × ( d+1 ) ( j ∈ [ N ] additionally indexes the first dimension of P i ) describes how the jth snippet template is aligned to form the ith snippet of the segment template . The weight αij ( j ∈ [ N ] additionally indexes the elements of αi ) controls the importance of jth snippet template for the ith snippet of the segment template . In other words , ( P i , αi ) describes how the N snippet templates are used to construct an l-long snippet and a SegCap requires S such parameters to describe an L-long segment template . Fig . 2 ( c ) illustrates the interpretation of P . B and ν are dependent on the input . B ∈ R ( d+1 ) × ( d+1 ) is a transformation on P , and ν ∈ [ 0 , 1 ] is the activation of the SegCap . The segment template P is fixed for a SegCap w.r.t the training domain . Segment Encoding/Decoding Assume we have M SegCaps with which we hope to reconstruct the low-level motion encoded in the SniCap parameters . This is equivalent to reconstructing all the data-dependent SniCap parameters [ CSni1 , . . . , CSniS ] , where CSnii = { ( Aij , µij ) | j = 1 , . . . , N } is the set of SniCap parameters for the ith snippet . To obtain the SegCap parameters , we first flatten each SniCap ’ s A into a vector and concatenate it with its corresponding µ . Then we encode the S-long sequence of flattened SniCap parameters with an LSTM model fLSTM shared by all SegCaps , and M fully-connected layers ( one for each SegCap ) to produce { B , ν } . Formally , h = fLSTM ( [ CSni1 , . . . , CSniS ] ) , { B ( k ) , ν ( k ) } = f ( k ) FC ( T , h ) , k = 1 , . . . , M , ( 3 ) where T = { Ti|i = 1 , . . . , N } , and superscript ( k ) refers to the kth SegCap . The transformation and activation parameters are then applied to P to reconstruct snippet parameters P̂ ( k ) ij = B ( k ) × P ( k ) ij , i = 1 , . . . , S , j = 1 , . . . , N , k = 1 , . . . , M , ĈSnii = ( Âi , µ̂i ) = ( M∑ k=1 ν ( k ) P̂ ( k ) i , M∑ k=1 ν ( k ) α ( k ) i ) , i = 1 , . . . , S , ( 4 ) where Âi ∈ RN× ( d+1 ) × ( d+1 ) and µ̂i ∈ RN are the reconstructed snippet transformation and activation of the snippet templates for the ith snippet . Note that S = L/l , which means fLSTM can have a much smaller footprint than a recurrent network that handles the whole L-long sequence . The above formulation enables SegCap to learn a transformation-invariant representation of motion . Intuitively , P describes snippet-segment relation , and B can be regarded as the spatial relation between a segment template pattern and the observed trajectory . The segment activation ν ∈ RM reveals the semantics of the input trajectory and can be used for self-supervised training .

Summary: This work proposes a motion representation learning method based on capsule autoencoders. It decomposes an input 2D keypoint trajectory into snippets and segments, and form a hierarchical representation that is also transformation invariant. The proposed motion representation achieves state-of-the-art motion classification results on a self-proposed synthetic 2D motion dataset as well as skeleton-based human action recognition.