Unsupervised Motion Representation Learning with Capsule Autoencoders

1 Introduction . Real-world movements contain a plethora of information beyond the literal sense of moving . For example , honeybees “ dance ” to communicate the location of a foraging site and human gait alone can reveal activities and identities [ 10 ] . Understanding these movements is vital for an artificial intelligent agent to comprehend and interact with the ever-changing world . Studies on social behavior analysis [ 8 , 9 ] , action recognition [ 59 , 64 ] , and video summarizing [ 60 ] have also acknowledged the importance of movement . A key step towards understanding movements is to analyze their patterns . However , learning motion pattern representations is non-trivial due to ( 1 ) the curse of dimensionality from input data , ( 2 ) difficulties in modeling long-term dependencies in motion sequences , ( 3 ) high intra-class variation as a result of subject or viewpoint change , and ( 4 ) insufficient data annotation . The first two challenges have been ameliorated by the advances in keypoint detection [ 57 ] , spatial-temporal feature extractors [ 38 , 43 , 50 ] , and hierarchical temporal models [ 13 , 49 , 56 ] . The third and the fourth nonetheless remain hurdles and call for unsupervised transformation-invariant motion models . Inspired by the viewpoint-invariant capsule-based representation for images [ 11 , 17 ] , we exploit capsule networks and introduce the Motion Capsule Autoencoder ( MCAE ) , an unsupervised capsule framework that learns the transformation-invariant motion representation for keypoints . MCAE models motion signals in a two-level snippet-segment hierarchy . A snippet is a movement of a narrow time span , while a segment consists of multiple temporally-ordered snippets , representing a longer-time motion . In both the lower and the higher levels , the snippet capsules ( SniCap ) and the segment capsules ( SegCap ) maintain a set of templates as their identities—snippet templates and 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . segment templates , respectively—and transform them to reconstruct the input motion signal . While the snippet templates are explicitly modeled as motion sequences , the SegCaps are built upon the SniCaps and parameterize the segment templates in terms of the snippet templates , resulting in fewer parameters compared with single-layer modeling . The SniCaps and SegCaps learn transformationinvariant motion representation in their own time spans . The activations of the SegCaps serve as a high-level abstraction of the input motion signal . The contributions of this work are as follows : • We propose MCAE , an unsupervised capsule framework that learns a transformationinvariant , discriminative , and compact representation of motion signals . Two motion capsules are designed to generate representation at different abstraction levels . The lower-level representation captures the local short-time movements , which are then aggregated into higher-level representation that is discriminative for motion of wider time spans . • We propose Trajectory20 , a novel and challenging synthetic dataset with a wide class of motion patterns and controllable intra-class variations . • Extensive experiments on both Trajectory20 and real-world skeleton human action datasets show the efficacy of MCAE . In addition , we perform ablation studies to examine the effect of different regularizers and some key hyperparameters of the proposed MCAE . 2 Related Works . Motion Representation A variety of methods have been proposed to learn ( mostly human ) motion representation from video frames [ 3 , 23 , 28 , 53 ] , depth maps [ 14 , 22 , 27 , 37 , 47 ] , keypoints/skeletons [ 4 , 21 , 24 , 26 , 29 , 33 , 41 , 52 , 55 , 58 ] , or point clouds [ 6 , 7 ] . Earlier works use handcrafted features like Fourier coefficients [ 47 ] , dense trajectory features [ 46 , 30 ] , and Lie group representations [ 44 ] . Some works use canonical human pose [ 32 ] or view-invariant short tracklets to learn robust feature for recognition [ 19 ] . The development of deep learning brings the usage of convolution networks ( ConvNet ) and recurrent networks for motion representation . Simonyan et al . [ 39 ] proposes a two-stream ConvNet which combines video frame with optical flow . C3D [ 43 ] proposes to use 3D convolution on the spatial-temporal cubes . Srivastava et al . [ 40 ] uses an Long Short-Term Memory ( LSTM ) -based encoder to map input frames to a fixed-length vector and apply task-dependent decoders for applications such as frame reconstruction and frame prediction . The combined use of convolution module and LSTM has also been proved effective in [ 3 , 38 , 51 ] . A series of works [ 15 , 18 , 20 , 45 , 48 ] have been proposed to address the problem of learning viewpoint-invariant motion representation from videos or keypoint sequences . MST-AOG [ 48 ] uses an AND-OR graph structure to separate appearance of mined parts from their geometry information . Li et al . [ 20 ] learn view-invariant representation by extrapolating cross-view motions . View-LSTM [ 18 ] defines a view decomposition , where the view-invariant component is learned by a Siamese architecture . While most of these works exploits multi-modal input of RGB frames , depth maps or keypoint trajectories , MCAE focuses on the pure keypoint motion . Capsule Network MCAE is closely related to the Capsule Network [ 11 ] , which is designed to represent objects in images using automatically discovered constituent parts and their poses . A capsule typically consists of a part identity , a set of transformation parameters ( i.e . pose ) , and an activation . The explicit modeling of poses helps learning viewpoint-invariant part features that are more compact , flexible , and discriminative than traditional ConvNets . Capsules can be obtained via agreement-based routing mechanisms [ 12 , 34 ] . More recently , Kosiorek et al . [ 17 ] proposed the unsupervised stacked capsule autoencoder ( SCAE ) , which uses feed-forward encoders and decoders to learn capsule representations for images . Apart from images , capsule network has been studied in other vision tasks . In [ 61 , 62 ] , capsule network is used for point cloud processing for 3D object classification and reconstruction . VideoCapsuleNet [ 5 ] proposes to generalize capsule networks from 2D to 3D for action detection in videos . Yu et al . [ 54 ] proposed a limited study on supervised skeleton-based action recognition using Capsule Network . Sankisa et al . [ 36 ] proposed to use Capsule Network for error concealment in videos . Despite the success of capsule networks in various vision tasks , the study of capsule networks on motion representation is scarce . In this work , MCAE performs unsupervised learning of motion rep- resented as coordinates rather than pixels . It aims at learning an appearance-agnostic transformationinvariant motion representation . We believe that introducing motion to the capsule network , or the other way round , provides ( 1 ) A new , robust , and efficient view into motion signals in any dimension space under the transformation-invariance assumption ( while the motion and transformation in dimension space could have semantics different from their 2D/3D counterparts ) , and ( 2 ) proof that disentangling identity from transformation variance works not only for vision problems but a possibly larger family of time series analysis problems . 3 Methodology . We consider a single point1 in d-dimension space . The motion of the point , i.e . a trajectory , is described by X = { xi|i = 1 , . . . , L } , where xi ∈ Rd is the coordinates at time i in a d-dimension space . Semantically , X belongs to a motion pattern , subject to an arbitrary and unknown geometric transformation . Given sufficient samples of X , we aim to learn a discriminative ( in particular , transformation-invariant ) representation for those motion samples without supervision . 3.1 Framework Overview . We solve this problem in two steps , namely snippet learning and segment learning . Snippets and segments correspond to the lower and higher levels of how MCAE views the motion signal . Both snippets and segments are temporally consecutive subsets ofX , but snippets have a shorter time span than segments . In the snippet learning step , the input X is first divided into S = L/l temporally non-overlapping snippets , where l is the length of snippets . Each of these snippets will be mapped into a semantic-agnostic representation by the Snippet Autoencoder . In the segment learning step , the snippet representations are combined and fed into the Segment Autoencoder , where the full motion is represented as a weighted mixture of the transformed canonical representations . The segment activations are used as the motion representation for downstream tasks . An overview of the framework is shown in Fig . 1 . In the following section , we delineate the details for each module and explain the training procedure . 3.2 Snippet Autoencoder . To encode the snippets ’ motion variation , we propose the Snippet Capsule ( SniCap ) , which we denote as CSni . SniCap is parameterized as CSni = { T , A , µ } , where T , A , and µ are the snippet template , snippet transformation parameter , and snippet activation , respectively . The snippet 1We show a way to generalize MCAE to multi-point systems in Section 4.2 template T = { ti|ti ∈ Rd , i = 1 , ... , l } describes a motion template of length l and is the identity information of a SniCap . A and µ depend on the input snippet . The transformation parameter A ∈ R ( d+1 ) × ( d+1 ) descries the geometric relation between the input snippet and the snippet template . The snippet activation µ ∈ [ 0 , 1 ] denotes whether the snippet template is activated to represent the input snippet . Snippet Encoding/Decoding For a given snippet xi : i+l , the snippet module performs the following steps : ( 1 ) encode motion properties with Snippet Encoder into SniCaps , and ( 2 ) decode SniCaps to reconstruct the original xi : i+l . For the encoding step , a 1D-ConvNet fCONV is used to extract the motion information from xi : i+l and predict SniCap parameters , i.e . { ( Aj , µj ) |j = 1 , . . . , N } = fCONV ( xi : i+l ) where N is the number of SniCaps . The range of µ is confined by a sigmoid activation function . For the decoding step , we first apply the transformation A to the snippet templates as ( t̂ij 1 ) = Ai ( tj 1 ) , i = 1 , . . . , N , j = 1 , . . . , l. ( 1 ) Then , the transformed templates from different SniCaps are mixed , according to their activations , and the corresponding reconstructed input is x̂j = N∑ i=1 µit̂ij , j = 1 , . . . , l , ( 2 ) where t̂ij indicates the transformed coordinate of the ith SniCap at jth time step . 3.3 Segment Autoencoder . The motion information encoded in SniCaps is agnostic to the segment level motion patterns . This makes it less biased towards the training data domain . However , its utility on high-level applications , such as activity analysis or motion classification , is greatly undermined . For example , consider Fig . 2 ( a ) as a reference “ triangle ” trajectory . Fig . 2 ( b ) illustrates a possible intra-class variation . Since the two trajectories differ greatly in their local movement , they could be considered as different classes without transformation-invariant information from the full trajectory . Hence , we introduce a segment encoder to gain a holistic understanding of motion and encapsulate such information in the segment capsules ( SegCap ) . A segment is a motion of length L ( generally the segment length do not have to be the same as the signal length ) and can be interpreted as S = L/l consecutive non-overlapping snippets . A SegCap is parameterized as CSeg = { P , B , ν } , where P , B , and ν are the segment template , segment transformation parameter , and segment activation , respectively . Different from the SniCap , whose template is explicitly a motion sequence , the SegCap parameterizes the segment template P in terms of the N snippet templates . Specifically , P = { ( P i , αi ) | i = 1 , . . . , S } , where P i ∈ RN× ( d+1 ) × ( d+1 ) and αi ∈ RN . Each P ij ∈ R ( d+1 ) × ( d+1 ) ( j ∈ [ N ] additionally indexes the first dimension of P i ) describes how the jth snippet template is aligned to form the ith snippet of the segment template . The weight αij ( j ∈ [ N ] additionally indexes the elements of αi ) controls the importance of jth snippet template for the ith snippet of the segment template . In other words , ( P i , αi ) describes how the N snippet templates are used to construct an l-long snippet and a SegCap requires S such parameters to describe an L-long segment template . Fig . 2 ( c ) illustrates the interpretation of P . B and ν are dependent on the input . B ∈ R ( d+1 ) × ( d+1 ) is a transformation on P , and ν ∈ [ 0 , 1 ] is the activation of the SegCap . The segment template P is fixed for a SegCap w.r.t the training domain . Segment Encoding/Decoding Assume we have M SegCaps with which we hope to reconstruct the low-level motion encoded in the SniCap parameters . This is equivalent to reconstructing all the data-dependent SniCap parameters [ CSni1 , . . . , CSniS ] , where CSnii = { ( Aij , µij ) | j = 1 , . . . , N } is the set of SniCap parameters for the ith snippet . To obtain the SegCap parameters , we first flatten each SniCap ’ s A into a vector and concatenate it with its corresponding µ . Then we encode the S-long sequence of flattened SniCap parameters with an LSTM model fLSTM shared by all SegCaps , and M fully-connected layers ( one for each SegCap ) to produce { B , ν } . Formally , h = fLSTM ( [ CSni1 , . . . , CSniS ] ) , { B ( k ) , ν ( k ) } = f ( k ) FC ( T , h ) , k = 1 , . . . , M , ( 3 ) where T = { Ti|i = 1 , . . . , N } , and superscript ( k ) refers to the kth SegCap . The transformation and activation parameters are then applied to P to reconstruct snippet parameters P̂ ( k ) ij = B ( k ) × P ( k ) ij , i = 1 , . . . , S , j = 1 , . . . , N , k = 1 , . . . , M , ĈSnii = ( Âi , µ̂i ) = ( M∑ k=1 ν ( k ) P̂ ( k ) i , M∑ k=1 ν ( k ) α ( k ) i ) , i = 1 , . . . , S , ( 4 ) where Âi ∈ RN× ( d+1 ) × ( d+1 ) and µ̂i ∈ RN are the reconstructed snippet transformation and activation of the snippet templates for the ith snippet . Note that S = L/l , which means fLSTM can have a much smaller footprint than a recurrent network that handles the whole L-long sequence . The above formulation enables SegCap to learn a transformation-invariant representation of motion . Intuitively , P describes snippet-segment relation , and B can be regarded as the spatial relation between a segment template pattern and the observed trajectory . The segment activation ν ∈ RM reveals the semantics of the input trajectory and can be used for self-supervised training .

Summary: The paper presents a method for incorporating transformation invariance using capsule networks in the unsupervised learning of motion representations. The proposed method solves the problem in two steps, snippet and segment learning, which correspond to a lower and higher level of motion signals (basic trajectories -> reconstruction/time -> Contrastive learning and reconstruction). The method was evaluated on a synthetic dataset and three skeleton datasets for action recognition achieving convincing results.