Temporal Alignment Prediction for Supervised Representation Learning and Few-Shot Sequence Classification

1 INTRODUCTION . Distance between sequences plays a crucial role in sequence classification ( Sakoe & Chiba , 1978 ) , retrieval ( Su et al. , 2019 ) , clustering ( Garcı́a-Garcı́a et al. , 2008 ) , etc . Measuring distance between sequences is difficult , since different sequences may have different sampling rates , execution speeds , local distortions , initial states , and elastic warpage . To tackle such temporal variances , existing sequence distances either encode each sequence into a feature vector invariant to temporal variances ( Abid & Zou , 2018 ; Lohit et al. , 2019 ) or employ alignment for temporal correspondence calibration ( Sakoe & Chiba , 1978 ; Su & Hua , 2019 ) . Typical feature-based methods use recurrent neural networks ( RNNs ) ( Ramachandran et al. , 2017 ) to encode sequences and measure the Euclidean distance between corresponding features . The feature of a sequence is fixed when compared with any other sequence . Although such methods are naturally learnable and only perform forward calculations in inference , they require large amounts of sequences to train complex RNNs . Moreover , how the learned features handle temporal variances and what kinds of variances can be handled are not clear . Alignment-based methods determine different optimal alignments for different sequence pairs . This is more intuitive and flexible because temporal variances may be different when comparing different sequences . The inferred alignments clearly indicate how and where the two sequences differ in temporal steps . Most alignment methods solve an optimization problem under pre-defined feasible constraints to infer the optimal alignment . E.g. , DTW ( Sakoe & Chiba , 1978 ) requires dynamic programming and OPW ( Su & Hua , 2019 ) employs fixed-point iterations . Such optimizations are often time-consuming and can not fully utilize GPU . Moreover , since inferring the alignment is itself an optimization problem and has its own objective , sequence distance-based end-to-end learning using other objectives becomes intractable . For ∗Corresponding author : Ji-Rong Wen . instance , learning discriminative temporal representations for elements in sequences often adopt the objective that sequences of different classes are better separated w.r.t . a sequence distance ( Mei et al. , 2014 ; Su & Wu , 2020b ) . Gradients of this overall objective are difficult to pass through alignments since they are latent variables determined by another optimization problem . In this paper , we propose a learnable alignment-based sequence distance , namely Temporal Alignment Prediction ( TAP ) . TAP simulates the optimization-based inference from the input sequence pair to the optimal alignment as a function . The function is modeled by a lightweight convolutional neural network to directly predict the optimal alignment . TAP can be applied in different distance-based machine learning tasks under different settings . As instances , we show the applications of TAP in supervised representation learning and few-shot learning for sequence data . For supervised learning , we employ metric learning losses by using TAP as the distance measure to learn the frame-wise temporal representations . For few-shot learning , we adopt TAP as the distance measure in the metric learning based paradigm to compare the query and support sequence samples . In both cases , owing to the straightforward structure of TAP , the alignment prediction network and the feature extraction or transformation network can be jointly trained in an end-to-end manner , resulting in principled yet straightforward supervised learning and few-shot learning solutions for sequences . We further show the application of TAP in self-supervised alignment learning in the appendix . The main contributions of this paper are three-fold . 1 . We propose TAP , a learnable and fast sequence distance that only requires straightforward calculations in inference . 2 . We show that TAP conveniently enables end-to-end supervised sequence representation learning with different losses . 3 . We adopt TAP to dynamically align the support and query sequences for few-shot action recognition , so that alignments and temporal representations can be jointly learned with the episode-training strategy . Experiments on seven datasets demonstrate the effectiveness of TAP for both supervised learning and few-shot learning . 2 RELATED WORK . Alignment-based sequence distance . Various variations of DTW have been proposed to speed up the inference ( Salvador & Chan , 2007 ; Al-Naymat et al. , 2009 ) , or adapt to additional or modified constraints ( Ratanamahatana & Keogh , 2004 ) , or tackle sequences from different modalities ( Zhou & Torre , 2009 ; Zhou & De la Torre , 2016 ; Trigeorgis et al. , 2016 ; 2017 ; Cohen et al. , 2021 ) . They rely on non-differential dynamic programming for inference . Soft-DTW ( Cuturi & Blondel , 2017 ) optimizes a differentiable loss by using the soft-minimum cost of all feasible alignments as the objective . TCOT ( Su et al. , 2019 ) and OPW ( Su & Hua , 2019 ) view the alignment problem as the optimal transport problem with temporal regularizations . These methods solve an optimization problem to infer the optimal alignment . In contrast , our method directly predicts the alignment using a neural network , does not have its own objective , and only involves differentiable calculations . Data-driven alignment methods . In Abid & Zou ( 2018 ) , Autowarp finds the warping distance that best mimics the distance between the features encoded by LSTM , which is difficult to be integrated into an end-to-end framework because the autoencoder needs to be pre-trained and a betaCV-based optimization is required . It is shown in Tallec & Ollivier ( 2018 ) that RNN can learn to warp sequences . In Lohit et al . ( 2019 ) ; Weber et al . ( 2019 ) , a fixed warping function is predicted for a given sequence when comparing with different sequences . This may be viewed as warping all sequences w.r.t . a fixed timing scale , which may bias to the training sequences . In contrast , since the relative local variations and correspondences between different sequences are different , our method predicts different alignments for different sequence pairs . Supervised representation learning for sequences . Generally , since alignment-based distances need to solve their own optimization problems to infer alignments in the latent space , the gradients of the supervised metric learning loss can not be directly propagated . Existing methods exploit either an approximate proxy loss ( Su et al. , 2018 ; Lajugie et al. , 2014 ) , or a time-consuming optimization method to iteratively infer alignments ( Mei et al. , 2014 ) , or both ( Su & Wu , 2020a ; b ) . In this paper , we show that existing deep metric learning methods can be readily applied to the proposed TAP distance in an end-to-end manner . Few-shot action recognition . Most existing few-shot action recognition methods follow the metric learning based paradigm . In Ben-Ari et al . ( 2021 ) and Tan & Yang ( 2019 ) , TAEN and FAN en- code each action into a representation with a fixed dimension and apply vector-wise metrics . Recent works identify the importance of temporal alignment for tackling non-linear temporal variations . Various alignment-based distances are proposed to compare sequence pairs , such as the temporal attentive relation network ( TARN ) ( Bishay et al. , 2019 ) , a variant of DTW used in OTAM ( Cao et al. , 2020 ) , the permutation-invariant spatial and temporal attention reweighted distance in ARN ( Zhang et al. , 2020 ) , the temporal CrossTransformer for comparing all possible subsequences in TRX ( Perrett et al. , 2021 ) , and the two-stage temporal alignment network ( TTAN ) ( Li et al. , 2021 ) . Following this spirit , we apply the proposed TAP to perform temporal alignment and measure the distance between support and query sequences without any complicated attention mechanism , yielding a simple yet effective few-shot sequence learning solution . 3 METHODOLOGY . 3.1 TEMPORAL ALIGNMENT PREDICTION . We first revisit the unified formulation of sequence distance in Su & Wu ( 2019 ) . Let X = [ x1 , · · · , xLX ] ∈ Rd×LX and Y = [ y1 , · · · , yLY ] ∈ Rd×LY be two sequences with lengths of LX and LY , respectively . The elements xi , i = 1 , · · · , LX of X and yj , j = 1 , · · · , LY of Y lie in a d-dimensional feature space Rd . Many alignment-based distances between X and Y can be unified as follows : d ( X , Y ) = 〈T ∗ , D〉 , ( 1 ) T ∗ = argmin T∈Φ 〈T , D〉+ R ( T ) , ( 2 ) where 〈T , D〉 = tr ( T TD ) is the Frobenius dot product . D : = [ e ( xi , yj ) ] ij ∈ RLX×LY is the matrix of pairwise distances between elements in X and Y , e ( xi , yj ) is a vector-wise distance between two elements xi and yj , which we use Euclidean distance in this paper . T is an alignment matrix whose element Tij indicates whether or how likely xi and yj are aligned . Φ is the set of all feasible T with some constraints , which is a subset of RLX×LY . R ( T ) is a regularization term on T . T ∗ is the optimal alignment which is the solution of the optimization problem in Eq . ( 2 ) . Different sequence distances impose different constraints on the feasible set , have different regularization terms , and use different optimization methods for inference . For instance , DTW optimizes Eq . ( 2 ) with R ( T ) = 0 and the boundary , continuity , and monotonicity constraints via dynamic programming , while OPW optimizes Eq . ( 2 ) with two temporal regularization terms and the coupling constraints via the Sinkhorn ’ s matrix scaling algorithm . Solving T ∗ by the optimization Eq . ( 2 ) not only requires a long inference time , but also makes it difficult to apply a loss on the sequence distance Eq . ( 1 ) for learning element representations , because T ∗ is a latent variable that needs to be inferred and its gradient can not be calculated . To avoid solving the optimization problem , we propose a feedforward framework for measuring the distances between sequences , namely Temporal Alignment Prediction ( TAP ) . Fig . 1 ( a ) illustrates the TAP framework . For two sequences X = [ x1 , · · · , xLX ] and Y = [ y1 , · · · , yLY ] , their TAP distance also has the form of Eq . ( 1 ) , i.e. , the Frobenius dot product of D and T ∗ . Different from other sequence distances which infer the alignment with predefined objectives and constraints , TAP uses an alignment prediction neural network f to directly predict the optimal T ∗ = f ( X , Y ) by taking the two sequences as inputs and learns f from data . The alignment prediction network f can be instantiated by different architectures . In this paper , we propose a lightweight convolutional neural network architecture for its simplicity and fast inference speed . Specifically , we reuse the spatial ground matrix D to measure all pairwise distances between elements in X and Y . The relative positions of x̂i and ŷj are i/LX and j/LY , respectively . To incorporate the temporal dissimilarities , TAP further calculates all pairwise Euclidean distances between the relative positions between xi and yj into a matrix Dt : = [ e ( i/LX , j/LY ) ] ij ∈ RLX×LY . Then , we concatenate D and Dt along the channel dimension to form Ds with a size of Lx×Ly×2 . We use a CNN g to predict the final alignment matrix from Ds . The CNN uses three convolutional layers each followed by ReLu . The kernel size in the three layers is 5 , 5 , and 3 , respectively , the stride is fixed to 1 , and the padding is set to 2 , 2 , and 1 , respectively , to keep the spatial size . The numbers of kernels in the three layers are set to 30 , 30 , and 1 , respectively . The learned kernels are expected to capture the local alignment patterns . The output of f is augmented by a residual connection with D , resulting in the similarity matrix S : = − ( D + g ( Ds ) ) ∈ RLX×LY . The attentions of elements in Y on xi are obtained by performing Softmax on the i-th row of S. The attentions on all elements in X form an attention matrix A , which can be obtained by performing Softmax along the 2nd dimension of S. To generate the predicted alignment T ∗ , TAP finally performs a global L1 normalization on A : A = [ exp ( Sij ) ∑LY k=1 exp ( Sik ) ] ij ∈ RLX×LY ; T ∗ = [ Aij∑LX i=1 ∑LY j=1 Aij ] ij ∈ RLX×LY . ( 3 ) T ∗ij indicates the probability of aligning xi and yj . The TAP distance between X and Y is calculated as in Eq . ( 1 ) . The prediction network f of TAP is lightweight since it only contains a few convolutional kernels of g as parameters . Limitations . 1 . The family of alignments for TAP is Φ : = { T ∈ RLx×Ly+ |T1Ly = 1Lx/Lx } , where 1L is a L-dimensional vector with all 1 elements . Since strict order-preserving is not guaranteed in Φ , the performance of TAP may be limited when data are strictly ordered . 2 . T T1Lx = 1Ly/Ly does not necessarily hold , hence TAP is asymmetric and is not a real metric . These limitations make TAP more flexible in turn . 1 . Without strict order-preserving constraint , TAP can tackle local temporal reorders and generalize to non-sequential ( e.g . spatial , cross-modal , etc ) correspondences . 2 . Asymmetric alignment distinguishes the source sequence X and the target sequence Y , where all elements in X must be transported with the same mass when aligning to different target sequences . To perform classification or retrieval , we can always set the test or query sequence as the source , which serves as a standard template to be aligned . The symmetric distance can be obtained by averaging d ( X , Y ) and d ( Y , X ) .

Summary: One-shot sequence alignment with CNN. Comparisons of sequence data are important for many tasks including action recognition and retrieval. Previous approaches include generating fixed-size feature vectors (e.g., RNN) or temporally aligning two sequences (e.g., DTW). Instead, the proposed method directly predict the alignment matrix T between two sequences X and Y with CNN. This is differentiable, and hence can be applied to many tasks such as supervised sequence classification/retrieval, and few-shot classification. Experiments uses a variety of sequence classification tasks, such as skeleton-based action recognition and spoken digit audio classification, and show a good performance of the proposed method.