Temporal Alignment Prediction for Supervised Representation Learning and Few-Shot Sequence Classification

1 INTRODUCTION . Distance between sequences plays a crucial role in sequence classification ( Sakoe & Chiba , 1978 ) , retrieval ( Su et al. , 2019 ) , clustering ( Garcı́a-Garcı́a et al. , 2008 ) , etc . Measuring distance between sequences is difficult , since different sequences may have different sampling rates , execution speeds , local distortions , initial states , and elastic warpage . To tackle such temporal variances , existing sequence distances either encode each sequence into a feature vector invariant to temporal variances ( Abid & Zou , 2018 ; Lohit et al. , 2019 ) or employ alignment for temporal correspondence calibration ( Sakoe & Chiba , 1978 ; Su & Hua , 2019 ) . Typical feature-based methods use recurrent neural networks ( RNNs ) ( Ramachandran et al. , 2017 ) to encode sequences and measure the Euclidean distance between corresponding features . The feature of a sequence is fixed when compared with any other sequence . Although such methods are naturally learnable and only perform forward calculations in inference , they require large amounts of sequences to train complex RNNs . Moreover , how the learned features handle temporal variances and what kinds of variances can be handled are not clear . Alignment-based methods determine different optimal alignments for different sequence pairs . This is more intuitive and flexible because temporal variances may be different when comparing different sequences . The inferred alignments clearly indicate how and where the two sequences differ in temporal steps . Most alignment methods solve an optimization problem under pre-defined feasible constraints to infer the optimal alignment . E.g. , DTW ( Sakoe & Chiba , 1978 ) requires dynamic programming and OPW ( Su & Hua , 2019 ) employs fixed-point iterations . Such optimizations are often time-consuming and can not fully utilize GPU . Moreover , since inferring the alignment is itself an optimization problem and has its own objective , sequence distance-based end-to-end learning using other objectives becomes intractable . For ∗Corresponding author : Ji-Rong Wen . instance , learning discriminative temporal representations for elements in sequences often adopt the objective that sequences of different classes are better separated w.r.t . a sequence distance ( Mei et al. , 2014 ; Su & Wu , 2020b ) . Gradients of this overall objective are difficult to pass through alignments since they are latent variables determined by another optimization problem . In this paper , we propose a learnable alignment-based sequence distance , namely Temporal Alignment Prediction ( TAP ) . TAP simulates the optimization-based inference from the input sequence pair to the optimal alignment as a function . The function is modeled by a lightweight convolutional neural network to directly predict the optimal alignment . TAP can be applied in different distance-based machine learning tasks under different settings . As instances , we show the applications of TAP in supervised representation learning and few-shot learning for sequence data . For supervised learning , we employ metric learning losses by using TAP as the distance measure to learn the frame-wise temporal representations . For few-shot learning , we adopt TAP as the distance measure in the metric learning based paradigm to compare the query and support sequence samples . In both cases , owing to the straightforward structure of TAP , the alignment prediction network and the feature extraction or transformation network can be jointly trained in an end-to-end manner , resulting in principled yet straightforward supervised learning and few-shot learning solutions for sequences . We further show the application of TAP in self-supervised alignment learning in the appendix . The main contributions of this paper are three-fold . 1 . We propose TAP , a learnable and fast sequence distance that only requires straightforward calculations in inference . 2 . We show that TAP conveniently enables end-to-end supervised sequence representation learning with different losses . 3 . We adopt TAP to dynamically align the support and query sequences for few-shot action recognition , so that alignments and temporal representations can be jointly learned with the episode-training strategy . Experiments on seven datasets demonstrate the effectiveness of TAP for both supervised learning and few-shot learning . 2 RELATED WORK . Alignment-based sequence distance . Various variations of DTW have been proposed to speed up the inference ( Salvador & Chan , 2007 ; Al-Naymat et al. , 2009 ) , or adapt to additional or modified constraints ( Ratanamahatana & Keogh , 2004 ) , or tackle sequences from different modalities ( Zhou & Torre , 2009 ; Zhou & De la Torre , 2016 ; Trigeorgis et al. , 2016 ; 2017 ; Cohen et al. , 2021 ) . They rely on non-differential dynamic programming for inference . Soft-DTW ( Cuturi & Blondel , 2017 ) optimizes a differentiable loss by using the soft-minimum cost of all feasible alignments as the objective . TCOT ( Su et al. , 2019 ) and OPW ( Su & Hua , 2019 ) view the alignment problem as the optimal transport problem with temporal regularizations . These methods solve an optimization problem to infer the optimal alignment . In contrast , our method directly predicts the alignment using a neural network , does not have its own objective , and only involves differentiable calculations . Data-driven alignment methods . In Abid & Zou ( 2018 ) , Autowarp finds the warping distance that best mimics the distance between the features encoded by LSTM , which is difficult to be integrated into an end-to-end framework because the autoencoder needs to be pre-trained and a betaCV-based optimization is required . It is shown in Tallec & Ollivier ( 2018 ) that RNN can learn to warp sequences . In Lohit et al . ( 2019 ) ; Weber et al . ( 2019 ) , a fixed warping function is predicted for a given sequence when comparing with different sequences . This may be viewed as warping all sequences w.r.t . a fixed timing scale , which may bias to the training sequences . In contrast , since the relative local variations and correspondences between different sequences are different , our method predicts different alignments for different sequence pairs . Supervised representation learning for sequences . Generally , since alignment-based distances need to solve their own optimization problems to infer alignments in the latent space , the gradients of the supervised metric learning loss can not be directly propagated . Existing methods exploit either an approximate proxy loss ( Su et al. , 2018 ; Lajugie et al. , 2014 ) , or a time-consuming optimization method to iteratively infer alignments ( Mei et al. , 2014 ) , or both ( Su & Wu , 2020a ; b ) . In this paper , we show that existing deep metric learning methods can be readily applied to the proposed TAP distance in an end-to-end manner . Few-shot action recognition . Most existing few-shot action recognition methods follow the metric learning based paradigm . In Ben-Ari et al . ( 2021 ) and Tan & Yang ( 2019 ) , TAEN and FAN en- code each action into a representation with a fixed dimension and apply vector-wise metrics . Recent works identify the importance of temporal alignment for tackling non-linear temporal variations . Various alignment-based distances are proposed to compare sequence pairs , such as the temporal attentive relation network ( TARN ) ( Bishay et al. , 2019 ) , a variant of DTW used in OTAM ( Cao et al. , 2020 ) , the permutation-invariant spatial and temporal attention reweighted distance in ARN ( Zhang et al. , 2020 ) , the temporal CrossTransformer for comparing all possible subsequences in TRX ( Perrett et al. , 2021 ) , and the two-stage temporal alignment network ( TTAN ) ( Li et al. , 2021 ) . Following this spirit , we apply the proposed TAP to perform temporal alignment and measure the distance between support and query sequences without any complicated attention mechanism , yielding a simple yet effective few-shot sequence learning solution . 3 METHODOLOGY . 3.1 TEMPORAL ALIGNMENT PREDICTION . We first revisit the unified formulation of sequence distance in Su & Wu ( 2019 ) . Let X = [ x1 , · · · , xLX ] ∈ Rd×LX and Y = [ y1 , · · · , yLY ] ∈ Rd×LY be two sequences with lengths of LX and LY , respectively . The elements xi , i = 1 , · · · , LX of X and yj , j = 1 , · · · , LY of Y lie in a d-dimensional feature space Rd . Many alignment-based distances between X and Y can be unified as follows : d ( X , Y ) = 〈T ∗ , D〉 , ( 1 ) T ∗ = argmin T∈Φ 〈T , D〉+ R ( T ) , ( 2 ) where 〈T , D〉 = tr ( T TD ) is the Frobenius dot product . D : = [ e ( xi , yj ) ] ij ∈ RLX×LY is the matrix of pairwise distances between elements in X and Y , e ( xi , yj ) is a vector-wise distance between two elements xi and yj , which we use Euclidean distance in this paper . T is an alignment matrix whose element Tij indicates whether or how likely xi and yj are aligned . Φ is the set of all feasible T with some constraints , which is a subset of RLX×LY . R ( T ) is a regularization term on T . T ∗ is the optimal alignment which is the solution of the optimization problem in Eq . ( 2 ) . Different sequence distances impose different constraints on the feasible set , have different regularization terms , and use different optimization methods for inference . For instance , DTW optimizes Eq . ( 2 ) with R ( T ) = 0 and the boundary , continuity , and monotonicity constraints via dynamic programming , while OPW optimizes Eq . ( 2 ) with two temporal regularization terms and the coupling constraints via the Sinkhorn ’ s matrix scaling algorithm . Solving T ∗ by the optimization Eq . ( 2 ) not only requires a long inference time , but also makes it difficult to apply a loss on the sequence distance Eq . ( 1 ) for learning element representations , because T ∗ is a latent variable that needs to be inferred and its gradient can not be calculated . To avoid solving the optimization problem , we propose a feedforward framework for measuring the distances between sequences , namely Temporal Alignment Prediction ( TAP ) . Fig . 1 ( a ) illustrates the TAP framework . For two sequences X = [ x1 , · · · , xLX ] and Y = [ y1 , · · · , yLY ] , their TAP distance also has the form of Eq . ( 1 ) , i.e. , the Frobenius dot product of D and T ∗ . Different from other sequence distances which infer the alignment with predefined objectives and constraints , TAP uses an alignment prediction neural network f to directly predict the optimal T ∗ = f ( X , Y ) by taking the two sequences as inputs and learns f from data . The alignment prediction network f can be instantiated by different architectures . In this paper , we propose a lightweight convolutional neural network architecture for its simplicity and fast inference speed . Specifically , we reuse the spatial ground matrix D to measure all pairwise distances between elements in X and Y . The relative positions of x̂i and ŷj are i/LX and j/LY , respectively . To incorporate the temporal dissimilarities , TAP further calculates all pairwise Euclidean distances between the relative positions between xi and yj into a matrix Dt : = [ e ( i/LX , j/LY ) ] ij ∈ RLX×LY . Then , we concatenate D and Dt along the channel dimension to form Ds with a size of Lx×Ly×2 . We use a CNN g to predict the final alignment matrix from Ds . The CNN uses three convolutional layers each followed by ReLu . The kernel size in the three layers is 5 , 5 , and 3 , respectively , the stride is fixed to 1 , and the padding is set to 2 , 2 , and 1 , respectively , to keep the spatial size . The numbers of kernels in the three layers are set to 30 , 30 , and 1 , respectively . The learned kernels are expected to capture the local alignment patterns . The output of f is augmented by a residual connection with D , resulting in the similarity matrix S : = − ( D + g ( Ds ) ) ∈ RLX×LY . The attentions of elements in Y on xi are obtained by performing Softmax on the i-th row of S. The attentions on all elements in X form an attention matrix A , which can be obtained by performing Softmax along the 2nd dimension of S. To generate the predicted alignment T ∗ , TAP finally performs a global L1 normalization on A : A = [ exp ( Sij ) ∑LY k=1 exp ( Sik ) ] ij ∈ RLX×LY ; T ∗ = [ Aij∑LX i=1 ∑LY j=1 Aij ] ij ∈ RLX×LY . ( 3 ) T ∗ij indicates the probability of aligning xi and yj . The TAP distance between X and Y is calculated as in Eq . ( 1 ) . The prediction network f of TAP is lightweight since it only contains a few convolutional kernels of g as parameters . Limitations . 1 . The family of alignments for TAP is Φ : = { T ∈ RLx×Ly+ |T1Ly = 1Lx/Lx } , where 1L is a L-dimensional vector with all 1 elements . Since strict order-preserving is not guaranteed in Φ , the performance of TAP may be limited when data are strictly ordered . 2 . T T1Lx = 1Ly/Ly does not necessarily hold , hence TAP is asymmetric and is not a real metric . These limitations make TAP more flexible in turn . 1 . Without strict order-preserving constraint , TAP can tackle local temporal reorders and generalize to non-sequential ( e.g . spatial , cross-modal , etc ) correspondences . 2 . Asymmetric alignment distinguishes the source sequence X and the target sequence Y , where all elements in X must be transported with the same mass when aligning to different target sequences . To perform classification or retrieval , we can always set the test or query sequence as the source , which serves as a standard template to be aligned . The symmetric distance can be obtained by averaging d ( X , Y ) and d ( Y , X ) .

Summary: This paper focuses on temporal sequence alignment, i.e. the task of finding an optimal alignment between sequences of different lengths. This task has been addressed by various traditional methods (e.g. DTW) that involve dynamic programming or other optimisations techniques that cannot be easily embedded in end-to-end learning frameworks.   This work proposes a method where the optimal temporal alignment is learnt by a CNN. The input is formed concatenating two matrices measuring element-wise distances between two sequences A and B. The network is trained to output a matrix whose (i, j)-th element indicates the likelihood of Ai being aligned to Bj.   The framework is evaluated with two tasks: supervised representation learning and few-shot action recognition. Tested on several datasets, the method is able to attain competitive performance and fast inference speed.