Elastic temporal alignment for few‐shot action recognition

IET Computer VisionVolume 17, Issue 1 p. 39-50 ORIGINAL RESEARCHOpen Access Elastic temporal alignment for few-shot action recognition Fei Pan, Fei Pan orcid.org/0000-0002-6361-0936 National Key Lab for Novel Software Technology, Nanjing University, Nanjing, ChinaSearch for more papers by this authorChunlei Xu, Chunlei Xu National Key Lab for Novel Software Technology, Nanjing University, Nanjing, ChinaSearch for more papers by this authorHongjie Zhang, Hongjie Zhang National Key Lab for Novel Software Technology, Nanjing University, Nanjing, ChinaSearch for more papers by this authorJie Guo, Jie Guo National Key Lab for Novel Software Technology, Nanjing University, Nanjing, ChinaSearch for more papers by this authorYanwen Guo, Corresponding Author Yanwen Guo [email protected] National Key Lab for Novel Software Technology, Nanjing University, Nanjing, China Correspondence Yanwen Guo, National Key Lab for Novel Software Technology, Nanjing University, No.163, Xianlin Avenue, Nanjing, China. Email: [email protected]Search for more papers by this author Fei Pan, Fei Pan orcid.org/0000-0002-6361-0936 National Key Lab for Novel Software Technology, Nanjing University, Nanjing, ChinaSearch for more papers by this authorChunlei Xu, Chunlei Xu National Key Lab for Novel Software Technology, Nanjing University, Nanjing, ChinaSearch for more papers by this authorHongjie Zhang, Hongjie Zhang National Key Lab for Novel Software Technology, Nanjing University, Nanjing, ChinaSearch for more papers by this authorJie Guo, Jie Guo National Key Lab for Novel Software Technology, Nanjing University, Nanjing, ChinaSearch for more papers by this authorYanwen Guo, Corresponding Author Yanwen Guo [email protected] National Key Lab for Novel Software Technology, Nanjing University, Nanjing, China Correspondence Yanwen Guo, National Key Lab for Novel Software Technology, Nanjing University, No.163, Xianlin Avenue, Nanjing, China. Email: [email protected]Search for more papers by this author First published: 04 August 2022 https://doi.org/10.1049/cvi2.12127AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Few-shot action recognition aims to learn a classification model with good generalisation ability when trained with only a few labelled videos. However, it is difficult to learn discriminative feature representations for videos in such a setting. The Elastic Temporal Alignment (ETA) for few-shot action recognition is proposed. First, a convolutional neural network is employed to extract feature representations of video frames sparsely sampled from videos. In order to obtain the similarity of two videos, a temporal alignment estimation function is utilised to estimate the matching score between each pair of frames from the two videos through an elastic alignment mechanism. The analysis shows that when we judge whether two frames from respective videos are matched, multiple adjacent frames in the videos should be considered, so as to embody the temporal information. Thus, before feeding per-frame feature vectors of videos into the temporal alignment estimation function, a temporal message passing function is leveraged to propagate the information of per-frame features in the temporal domain. The method has been evaluated on four action recognition datasets, including Kinetics, Something-Something V2, HMDB51, and UCF101. The experimental results verify the effectiveness of ETA and show its superiority over state-of-the-art methods. 1 INTRODUCTION Video action recognition is a hot research topic in the field of computer vision with wide applications, such as intelligent video surveillance, patient monitoring system, and video retrieval. Various methods have been proposed recently [1-6]. Although these approaches are capable of learning effective feature representations of videos, they generally need a large number of labelled videos for training. In contrast to labelled images, which are relatively easy to obtain, it is undoubtedly more expensive and time-consuming to collect and label large-scale video data, which require a lot of labour cost. Few-shot learning, which aims to train a model that can recognise unseen classes when just a small number of labelled examples are given, has attracted increasing attention in the machine learning and computer vision communities. Recently, works related to few-shot learning mainly focus on the image domain [7-12]. The few-shot setting poses more challenges to video action recognition. Videos are composed of continuous frames in the temporal domain. The key to few-shot action recognition lies in how to capture the spatio-temporal information effectively, given only a limited number of labelled examples. To tackle the issue of few-shot action recognition, several approaches have been proposed [13-15]. They generally first learn the similarities between the support and query videos and then predict the labels of the query videos based on the learnt similarity scores. It is observed that videos in real-world scenarios always exhibit large intra-class variations, which arise from various action styles, background clutter, and viewpoint changes [16]. Although some promising results have been achieved, these methods still have some limitations due to large intra-class variations of videos. In order to obtain the similarity between videos, Compound Memory Network (CMN) [13] and Action Relation Network (ARN) [14] concentrate on learning global video representations without exploring the relationship between frames from different videos. This will unavoidably result in the loss of information, which is severe in the few-shot setting. Cao et al. [15] propose a Temporal Alignment Module (OTAM) that calculates the distance between videos by a differentiable variant of the Dynamic Time Warping algorithm (DTW). However, the alignment mechanism in OTAM is too strict for some videos taken in real-world scenarios. Please refer to Figure 1 for an example in such a circumstance. FIGURE 1Open in figure viewerPowerPoint An example of temporal alignment between two videos belonging to the same category in real-world scenarios. The blue solid line indicates the match between two video frames. (a) The elastic alignment. (b) The strict alignment. The concepts of the elastic alignment and the strict alignment will be elaborated in Section 3. As shown in this example, the temporal ordering of frames in videos belonging to the same category may also vary. In addition, there are some noise frames in the videos, which usually appear at the beginning and end of videos. As a results, the elastic alignment is more applicable to videos in such a scenario compared with the strict alignment To address these problems, we propose the Elastic Temporal Alignment (ETA) for few-shot action recognition. The key to our method is a sequence similarity learning module that learns the similarities of the query videos with respect to the videos in the support set. This module comprises a temporal message passing function and a temporal alignment estimation function. The temporal alignment estimation function is leveraged to estimate the matching score of a pair of frames from two videos. In some situations, we need multiple adjacent frames to determine the matching degree between two video frames. For instance, the labels of videos in the Something-Something V2 dataset [17] contain information about the direction of the motion, for example, pushing something from left to right and pushing something from right to left. Contextual information in the temporal domain is crucial to judge whether two frames from respective videos are matched in this dataset. Obviously, if the temporal alignment estimation function takes only feature vectors extracted from video frames as inputs, it is difficult to obtain the alignment between videos accurately. Thus, we first employ the temporal message passing function to process features of video frames, such that the per-frame feature vectors are equipped with the context information in the temporal domain. Then, the temporal alignment estimation function is utilised to obtain the matching score of each pair of frames from two videos through an elastic alignment mechanism. Besides, inter-frame similarities are computed through leveraging the per-frame feature representations. Finally, we compute the similarity score between the two videos with the estimated matching scores as well as the inter-frame similarities. Elastic Temporal Alignment is implemented by deep neural networks and can be optimised in an end-to-end manner. Benefited from the sequence similarity learning module, ETA is capable of learning the similarity between videos effectively by exploring the relationship between frames from different videos. In this way, we increase the efficiency of data utilisation, which is significant in the few-shot setting. Besides, as opposed to OTAM, the proposed elastic alignment method does not impose strict order preservation to the alignment between videos. This relieves the limitation of OTAM and makes our method better accommodate complicated real-world situations. Our paper makes three main contributions. We propose an elastic alignment method, called ETA, for few-shot action recognition. We take advantage of a sequence similarity learning module to learn the similarity between videos effectively, which explores the relationship between frames from different videos. We evaluate ETA on four challenging datasets for action recognition (i.e., Kinetics, Something-Something V2, HMDB51, and UCF101) and achieve state-of-the-art performance in the experiments. 2 RELATED WORK Action recognition Action recognition methods based on deep learning can be roughly divided into two categories: two-stream ConvNets-based methods and 3D CNNs-based methods. Two-stream ConvNets [18] utilises the spatial network to extract the appearance features and the temporal network for capturing motion information and makes predictions by fusing the outputs of the two networks at test time. Based on two-stream ConvNets, Wang et al. [19] propose Temporal Segment Networks (TSN) that uses a sparse sampling strategy to capture long-range temporal dependencies in videos. Zhao et al. [5] propose Multi-mode Neural Network that takes advantage of multi-mode video data for action recognition. Wang et al. [6] introduce Temporal Difference Network that employs a temporal difference module to model motions in videos. C3D [20] employs 3D convolutional filters to learn spatio-temporal features for action recognition. Wang et al. [21] propose a spatio-temporal non-local module, which calculates the feature of a target position through the weighted sum of features at all positions. Tran et al. [1] propose an R(2 + 1)D structure, which factorises the 3D convolutional filters into the spatial and temporal parts. Few-shot learning Vinyals et al. [22] propose Matching Network, which generates a weighted nearest neighbour classifier based on the examples in the support set. Snell et al. [23] propose the Prototypical Network, which calculates the prototype of each category in the support set and generates distributions over categories for the query examples based on the prototypes. Finn et al. [24] propose Model Agnostic Meta Learning, a meta-learning strategy that searches for good initialisation for learnable parameters of deep neural networks which can adapt to new tasks quickly. Liu et al. [25] propose Transductive Propagation Network that learns to propagate labels from the labelled examples in the support set to the unlabelled query examples. Ziko et al. [8] propose a transductive inference algorithm that introduces a unary term based on the class prototypes and a pairwise Laplacian term encouraging similar query examples to have the same labels. Wang et al. [11] introduce dense Gaussian processes for few-shot learning. Lai et al. [12] propose a meta classifier-predictor module to generate the weights of the classification network in a new task, so as to better capture the characteristics of examples in the task. Zhang et al. propose DeepEMD [26] for few-shot learning that computes the similarity between a pair of examples with the optimal matching cost between two sets of vectors. Compared with DeepEMD that denotes an example as a set of vectors, our method represents an example as a sequence of vectors. It highlights the importance of sequence modelling of videos. Few-shot action recognition Zhu et al. [13] propose CMN that uses a key-value memory network to store the video level features and classifies test videos by looking up on the memory network. Zhang et al. [14] propose ARN consisting of an encoder, a comparator and an attention module to model the temporal structures of videos and solve the distribution shift problems by the attention by an alignment mechanism. Cao et al. [15] propose Temporal Alignment Module (OTAM) that calculates the distance between videos with the DTW [27] and continuously relaxes the discrete operations in DTW to facilitate the end-to-end training of the model [28]. Fu et al. [29] propose a video segment augmentation method to generate new videos for few-shot action recognition. AMeFu-Net [30] utilises depth information to alleviate the problem of data scarcity and fuses the features of the red-green-blue and depth streams by an instance normalisation module. ITANet [31] presents an implicit temporal alignment method for few-shot action recognition. Perrett et al. [32] propose Temporal-Relational CrossTransformers (TRX) that compares pairs of temporally ordered frames in a query video to sub-sequences of all support videos. Compared to the conventional query centred learning, Zhu et al. [33] propose a prototype-centered attentive learning (PAL) model for few-shot action recognition. Li et al. [34] solve the problem of action duration misalignment and action evolution misalignment by a two-stage action alignment network (TA2N). 3 CONCEPT DESCRIPTION In this section, we describe the problem of few-shot action recognition and elaborate on the concepts of the elastic alignment and the strict alignment. We adopt an episodic training strategy [22] that learns from diverse tasks over a large number of episodes for few-shot video action recognition. In a N-way K-shot problem, for each episode T $\mathcal{T}$ , the support set S $\mathcal{S}$ contains videos with N categories and each category has only K examples. We should classify the videos in the query set Q $\mathcal{Q}$ to one of the N categories in the support set. Besides, the episodes are randomly sampled from a meta set. In our setting, there are three meta sets, which have no overlapping categories, namely the meta training set, the meta validation set and the meta test set. We train a model with the meta training set, which will be evaluated on the meta test set. The meta validation set is used for tuning the hyper-parameters. During training, the optimisation is performed based on the prediction results of the query examples in the episode. Specifically, we predict the labels of videos in the query set Q $\mathcal{Q}$ based on the video and label pairs in the support set S $\mathcal{S}$ . Then, we calculate the loss values of all the query videos by a loss function L $\mathcal{L}$ . In order to search for the optimal model parameterised by w, we minimise the averaged loss values over multiple episodes: min w E T 1 n q ∑ j = 1 n q L V j , y j ; S , w , $\underset{\boldsymbol{w}}{\mathrm{min}\,}{\mathbb{E}}_{\mathcal{T}}\left[\frac{1}{{n}_{q}}\sum\limits _{j=1}^{{n}_{q}}\mathcal{L}\left({V}_{j},{y}_{j};\mathcal{S},\boldsymbol{w}\right)\right],$ (1)where nq denotes the number of examples in the query set Q $\mathcal{Q}$ . Besides, Vj and yj represent a video and the corresponding label in Q $\mathcal{Q}$ , respectively. The key to the above process is to compute the similarity score between two videos, which can be obtained through temporal alignment. Figure 1 demonstrates an example of temporal alignment between two videos belonging to the same category. The blue solid line in the figure indicates the match between two video frames. Since the elastic alignment is compared to the strict alignment, we first introduce the concept of the strict alignment. Given two videos, if the ith frame of the first video matches the jth frame of the second video, we add the index pair (i, j) to the matching set. We add the index pairs corresponding to all the inter-frame matchings of the two videos into the matching set and arrange the elements in the set. The strict alignment means that the temporal ordering is strictly preserved when matching. In other words, there exists an arrangement ((i1, j1), (i2, j2), …, (im, jm)) such that ik ≤ ik+1 and jk ≤ jk+1, where m represents the number of elements in the matching set and k ∈ {1, …, m − 1}. As shown in Figure 1b, the OTAM algorithm [15] preserves the temporal ordering strictly and adds pseudo nodes at the beginning and end of the second video. In addition, OTAM also imposes other constraints on the alignment to reduce computational cost. Since videos in real-world scenarios have large intra-class variations, the temporal ordering of frames in videos belonging to the same category may also vary. In this case, the matching results of video frames obtained through the strict alignment may be inaccurate. As illustrated in Figure 1a, the elastic alignment means that the temporal ordering does not need to be strictly preserved when matching and is allowed to be locally inconsistent. Compared with the strict alignment, the elastic alignment is more applicable to videos in real-world scenarios and can generate more accurate inter-frame matching results. 4 PROPOSED METHOD The proposed method can be briefly described as follows. We first sample a few frames from videos and extract per-frame features with a convolutional neural network. Then, we learn the similarities of a query video with respect to the videos in the support set. Figure 2 shows the outline of similarity learning between two videos. Moreover, for each category, we compute the mean value of the similarities of the query video with respect to the videos belonging to this category in the support set. Finally, we apply the softmax operator to the mean values to obtain the probability prediction of the category the query video belongs to and adopt the negative log-likelihood loss function for model training. FIGURE 2Open in figure viewerPowerPoint The outline of similarity learning between two videos. We first extract feature vectors of video frames sampled from a pair of videos (Va and Vb) with a feature extraction function fθ to obtain two sequences of vectors, which are denoted by A and B, respectively. Then, we apply the temporal message passing function gα on A and B to encode the temporal information and get the sequences of vectors X and Y. Next, the temporal alignment estimation function hβ is utilised to estimate the alignment score between each pair of vectors from the sequences X and Y. Besides, we compute the inter-timestamp similarities of the two sequences X and Y with the cosine similarity method. Finally, the similarity score of the two videos (s(Va, Vb)) is obtained with the matrices S and P generated by the preceding two steps. The symbol ⊗ in the figure denotes the inner product operation Our method takes advantage of a sequence similarity learning module to calculate the similarity value between videos, such that the temporal information in videos can be fully exploited for label prediction. This section first describes the sampling and feature extraction of video frames, then elaborates on the sequence similarity learning module, and finally introduces the objective function in the training phase. Frame sampling and feature extraction Videos usually consist of hundreds of video frames with high redundancy among consecutive frames. Thus, it is usually necessary to sample a few frames spanning the whole video. We follow the sparse sampling strategy as described in TSN [19]. For a given video V, we divide it into T segments of equal durations and then randomly sample a single frame from each segment. Sparse sampling enables videos of varying temporal durations to be expressed as sequences with a fixed length. The sparse sampling procedure is denoted as a function φT in this paper. After the video V is sparsely sampled, a sequence of images φT(V) = (x1, …, xT) is obtained where xi represents an image sampled from V. We take advantage of a convolutional neural network to generate compact feature representations of size Df for images, which is denoted as a function fθ parameterised by θ. We leverage fθ to extract feature vectors of images in φT(V) and get a sequence of vectors fθ(φT(V)) = (fθ(x1), …, fθ(xT)). Sequence similarity learning module In the following, we represent a sequence of vectors as a matrix, where the ith column of the matrix is equal to the ith vector in the sequence. For example, we denote a sequence of length T with each element being a D-dimensional vector as a matrix Z ∈ R D × T $\boldsymbol{Z}\in {\mathbb{R}}^{D\times T}$ . Besides, Z [ i ] ∈ R D ${\boldsymbol{Z}}_{[i]}\in {\mathbb{R}}^{D}$ represents the ith column of the matrix Z, i ∈ {1, …, T}. Given two videos Va and Vb, we first sample video frames sparsely with φT and then feed them into the feature extraction function fθ. We thus have: A = f θ φ T V a , B = f θ φ T V b , A , B ∈ R D f × T . $\boldsymbol{A}={f}_{\boldsymbol{\theta }}\left({\varphi }_{T}\left({V}_{a}\right)\right),\boldsymbol{B}={f}_{\boldsymbol{\theta }}\left({\varphi }_{T}\left({V}_{b}\right)\right),\boldsymbol{A},\boldsymbol{B}\in {\mathbb{R}}^{{D}_{f}\times T}.$ (2) In this subsection, we compute the similarity of the sequences A and B by a sequence similarity learning module as the similarity between the videos Va and Vb. We begin by introducing the DTW algorithm [27], which can be used to obtain the optimal alignment path between two sequences of vectors E and F E , F ∈ R D × T $\left(\boldsymbol{E},\boldsymbol{F}\in {\mathbb{R}}^{D\times T}\right)$ . The goal of DTW is to search for the optimal π* such that π ∗ = arg min π ∑ ( i , j ) ∈ π d E [ i ] , F [ j ] , ${\boldsymbol{\pi }}^{\ast }=\mathrm{arg}\,\underset{\boldsymbol{\pi }}{\mathrm{min}\,}\sum\limits _{(i,j)\in \boldsymbol{\pi }}d\left({\boldsymbol{E}}_{[i]},{\boldsymbol{F}}_{[j]}\right),$ (3)where π is an alignment path with a sequence of m index pairs: π = (π1, π2, …, πm) = ((i1, j1), (i2, j2), …, (im, jm)). Besides, the function d returns the distance between two vectors. In order to be considered admissible, the path should satisfy the following conditions: the beginning and end of the two sequences are matched: π1 = (1, 1), πm = (T, T); each component of the index pair in π is monotonically increasing: ik ≤ ik+1 ≤ ik + 1, jk ≤ jk+1 ≤ jk + 1, k ∈ {1, …, m − 1}. The Dynamic Time Warping algorithm is solved through dynamic programming: R i , j = d E [ i ] , F [ j ] + min R i − 1 , j , R i , j − 1 , R i − 1 , j − 1 , ${R}_{i,j}=d\left({\boldsymbol{E}}_{[i]},{\boldsymbol{F}}_{[j]}\right)+\mathrm{min}\left\{{R}_{i-1,j},{R}_{i,j-1},{R}_{i-1,j-1}\right\},$ (4)where Ri,j is the distance of the optimal alignment path between the sequence E observed up to timestamp i and the sequence F observed up to timestamp j. After solving DTW, the distance between the sequences E and F can be calculated by d i s t ( E , F ) = R T , T = ∑ i ∗ , j ∗ ∈ π ∗ 1 − E i ∗ T F j ∗ ‖ E i ∗ ‖ 2 ‖ F j ∗ ‖ 2 . $\begin{array}{rl}\hfill dist(\boldsymbol{E},\boldsymbol{F})& ={R}_{T,T}\hfill \\ \hfill & =\sum\limits _{\left({i}^{\ast },{j}^{\ast }\right)\in {\boldsymbol{\pi }}^{\ast }}1-\frac{{\boldsymbol{E}}_{\left[{i}^{\ast }\right]}^{T}{\boldsymbol{F}}_{\left[{j}^{\ast }\right]}}{\Vert {\boldsymbol{E}}_{\left[{i}^{\ast }\right]}{\Vert }_{2}\Vert {\boldsymbol{F}}_{\left[{j}^{\ast }\right]}{\Vert }_{2}}.\hfill \end{array}$ (5)Note that the function d in Equation (3) is instantiated by the cosine distance in Equation (5). After short calculation, Equation (5) can be rewritten as d i s t ( E , F ) = m − s ′ ( E , F ) , $dist(\boldsymbol{E},\boldsymbol{F})=m-{s}^{\prime }(\boldsymbol{E},\boldsymbol{F}),$ (6) w i t h s ′ ( E , F ) = ∑ i ∗ , j ∗ ∈ π ∗ E i ∗ T F j ∗ ‖ E i ∗ ‖ 2 ‖ F j ∗ ‖ 2 . $\mathrm{w}\mathrm{i}\mathrm{t}\mathrm{h}\ {s}^{\prime }(\boldsymbol{E},\boldsymbol{F})=\sum\limits _{\left({i}^{\ast },{j}^{\ast }\right)\in {\boldsymbol{\pi }}^{\ast }}\frac{{\boldsymbol{E}}_{\left[{i}^{\ast }\right]}^{T}{\boldsymbol{F}}_{\left[{j}^{\ast }\right]}}{\Vert {\boldsymbol{E}}_{\left[{i}^{\ast }\right]}{\Vert }_{2}\Vert {\boldsymbol{F}}_{\left[{j}^{\ast }\right]}{\Vert }_{2}}.$ (7) Then, we introduce a binary matrix I ∈ RT×T. For all i, j ∈ {1, …, T}, we set Iij to 1 if (i, j) belongs to π* and 0 otherwise. With the matrix I, the computation of the similarity value s′(E, F) defined in Equation (7) can be rewritten as s ′ ( E , F ) = ∑ i = 1 T ∑ j = 1 T I i j E [ i ] T F [ j ] ‖ E [ i ] ‖ 2 ‖ F [ j ] ‖ 2 . ${s}^{\prime }(\boldsymbol{E},\boldsymbol{F})=\sum\limits _{i=1}^{T}\sum\limits _{j=1}^{T}{\boldsymbol{I}}_{ij}\frac{{\boldsymbol{E}}_{[i]}^{T}{\boldsymbol{F}}_{[j]}}{\Vert {\boldsymbol{E}}_{[i]}{\Vert }_{2}\Vert {\boldsymbol{F}}_{[j]}{\Vert }_{2}}.$ (8) Next, we introduce the sequence similarity learning module in our method, the pipeline of which is summarised in Figure 3. This module is composed of two components: a temporal message passing function and a temporal alignment estimation function. The temporal message passing function transforms the feature vectors of the sampled video frames such that the features at each timestamp contain information of neighbouring frames. Besides, the temporal alignment estimation function judges whether two frames from respective videos are matched. FIGURE 3Open in figure viewerPowerPoint The pipeline of a sequence similarity learning module. It takes two sequences of vectors as input and outputs the similarity score of the two sequences. In this figure, A and B represent two sequences of length 5. Besides, gα and hβ denote a temporal message passing function and a temporal alignment estimation function, respectively. The constructions of the matrices X, Y, S, P will be explained in the main text We utilise Bidirectional Gated Recurrent Unit (BiGRU) that combines Bidirectional RNNs [35] with GRU [36] as the temporal message passing function. GRU is formulated by the following iterative equations: z t = σ W z x t + U z h t − 1 + b z , r t = σ W r x t + U r h t − 1 + b r , h ˆ t = ϕ h W h x t + U h r t ⊙ h t − 1 + b h , h t = 1 − z t ⊙ h t − 1 + z t ⊙ h ˆ t . $\begin{array}{rl}\hfill {\boldsymbol{z}}_{t}& =\sigma \left({\boldsymbol{W}}_{z}{\boldsymbol{x}}_{t}+{\boldsymbol{U}}_{z}{\boldsymbol{h}}_{t-1}+{\boldsymbol{b}}_{z}\right),\hfill \\ \hfill {\boldsymbol{r}}_{t}& =\sigma \left({\boldsymbol{W}}_{r}{\boldsymbol{x}}_{t}+{\boldsymbol{U}}_{r}{\boldsymbol{h}}_{t-1}+{\boldsymbol{b}}_{r}\right),\hfill \\ \hfill {\widehat{\boldsymbol{h}}}_{t}& ={\phi }_{h}\left({\boldsymbol{W}}_{h}{\boldsymbol{x}}_{t}+{\boldsymbol{U}}_{h}\left({\boldsymbol{r}}_{t}\odot {\boldsymbol{h}}_{t-1}\right)+{\boldsymbol{b}}_{h}\right),\hfill \\ \hfill {\boldsymbol{h}}_{t}& =\left(1-{\boldsymbol{z}}_{t}\right)\odot {\boldsymbol{h}}_{t-1}+{\boldsymbol{z}}_{t}\odot {\widehat{\boldsymbol{h}}}_{t}.\hfill \end{array}$ In GRU, xt and ht represent the input vector and the output vector at timestamp t, respectively. σ and ϕh are the sigmoid function and the hyperbolic tangent function, respectively. Besides, Wz, Uz, bz, Wr, Ur, br, Wh, Uh, bh are the learnable parameters. BiGRU processes the input vectors with two separate recurrent layers in both the forward and backward directions. The output vectors of the recurrent layers from the two directions are then concatenated as the output of BiGRU. As a consequence, BiGRU can access long-range context in two opposite directions. We denote BiGRU as the function g α : R D f × T → R D g × T ${g}_{\boldsymbol{\alpha }}:{\mathbb{R}}^{{D}_{f}\times T}\to {\mathbb{R}}^{{D}_{g}\times T}$ , where α is the concatenation of all learnable parameters and Dg is the dimension of its output features. We thus have: X = g α ( A ) , Y = g α ( B ) , X , Y ∈ R D g × T . $\boldsymbol{X}={g}_{\boldsymbol{\alpha }}(\boldsymbol{A}),\boldsymbol{Y}={g}_{\boldsymbol{\alpha }}(\boldsymbol{B}),\boldsymbol{X},\boldsymbol{Y}\in {\mathbb{R}}^{{D}_{g}\times T}.$ (9) Subsequently, a pair of columns from X and Y at respective timestamps are concatenated and passed into the temporal alignment estimatio