Temporal Alignment Prediction for Supervised Representation Learning and Few-Shot Sequence Classification
Authors: Bing Su
Ji-Rong Wen
Venue: ICLR 2022 Poster (2022)
Keywords: Temporal Alignment
Supervised Representation Learning
Few-shot Action Recognition
Alignment Prediction
Sequence Classification

Abstract: Explainable distances for sequence data depend on temporal alignment to tackle sequences with different lengths and local variances. Most sequence alignment methods infer the optimal alignment by solving an optimization problem under pre-defined feasible alignment constraints, which not only is time-consuming, but also makes end-to-end sequence learning intractable. In this paper, we propose a learnable sequence distance called Temporal Alignment Prediction (TAP). TAP employs a lightweight convolutional neural network to directly predict the optimal alignment between two sequences, so that only straightforward calculations are required and no optimization is involved in inference. TAP can be applied in different distance-based machine learning tasks. For supervised sequence representation learning, we show that TAP trained with various metric learning losses achieves completive performances with much faster inference speed. For few-shot action classification, we apply TAP as the distance measure in the metric learning-based episode-training paradigm. This simple strategy achieves comparable results with state-of-the-art few-shot action recognition methods.

Published as a conference paper at ICLR 2022
TEMPORAL ALIGNMENT PREDICTION FOR SUPER- VISED REPRESENTATION LEARNING AND FEW-SHOT SEQUENCE CLASSIFICATION
Bing Su Ji-Rong Wen'
Beijing Key Laboratory of Big Data Management and Analysis Methods Gaoling School of Artificial Intelligence, Renmin University of China, Beijing 100872, China sabingats@gmall com; jrwen& ruc. edu. ahu..n cn
ABSTRACT
Explainable distances for sequence data depend on temporal alignment to tackle sequences with different lengths and local variances. Most sequence alignment methods infer the optimal alignment by solving an optimization problem under pre-defined feasible alignment constraints, which not only is time-consuming, but also makes end-to-end sequence learning intractable. In this paper, we propose learnable sequence distance called Temporal Alignment Prediction (TAP). TAP employs a lightweight convolutional neural network to directly predict the opti- mal alignment between two sequences, that only straightforward calculations are required and no optimization is involved in inference. TAP can be applied in different distance-based machine learning tasks. For supervised sequence repre- sentation learning, we show that TAP trained with various metric learning losses achieves completive performances with much faster inference speed. For few- shot action classification, we apply TAP as the distance measure in the metric learning-base baased episode-training paradigm. This simple strategy achieves compa- rable results with state-of-the-ard few-shot action recognition methods.
INTRODUCTION
Distance between sequences plays a crucial role in sequence classification (Sakoe Chiba 1978), retrieval (Su al. 2019), clustering Garrcaa-Garefa al. 22008), etc. Measuring distance between sequences is difficult, since different sequences may have different sampling rates, execution speeds, local distortions, initial states, and elastic warpage. To tackle such temporal variances, existing sequence distances either encode each sequence into feature vector invariant to temporal vari- ances (Abid & Zou 2018 Lohit et al., 2019) or employ alignment for temporal correspondence calibration (Sakoe &
[Su & Hua 2019). Typical feature-based methods use recurrent al., (017) to encode sequences and measure the Fu- clidean distance between corresponding features. The feature of sequence is fixed when compared with any other sequence. Although such methods are naturally learnable and only perform forward calculations in inference, they require large amounts of sequences to train complex RNNs. More- over, how the learned features handle temporal variances and what kinds of variances can be handled not clear.
neural networks (RNNs) (Ramachandran
Alignment- basedd methods determine different optimal alignments for different sequence pairs. This intuitive and flexible because temporal variances may be different when comparing different sequences. The inferred alignments clearly indicate how and where the two sequences differ in temporal steps. Most alignment methods solve an optimization problem under pre-defined feasible constraints to infer the optimal alignment. E.g., DTW (Sakoe & Chiba 1978) requires dynamic programming and OPW (Su Hual 2019) employs fixed-point iterations. Such optimizations are often time-consuming and cannot fully utilize GPU.
Moreover, since inferring the alignment is itself an optimization problem and has its own objec- tive, sequence distance-based and-to-end learning using other objectives becomes intractable. For
"Corresponding author; Ji-Rong Wen.

---

Published as a conference paper at ICLR 2022
instance, learning discriminative temporal representations for elements in sequences often adopt the objective that sequences of different classes are better separated w.r.t. a sequence distance (Mei et al.) 2014 Su & Wu 2020b). Gradients of this overall objective are difficult to pass through alignments since they are latent variables determined by another optimization problem.
In this paper, propose a learnable alignment-base sequence distance, namely Temporal Align- ment Prediction (TAP). TAP simulates the optimization based inference from the input sequence pair to the optimal alignment as a function. The function is modeled by lightweight convolutional neu- ral network to directly predict the optimal alignment. TAP can be applied in different distance-based machine learning tasks under different settings. As instances, we show the applications of TAP in supervised representation learning and few-shot learning for sequence data. For supervised learn- ing, we employ metric learning losses by using TAP as the distance measure to learn the frame- sisee temporal representations. For few-shot learning, we adopt TAP as the distance measure in the metric learning based paradigm to compare the query and support sequence samples. In both cases, owing to the straightforward structure of TAP, the alignment prediction network and the feature extraction or transformation network can be jointly trained in an end-to-end manner, resulting in principled yet straightforward supervised learning and few-shot learning solutions for sequences. We further show the application of TAP in self-supervised alignment learning in the appendix.
The main contributions of this paper three-fold. 1. We propose TAP, learnable and fast quence distance that only requires straightforward calculations in inference. 2. We show that TAP conveniently enables end-to-end supervised sequence representation learning with different losses. 3. We adopt TAP to dynamically align the support and query sequences for few-shot action recogni- tion, so that alignments and temporal representations can be jointly learned with the episode-training strategy. Experiments on seven datasets demonstrate the effectiveness of TAP for both supervised learning and few-shot learning.
2 RELATED WORK
Alignment-based sequence distance.
Various variations of DTW have been proposed to speed up
the inference (Salvador & Chan 2007
Al-Naymat al., 2009 , or adapt to additional or modified
constraints (Ratanamahatana & Keogh 2004), or tackle sequences from different modalities (Zhou
& Torre 2009; Zhou & De Torre Trigeorgis al.) 2016 2017 Cohen et al., (20TT). They
rely on non-differential dynamic programming for inference. Soft-DTW (Cuturi & Blondel| 2017) optimizes a differentiable loss by using the soft-minimum cost of all feasible alignments as the objective. TCOT (Su et al., 2019) and OPW (Su & Hual 2019) view the alignment problem as the optimal transport problem with temporal regularizations. These methods solve an optimization problem to infer the optimal alignment. In contrast, our method directly predicts the alignment using a neural network, does not have its own objective, and only involves differentiable calculations.
Data-driven alignment methods. In Abid & Zou (2018), Autowarp finds the warping distance that best mimics the distance between the features encoded by LSTM, which is difficult to be integrated into an end-to-end framework because the autoencoder needs to be pre-trained and a betaC b-bsed optimization is required. It is shown in Tallec & Ollivier (2018) that RNN can learn to warp se- quences. In Lohit et al. 2019): Weber et al. (2019). a fixed warping function is predicted for a given sequence when comparing with different sequences. This may be viewed as warping all sequences w.r.t. a fixed timing scale, which may bias to the training sequences. In contrast, since the relative local variations and correspondences between different sequences are different, our method predicts different alignments for different sequence pairs.
Supervised representation learning for sequences. Generally, since alignment-based distances need to solve their own optimization problems to infer alignments in the latent space, the gradients of the supervised metric learning loss cannot be directly propagated. Existing methods exploit either an approximate proxy loss (Su ct al., 20188) Lajugie et al. 20014), or a time-consuming optimization method to iteratively infer alignments (Mei et al. 22014,, or both (Su & Wu 2020a b). In this paper, we show that existing deep metric learning methods can be readily applied to the proposed TAP distance in end-to-end manner.
Few-shot action recognition. Most existing few-shot action recognition methods follow the metric learning based paradigm. In Ben-Ari et al. (2021) and Tan & Yang (2019), TAEN and FAN en-
2

---

Published as a conference paper at ICLR 2022
D
D
Aligament Prediction
X
T
S
CNN B
Concul cnation
SoftMMe La Nem
Frohenius dot product
T
Y
Dr
Distunce
Aligument
(a)
Figure 1: (a) The TAP framework. (b) The alignment prediction network.
code each action into a representation with a fixed dimension and apply vector-wise metrics, Recent works identify the importance of temporal alignment for tackling non-linear temporal variations. Various alignment-base based distances are proposed to compare sequence pairs, such as the temporal at- tentive relation network (TARN) (Bishay et al. 2019), variant of DTW used OTAM (Cao al. 2020). the permutation-invarian spatial and temporal attention reweighted distance in ARN (Zhang al. 2020). the temporal Troosorranerr for comparing all possible subsequences TRX (Per- rett al. (021), and the two-stage temporal alignment network (TTAN) (Li al. 2021}. Following this spirit, we apply the proposed TAP to perform temporal alignment and measure the distance be- tween support and query sequences without any complicated attention mechanism, yielding simple yet effective few-shot sequence learning solution.
3 METHODOLOGY
3.1 TEMPORAL ALIGNMENT PREDICTION
We first revisit the unified formulation of sequence distance in Su & Wu (201)). Let X [m1,... .xLx Rd and Y .',,... yyyy E Rd be sequences with lengths of Lx and Ly, respectively. The elements 2;, = 1, ... Lx of X and yj. = 1, Ly of Y lie a d-dimensional feature space Rd. Many alignment- based distances between X and Y can be
unified follows:
d(X,)) = (T*,
(1)
T arg min (T, D) + (T),
TEB
where (T, tr("" D) is the Frobenius dot product. D = [e yy)kk R4xx is the matrix of pairwise distances between elements in X and Y, e(a; yj) is a vector-wise distance between elements x; and yj- which matrix whose element Ti indicates whether or how likely x; and yj are aligned. & is the set of all feasible T with some constraints, which a subset of R xLx (T) is a regularization term on T. T is the optimal alignment which is the solution of the optimization problem in Eq.
Euclidean distance in this paper. T is alignment
Different sequence distances impose different constraints on the feasible set, have different regular- ization terms, and use differentt optimization methods for inference. For instance, DTW optimizes Eq. (2) with & (T) = and the boundary, continuity, and monotonicity constraints via dynamic pro- gramming, while OPW optimizes Eq. (2) with two temporal regularization terms and the coupling constraints via the Sinkhorn's 's matrix scaling algorithm. Solving T" by the optimization Eq. ( not only requires long inference time, but also makes difficult apply loss on the sequence distance Eq. (1) for learning element representations, because T is a latent variable that needs to be inferred and its gradient cannot be calculated.
To avoid solving the optimization problem, we propose feedforward framework for measuring the distances between sequences, namely Temporal Alignment Prediction (TAP). Fig. [(a) illustrates the TAP framework. For two sequences X [..,.... x xxxx] and Y "y..... VLy J. their TAP distance also has the form Eq. I i.e., the Frobenius dot product D and T". Different from other sequence distances which infer the alignment with predefined objectives and constraints, TAP uses an alignment prediction neural network f to directly predict the optimal T f(X. Y) by
taking the sequences inputs and learns from data.
3

---

Published as a conference paper at ICLR 2022
The alignment prediction network f can be instantiated by different architectures. In this paper, propose a lightweight convolutional neural network architecture for its simplicity and fast inference speed. Specifically, we reuse the spatial ground matrix D to measure all pairwise distances between elements in X and Y. The relative positions of d and yjj are i/ lx and j/ J/Ly, respectively. To incorporate the temporal dissimilarities, TAP further calculates all pairwise Euclidean distances be- tween the relative positions between 2; and y, into matrix D, := [e(i/ Lx Lyy E xxx
Then, concatenate D and D, along the channel dimension to form D, with a size of La x X2. We use a CNN to predict the final alignment matrix from D.. The CNN uses three convolutional layers each followed by ReLu. The kernel size the three layers is 5, 5, and 3. respectively, the stride is fixed to 1, and the padding is set to 2, 2, and 1, respectively, to keep the spatial size. The numbers of kernels in the three layers are set to 30, 30, and 1, respectively. The learned kernels are expected to capture the local alignment patterns. The output of f is augmented by residual connection with D, resulting in the similarity matrix == -D gD.)) E LxLxxLx
x2.
The attentions of elements in Y on æ; are obtained by performing Softmax the i-th row of S. The attentions on all elements in form an attention matrix A, which can be obtained by performing Softmax along the 2nd dimension of S. To generate the predicted alignment T", TAP finally performs global L1 normalization on A:
exp(S (g)) Jis E ER2xLLr T J.y k= exp(S
A
y by Jss E RLXXLY
A
(3)
Tÿ indicates the probability of aligning and yj. The TAP distance between X and Y is caleu- lated as in Eq. (T. The prediction network f of TAP is lightweight since it only contains a few convolutional kernels of g as parameters.
Limitations. 1. The family of alignments for TAP +b == (T E R R where 1, is L-dimensional vector with all 1 elements. Since strict order-preserving is not guaranteed in +, the performance of TAP may be limited when data are strictly ordered. 2. T 1,, = L,, Ly does not necessarily hold, hence TAP is asymmetric and is not real metric. These limitations make TAP more flexible in turn. Without strict order-pred ander-preserving constraint, TAP can tackle local temporal reorders and generalize to non-sequential (e.g, spatial, modal, etc) correspondences.
1t. Lz).
Asymmetric alignment distinguishes the source sequence and the target sequence Y, where all elements in X must be transported with the same mass when aligning to different target sequences. To perform classification or retrieval, we can always set the test or query sequence the source, which serves standard template to be aligned. The symmetric distance can be obtained by averaging d( X, Y) and d(Y, X).
3.2 SUPERVISED REPRESENTATION LEARNING WITH TAP
The performance of conventional alignment-baut based distances and TAP highly depends on D, where D depends on the representation space or the ground metric for elements in sequences. Therefore, learning more discriminative representations or ground metrics is of great interest. We transform wlement-wise features z; of X into new space by transformation network h. We denote the sequence of transformed features (x;;) by (X).. When labeled training sequences are available, the objective is often to make the distance better separate sequences from different classes. However, such objectives are difficult to optimize for conventional alignment- based distances because they solve another optimization problem obtain the alignments, as presented in Appendix A.2
In contrast, TAP enables supervised learning of temporal representations without sceking surrogates objectives. Given batch of B training sequences X", n = 1,... B with corresponding labels v",n n = 1,... B, for each sequence X", generate its augmentation X", whose class label the same X", and calculate its TAP distances to all other B sequences including X" after transformation by h. We obtain its largest TAP distance to h h(x"') of the same class dp ((("")) J: and its smallest TAP distance to ((x")) of different classes as d' = min #yn d(h(X (X") h(x f).
max, m'x"""""''p" - d(b
Any metric learning losses can be used to train TAP. We employ three metric learning losses to learn h and / jointly, including the triplet loss with hard-mining (HM) (Hermans al. 2017). the lifted
4

---

Published as a conference paper at ICLR 2022
structure (Lifted) loss (Song al. 2016), and binomial deviance loss (Binomial) Yi et al. 2014). For the triplet loss, we minimize the TAP distances to sequences of the same class that are larger than d' m, and maximize those different classes that are smaller than d' m. where is margin and set to 0.1. The objective to be minimized over f and h is formulated as follows:
min f.h n=1
max(d(h(XX)) A(x");;; mxx
+m.
n y
(4)
max(dj +m +m=d(m("")))xx>
a aay #3"
For the lifted structure loss, minimize the following objective to utilize the TAP distances among all the positive and negative pairs:
B min Σ $ log( E f.A
(,0I((") log( Σ n'ay" 7y"
(5)
n":g"==
For the binomial deviance loss, for the 1eth sequence, we first perform hard mining to select its hard positive and negative samples within the batch:
gon [n'ly" d(h(((">"">> h(x");; f)>d"->>-- N In'ly" vy" d((")) h(X"""")""""")) f: < +m}
(6)
We then minimize the following objective:
min Σ f.h
Σ
log(1 + (xx_
n'
Σ
log(1 + ==((d((("") -10
a.." a'E.Y'
where | and | denote the number of mined positive and negative sequences, respectively. In all losses, a and 3 are hyper- parameters controlling the weights positive and negative pairs, respectively. For all the losses, also incorporate the self-supervised loss of minimizing the TAP distance between each sequence and its augmentation, as presented in the appendix, where the aug- mentation is obtained by random blurring and random merge. For any supervised loss, the feature transformation network h and the alignment prediction network f in TAP are learned jointly by back procpapppggttion in an end-to-end manner.
3.3 FEW-SHOO ACTION RECOGNITION WITH TAP
In this subsection, we demonstrate the application of TAP in few-shot action recognition. For a N- way K-shot classification task, there are action classes and each class has support videos. The task aims to classify each query video into one of the N classes. We follow the episode training and evaluation. The video dataset is divided into a training set, a validation set, and test set. Classes in randomly construct one N-way K-shot task in
the three sets are disjoint. In the training phase, each episode to train the model. The best model selected according to the performance on tasks constructed from the validation set and evaluated on tasks constructed from the test set.
We adopt the metric learning strategy with the TAP distance. For each N-way K-shot classification task, the support videos denoted by V
1,.... N; 1.... K and the query videos denoted by V₂-4 = 1,... Q, where Q is the number of query videos. We follow (Colo//Cooaa et al. (2020); sample a fixed number L of frames per video. We use a feature embedding
Li et al. (022 backbone h to represent each frame Uú by a d-dimensional feature vector æ; = h(v;), thus each video V is represented by a feature sequence X h(V)== [x1.... æz, E _ self-attention (SA) module can be performed to enhance X = SA(X) as presented in Appendix A..
To classify query video Vg. we calculate the TAP distances between X, and all the support se- quences. Since TAP is non-symmetric, we add di d(X,. X f) and d(X X): f) as the dissimilarity
between X,, and the k-thh support sequence of the n-th class XF. The similarity between V,, and the nith class is then obtained as:
K
d X g.
X2: - -(;
(8)
5

---

Published as a conference paper at ICLR 2022
Xg is classified to the class with the largest similarity.
In the training phase, we jointly learn the backbone h and the prediction network ff of TAP with the cross-entropy loss. The loss over single episode is formulated as follows:
Q
exp((s 3a
C= Σ log
(9)
2n
exp( 8q)
where is the ground-truth class label of the query video V. SA is applied to query and support sequences before performing alignment by TAP, the method denoted by SA-TAP
4 EXPERIMENTS
4.1 DATASETS
For supervised representation learning, we evaluate TAP on four sequence datasets. MSR Ac- tion3D (Li et al. 2010 Wang et al., contains 557 skeleton sequences from 20 action classes. We follow the splitting of the dataset in Wang et al. (2012); Wang & Wu (013). We use the 192-dimensional element-wise features in Wang et al. (2012); Wang & Wu (2013). MSR Daily
Activity3D (Wang et al. 2012) contains 320 skeleton sequences from 16 activity classes. We also follow the splitting of the dataset in Wang et al. (212);; Wang & Wu (013) and use the 390-
dimensional element-wise features in Wang et al. 2012): Wang & (203). The "Spoken Ara- bic Digits (SAD)" dataset from the UCI Machine Learning Repository (Bache & Lichman 2013)
contains 8,800 sequences of 13-dimensional mel-frequency cepstrum coefficients from 10 classes. The dataset has 6,600 training sequences and 2,200 testing sequences. ChaLearn (Escalera al., 2013bla) contains 955 Italian gesture sequences from 20 classes. The dataset is split into training, validation, and test sets. Following/Fernane
al. 2013), we conduct experiments on the segmented sequences where each sequence contains one gesture. We use the 100-dimensional element- wiie features in Fernando et al. (2015). On datasets, the lengths of different sequences are different.
For few-shot action recognition, we evaluate TAP on three video datasets. UCF101 (Soomro et al.) 2012) contains 13,320 videos of 101 classes. We follow the protocol used in Zhang al. (2020) to split the dataset, where the training, validation, and test sets contain 70, 10, and 21 classes with 9154, 1421, and 2745 videos, respectively. HMDB51 (Kuehne et al. 201 I) consists of 6,840 videos from 51 classes. We also follow the protocol used in Zhang et al. (2020) to split the dataest, where the training. validation, and test sets contain 31, 10. and 10 classes with 4280, 1 1194, and 1292 videos, respectively. Donnething-Somethinn--
(SSv2) (Goyal al. 2017) contains 220,847 videos of 174 classes. We follow the protocol in Cao et al. (2020) to split the dataset, where the training, validation, and test sets contain 64, 12, and 24 classes with 77,500, 1925, and 2854 videos, respectively.
4.2 IMPLEMENTATION DETAILS
The structure and hy hyper-parameters of the alignment prediction network are fixed in TAP, as intro- duced in Section 3.1 For supervised representation learning, perform tasks each dataset: sequence classification and sequence retrieval. For classification, we use the nearest neighbor (1- NN) classifier. For each test sequence, we calculate the distances all training sequences w.r.t. the evaluated distance and its label is determined by the training sequence with the smallest distance. We use accuracy as the performance measure. For retrieval, we use each test sequence as query and rank all training sequences according to their distances to the query in ascending order. We use mean average precision (MAP) as the performance measure.
Following Su & Wu (020b)) we use three-layer perceptron as the transformation network h, where the two hidden layers have 1024 nodes followed by ReL. activations and the output has the same dimension with the input element-wing features. For learning the TAP distance,
the Adam optimizer with a momentum of 0.9 and a weight decay of le and train a maximum of 100 epochs. The learning rate is fixed to 0.0001 on all datasets. The batch size is set 64 except for the MSR Activity3D dataset, where is reduced to 16 due to memory limitation. We fix the and 3 in the lifted structure loss and the binomial deviance loss to 40 and respectively.
To evaluate TAP on few-shot action recognition, we perform 5-way 1-shot and 5-way 5-shot clas- sification tasks. We increase the numbers of kernels in the middle layers of f from 30 64. Fol-

---

Published as a conference paper at ICLR 2022
Table 1: Comparison of supervised representation learning methods.
Action3D
Activity3D
SAD
ChaLeern
Method
MAP
1-NN
MAP
I-NN
MAP
1-NN
MAP
1-NN
LDMLT+DTW Mei
2014
0.643
84,98
0.366
55.00
0.595
96.50
0.213
84.37
I.M
Mci cl al.
2014)
0.536
80.59
0.348
54.37
0.611
96.73
0.216
82.74
SWMD+DTW
Huang el al
2016
0.597
80.95
0.378
61.25
0.524
93.95
0.144
64.45
SWMD+OPW HHang
al.
2016
0.432
66.67
0,356
55.00
0.580
95,41
0.154
60.31
RVSML+DTW [Su & Wu
2019)
0.593
82.78
0,422
62.50
0.602
96.23
0.338
87.38
RVSML+OPW (Su
2019
0.475
76.56
0.366
57.50
0.656
97.09
0.331
83.82
Binomial+DTW ct al.
2014
0.481
60.81
0,568
62.50
0.482
91.73
0.182
69.72
Binomial-DTW
Yi ct al.
2014
0,500
62.64
0.616
60.62
0.602
95.77
0.199
69.17
Lifted-DTW (Song et al.
2016
0.440
67.40
0,484
62.50
0.464
89.64
0.130
66.85
Lifted+OPW (Song et al.
2016
0.504
65.20
0.596
65.62
0.548
94.86
0.146
67.66
HM+DTW Hermans et al.
2017
0.432
57.51
0.589
65.62
0.656
92.68
0.237
67.28
HM+OPW Hermans ct al. 2017
0.438
55.31
0.613
63.12
0.738
95.95
0.256
68.41
MS+DTW Wang et al.
2019)
0.344
45,42
0.479
51.25
0.580
85.64
0.181
65,95
MS+OPW Wang
2019)
0.342
40.66
0.480
50.00
0.631
92.27
0.197
63.67
D-R+DTW
0.617
79.49
0.531
61.88
0.782
97.32
0.440
81.15
D-R+OPW (Su & Wu
20/20b
0.691
74.36
0.689
65.62
0.831
98.91
0.463
79.91
HM
0.818
87.91
0.585
69.38
0.959
99.55
0.578
87.12
Lifted-SoftDDT
0.074
4.03
0.108
6.25
0.123
10.00
0.560
80.66
Binom
0.806
89.74
0.108
6.25
0.123
10.00
0.524
81.38
HM+TAP (Ours)
0.798
87.18
0.701
74.38
0.961
98.82
0.614
85.06
Lifted+TAr TAP (Ours)
0.772
82.05
0.728
73.75
0.936
98.14
0.469
74.64
Binomial+TAP (Ours)
0.826
87.91
0.678
73.13
0.952
98.64
0.614
87.17
lowing Cao et al. 2020); Zhang et al. (2020). we uniformly sample = 8 frames per video, resize each frame to 256 256, augment the frames with random horizontal flipping, perform 224 224 random crops for training and center crops for testing, and use the pretrained ResNet-50 as the backbone. We jointly tune the backbone and train TAP by the SGD optimizer with learning rate of 0.001. Each episode contains task due to the memory constraint. We adapt the toolbox and all other spperr-prramters remain the same. We train a maximum of 100000 iterations on SSv2 and 35000 iterations on other datasets. We select the best iteration every 5000 iterations according to the performance on the validation set. For I-shot task on HMDB51, the validation accuracy starts to fluctuate after 5000 iterations, so we train 5000 iterations and select every 1000 iterations. We randomly sample 10,000 tasks from the test set and report the average accuracy.
4.3
COMPARISON WITH SUPERVISED REPRESENTATION LEARNING METHODS
TAP can be used to learn temporal representations a supervised manner as presented in Sec- Comparisons with other supervised sequence representation learning methods the four datasets are shown in Table where we follow the setting and compare with the results reported in Su & Wu (2020b). LDMLT (Mei et al.) 2014),, SWMD (Huang al., 2016), and RVSML (Su & Wu 2019) learn linear transformations for elements in sequences, while deep metric learning meth- ods with binomial deviance loss (Binomial) (Yi et al. (0044), lifted structured loss (Lifted) (Song et al. 2016). triplet loss with hard-mining (HM) (Hermans et al.) (017). and multi-similarity loss (MS) (Wang al. 2019) learn non-linear temporal representations by treating elements in each se- quence as independent samples of the same class. DecpRVSML (D-R) (Su & Wu 2020b) is the deep extension of RVSML. All methods use the same transformation network. TAP generally out- performs these methods on all datasets. Especially, TAP achieves much higher MAP, which reflects that the TAP distances between sequences from the same class are generally smaller than those from different classes. This shows the effectiveness of the end-to-end joint training of the feature transformation network and the alignment prediction network of TAP.
tion 3.2]
SoftDTW is differentiable and also be combined with different losses train h. This can be achieved by replacing the TAP distance d(h(X"), h(x");; ff with the SoftDTW distance in Eq. (4),
7

---

Published as a conference paper at ICLR 2022
Table 2:
Comparison of the training time per
Test sequence
epoch (middle) and running time (bottom).
Dataset
Action3D
SAD
Chal.cam
Neavest tralinng requence TAP
1
39.64
39.81
39.72
SofDDW
23.28
508.39
544.30
Neuwent sepuence Sofoor
TAP
35.78
876.75
901.46
DTW
0.9212
29.568
28.9270
OT
279.6414
794.9984
2614.677
OPW
0.8029
25.309
32.7774
Figure 2: TAP (left) alignment and Soft-DTW (right) alignment.
SoftDTW
0.217
4.922
4.959
TAP 0.138
3.009
3.033
Eq. (5), and Eq. ( respectively. Comparisons between TAP and SoftDTW for supervised represen- tation learning are also presented in Tab. [ We employ the PyTorch implementation of ToftDTW. The triplet loss is relatively straightforward, since the sequence distance does undergo linear function. SoftDTW outperforms TAP when the HM loss is used on some datasets. In the other two losses, exponential and logarithmic transformations are applied the sequence distance, so the backpropagation through the alignment variable is more complicated. For SoftDTW, using a large will lead to infinite initial losses, so we reduce a and 8 to 2 and 1, respectively, When using the lifted structure loss and binomial deviance loss to optimize the transformation network h, the results of SoftDTW are inferior those of TAP on most datasets. In addition, it is easy to fall into trivial solution when incorporating SoftDTW into complex losses some datasets. This may be because that gradients backpropagated through SoftDTW cannot automatically calculated and such manually derived backward processes with respect to complex losses may be more unstable. On all datasets, TAP outperforms all other methods by a significant margin on MAP. In cases where the accuracy of TAP is not the best, it is also comparable with the best result. These results show that TAP can be better learned with different losses, the training of TAP is easier and more consistent, and TAP has better potential to be applied to more complex tasks. In addition, TAP can be general- ized to tackle non-sequential data, e.g., determining the spatial or set-to-set correspondences; while SoftDTW is more restricted strictly order-preserving sequence data.
4.4 EVALUATION ON COMPUTATIONAL TIMES AND ALIGNMENTS
We compare the average training time per epoch of SoftDTW and TAP on single TITAN V GPU by using the triplet loss in the middle part of Table Since the asymmetric TAP distance between any two sequences in batch is calculated twice, the training time of TAP is longer than SoftDTW. Table (bottom) compares the average running times for computing the distances between one test sequence and all training sequences with different distances including DTW, optimal transport (OT), OPW, SoftDTW, and TAP. 7 in the top row indicates the average length of sequences in the dataset. TAP runs faster than OT, DTW, and OPW by at least an order of magnitude. The larger the dataset, the more significant the advantage. This is because TAP avoids complex optimizations required by other distances and can be accelerated conveniently by GPU. When both running on GPU, TAP is faster than SoftDTW. As shown in Appendix AA.5] the advantage increases as increases.
2ig[2 shows visualizations on the MSR Action3D dataset. For the test sequence of "draw tick", the nearest training sequence by TAP comes from the same class, but the one by Soft-DTW comes from another class "horizontal arm wave". Actions contain different movement cycles and similar poses. Each alignment path of Soft-DTWW is strictly prder-preserving causing some meaningless alignments. The alignments of TAP show periodicity while preserving the general orders.
In the alignment prediction network f. implement g by CNN to model the temporal constraints since multiple convolution kernels in different layers are suitable for capturing different local align- ment patterns and variances of local motions. The local receptive field of CNN can be viewed as imposing a constraint on the local temporal warping. Ablation studies on the architecture and hyper- parameters of g are presented in Appendix A.7 To evaluate the effect of g( (D.), we apply the trained
In Fig. 2
we
the implementation
(ttp:://ggttaa..commmminnneeeee for calculating the SoftDTW alignment.
8

---

Published as a conference paper at ICLR 2022
Table 5: Comparison of few-shot action recognition methods.
UCFI01
HMDB51
SSv2
Method
1-shot
5-shot
1-shot
5-shot
1-shot
5-shot
ARN
66.3.10.99
83.1100.70
45.5
60.6.10.86
CMN-J
34.4
43.8
OTAM
79.9
88,9
54.5
66.1
42.8
52.3
TRX(2 1)
79.0
95.9
52.1
76.1
38.8
60.6
TRX(2 2,3)
78.2
96.1
53.1
75.6
42.0
64.6
TTAN
80.9
93.2
57.1
74.0
SoftDTW
51.4+-0.4
62.1+0.4
36.0+E0.4
46.0==0.4
23.9++0.3
31.5+00.4
TAP
83.5.0.3
94,510.2
54.01004
72.510.4
43.5110.4
61.310.4
SA-TAP
83.910.3
95,4110.2
57.5110.
74.2.10.4
45.2110.4
63.010.4
ТAР model on each test sequence and calculate the Frobenius norm ratio:
[|x(D.)|E
The average
||D]i
ratios of all test sequences with different losses are shown in Tab. showing that (D+)) contributes more than D.
The ratio ranges from 2 to 21,
Table 3: Frohenius Norm ratios.
Table 4: MSE between predicted alignments and con-
Method
Action3D
Activity3D
structed ground-truth alignments.
HM+TAP 3.5539
10.2145
Dataset
SoftDWWs
TAP-s
SoftDTW
TAP
lifted+TAP
21.2767
15.6680
Action3D
0.0165
0.0131
0.0163
0.0062
Binomial+TAP
2.1119
13.8124
Activity3D
0.0040
0.0037
0.0037
0.0026
To quantitatively evaluate the alignments. we generate augmentations and corresponding ground- truth alignments for each sequence in the test set. Comparison on MSE between ground-truths and predicted alignments by the supervised learned TAP (TAP-s) and SoftDTW (SoftDTW-s) is shown in the left part of Tab. [4] f in TAP can also be directly learned by minimizing MSE on the training set via augmentations. We directly apply TAP learned in this way and SoftDTW align the original test sequences and their augmentations. Results are shown in the right part of Tab. [ Details of the evaluation methods are presented in Appendix A.8 In both cases, TAP achieves lower MSEs.
4.5
EVALUATION ON FEW-SHOT ACTION RECOGNITION
We compare TAP and SA-TAP with state-of-the-an few shot action recognition methods including
ARN Zhang et al. (2020). CMN-J Zhu & Yang 2018 2020), OTAM (Cao et (020), TRX (Perrett 2021). Results on 5-way 1-shot and 5-way 5-shot tasks are shown Table 5 where the results of OTAM and TRX UCF101 and HMDB51 1 reported in Li cet al. 202T). OTAM employs a modified DTW as the distance. TAP generally outperforms OTAM and SohDTW significantly. When only support sample per class is available, TAP also achieves better results than TRX. TAP with much fewer parameters outperforms TTAN consisting of two stages on UCFI01, TAP has tighter 95% confidence intervals and more stable than SoftDTW and ARN. SA-TAP further improves TAP. It outperforms all other methods in 1-shot tasks on UCF-101 and HMDB-51. On SSv2, SA-TAP achieves comparable results TTAN 1-shot learning and comparable results to TRXs in 5-shot learning, while significantly outperforming other methods. These results validate the effectiveness of TAP and SA-TAP on few-shot action recognition.
ct al. 2021), and ITAN Li al.
5 CONCLUSION
In this paper, we present TAP, which is learnable distance for sequences. TAP calculates the opti- mal alignment from the input sequences using a prediction network. We demonstrate the application of TAP in supervised representation learning for sequence data and few-shot action recognition. For supervised learning, TAP can learned with various metric learning-bate based losses. Experimental results four real-world datasets show that TAP achieves competitive performances with much faster inference speed. For few-shot learning, simply applying TAP in the metric- metric-based strategy achieves comparable results with some state-of-the-art few-shot action recognition methods.

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
This paper proposes a learnable sequence distance that employs a lightweight neural network to predict the optimal alignment between two sequences and applications in supervised learning, few-shot learning, and self-supervised learning (in the appendix) for sequence data. The proposed distance can be applied to any sequence data to speed up the inference of optimal alignment and improve the discrimination ability. Because this work presents such a general distance measure and its learning methods under different conditions, we did not see any particular foresceable negative ethics impacts.
REPRODUCIBILITY STATEMENT
The architecture and hyper-parametent of the alignment prediction network in TAP are fixed and specified in Section 3.1 We specify all the training details on hyper-parameters and experimental
settings in Section 4.2
Our code available at htttttts//gggeeeee wwt.coo/////.//. com/BingSu12/
ACKNOWLEDGMENTS
This work was supported in part by the National Natural Science Foundation of China No. 61976206 and No. 61832017, Beijing Outstanding Young Scientist Program NO. Beijing Academy of Artificial Intelligence (BAAI). China Unicom Innovation Ecological Cooperation Plan, the Fundamental Research Funds for the Central Universi- ties, the Research Funds of Renmin University of China 21XNLG05, and Public Computing Cloud, Renmin University of China. This work was also supported part by Intelligent Social Governance Platform, Major Innovation & Planning Interdisciplinary Platform for the "Double-Fin Class" Ini- tiative, Renmin University of China, and Public Policy and Decision maainng Research Lab of Ren- min University of China.
JUWZYYJ01121110002220
REFERENCES
Abubakar Abid and James Zou. Autowarp: learning a warping distance from unlabeled time series using sequence autoencoders. In Proceedings of the 32nd International Conference Neural Information Processing Systems, pp.
2018
Ghazi Al-Naymat, Sanjay Chawla, and Javid Taheri. Sparsedtw: novel approach to speed up dynamic time warping. In Proceedings of the Eighth Australasian Data Mining Conference- Volume 101, pp. 117-127. Australian Computer Society, Inc., 2009.
K. Bache and M. Lichman. UCI Machine Learning Repository. http://archive.ie.ucii.oo eci.ed//ml Uni- versity of California, Irvine, School of Information and Computer Sciences, 2013.
Rami Ben-Ari, Mor Shpigel Nacson, Ophir Azulai, Udi Barzelay, and Daniel Rotman. Taen: Tem- poral aware embedding network for sho- action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2786-2794, 2021.
Mina Bishay, Georgios Zoumpourlis, and Ioannis Patras. Tarn: Temporal attentive relation network for few-shot and zero-shot action recognition. arXiv preprint arXiv:1907.09000 2019.
Kaidi Cao, Jingwei Ji, Zhangjie Cao, Chien-Yi Chang, and Juan Carlos Niebles. Few-shot video classification via temporal alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10618-10627, 2020.
Samuel Cohen, Giulia Luise, Alexander Terenin, Brandon Amos, and Mare Deisenroth. Aligning time series on incomparable spaces. In International Conference on Artificial Intelligence and Statistics, pp. 1036-1044. PMLR, 2021.
Marco Cuturi. Sinkhor distances:
Lightspeed computation of optimal transport. In Advances in neural information processing systems, pp. 2292-2300, 2013.
10

---

Published as a conference paper at ICLR 2022
Marco Cuturi and Mathieu Blondel. Soft-dtw: differentiable loss function for time-series. In International Conference on Machine Learning, pp. 894 894-903, 903, 2017.
Sergio Escalera, Jordi Gonzålez, Xavier Baró, Miguel Reyes, Isabelle Guyon, Vassilis Athitsos, Hugo Escalante, Leonid Sigal, Antonis Argyros, Cristian Sminchisescu, al. Chalearn multi- modal gesture recognition 2013: grand challenge and workshop summary. In Proceedings of the 15th ACM on International conference on multimodal interaction, pp. 365-368. ACM, 2013a.
Sergio Escalera, Jordi Gonzàlez, Xavier Baró, Miguel Reyes, Oscar Lopes, Isabelle Guyon, Vassilis Athitsos, and Hugo Escalante. Multi-modal gesture recognition challenge 2013: Dataset and results. In Proceedings of the 15th ACM on International conference on multimodal interaction, pp. 445-452. ACM, 2013b.
Basura Fernando, Efstratios Gavves, José Oramas M., Amir Ghodrati, and Tinne Tuytelaars. Model- ing video evolution for action recognition. In Proc. IEEE Int'l Conf. Computer Vision and Patlern Recognition, 2015.
Darío Gareía Garefa-García, Emilio Parrado Hernández, and Fernando Díaz-de María. A new distance measure for model-based sequence clustering. IEEE transactions on pattern analysis and machine intelligence, 31(7):1225-1331 2008.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The something something video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pp. 5842-5850, 2017.
Alexander Hermans, Lucas Beyer, and Bastian Leibe. In defense of the triplet loss for person re- identification. arXiv preprini arXiv: 1ttiv:7700..0 1000:077377 2017.
Gao Huang, Chuan Guo, Matt J Kusner, Yu Sun, Fei Sha, and Kilian Q Weinberger. Supervised word mover's distance. In Advances in Neural Information Processing Systems, pp. 4862-4870, 2016.
Soheil Kolouri, Se Rim Park, Matthew Thorpe, Dejan Slepcev, and Gustavo K Rohde. Optimal mass transport: Signal processing and machine-learning applications. IEEE Signal Processing Magazine, 34(4):43-555 59, 2017.
Hildegard Kuehne, Hueihan Jhuang, Estibaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: large video database for human motion recognition. In 2011 International conference on computer vision, pp. 2556-2563. IEEE, 2011.
Rémi Lajugie Damien Garreau, Francis Bach, and Sylvain Arlot. Metric learning for temporal sequence alignment. In Advances in Neural Information Processing Systems (NIPS), pp. 1817- 1825, 2014.
Shuyuan Li, Huabin Liu, Rui Qian, Yuxi Li, John See, Mengjuan Fei, Xiaoyuan Yu, and Weiyao Lin. Ttan: Two-stage temporal alignment network for few-shot action recognition. arXiv preprint arXiv:2107.04782 2021.
Wanqing Li, Zhengyou Zhang, and Zicheng Liu. Action recognition based on bag of 3d points. In IEEE Int'l Workshop on CVPR for Human Communicative Behavior Analysis, 2010.
Suhas Lohit, Qiao Wang, and Pavan Turaga. Temporal transformer networks: Joint learning of invariant and discriminative time warping. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 12435-12435. 2019.
Jiangyuan Mei, Meizhu Liu, Hamid Reza Karimi, and Huijun Gao. Logdet divergence-based metric learning with triplet constraints and its applications. IEEE Transactions on Image Processing, 23 (11::492-49931 2014.
Toby Perrett, Alessandro Masullo, Tilo Burghardt, Majid Mirmehdi, and Dima Damen. Temporal- relational crosstransformers for few-shot action recognition. In Proceedings of the IEEÉ/CVF Conference Computer Vision and Pattern Recognition, pp. 475-484, 2021.

---

Published as a conference paper at ICLR 2022
Prajit Ramachandran, Peter J Liu, and Quoe Le. Unsupervised pretraining for sequence sequence learning. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 183-391,200 2017.
Chotirat Ann Ratanamahatana and Eamonn Keogh. Making time-series classification more accurate using learned constraints. In Proc. SIAM Ini. Conf. Data Mining, pp. 11-22. Lake Buena Vista, Florida, 2004.
H. Sakoe and S. Chiba. Dynamic Programming Algorithm Optimization for Spoken Word Recog- nition. IEEE Trans. Acousties, Speech and Signal Processing, 26(1):43-49, 1978.
Stan Salvador and Philip Chan. Toward accurate dynamic time warping in linear time and space. Intelligent Data Analysis, 1165):611-588 2007.
Hyun Oh Song. Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition, pp. 4004-4012, 2016.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint 1rXiv:1212.0402, 2012.
Bing Su and Gang Hua. Order-preserving optimal transport for distances between sequences. IEEE transactions on pattern analysis and machine intelligence, 41(12)2961-2977,, 2019.
Bing Su and Ying Wu. Learning distance for sequences by learning a ground metric. In International Conference on Machine Learning, pp. 6015 6025, 2019.
Bing Su and Ying Wu. Learning low-dimensional temporal representations with latent alignments.
IEEE transactions on pattern analysis and machine intelligence, 42(1 11:2842-887 2020a.
Bing Su and Ying Wu. Learning meta-distance for sequences by learning a ground metrie via virtual sequence regression. IEEE Transactions Pattern Analysis and Machine Intelligence, 2020b.
Bing Su, Xiaoqing Ding, Hao Wang, and Ying Wu. Discriminative dimensionality reduction for multi-dimensional sequences. IEEE Trans. Pattern Anal. Mach. Intell., 40(1):77-91, 2018.
Bing Su, Jiahuan Zhou, and Ying Wu. Order-preserving wasserstein discriminant analysis. In Proceedings of the IEEE International Conference on Computer Vision, pp. 9885-9894, 2019,
Corentin Tallec and Yann Ollivier. Can recurrent neural networks warp time? In International Conference on Learning Representation 2018, 2018
Shaoqing Tan and Ruoyu Yang. Learning similarity: Feature-aligning network for few-shot action recogmition. In 2019 International Joint Conference on Neural Networks (IJCNN), pp. 1-7. IEEE, 2019.
George Trigeorgis, Mihalis Nicolaou, Stefanos Zafeiriou, and Bjorn Schuller. Deep canonical time warping. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion, pp. 10-5118 2016.
George Trigeorgis, Mihalis A Nicolaou, Björ W Schuller, and Stefanos Zafeiriou. Deep canon- ical time warping for simultaneous alignment and representation learning of sequences. IEEE transactions on pattern analysis and machine intelligence, 40(5):1128-111 1138, 2017.
Jiang Wang and Ying Wu. Learning maximum margin temporal warping for action recognition. In Proc. IEEE Int'l Conf. Computer Vision, 2013.
Jiang Wang, Zicheng Liu, and Ying Wu. Mining actionlet ensemble for action recognition with depth cameras. In Proc. IEEE Int') Conf. Computer Vision and Pattern Recognition, 2012.
Xun Wang, Xintong Han, Weilin Huang, Dengke Dong, and Matthew R Scott. Multi-similarity loss with general pair weighting for deep metric learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5022-5030, 2019.
12

---

Published as a conference paper at ICLR 2022
Ron Shapira Weber, Matan Eyal, Nicki Skafte Detlefsen, Oren Shriki, and Oren Freifeld. Diffeo- morphic temporal alignment nets. In NeurIPS, pp. 1570-6581,22 2019.
Dong Yi, Zhen Lei, Shengcai Liao, and S. Li. Deep metric learning for person re-identification. In International Conference on Pattern Recognition, pp. 34-39, 2014.
Hongguang Zhang, Li Zhang, Xiaojuan Qi, Hongdong Li, Philip HS Torr, and Piotr Koniusz. Few- shot action recognition with permutation-invarian attention. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V 16, pp. 525- 542. Springer, 2020.
Feng Zhou and Fernando De la Torre. Generalized canonical time warping. IEEE transactions on pattern analysis and machine intelligence, 38(2):279-22) 2016.
Feng Zhou and Fernando Torre. Canonical time warping for alignment of human behavior. In Proc. Advances in Neural Information Processing Systems, pp. 2286-2294, 2009.
Linchao Zhu and Yi Yang. Compound memory networks for few-shot video classification. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 751-766, 2018.
Linchao Zhu and Yi Yang. Label independent memory for semi-supervised few-shot video classifi- cation. IEEE Transactions Putlern Analysis and Machine Intelligence, 2020.
A APPENDIX
A.1 SELF-SUPERVISED LOSS FOR TAP
We show that the proposed TAP distance can also be learned in a self-supervised manner with unlabeled sequence samples. We learn the prediction network of TAP by minimizing the TAP dis- tances between sequences and their corresponding augmentations. Data of specific modalities such as images and videos can be augmented through low-level transformations, where self-supervised learning methods learn the feature embedding network by encouraging consistent features extracted from original data and their augmentations. Differently, we focus on learning the distance between sequences, where the element-wise features are already given no matter where and how the fea- tures are extracted, so low-level transformations are not applicable. To this end, we propose two augmentation methods for sequences: random blur and random merge.
Random blur. Random blur constructs a blur kemel with a randomly selected size. The values in the kernel are also randomly selected from the range of [O, 1] so that values in the kernel decrease from middle to both sides. Given a sequence X = [21, ... Lxx E RdxLx, random blur generates an augmented sequence X by convolving X with K. The stride along the 1st dimension is set to 1. The stride along the 2nd dimension is randomly selected from 1, 2, No padding is applied. Thus, each column element 2;; of X the weighted average of several successive elements of X. The ground-truth alignment values between 2; and the N₂ elements are set as the values in K, while those between 2; and all other elements X are set to 0. In this way, we construct the ground-truth alignment matrix between X and X.
Random merge. For X with a length of Lx, random merge first randomly selects an integer i. from [0.5Lx,
X is then randomly segmented into L parts, where each part has at least one element. For the k-th part with La elements, Lg values are randomly sampled from a standard Gaussian distribution and then normalized by softmax to form La weights. The La elements are averaged with the weights to form new element Te.. The augmented sequence consists of all the generated elements for all parts sequentially: X [ :igh The ground-truth alignment values between zgg and the L elements in the k-th part of X are set as the L, corresponding weights, while those between ai and elements in other parts set to 0. In this way, the ground- truth alignment matrix T between X and X is constructed.
0. 0.8Lxx
Self-supervised learning of TAP. Given N unlabeled training sequences X", 12 = 1,... ,,, the goal is learn the TAP distance by learning the prediction network. For each X", we first ran- domly select of the methods generate its augmented sequence X and the corresponding
13

---

Published as a conference paper at ICLR 2022
ground-truth alignment T" We then calculate the TAP distance d d(x", :x:;f) and the correspond-
ing non-normalized alignment A in Eq. (3) between X and X". Since TAP is asymmetric, also calculate the dual TAP distance d( X X"; f) and dual alignment A between X and X". We minimize the TAP and its dual distances. We also minimize the Mean Squared Errors (MSEs) between the predicted dual alignments and the ground-truth alignments, which indirectly forces the alignment and the dual alignment to be consistent. The objective for a batch of B sequences is:
B d(x"
min B
+(
;";; MSE TT) + M. T))
(10)
1=1
Parameters of the alignment prediction network f in TAP are learned by back- pock-propagation. Since D is fixed with given sequences and T is obtained after softmax and L1 normalization, the TAP dis- tance between two sequences is bounded. Therefore, f has no trivial collapse solution, so negative required. Once is learned, TAP difference between two sequences by feedforward calculations. In Section 3.2 the self-supervised loss 10 is added to all supervised losses improve the stability of the learned TAP.
sequence samples
naturally
be applied to measure the
Table 6: Results the MSR Action3D dataset.
Table 7: Results MSR Activity3D dataset.
Distance
MAP
1-NN
5-NN
30-NN
Distance
MAP
I-NN
5-NN
30-NN
DTW
0.590
81.32
80.95
72.53
DTW
0.338
58.75
49.38
31.87
IDTW
0.567
82.78
79.12
64.84
IDTW
0.288
50.00
50.00
30.00
nDTW
0.565
79.85
76.92
67.40
nDTW
0.306
55.63
52.50
30.00
OT
0.544
78.02
75.09
59.34
OT
0.308
55.63
50.62
25.00
Sinkhomn
0.546
78.02
74.73
60.44
Sinkhorn
0.307
54.37
50.62
25.62
TCOT
0.578
80.59
79.49
67.03
TCOT
0.321
59.38
51.88
28.13
OPW
0.587
84.25
80.22
67.03
OPW
0,346
58.13
53.75
30.63
SoftDTW
0.589
81.32
80.95
72.89
SoftDTW
0.339
60.00
50.00
33.75
SoftDTW
0.076
4.03
5,50
5.13
SofDTW"
0.108
6.25
6.25
6.25
TAP
0.557
79.85
78.02
67.77
TAP
0.362
61.25
55.63
36.25
Comparison with alignment-based distances. We compare the self-supervised learned TAP with 8 alignment-based based sequence distances in the retrieval and classification tasks, including DTW, length-normalized DTW (IDTW), matching step-normalized DTW (nDTW), optimal transport (OT) Kolouri al. 2017), Sinkhorn (Cuturi. 2013). TCOT (Su & Hua 2019), OPW (Su & Hua 2019 and SoftDTW (Cuturi & Blondel, 2017). We follow the setting in (Su & Hual 2019) and directly compare with the results reported in Su & Hua (2019). The comparisons on the MSR Ac- tion3D and MSR Activity3D datasets are shown in Table 6 and Table
7 respectively. We observe that TAP generally outperforms other distances on the MSR Activity31 dataset, performs worse on the MSR Action3D dataset.
SoftDTW can be used as a differentiable loss on top of single-layer embedding. SoftDTW* trains the additional linear layer in self-supervised manner by minimizing the SoftDTW distances be- tween the original training sequences and their augmented ones, but the effect equivalent to ran- dom classification. Without negative sample sampling, SoftDTW* falls into the trivial solution of all zeros. self-supervised learning, TAP naturally avoids the trivial solution since it calculates D from the original sequences and performs Softmax on T.
A.2 PROBLEM OF SUPERVISED REPRESENTATION LEARNING WITH ALIGNMENT LAIGNMENT-BASED DISTANCES
For supervised learning, the objective often to make the sequence distance better separate se- quences from different classes. However, such objectives difficult to optimize for conventional alignment-base distances because they solve another optimization problem to obtain the align- ments. For example, if we apply the triplet loss to the OPW distance, the objective becomes:
14

---

Published as a conference paper at ICLR 2022
min Σ Σ max( (Tin's ((((""), -dy + m. m)
n'yy"==y"
Σ
max(df + (Ti "Tin, ((xxx"
(1I)
n' n/ $
8.1. T nn' argmin (T. (II(X")))(X")))) + RTT)
TE$
The gradients of this objective with respect to the parameters of h rely on Tin' which is determined by another optimization problem with additional constraints on T and this optimization problem depends on h in turn. Therefore, the gradients cannot be explicitly formulated and directly obtained. The overall optimization problem thus cannot optimized via back propagation in an end-to-end manner. An alternative solution is to alternatively update 7 and h by fixing the other in an iterative way, which is one-consumming and there is no theoretical guarantee of convergence. Existing meth- ods (Mei et al. [2014: Su & Wu 2020b) optimize proxy approximate objective through complex iterative solutions.
A.3 SELF-ATTENTION MODULE IN SA-TAP
In few-shot action recognition. TAP can be combined with the self-attention mechanism. Especially, the proposed TAP focuses on aligning different sequences, but cannot fully utilize the temporal dependencies among frames in the same sequence. This can be alleviated by applying a single-head self-attention (SA) module to each sequence before performing alignment by TAP. Specifically, after obtaining the feature sequence X for a video, we first obtain X, = X + P. (X), where (X is the sequence of position embeddings for elements in X. We then employ the self-attention module to transform X, into the following sequence:
W 9
X Softmar
' W.
(12)
Vd
where W,, We. and W, are three linear projections with the size of x d. The sequence X is finally transformed into:
x LN + X,)
(13)
Where LN is the layer normalization. calculating their TAP distance in Eq. We denote this method by SA-TAP.
Both the support and query sequences are enhanced before
A.4 COMPLEXITY AND MODEL SIZE
The complexities for calculating D. performing convolutions with few layers and small ker- nels, calculating S, and performing Softmax and L, normalization are O(Lx Lyd), O Lyd), O(Lx and Lv),, respectively, where the number of layers, the size of kernels, and the number of kernels are omitted since they are small. The overall complexity of TAP is Ly
TAP only contains three convolution layers and hence the model size is quite small. For supervised learning, the feature transformation network is the same for all methods. DTW, OPW, and Soft-DTW parameter-free but require solving optimization problem for inferring the optimal alignment, so they introduce no additional parameters. In the three layers of TAP, we fix the kernel sizes to 5, 5, and 3, respectively, and we fix the number of filers to 30. 30, and 1, respectively. Therefore, TAP only introduces 30 x
5 x 5 2 + 30 x 5 x 5 x 30 + 1 x 3 x 3 x 30 0.0231 M parameters.
For few-shot action recognition, the feature extraction backbone is fixed ResNet-50 for all meth- ods. OTAM employs variant of DTW as the distance measure and hence does not introduce de projections and one
x d, projection, where Ω is the length of tuple, d = 2048 is the dimension of frame-wide features and d, d, is set to 1 1152. Therefore, TRX introduces 3 x 2 x 2048 x 1152 x Q parameters, S2 often set to 2 for using frame pairs and 3 for frame tuples, the number of parameters ranges from 27M to 40.5M. Moreover, TRX requires an exhaustive traversal of pairs and triplets of frames with high complexity. In TTAN, the location network in the temporal transform module takes feature sequence input
additional parameters. TRX contains two x
15

---

Published as a conference paper at ICLR 2022
0.35
0.30
0.00
Rurning Lime
0.43
Running time
0.20
2.09
0.55
0.04
0:50
042
000
100
Length sequences
Dimensionalitt vectors sequences
(a)
(b)
Figure 3: (a) Running times of SoftDTW and TAP as functions of the length of sequences. (b)
Running times SoftDTW and TAP as functions of the dimensionality of features in sequences.
Table 8: Comparison of TAP with/without data augmentation for supervised learning.
Action3D
Activity3D MAP
Method
MAP
I-NN
1-NN
HM+TAP w/o DA
0.839
88.28
0.690
71.25
HM+TAP
0.798
87.18
0.701
74.38
Lifted+TAP w/o DA
0.820
86.81
0.723
70.00
Lifted+TAP
0.772
82.05
0.728
73.75
Binomial+TAP w/o DA
0.802
86.45
0.689
71.88
Binomial+TAP
0.826
87.91
0.678
73.13
and outputs indexes with several learnable layers, but the detailed network structure is not spec- ified; the temporal attentive module contains 3 projections and has 12M parameters; the late fusion module introduces another d projections with 12M parameters. In contrast, TAP-CNN also only introduces 64 x 5 x 5 x 2 + 64 x 5 x 5 x 64 + 1+11 x 3 x 3 x 64 = 0. 0.1013M parameters since the kernel sizes are enlarged to 64, 64, and 1, respectively. For SA-TAP, the self-attention layer contains 3 d x d projections, Therefore, SA-TAP introduces 3d² + 0. 0.1013M = 12.1013M parameters. The model sizes of both TAP and SA-TAP are smaller than TRX and TTAN.
A.5 COMPARISON OF TAP AND SOFTDTW ON RUNNING TIMES
We study the scalability of the proposed TAP in comparison with SoftDTW the length and dimen- sionality increase. We randomly generate two sequences of length L, both consists of d-dimensional features. For different L and we compute the SoftDTW distance and the TAP distance with ran- dom parameters between the two sequences, respectively, and compare the corresponding running times. We first fix d to 500, and increase from 100 to 1000 with an interval of 100. The run- ning times of SoftDWW and TAP as functions of T. are compared in Fig. 3(a) The running time of SoftDTW grows rapidly as the length increases due to the dynamic programming and backtracking processes. In contrast, since only the forward matrix operations required, the running time of TAP does not change much as the length increases. The longer the sequences, the more obvious the speed advantage of the proposed TAP.
We then fix L to 500, and increase d from 100 to 1000 with an interval of 100. The running times of SoftDTW and TAP functions of d compared in Fig. (3bb We observe that the proposed TAP consistently about 10 times faster than SoftDTW. These results demonstrate that TAP can better scale to long and high-dimensional sequences.
16

---

Published as a conference paper at ICLR 2022
Table 9: Comparison of using transformer and CNN.
Action3D
Activity3D
Method
MAP
I-NN
MAP
1-NN
HM+TAP-Trans
0.751
81.32
0.691
72.50
HM+TAP(CNN)
0.798
87.18
0.701
74.38
Lifted+TAP- Trans
0.740
77.29
0.697
69.38
AAfted+TAP(CNN)
0.772
82.05
0.728
73.75
Binomial+TAP-TII
0.746
80.22
0.705
71.25
TAPCNNN)+
0.826
87.91
0.678
73.13
A.6 EFFECT OF DATA AUGMENTATION SUPERVISED LEARNING
We apply the data augmentations, i.e., random blur and random merge, in supervised learning only for obtaining more stable results. For example, in cases, all sequences in small batch satisfy the constraints in Eq. (4) and Eq. (6). where data augmentation can enlarge the within-class distance. Comparison results of TAP with and without data augmentations on the MSR Action3D and MSR Activity3D datasets are shown in Tab. 8 In some cases, the data augmentation only brings minor improvements, while in other cases, the data augmentation even causes performance degradation, which may be because augmented sequences obtained by such temporal blur and merge do not reflect the real intra-class variances. Generally, in most cases, since the augmented sequence often has small distance from the original sequence, it will not change the selection of hard negatives and hard positives in Eq. (6). Without the proposed data augmentations, TAP still outperforms all other methods where it performs best in Tab. and all conclusions still hold.
A.7 ABLATION ON THE ARCHITECTURE OF THE ALIGNMENT PREDICTION NETWORK
The intuition behind using CNN as in the alignment prediction network f is that CNN can im- plicitly model the temporal constraints and capture local aligmment patterns. Temporal constraints for performing temporal alignment are often reflected as follows: the probability of two frames be- ing aligned is not only related to the distance between the two elements and the difference between their relative positions, but also depends on the distances and alignment probabilities between their adjacent frames. Convolution kernels aim capturing such local information. There may exist dif- ferent local alignment patterns at different levels. Multiple convolution kernels in different layers are suitable for capturing these different local patterns and variances of local motions.
A transformer-Hi like cross- cooss-attentinn- baasd architecture can also be used to predict temporal alignment. Specifically, for aligning Y to X, we use linear projection W, IR xd to transform all elements a;; of X a;; WW resulting in transformed sequence X [ 1,... Ix] W X, where 30 is the transformed dimensionality. Similarly, is transformed into Y [ 1,... Lr] Y by another projection WE. We use each aē; as the query, use új. j 1, LY as keys, and calculate all pairwise Euclidean distances between 2;, 1,... Lx and Dj.j = 1, 1,... Ly into a matrix D := [e(2,, V E R2x XLY, The similarity matrix S is calculated as: S S- -D, + >D,), where the hyper-parameter is set to 50. Similarly with TAP, we perform Softmax and L1 normalization in (3) to 5 to obtain the final alignment T. We denote this method by trPst Comparisons of TAP and TAP-trans are shown Tab. [ We observe that CNN outperforms the Transformer-like attention-based network. This may indicate that CNN indeed can capture the local alignment patterns by integrating the local contexts.
The local receptive field of CNN can be viewed as imposing constraint on the temporal deviation: only local warping within the receptive field considered since extremely unbalanced warping is not preferred. Increasing the kernel sizes or using more convolution layers gains larger receptive fields, hence larger warping patterns are allowed and long-range dependencies can be utilized. We have added comparisons with two variants. namely TAP (I) and TAP (d), to evaluate the effects of using larger kernels and using more convolution layers, respectively. In TAP (I), the kernel size in the three convolution layers is increased 7, 7, and with paddings 3, 3, and 2 to keep the spatial
17

---

Published as a conference paper at ICLR 2022
Table 10: Effects of using larger kernels and more convolutional layers in the alignment prediction network of TAP.
Action3D
ACtivity3D MAP
Method
MAP
I-NN
1-NN
HM+TAP
0.798
87.18
0.701
74.38
HM+TAP()
0.834
90.48
0.671
73.13
HM+TAP(d)
0.846
91.58
0.693
70.63
LiP
0.772
82.05
0.728
73.75
LiPted+TAP()
0,805
85.35
0.743
73.75
Lifted+TAP(L
0.779
83.52
0.743
71.25
Binomial+TAP
0.826
87.91
0.678
73.13
Binomial+TAP(I)
0.838
85,348
0.701
72.50
Binomial+TAP(d)
0.856
89.377
0.678
74.38
size, respectively. In TAP (d), another convolution layer is inserted after the second layer, where the kernel size, padding, and the number of kernels are set to 5, 2, and 30, respectively.
Comparison results are shown in Tab. TO Generally, using larger convolution kernels or adding more convolution layers to relax the restriction on warping and model more levels of alignment patterns can further increase the performance of TAP. We use the CNN architecture consisting of 3 convolution layers with small kernels to impose strong inductive bias and keep the alignment prediction network lightweight.
A.8 EVALUATION METHODS ON MSE BETWEEN PREDICTED AND GROUND-TRUTH ALIGNMENTS.
For each test sequence, we randomly generate 5 augmentations with the proposed augmentation methods (random blur and random merge) and approximately construct the ground-truth align- ment between the original sequence and its augmentations, respectively. To evaluate the predicted alignments by supervised trained TAP (TAP-s), to transform the sequence and each of its augmentations, and then predict their un-normalized alignments by the trained alignment prediction network f. For supervised learning with SoftDTW (SoftDTW-s), after transforming the sequences with the trained h. the alignments are directly in- ferred by SoftDW.. The MSE between the predicted alignments and the ground-truth alignments of all test-augmentation sequence pairs is calculated as the performance measure, Comparisons of TAP-s and SoftDTW-s by using the triplet loss are shown in the left part of Tab,
first apply the trained transformation network
We can also directly train TAP without jointly learning the transformation network h by constraining the predicted alignments. We generate augmentations for each training sequence and construct the corresponding ground-truth alignment. The prediction network f is trained by minimizing the MSE loss between the predicted un-normalized alignments by TAP and the constructed ground- truth alignments. In the testing stage, we directly apply SofDTW or the trained TAP to predict alignments between test sequences and their corresponding augmentations. Comparisons on the MSE between the ground-truth alignments and the alignments predicted by SoftDTW and TAP on the test set are shown in the right part of Tab. 4 We observe that both cases, TAP achieves lower MSEs, which may indicate that TAP can indeed learn alignment patterns from the training sequenees to better approximate the ground-trutt alignments.
A.9 EVALUATING THE LEARNED REPRESENTATIONS WITH LINEAR CLASSIFIER
In supervised representation learning. we jointly learn the frame-wise feature extraction network h and the alignment prediction network f. We further use a linear classifier to evaluate the learned representations. For each sequence, we freeze the backbone extract frame-wise features and then encode them into a vector representation by mean pooling. We then train a linear classifier on these vector representations. We use the Adam optimizer, where the syper-parameters of the optimizer remain the same as in supervised representation learning. The number of epochs is fixed to 3000. Comparisons of TAP and SoftDTW with the linear classifier shown in Tab. 11
18

---

Published as a conference paper at ICLR 2022
Table 11: Comparison of SoftDTTW and TAP with the linear classifier.
Action3D
Activity3D
Method
Top-1
Top-5
Top-1
Top-5
HM+SoftDTW
56.41
86.81
62.50
90.63
HM+TAP
67.03
90.84
63.75
91.25
Lifted+SofDDTW
5.50
24.54
6.25
31.25
Lifted+TAP
51.65
82.78
64.38
86.25
53.48
85.35
6.25
31.25
Binomial+TAP
69.23
90.11
65.63
91.25
We observe that TAP always outperforms SoftDTW with the linear classifier no matter what super- vised representation learning loss is adopted and whether SoftDTW falls into the trivial solution. This shows that the representations learned by TAP are more discriminative. The results obtained by the linear classifier are worse than the 1-NN classifier, which indicates that the mean pooling loses useful temporal information and TAP can take advantage of such information.
19


Reviews:
[{"review_id": "jD1AcE8hTkY", "reviewer": "Reviewer_1HmZ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well motivated and focuses on an important problem. The method itself is simple and makes use of previous contributions (loss functions, alignment formulation and models), however its overall combined architecture and approach is novel and can provide inspiration for future work. \n\nNevertheless, I have some questions regarding the very nature of the problem. In fact, the model is trained to learn an optimal alignment between temporal sequences, yet there is no specific evaluation of the model for this very task. The authors provide a good number of experiments for two different settings, however there is no study showing how well the model is able to predict the optimal alignment between two sequences. While this may not be a major concern per se, it seems a little confusing that the method is trained for sequence alignment but it is not evaluated for that. In fact, the alignment here is proposed as a “proxy distance” to be used for other tasks. There is nothing particularly wrong with that, but again this missing study raises some confusion. I appreciate the examples in Figure 2 showing some alignments, but I reckon a quantitative analysis is needed.\n\nThe choice of CNN as the main model is a little questionable in my opinion. By design, CNNs work on fixed-size input. In fact, when the method is evaluated on few-show action recognition, a fixed number of frames is sampled from all sequences. For supervised representation learning it is not clear whether the input distance matrices are padded to have consistent input shapes or whether a fixed number of frames were sampled here too. While using fixed representation/samples for sequence alignment is not uncommon, the main reason why a CNN was preferred was its simplicity and interpretability. The latter is often mentioned in the paper, yet there is no analysis of the presented experiments with regards to interpretability. Considering this, the argument to favour CNNs becomes weaker and I wonder how other architectures better suited for sequence modelling (e.g. RNNs or Transformers) would work within the framework. \n\nRegarding supervised representation learning, it is not clear why a simple 1-NN classifier was used. In fact, the typical approach in this case would be to train the backbone with metric learning, freeze then the backbone and train a linear classifier on the frozen distribution. \n\nFinally, regarding running time, the authors compare inference speed for several methods, showing that theirs is fastest. I believe that reporting training time would be fair too or at very least interesting: since most of the methods compared against does not require training, it would be good to know what’s the training overhead of this model.", "summary_of_the_paper": "This paper focuses on temporal sequence alignment, i.e. the task of finding an optimal alignment between sequences of different lengths. This task has been addressed by various traditional methods (e.g. DTW) that involve dynamic programming or other optimisations techniques that cannot be easily embedded in end-to-end learning frameworks. \n\nThis work proposes a method where the optimal temporal alignment is learnt by a CNN. The input is formed concatenating two matrices measuring element-wise distances between two sequences A and B. The network is trained to output a matrix whose (i, j)-th element indicates the likelihood of Ai being aligned to Bj. \n\nThe framework is evaluated with two tasks: supervised representation learning and few-shot action recognition. Tested on several datasets, the method is able to attain competitive performance and fast inference speed.", "summary_of_the_review": "The paper is well written, addresses an interesting problem and proposes a novel approach for end-to-end temporal alignment. However the evaluation of the method, while comprehensive in some ways, has some holes. This may be a concern since the very nature of the method, temporal alignment, is not evaluated.\n\n# Update after rebuttal\n\nI believe the authors did a good job with their rebuttal. All my concerns have been well addressed and the paper is know more polished. I believe the quality of this work is good enough for publication.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QbpMfAE6dZg", "reviewer": "Reviewer_tA1W", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. Using learnable sequence distance removes ordering constraints in DTW-based alignments, and this may be a better setting for actions that do not enforce strict ordering of motions.\n\n2. As shown in Table 2, the inference speed of TAP is better than previous approaches.\n\n3. The technical part is well written and easy to understand. \n\n\nWeaknesses:\n\n1. No significant improvement on few-shot action recognition performance. As shown in Table 3, TAP only outperforms prior work on UCF101 1-shot accuracy, while falls behind on other metrics. Moreover, many exisiting methods (e.g. OTAM, ARN, CMN, TTAN) benchmark on much larger action recognition datasets including Kinetics and Something-something, while there is no reported experimental results of TAP on these datasets. As those prior work of sequence distances focuses on few-shot action recognition, it is hard to convince me that TAP is superior than the prior work.\n\n2. No ablation on the data augmentation. The authors also propose two augmentation methods, random blur and random range, and have used them in their experiments. However I don't see ablation study examining the contributions of the data augmentation. I can't tell if the performance improvement is mainly from TAP or the data augmentation.\n\n3. Missing reference: Zhu and Yang, Compound Memory Networks for Few-shot Video Classification. ECCV 2018\n\n\nQuestions:\n\n1. As shown in Table 1, TAP does better in terms of MAP, but not as good as the prior work in terms of 1-NN. Could you explain why TAP typically performs better on one metric over another?\n\n2. If there's no data augmentation used, what is TAP's performance on both tasks?\n\n3. TAP has an extra neural net for predicting alignment. What's the overall model size of TAP compared to prior approaches?", "summary_of_the_paper": "This paper proposes a sequence distance metric named temporal alignment prediction (TAP) which leverages a CNN to predict alignment between two sequences, in contrast to other non-learnable alignment algorithms in prior work. The authors have benchmarked the TAP on two video tasks of supervised video representation learning and few-shot action recognition.", "summary_of_the_review": "Although the proposed way of computing distances between sequences is learnable, flexible, and light-weighted, the experiments on few-shot action recognition could not support the supremacy of TAP over existing methods. The paper also lacks necessary ablation study on the data augmentation used in TAP experiments. Overall, I think the quality of the current draft is marginally lower than the acceptance threshold.\n\n====== Post-rebuttal ======\nThe authors have addressed my concerns and questions in the rebuttal. I'm raising my rating to 6.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AJYMDCQiRYT", "reviewer": "Reviewer_UnWN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is interesting because it attempts to solve the alignment problem directly by using CNN. The problem is formulated as in Eqs. (1) and (2), and what we have to do is to find the optimal alignment matrix T*. This is usually obtained by solving an optimization problem, which prevents using backprop in an end-to-end manner.\n\nThe alignment matrix T is a rectangular matrix, so the proposed method directly estimates this matrix with CNN, whose inputs are matrix D (pairwise distance matrix of sequences X and Y) and distance matrix Dt of relative positions (larger when indices are different). These are stacked to Ds and fed to a small network g to generate g(Ds). With a residual connection, S=D+g(Ds) is further normalized to obtain an estimate of T*.\n\nThis is a quite simple idea, but sounds reasonable if we have a large amount of data. There could be a variety of potential applications.\n\n\nThere are some concerns.\nFirst, the CNN g looks to have a relatively small receptive field compared to the length of sequences. In p.4, the CNN g has three layers with kernel sizes of 5,5,3, which indicates that the size of the receptive field is about 10 (because the spatial size is kept). However the datasets used in the experiments have much longer sequence lengths, as illustrated in Figure 2. This is strange when we consider the good performance of the proposed model.\n(note: this applies only for the supervised learning, because few-shot learning uses only 8 frames)\n\nSecond, relating to the above, the CNN g might not work as expected. The residual connection D + g(Ds) works as shown in experiments, and it is not clear how much the residual g(Ds) contributes the results. If g(Ds) is small compared to D, the performance is mainly due to D, a simple distance matrix.\nSince the backbone network h is jointly trained, the good performances mainly come from the backbone, not the alignment prediction network f.\n\n\n\nminor issues:\nIt is interesting to see how the alignment prediction network correctly predicts alignments. It is possible to compare the prediction and the ground truth by solving optimization. Of course the correct prediction might not lead to higher performances, but checking the \"correctness\" of the prediction may show show some insights how the network behaves.\n", "summary_of_the_paper": "One-shot sequence alignment with CNN.\nComparisons of sequence data are important for many tasks including action recognition and retrieval. Previous approaches include generating fixed-size feature vectors (e.g., RNN) or temporally aligning two sequences (e.g., DTW).\nInstead, the proposed method directly predict the alignment matrix T between two sequences X and Y with CNN.\nThis is differentiable, and hence can be applied to many tasks such as supervised sequence classification/retrieval, and few-shot classification.\nExperiments uses a variety of sequence classification tasks, such as skeleton-based action recognition and spoken digit audio classification, and show a good performance of the proposed method.", "summary_of_the_review": "The concept is interesting, and potential applications are many.\nHowever the proposed concept has not been well studied.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper has been evaluated by three reviewers with 2 borderlines leaning towards the accept, and with 1 accept. The reviewers have noted that the idea of alignment is not particularly novel per se. Nonetheless, they found some merit in the use of a network learning the alignment and they liked experiments.

AC has however some concerns about this work. Firstly, it is not clear why Lifted+SoftDTW and Binomial+SoftDTW completely fail in Table 1, and in Table 5, SoftDTW is worse by 30% than TAP. Is soft-DTW set up properly in these experiments (the softmax temperature, the base distance used, the maximum number of steps away from the main path etc.)?

AC is also not convinced about the principled nature of the proposed alignment. Eq. 3 and the residual design above seem more as heuristics than a principled OT transport as Eq. 1 and 2 set out to suggest. With concatenation of distances between sequence features and positional encoding, the proposed alignment seems more similar to attention and transformers than OT.
Decision: Accept (Poster)