Drop-DTW: Aligning Common Signal Between Sequences While Dropping Outliers

1 Introduction . The problem of sequence-to-sequence alignment is central to many computational applications . Aligning two sequences ( e.g. , temporal signals ) entails computing the optimal pairwise correspondence between the sequence elements while preserving their match orderings . For example , video [ 1 ] or audio [ 2 ] synchronization are important applications of sequence alignment in the same modality , while the alignment of video to audio [ 3 ] represents a cross-modal task . Dynamic Time Warping ( DTW ) [ 2 ] is a standard algorithm for recovering the optimal alignment between two variable length sequences . It efficiently solves the alignment problem by finding all correspondences between the two sequences , while being robust to temporal variations in the execution rate and shifts . A major issue with DTW is that it enforces correspondences between all elements of both sequences and thus can not properly handle sequences containing outliers . That is , given sequences with interspersed outliers , DTW enforces matches between the outliers and clean signal , which is prohibitive in many applications . A real-world example of sequences containing outliers is instructional videos . These are long , untrimmed videos depicting a person performing a given activity ( e.g. , making a latte ) following a pre-defined set of ordered steps ( e.g. , a recipe ) . Typically , only a few frames in the video correspond to the instruction steps , while the rest of the video frames are unrelated to the main activity ( e.g. , the person talking ) ; see Figure 1 for an illustration . In this case , matching the outlier frames to instruction steps will not yield a meaningful alignment . Moreover , the match score 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . between such sequences , computed by DTW , will be negatively impacted by “ false ” correspondences and therefore can not be used reliably for downstream tasks , e.g. , retrieval or representation learning . In this paper , we introduce Drop-DTW to address the problem of matching sequences that contain interspersed outliers as illustrated in Figure 1 ( right ) and compared to standard DTW ( on the left ) . While various improvements to DTW have been previously proposed ( e.g. , [ 1 , 4–8 ] ) , Drop-DTW is the first to augment DTW with the ability to flexibly skip through irrelevant parts of the signal during alignment , while still allowing one-to-many matching . Rather than relying on a two-step greedy approach , where elements are first dropped before aligning the remaining signal , Drop-DTW achieves this in a unified framework that solves for the optimal temporal alignment while jointly detecting outliers . Drop-DTW casts sequence alignment as an optimization problem with a novel component specifying the cost of dropping an element within the optimization process . It is efficiently realized using a dynamic program that naturally admits a differentiable approximation and can be efficiently used at training and inference time . Contributions . • We propose an extension of DTW that is able to identify and align the common signal between sequences , while simultaneously excluding interspersed outliers from the alignment . • The proposed Drop-DTW formulation naturally admits a differentiable approximation and we therefore demonstrate its usage as a training loss function . • We demonstrate the utility of Drop-DTW , for both training and inference for multi-step localization in instructional videos , using only ordered steps as weak supervision . We achieve state-of-the-art results on the CrossTask [ 9 ] dataset and are the first to tackle the COIN [ 10 ] and YouCook2 [ 11 ] datasets , given only ordered steps ( i.e. , no framewise labels are used ) . • We employ Drop-DTW as a loss function for weakly-supervised video representation learning on the PennAction dataset [ 12 ] , modified to have interspersed outliers , and unsupervised audio-visual representation learning on the AVE dataset [ 13 ] . Compared to the baselines , Drop-DTW yields superior representations , as measured by its performance on various downstream tasks . Our code is available at : https : //github.com/SamsungLabs/Drop-DTW . 2 Related work . Sequence alignment . The use of aligned sequences in learning has recently seen a growing interest across various tasks [ 14 , 4 , 5 , 15 , 1 , 6 , 16 , 17 ] . While some methods start from aligned paired signals [ 14 , 18 ] , others directly learn to align unsynchronized signals [ 15 , 5 , 1 , 6 ] . One approach tackles the alignment problem locally by maximizing the number of one-to-one correspondences using a soft nearest neighbors method [ 15 ] . More closely related are methods [ 4 , 5 , 1 , 16 ] that seek a global alignment between sequences by relying on Dynamic Time Warping ( DTW ) [ 2 ] . To handle noise in the feature space some methods use Canonical Correlation Analysis ( CCA ) with standard DTW [ 19 , 20 ] . To use DTW for end-to-end learning , differentiable approximations of the discrete operations ( i.e. , the min operator ) in DTW have been explored [ 4 , 1 ] . One of the main downsides of standard DTW-based approaches is that they require clean , tightly cropped , data with matching endpoints . In contrast to such methods , Drop-DTW extends DTW and its differentiable variants with the ability to handle outliers in the sequences by allowing the alignment process to automatically skip outliers ( i.e. , match signal while dropping outliers ) . While previous work [ 21 , 6 , 22 ] targeted rejecting outliers limited to the start or end of the sequences , our proposed Drop-DTW is designed to handle outliers interspersed in the sequence , including at the endpoints . Other work proposed limited approaches to handle such outliers either using a greedy two-step approach [ 7 ] , consisting of outlier rejection followed by alignment , or by restricting the space of possible alignments [ 8 , 23 ] ( e.g. , only allowing one-to-one matches and individual drops ) . In contrast , Drop-DTW solves for the optimal temporal alignment while simultaneously detecting outliers and allowing for one-to-many matches . As shown in our experiments , the ability of Drop-DTW to flexibly skip outliers during alignment can in turn be used at inference time to localize the start and end times of inlier subsequences . Representation learning . Self- and weakly-supervised approaches are increasingly popular for representation learning as they eschew the need of dense labels for training . Such approaches are even more appealing for video-based tasks that typically require inordinately large labeling efforts . A variety of ( label-free ) proxy tasks have emerged for the goal of video representation learning [ 24–41 ] . More closely related are methods using sequence alignment as a proxy task [ 5 , 15 , 1 , 6 , 16 , 17 ] . Our approach has the added advantage of handling interspersed outliers during alignment . Also related are methods that leverage multimodal aspects of video , e.g. , video-audio [ 42 , 43 ] or video-language alignment [ 44–47 ] . Our step localization application builds on strong representations learned via the task of aligning vision and language using narrated instructional videos [ 45 ] . However , we augment these representations with a more global and robust approach to the alignment process , thereby directly enabling fine-grained applications , such as multi-step localization . Step localization . Temporal step localization consists of determining the start and end times of one or more activities present in a video . These methods can be broadly categorized into two classes based on the level of training supervision involved . Fully supervised approaches ( e.g. , [ 48 , 49 , 10 ] ) rely on fine-grained temporal labels indicating the start and end times of activities . Weakly supervised approaches eschew the need for framewise labels . Instead , they use a video-level label corresponding to the activity category [ 50 , 51 ] , or sparse time stamps that provide the approximate ( temporal ) locations of the activities . More closely related are methods that rely on the order of instruction steps in a clip to yield framewise step localization [ 52–54 , 5 , 9 ] . In contrast , we do not rely on categorical step labels from a fixed set of categories and demonstrate that our approach applies to any ordered sequence of embedded steps , such as embedded descriptions in natural language . 3 Technical approach . 3.1 Preliminaries : Dynamic time warping . Dynamic Time Warping ( DTW ) [ 2 ] computes the optimal alignment between two sequences subject to certain constraints . Let X = [ x1 , . . . , xN ] > ∈ RN×d and Z = [ z1 , . . . , zK ] > ∈ RK×d be the input sequences , where N , K are the respective sequence lengths and d is the dimensionality of each element . The valid alignments between sequences are defined as a binary matrix , M ∈ { 0 , 1 } K×N , of the pairwise correspondences , where Mi , j = 1 if zi is matched to xj , and 0 otherwise . Note that Z corresponds to the first ( row ) index of M and X the second ( column ) . Matching an element zi to element xj has a cost Ci , j which is typically a measure of dissimilarity between the elements . DTW yields the optimal alignment M∗ between sequences Z and X that minimizes the overall matching cost : M∗ = arg min M∈M 〈M , C〉 = arg min M∈M ∑ i , j Mi , jCi , j , ( 1 ) where 〈M , C〉 is the Frobenius inner product andM is the set of all feasible alignments that satisfy the following constraints : monotonicity , continuity , and matching endpoints ( M1,1 = MK , N = 1 , ∀M ∈ M ) . Fig . 2 ( a ) provides an illustration of feasible and optimal alignments . The cost of aligning two sequences with DTW is defined as the cost of the optimal matching : c∗ = 〈M∗ , C〉 . DTW proposes an efficient dynamic programming algorithm to find the solution to ( 1 ) . In the case of sequences containing outliers , the matching endpoints and continuity constraints are too restrictive leading to unmeaningful alignments . Thus , we introduce Drop-DTW that admits more flexibility in the matching process . 3.2 Sequence alignment with Drop-DTW . Drop-DTW extends consideration of feasible alignments , M , to those adhering to the monotonicity constraint only . Consequently , unlike DTW , elements can be dropped from the alignment process . Before introducing Drop-DTW , let us discuss a naive solution to the problem of outlier filtering in DTW . One could imagine a greedy two-step approach where one first ( i ) drops the outliers , and then ( ii ) aligns remaining elements with standard DTW . Since step ( i ) ( i.e. , dropping ) is performed independently from step ( ii ) ( i.e. , alignment ) , this approach yields a sub-optimal solution . Critically , if an important element is erroneously dropped in step ( i ) , it is impossible to recover it in step ( ii ) . Moreover , the outlier rejection step is order agnostic and results in drops broadly scattered over the entire sequence , which makes this approach inapplicable for precise step localization . To avoid such issues , it is critical to jointly address ( i ) outlier detection and ( ii ) sequence alignment . For this purpose , we propose Drop-DTW , a unified framework that solves for the optimal temporal alignment while jointly enabling element dropping by adding another dimension to the dynamic programming table . Specifically , dropping an element xj means there is no element in Z that has been matched to xj . This is captured in the j-th column of the correspondence matrix M containing only zeros , i.e. , M : ,j = 0 . To account for unmatched elements in the alignment objective , we extend the set of costs beyond pairwise matching , Ci , j , used in DTW , with novel drop costs dzi ∈ RK and dxj ∈ RN , for elements zi ∈ Z and xj ∈ X , respectively . The optimal matching can be then defined as follows : M∗ = arg min M∈M̄ 〈M , C〉+ Pz ( M ) · ( dzi ) + Px ( M ) · ( dxj ) , ( 2 ) where · denotes the inner product , M̄ is the set of binary matrices satisfying just the monotonicity constraint , and Px ( M ) is a vector with the j-th element equal to one if the M : ,j = 0 and zero otherwise ; Pz ( M ) is defined similarly , but on rows . For clarity , in Algorithm 1 , we describe the dynamic program for efficiently computing the optimal alignment and its cost with dropping elements limited to X , i.e. , dzi =∞ . Our general Drop-DTW algorithm that drops elements from both sequences , X and Z , is given in the supplemental .

Summary: I really enjoyed reading this paper. It was very well written and positioned against existing works, and the technical aspects explained clearly. I have a few concerns but overall it would make for an excellent inclusion into the NeurIPS program.  The paper addresses the problem of aligning two contiguous temporal sequences, e.g., an untrimmed video and a given activity to match. Unlike previous approaches that make use of dynamic time warping to align the sequences, this paper proposes to model and remove outliers from the matching process. This could be done as a two-stage process: first find and remove outlier elements and then, second, apply dynamic time warping to the remaining sub-sequences. Rather than doing this the paper formulates a joint objective that aligns inliers (enforcing the monotonicity constraint) while allowing outliers to not be matched (i.e., not incur a mismatch penalty).  The approach is differentiable and is used as the loss function of a deep learning model. Experiments show that the method outperforms previous approaches on some standard tasks. In particular it does better than the two-stage baseline of first removing outliers and then matching as two disjoint steps.