Drop-DTW: Aligning Common Signal Between Sequences While Dropping Outliers

1 Introduction . The problem of sequence-to-sequence alignment is central to many computational applications . Aligning two sequences ( e.g. , temporal signals ) entails computing the optimal pairwise correspondence between the sequence elements while preserving their match orderings . For example , video [ 1 ] or audio [ 2 ] synchronization are important applications of sequence alignment in the same modality , while the alignment of video to audio [ 3 ] represents a cross-modal task . Dynamic Time Warping ( DTW ) [ 2 ] is a standard algorithm for recovering the optimal alignment between two variable length sequences . It efficiently solves the alignment problem by finding all correspondences between the two sequences , while being robust to temporal variations in the execution rate and shifts . A major issue with DTW is that it enforces correspondences between all elements of both sequences and thus can not properly handle sequences containing outliers . That is , given sequences with interspersed outliers , DTW enforces matches between the outliers and clean signal , which is prohibitive in many applications . A real-world example of sequences containing outliers is instructional videos . These are long , untrimmed videos depicting a person performing a given activity ( e.g. , making a latte ) following a pre-defined set of ordered steps ( e.g. , a recipe ) . Typically , only a few frames in the video correspond to the instruction steps , while the rest of the video frames are unrelated to the main activity ( e.g. , the person talking ) ; see Figure 1 for an illustration . In this case , matching the outlier frames to instruction steps will not yield a meaningful alignment . Moreover , the match score 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . between such sequences , computed by DTW , will be negatively impacted by “ false ” correspondences and therefore can not be used reliably for downstream tasks , e.g. , retrieval or representation learning . In this paper , we introduce Drop-DTW to address the problem of matching sequences that contain interspersed outliers as illustrated in Figure 1 ( right ) and compared to standard DTW ( on the left ) . While various improvements to DTW have been previously proposed ( e.g. , [ 1 , 4–8 ] ) , Drop-DTW is the first to augment DTW with the ability to flexibly skip through irrelevant parts of the signal during alignment , while still allowing one-to-many matching . Rather than relying on a two-step greedy approach , where elements are first dropped before aligning the remaining signal , Drop-DTW achieves this in a unified framework that solves for the optimal temporal alignment while jointly detecting outliers . Drop-DTW casts sequence alignment as an optimization problem with a novel component specifying the cost of dropping an element within the optimization process . It is efficiently realized using a dynamic program that naturally admits a differentiable approximation and can be efficiently used at training and inference time . Contributions . • We propose an extension of DTW that is able to identify and align the common signal between sequences , while simultaneously excluding interspersed outliers from the alignment . • The proposed Drop-DTW formulation naturally admits a differentiable approximation and we therefore demonstrate its usage as a training loss function . • We demonstrate the utility of Drop-DTW , for both training and inference for multi-step localization in instructional videos , using only ordered steps as weak supervision . We achieve state-of-the-art results on the CrossTask [ 9 ] dataset and are the first to tackle the COIN [ 10 ] and YouCook2 [ 11 ] datasets , given only ordered steps ( i.e. , no framewise labels are used ) . • We employ Drop-DTW as a loss function for weakly-supervised video representation learning on the PennAction dataset [ 12 ] , modified to have interspersed outliers , and unsupervised audio-visual representation learning on the AVE dataset [ 13 ] . Compared to the baselines , Drop-DTW yields superior representations , as measured by its performance on various downstream tasks . Our code is available at : https : //github.com/SamsungLabs/Drop-DTW . 2 Related work . Sequence alignment . The use of aligned sequences in learning has recently seen a growing interest across various tasks [ 14 , 4 , 5 , 15 , 1 , 6 , 16 , 17 ] . While some methods start from aligned paired signals [ 14 , 18 ] , others directly learn to align unsynchronized signals [ 15 , 5 , 1 , 6 ] . One approach tackles the alignment problem locally by maximizing the number of one-to-one correspondences using a soft nearest neighbors method [ 15 ] . More closely related are methods [ 4 , 5 , 1 , 16 ] that seek a global alignment between sequences by relying on Dynamic Time Warping ( DTW ) [ 2 ] . To handle noise in the feature space some methods use Canonical Correlation Analysis ( CCA ) with standard DTW [ 19 , 20 ] . To use DTW for end-to-end learning , differentiable approximations of the discrete operations ( i.e. , the min operator ) in DTW have been explored [ 4 , 1 ] . One of the main downsides of standard DTW-based approaches is that they require clean , tightly cropped , data with matching endpoints . In contrast to such methods , Drop-DTW extends DTW and its differentiable variants with the ability to handle outliers in the sequences by allowing the alignment process to automatically skip outliers ( i.e. , match signal while dropping outliers ) . While previous work [ 21 , 6 , 22 ] targeted rejecting outliers limited to the start or end of the sequences , our proposed Drop-DTW is designed to handle outliers interspersed in the sequence , including at the endpoints . Other work proposed limited approaches to handle such outliers either using a greedy two-step approach [ 7 ] , consisting of outlier rejection followed by alignment , or by restricting the space of possible alignments [ 8 , 23 ] ( e.g. , only allowing one-to-one matches and individual drops ) . In contrast , Drop-DTW solves for the optimal temporal alignment while simultaneously detecting outliers and allowing for one-to-many matches . As shown in our experiments , the ability of Drop-DTW to flexibly skip outliers during alignment can in turn be used at inference time to localize the start and end times of inlier subsequences . Representation learning . Self- and weakly-supervised approaches are increasingly popular for representation learning as they eschew the need of dense labels for training . Such approaches are even more appealing for video-based tasks that typically require inordinately large labeling efforts . A variety of ( label-free ) proxy tasks have emerged for the goal of video representation learning [ 24–41 ] . More closely related are methods using sequence alignment as a proxy task [ 5 , 15 , 1 , 6 , 16 , 17 ] . Our approach has the added advantage of handling interspersed outliers during alignment . Also related are methods that leverage multimodal aspects of video , e.g. , video-audio [ 42 , 43 ] or video-language alignment [ 44–47 ] . Our step localization application builds on strong representations learned via the task of aligning vision and language using narrated instructional videos [ 45 ] . However , we augment these representations with a more global and robust approach to the alignment process , thereby directly enabling fine-grained applications , such as multi-step localization . Step localization . Temporal step localization consists of determining the start and end times of one or more activities present in a video . These methods can be broadly categorized into two classes based on the level of training supervision involved . Fully supervised approaches ( e.g. , [ 48 , 49 , 10 ] ) rely on fine-grained temporal labels indicating the start and end times of activities . Weakly supervised approaches eschew the need for framewise labels . Instead , they use a video-level label corresponding to the activity category [ 50 , 51 ] , or sparse time stamps that provide the approximate ( temporal ) locations of the activities . More closely related are methods that rely on the order of instruction steps in a clip to yield framewise step localization [ 52–54 , 5 , 9 ] . In contrast , we do not rely on categorical step labels from a fixed set of categories and demonstrate that our approach applies to any ordered sequence of embedded steps , such as embedded descriptions in natural language . 3 Technical approach . 3.1 Preliminaries : Dynamic time warping . Dynamic Time Warping ( DTW ) [ 2 ] computes the optimal alignment between two sequences subject to certain constraints . Let X = [ x1 , . . . , xN ] > ∈ RN×d and Z = [ z1 , . . . , zK ] > ∈ RK×d be the input sequences , where N , K are the respective sequence lengths and d is the dimensionality of each element . The valid alignments between sequences are defined as a binary matrix , M ∈ { 0 , 1 } K×N , of the pairwise correspondences , where Mi , j = 1 if zi is matched to xj , and 0 otherwise . Note that Z corresponds to the first ( row ) index of M and X the second ( column ) . Matching an element zi to element xj has a cost Ci , j which is typically a measure of dissimilarity between the elements . DTW yields the optimal alignment M∗ between sequences Z and X that minimizes the overall matching cost : M∗ = arg min M∈M 〈M , C〉 = arg min M∈M ∑ i , j Mi , jCi , j , ( 1 ) where 〈M , C〉 is the Frobenius inner product andM is the set of all feasible alignments that satisfy the following constraints : monotonicity , continuity , and matching endpoints ( M1,1 = MK , N = 1 , ∀M ∈ M ) . Fig . 2 ( a ) provides an illustration of feasible and optimal alignments . The cost of aligning two sequences with DTW is defined as the cost of the optimal matching : c∗ = 〈M∗ , C〉 . DTW proposes an efficient dynamic programming algorithm to find the solution to ( 1 ) . In the case of sequences containing outliers , the matching endpoints and continuity constraints are too restrictive leading to unmeaningful alignments . Thus , we introduce Drop-DTW that admits more flexibility in the matching process . 3.2 Sequence alignment with Drop-DTW . Drop-DTW extends consideration of feasible alignments , M , to those adhering to the monotonicity constraint only . Consequently , unlike DTW , elements can be dropped from the alignment process . Before introducing Drop-DTW , let us discuss a naive solution to the problem of outlier filtering in DTW . One could imagine a greedy two-step approach where one first ( i ) drops the outliers , and then ( ii ) aligns remaining elements with standard DTW . Since step ( i ) ( i.e. , dropping ) is performed independently from step ( ii ) ( i.e. , alignment ) , this approach yields a sub-optimal solution . Critically , if an important element is erroneously dropped in step ( i ) , it is impossible to recover it in step ( ii ) . Moreover , the outlier rejection step is order agnostic and results in drops broadly scattered over the entire sequence , which makes this approach inapplicable for precise step localization . To avoid such issues , it is critical to jointly address ( i ) outlier detection and ( ii ) sequence alignment . For this purpose , we propose Drop-DTW , a unified framework that solves for the optimal temporal alignment while jointly enabling element dropping by adding another dimension to the dynamic programming table . Specifically , dropping an element xj means there is no element in Z that has been matched to xj . This is captured in the j-th column of the correspondence matrix M containing only zeros , i.e. , M : ,j = 0 . To account for unmatched elements in the alignment objective , we extend the set of costs beyond pairwise matching , Ci , j , used in DTW , with novel drop costs dzi ∈ RK and dxj ∈ RN , for elements zi ∈ Z and xj ∈ X , respectively . The optimal matching can be then defined as follows : M∗ = arg min M∈M̄ 〈M , C〉+ Pz ( M ) · ( dzi ) + Px ( M ) · ( dxj ) , ( 2 ) where · denotes the inner product , M̄ is the set of binary matrices satisfying just the monotonicity constraint , and Px ( M ) is a vector with the j-th element equal to one if the M : ,j = 0 and zero otherwise ; Pz ( M ) is defined similarly , but on rows . For clarity , in Algorithm 1 , we describe the dynamic program for efficiently computing the optimal alignment and its cost with dropping elements limited to X , i.e. , dzi =∞ . Our general Drop-DTW algorithm that drops elements from both sequences , X and Z , is given in the supplemental .

Summary: This paper proposed an algorithm named Drop-DTW, which enables DTW to drop outliers from the sequence-to-sequence matching. A differentiable Drop-DTW is trained for temporal step localization tasks, representation learning, audio-visual retrieval and localization. Drop-DTW gets good performance on these tasks.