SoftLoc: Robust Temporal Localization under Label Misalignment

1 INTRODUCTION The surge of deep neural networks ( LeCun et al. , 2015 ; Schmidhuber , 2015 ) has accentuated the evergrowing need for large corpora of data ( Banko & Brill , 2001 ; Halevy et al. , 2009 ) . The main bottleneck for the efficient creation of datasets remains the annotation process . Over the years , while new labeling paradigms have emerged to alleviate this issue ( e.g. , crowdsourcing ( Deng et al. , 2009 ) or external information sources ( Abu-El-Haija et al. , 2016 ) ) , these methods have also highlighted , and emphasized , the prevalence of label noise . Deep neural networks are unfortunately not immune to these perturbations as their intrinsic ability to memorize and learn label noise ( Zhang et al. , 2017 ) can be the cause of training robustness issues and poor generalization performance . In this context , the development of models robust to label noise is essential . This work tackles the problem of precise temporal localization of events ( i.e. , determining when and which events occur ) in sequential data ( e.g . time series , video or audio sequences ) despite only having access to poorly aligned annotations for training ( see Figure 1 ) . This task is characterized by the discrepency between the precision required of the predictions during inference and the noisiness of the training labels . Indeed , while models are trained on inaccurate data , they are evaluated on their ability to predict event occurences as precisely as possible with respect to the ground-truth . In such a setting , effective models have to infer event locations more accurately than the labels they relied on for training . This requirement is particularly challenging for most classical approaches that are designed to learn localization by strictly mimicking the provided annotations . Indeed , as the training labels themselves do not accurately reflect the event location , focusing on replicating these unreliable patterns is incompatible with the overall objective of learning the actual ground-truth . These challenges highlight the need for more relaxed learning approaches that are less dependent on the exact location of labels for training . The presence of temporal noise in localization tasks is ubiquitous given the continuous nature of the perturbation , in contrast to classification noise where only a fraction of the samples are misclassified . Temporal labeling is further characterized by an inevitable trade-off between annotation precision and time investment . For instance , while a coarse manual transcription of a minute of complex piano music might be achieved within a moderate time frame , a millisecond precision requirement — a common assumption for deep learning models — significantly increases the annotation burden . In this respect , models alleviating the need for costly annotations are key for a wide and efficient deployment of deep learning models in temporal localization applications . This work introduces a novel model-agnostic loss function that relaxes the reliance of the learning process on the exact temporal location of the annotations . This softer learning approach inherently makes the model more robust to temporally misaligned labels . Contributions This work : a ) proposes a novel loss function for robust temporal localization under label misalignment , b ) presents a succinct analysis of the loss ’ properties , c ) evaluates the robustness of state-of-the-art localization models to label misalignment , and d ) demonstrates the effectiveness of the proposed approach in various experiments . 2 PROBLEM FORMULATION . The main assumption of this work is the instantaneous nature ( i.e. , lasting only one time-step in discrete time settings ) of the events of interest . ( Durations can nevertheless be modelled in such a framework by labeling the beginning and end of each event class as two separate channels . ) Thus , for each sample , the ground-truth T G : = { ( tm , cm ) | m ≤M } consists of M event occurrences each defined by its exact timestamp ( tm ∈ R≥0 ) and its class ( cm ∈ [ 1 , ... , d ] , with d event classes ) . In this work , temporal label misalignment is then modelled by adding perturbations to the ground-truth timestamps : T L : = { ( tm + m , cm ) | m ≤M } , where m iid∼ E. ( 1 ) Although commonly defined as a normal distribution N ( 0 , σ2i ) ( see experiments in Section 5.1 and 5.2 ) , the noise distribution E can also represent a wider range of perturbations ( see experiments in Sections 5.2 and 5.3 ) . The training dataset D : = { ( Xi , T Li ) : 0 < i ≤ N } is comprised of N pairs with model input ( Xi ) and misaligned labels ( T Li ) . The aim of this work is the following : Objective Estimate the true event occurrence times T G of an unseen input sequence X using only the noisy data D for training . From a practical standpoint , although not necessary for the use of our loss , time is generally discretized . In such a discrete setting , each predictor Xi of the training data D : = { ( Xi , Yi ) : 0 < i ≤ N } is an observable temporal sequence of length Ti ( i.e. , Xi = ( xi ( t ) ) Tit=1 ∈ IR Ti×λ ) such as a DNNlearned representation , a spectrogram or any other λ-dimensional time-series . The label sequence Yi = ( yi ( t ) ) Ti t=1 ∈ { 0 , 1 } Ti×d is then the discrete equivalent of T L. ( Note that this last statement assumes that only one event per class can occur at each time-step ; in cases where this assumption is violated , the use of smaller temporal granularity solves this issue . ) 3 RELATED WORKS . Temporal Localization Under Label Misalignment The literature on temporal noise robustness is limited despite the critical relevance of this issue . First , Yadati et al . ( 2018 ) propose solutions combining noisy and expert labels ; however , unlike our approach , these methods require a sizable clean subset of annotations . Second , while Adams & Marlin ( 2017 ) achieve increased robustness by augmenting simple classifiers with an explicit probabilistic model of the noise structures , the effectiveness of the approach on more complex temporal models ( e.g. , LSTM ) still needs to be demonstrated . Finally , Lea et al . ( 2017 ) perform robust temporal action segmentation by introducing an encoder-decoder architecture . However , the coarse temporal encoding comes at the expense of finer-grained temporal information , which is essential for the precise localization of short events ( e.g. , drum hits ) . In this paper , rather than a new architecture , we propose a novel and flexible loss function — agnostic to the underlying network — which allows the robust training of temporal localization networks even in the presence of extensive label misalignment . Classical Heuristic Our approach is closely linked to the more classical trick of label smoothing or target smearing ( e.g. , applying a σ̃2-Gaussian filter Φ̃σ2 to the labels ) which has been considered to increase robustness to temporal misalignment of annotations ( Schlüter & Böck , 2014 ; Hawthorne et al. , 2017 ) . This slight modification of the input data converts the original point prediction problem into a distribution prediction problem . Indeed , the smoothing of the labels transforms the point labels into distributions . The algorithm is then trained to predict these distributions , which eventually have to be transformed back to point predictions using hand-crafted peak picking heuristics ( see Figure 3 ( left ) ) . This methodology is also very common in 2D image keypoint detection applications which deal with spatial uncertainty , e.g . human pose estimation ( Tompson et al. , 2014 ; 2015 ) or facial landmark detection ( Merget et al. , 2018 ) . However , despite its intuitive nature , this traditional solution presents several inherent drawbacks ( see Figure 2 ) : ( Issue 1 ) Even in a noise-free setting , by transforming the impulse-like target into a distribution , the optimal model predictions ( with respect to the training loss ) differs from the actual goal of the pipeline ( i.e. , precise localization indicated by the original event label ) . ( Issue 2 ) As the model learns by mimicking the smoothed target throughout the learning phase , the predictions themselves will be spread out over several time-steps . Hence , additional tailored heuristics , such as peak picking ( Böck et al. , 2013 ) or complex thresholding , are required to achieve precise temporal localization . ( Issue 3 ) Even advanced peak picking approaches struggle to disentangle close events . For instance , a unique maximum might emergence in the middle of two events , thus significantly disturbing the timeliness of the final predictions . ( Issue 4 ) Having the label mass dispersed temporally both before and after the event occurrences is problematic not only for causal models ( i.e. , models that make predictions at time t only with data up to time t− 1 ) but also for one-sided recurrent networks and fully convolutional architectures with limited receptive fields . Indeed , all these models have to estimate the left tail of the label distribution before even seeing the event occur . This requirement compels the model to find some structure before the actual event occurrence , leading to poor generalization performance . Although bidirectional networks do not suffer from it , this issue limits the range of possible architectures . The presence of strong label misalignment further worsens these four issues as increased noise commonly warrants increased smoothing , dispersing the label ( and consequently prediction ) mass even more . Overall , experimental evidence ( e.g . Section 5.1.1 ) shows that the accumulation of these issues proves to be very detrimental to the noise robustness of these classical approaches . In contrast , this work presents a novel paradigm for dealing with temporal label uncertainty . The main idea consists in directly inferring point predictions rather than resorting to distributions or heatmaps by fully integrating the modelling of the noise into the loss function . Such a direct approach allows for an end-to-end learning of localization without the need for additional hand-crafted components . In addition , the systematic and standalone loss function proposed in this regard ( Section 4.2 ) not only solves all the above-mentioned issues but also scales well to extensive label misalignment . Weakly-Supervised Learning Some weakly-supervised models leverage weaker annotations to infer more fine-grained concepts . In such frameworks , noisy labels are implicitly bypassed by the use of higher-level labels — which are more invariant to perturbations . For instance , some works achieve object detection ( Fergus et al. , 2003 ; Bilen & Vedaldi , 2016 ) or temporal localization ( Kumar & Raj , 2016 ; Wang et al. , 2017 ) using only class-level annotations . However , finer-grained labels , even noisy ones , often contain some additional information that is essential for optimal performance . 4 SOFTLOC MODEL 4.1 SOFT LOCALIZATION LEARNING LOSS . The general principle of relaxing the localization learning is intuitive and potentially powerful if carefully implemented . However , by smoothing the label only , classical approaches transform the original point prediction problem into a distribution prediction problem which eventually causes issues ( see Section 3 ) . Many of the drawbacks arising from the asymmetric nature of the one-sided smoothing can however be alleviated by filtering not only the labels ( i.e. , ΦSi2 ∗Yi ( · ) ) , but also the predictions ( i.e. , ΦSi2 ∗ Ŷi ( · ) ) with a unique softness parameter Si . The comparison of these two smoothed processes yields a relaxed loss function for the soft learning of the location that deals on its own with the temporal uncertainty of the labels . Indeed , in such a setting , the model is given input sequences of point-like events and directly infers point predictions without having to resort to distributions or heatmaps ( see Figure 3 ( right ) ) ; it is only the loss function that views these point labels and predictions as smoothed processes . In discrete time settings , the loss can be written as LSLL ( θ ) = ∑ i L ( ΦSi2 ∗ ŷi , θ ( · ) , ΦSi2 ∗ yi ( · ) ) , with ΦSi2 a Si 2-Gaussian filter , ( 2 ) where L ( · , · ) can be any measure of distance . The learning is characterized as soft since the loss is not strictly constraining in terms of precision or mass concentration . Indeed , the mass of each event can be both scattered over numerous time-steps and slightly shifted temporally without any abrupt increase in loss . Thus , the model ’ s reliance on exact label locations is relaxed . Measure L In noise-free settings , the average stepwise cross-entropy is a common choice of loss function for state-of-the-art models ( Lea et al. , 2017 ; Wu et al. , 2018 ; Hawthorne et al. , 2019 ) . While a potentially unbounded penalization of false predictions might be ideal when training on clean datasets , such behavior can be highly detrimental when labels are subject to temporal misalignment . Therefore , for all experiments in Section 5 , L is set to the ( bounded ) average local mean-squared error . Properties Symmetrically smoothing both the labels and predictions solves several of the issues highlighted in the previous section ( see Figure 4 ) . First , in a noise-free setting , the optimal predictions with respect to LSLL are the original annotations themselves . ( Solves 1 ) . Second , since the predictions are also smoothed over time , each trigger adds detection mass not only after , but also before the prediction time . Therefore , the model is not required the estimate the left-tail of the label distribution before the actual event occurrence ( Solves 4 ) . The prediction mass for a particular event is not necessarily dispersed over time anymore . For instance , in noise-free settings , the point-like targets themselves are the solution to the optimization problem . However , LSLL does not strictly constrain the mass of each event to be contained in a single time-step ( Partially Solves 2 & 3 ) . 4.2 SOFTLOC LOSS The potential dispersion of the prediction mass and its direct consequences on localization performance still need to be addressed . To that end , we propose to leverage the properties of the weaklysupervised model defined in ( Schroeter et al. , 2019 ) , which achieves precise temporal localization using only occurrence counts for training . Aside from exhibiting strong localization performance , the loss introduced in that work possesses an implicit mass convergence property , which concentrates the scattered prediction mass toward well-defined single points in time : LMC ( θ ) = − ∑ i log ∑ A∈F ∏ l∈A ŷi , θ ( l ) ∏ j∈Ac ( 1− ŷi , θ ( j ) ) , ( 3 ) where F is the set of all subsets of { 1 , 2 , ... , Ti } of size ∑ k yi ( k ) . Full SoftLoc Model Incorporating this mass convergence loss as a regularizer to our soft localization learning loss LSLL allows the model to directly achieve precise impulse-like localization , without weakening its noise robustness properties . Thus , this eliminates prediction ambiguity , as only a single point prediction is outputted per event occurrence ( Solves 2 & 3 ) . Overall , when trained with the SoftLoc loss , LSoftLoc ( θ ) = ( 1− ατ ) LSLL ( θ ) + ατLMC ( θ ) , ( 4 ) the model simultaneously softly learns to mimic the localization annotation , while converging the scatter mass toward impulse-like predictions . In this equation , ατ regulates the predominance of the mass convergence against the soft learning ( for training iteration τ ) . From a practical standpoint , starting with a moderate ατ allows an initial relaxed localization learning , before performing stronger mass convergence ( see Section 5 for the specific settings used in this paper ) . End-to-end Learning of Localization One of the key factors of the predominance of the deep learning models over classical ones relies on their ability to solve problems in an end-to-end fashion ( Collobert et al. , 2011 ; Krizhevsky et al. , 2012 ) , without the need to resort to partial optimization or hand-crafted heuristics . In contrast to more classical approach ( see Section 3 ) , our proposed method is an end-to-end solution to the problem of temporal localization in the presence of misaligned labels ( see Issue 2 ) . This solution eliminates the need for hand-crafted components ( e.g . peak picking ) and is expected to better serve the task at hand . Continuous Setting While all experiments in Section 5 and most state-of-the-art temporal localization models perform a discretization of time , the loss definition can easily be adapted to suit continuous-time frameworks .

Summary: This paper deals with temporal label noise, or label misalignment problem in localization, which is the long-standing problem of robust event localization regarding the temporally misaligned labels in sequential data.  Some existing works are constrained to hardly fit the annotations of data during training. The work models the temporal label misalignment with perturbations to the ground-truth timestamps, and presents a soft localization learning scheme which relaxes the reliance on the exact position of labels. The idea introduced to temporal data is well-motivated since the label for temporal data is often dispersed. Technically speaking, the proposed SoftLoc loss comprises two terms: 1). $\mathscr{L}_{SLL}$: a soft learning loss that relaxes the prediction mass concentration, by symmetrically filtering the labels and predictions. This term helps to relax the model’s reliance on exact label locations. 2). $\mathscr{L}_{MC}$: a mass convergence loss that acts as a regularizer to facilitate the model with precise impulse-like localizations. With a trade of factor to balance two terms, the SoftLoc model can achieve precise impulse-like localization performance without weakening the model robustness. Various applications, such as PIANO ONSET, DRUM DETECTION and TIME SERIES DETECTION are performed to verify the effectiveness of the proposed method. And state-of-the-art performance is achieved.