SoftLoc: Robust Temporal Localization under Label Misalignment
Authors: Julien Schroeter
Kirill Sidorov
Dave Marshall
Venue:  (2020)
Keywords: deep learning
temporal localization
robustness
label misalignment
music
time series

Abstract: This work addresses the long-standing problem of robust event localization in the presence of temporally of misaligned labels in the training data. We propose a novel versatile loss function that generalizes a number of training regimes from standard fully-supervised cross-entropy to count-based weakly-supervised learning. Unlike classical models which are constrained to strictly fit the annotations during training, our soft localization learning approach relaxes the reliance on the exact position of labels instead. Training with this new loss function exhibits strong robustness to temporal misalignment of labels, thus alleviating the burden of precise annotation of temporal sequences. We demonstrate state-of-the-art performance against standard benchmarks in a number of challenging experiments and further show that robustness to label noise is not achieved at the expense of raw performance. 

Under review as a conference paper at ICLR 2020
SOFTLOC: ROBUST TEMPORAL LOCALIZATION UNDER LABEL MISALIGNMENT
Anonymous authors Paper under double-blind review
ABSTRACT
This work addresses the long-standing problem of robust event localization in the presence of temporally misaligned labels in the training data. We propose novel versatile loss function that generalizes a number of training regimes from standard fully-supervised cross-entropy to count-based weakly-supervise learning. Unlike classical models which are constrained to strictly fit the annotations during training, our soft localization learning approach relaxes the reliance on the exact position of labels instead. Training with this new loss function exhibits strong robustness to temporal misalignment of labels, thus alleviating the burden of precise annotation of temporal sequences. We demonstrate state-of-the-ar the performance against stan- dard benchmarks in number of challenging experiments and further show that robustness to label noise is not achieved at the expense of raw performance.
INTRODUCTION
The surge of deep neural networks (LeCun et al. 2015 Schmidhuber 2015) has accentuated the evergrowing need for large corpora of data (Banko & Brill 2001 Halevy et al. 2009). The main bottleneck for the efficient creation of datasets remains the annotation process. Over the years, while new labeling paradigms have emerged to alleviate this issue (e.g., crowdsourcing (Deng et al. 2009) or external informa- tion sources (Abu-El-Haja al. 2016)) these methods have also highlighted, and emphasized, the prevalence of label noise. Deep neural networks are unfortunately not immune to these perturbations as their intrinsic ability to memorize and learn label noise (Zhang al., (017) can be the cause of training robustness issues and poor generalization perfor- mance. In this context, the development of models robust to label noise is essential.
Ground Truth
Ground
Training
Truth
Data
Labels
Noise
Noise
Ground
Inference
Trush
Data
This work tackles the problem of precise temporal localiza- tion of events (i.e., determining when and which events occur) in sequential data (e.g. time series, video or audio sequences) despite only having access to poorly aligned annotations for training (see Figure []] This task is characterized by the dis- crepency between the precision required of the predictions during inference and the noisiness of the training labels. In- deed, while models are trained on inaccurate data, they are evaluated on their ability to predict event occurences as pre- cisely as possible with respect the ground-trutie setting, effective models have to infer event locations more accurately than the labels they relied on for training. This requirement is particularly challenging for most classical approaches that are designed to learn localization by strictly mimicking the provided annotations. Indeed, as the training labels themselves do not accurately reflect the event location, focusing on replicating these unreliable patterns is incompatible with the overall objective of learning the actual ground-truth truth. These challenges highlight the need for more relaxed learning approaches that are less dependent on the exact location of labels for training.
Predictions
Figure 1: Temporal localization der label misalignment. Models are trained with noisy labels that differ from the actual ground-truth, while the final inference objective the pre- cise localization of events.
such

---

Under review as a conference paper at ICLR 2020
The presence of temporal noise in localization tasks is ubiquitous given the continuous nature of the perturbation, in contrast to classification noise where only a fraction of the samples are misclassified. Temporal labeling is further characterized by inevitable trade-off between annotation precision and time investment. For instance, while a coarse manual transcription of a minute of complex piano music might be achieved within moderate time frame, millisecond precision requirement common assumption for deep learning models - significantly increases the annotation burden. In this respect, models alleviating the need for costly annotations are key for a wide and efficient deployment of deep learning models in temporal localization applications.
This work introduces novel model-agnostic loss function that relaxes the reliance of the learning process on the exact temporal location of the annotations. This softer learning approach inherently makes the model more robust temporally misaligned labels.
Contributions This work: a) proposes a novel loss function for robust temporal localization under label misalignment, b) presents a succinct analysis of the loss' properties, c) evaluates the robustness of oftte-of-thh-ar localization models to label misalignment, and demonstrates the effectiveness of the proposed approach in various experiments.
PROBLEM FORMULATION
The main assumption of this work is the instantaneous nature (i.e., lasting only one time-step in discrete time settings) of the events of interest. (Durations can nevertheless be modelled in such a framework by labeling the beginning and end of each event class two separate channels.) Thus, for each sample, the ground-trute truth 7 := {(Im. 6m) m S M) consists of M event occurrences each defined by its exact timestamp (4 e Rzo) and its class (cm [L....,d] d]. with d event classes). this work, temporal label misalignment is then modelled by adding perturbations to the ground-truth timestamps:
The == {(tm + Cm, Cm) m S M), where 6y did E.
Although commonly defined as a normal distribution (0, of) (see experiments in Section 5.I and 5.2), the noise distribution F. can also represent wider range of perturbations (see experiments in Sections and and The training dataset D == { (X,, T : < i S N) is comprised of N pairs with model input (X.) and misaligned labels (T The aim of this work is the following:
Objective Estimate the true event occurrence times T of an unseen input sequence X using only the noisy data D for training.
From a practical standpoint, although not necessary for the use of our loss, time is generally dis- cretized. In such a discrete setting, each predictor X, of the training data D (X,, Y.) : < <is N} is an observable temporal sequence of length T; (i.c.. X; = (x) e IR T such as a DNN- learned representation, a spectrogram or any other X-dimensional time-series. The label sequence w
Y, (t)) E {0, 1) is then the discrete equivalent of TL. (Note that this last statement assumes that only one event per class can occur each time-step; in cases where this assumption is violated, the use of smaller temporal granularity solves this issue.)
RELATED WORKS
Temporal Localization Under Label Misalignment The literature on temporal noise robustness is limited despite the critical relevance of this issue. First, Yadati et al.) (2018) propose solutions combining noisy and expert labels; however, unlike our approach, these methods require a sizable clean subset of annotations. Second, while AAdms & Marlin (2017) achieve increased robustness by augmenting simple classifiers with an explicit probabilistic model of the noise structures, the effectiveness of the approach on more complex temporal models (e.g., LSTM) still needs to be demonstrated. Finally, Lea et al. (2017) perform robust temporal action segmentation by introducing an encoder-decodee architecture. However, the coarse temporal encoding comes the expense of finer- -rrinne temporal information, which is essential for the precise localization of short events (e.g.. drum hits). In this paper, rather than new architecture, propose novel and flexible loss function
2

---

Under review as a conference paper at ICLR 2020
?
lesuue
Issuc
Prediction
Oplimad Prediction
Prediction
Groad
Trcch
Figure 2: Inherent drawbacks of the classical trick of smoothing the labels only. Issue 2: ambiguous predictions of event locations require the use of additional heuristics. Issue 3: close events cannot be easily disentangled. Issue 4: the need to estimate the left-tail of label distributions compels the model to detect events before their actual occurrence, which might lead to overfitting.
agnostic to the underlying network - which allows the robust training of temporal localization networks even in the presence of extensive label misalignment.
Classical Heuristic Our approach is closely linked to the more classical trick of label smoothing or target smearing (e.g. applying a a²-Gaussian filter dy2 to the labels) which has been considered to increase robustness to temporal misalignment of annotations (Schlüter & Böck 2014 Hawthorne iet al., 2017). This slight modification of the input data converts the original point prediction problem into distribution prediction problem. Indeed, the smoothing of the labels transforms the point labels into distributions. The algorithm is then trained to predict these distributions, which eventually have to be transformed back to point predictions using hand-crafted peak picking heuristics (see Figure (left)) This methodology is also very common in 2D image keypoint detection applications which deal with spatial uncertainty, e.g. human pose estimation (Tompson
al.) 2014, 2015) or facial landmark detection (Merget et al.] 2018). However, despite its intuitive nature, this traditional solution presents several inherent drawbacks (see Figure (Issue 1) Even in a noise-free setting, by transforming the impulse-like target into distribution, the optimal model predictions (with respect to the training loss) differs from the actual goal of the pipeline (i.e., precise localization indicated by the original event label). (Issue 2) As the model learns by mimicking the smoothed target throughout the learning phase, the predictions themselves will be spread out over several time-steps. Hence, additional tailored heuristies, such peak picking (Böck al. (013)) complex thresholding, are required to achieve precise temporal localization. (Issue 3) Even advanced peak picking approaches struggle disentangle close events. For instance, a unique maximum might emergence in the middle of two events, thus significantly disturbing the timeliness of the final predictions. (Issue 4) Having the label mass dispersed temporally both before and after the event occurrences is problematic not only for causal models (i.e., models that make predictions at time t only with data up to time f 1) but also for one-sided recurrent networks and fully convolutional architectures with limited receptive fields. Indeed, all these models have to estimate the left tail of the label distribution before even seeing the event occur. This requirement compels the model to find some structure before the actual event occurrence, leading to poor generalization performance. Although bidirectional networks do not suffer from it, this issue limits the range of possible architectures. The presence of strong label misalignment further worsens these four issues increased noise commonly warrants increased smoothing, dispersing the label (and consequently prediction) mass even more. Overall, experimental evidence (e.g. Section 5.1.1) shows that the accumulation of these issues proves to be very detrimental to the noise robustness of these classical approaches.
In contrast, this work presents novel paradigm for dealing with temporal label uncertainty. The main idea consists in directly inferring point predictions rather than resorting to distributions or heatmaps by fully integrating the modelling of the noise into the loss function. Such direct approach allows for end-to-end learning of localization without the need for additional hand-crafted components. In addition, the systematic and standalone loss function proposed in this regard (Section 44.) not only solves all the above-mentionel issues but also scales well to extensive label misalignment.
Some supeev-uueer models leverage weaker annotations to infer more fine-graing concepts. In such frameworks, noisy labels are implicitly bypassed by the use of higher-level labels which are more invariant to perturbations. For instance, some works achieve object detection (Fergus et al. 2003 Bilen & Vedaldi 2016) or temporal localization (Kumar
Learning

---

Under review as a conference paper at ICLR 2020
Classical approach
Ours (end-to-end)
loss
SoftLoe
DANS
smoothing
Peak-picking
smoothing
smoothing
TINN
Labols
Data
Prodictions
Labela
Data
Prodictions
Figure 3: Modelling novelty. By smoothing both the labels and predictions, the model directly infers point predictions rather than distributions. Among other things, this modification allows for end-to-end learning of localization and alleviates the need for peak-picking, (left) Classical approach of smoothing the labels only. (right) Our novel end-to-end approach and its SoftLoe loss function.
& Raj 2016 Wang et al. 2017) using only class-level annotations. However, finer-grained labels, even noisy ones, often contain some additional information that is essential for optimal performance.
4 SOFTLoC MODEL
4.1 SOFT LOCALIZATION LEARNING Loss
The general principle of relaxing the localization learning is intuitive and potentially powerful if carefully implemented. However, by smoothing the label only, classical approaches transform the original point prediction problem into distribution prediction problem which eventually causes issues (see Section 3) Many of the drawbacks arising from the asymmetric nature of the one-sided smoothing can however be alleviated by filtering not only the labels (i.c., Lg2 Y,))), but also the predictions (i.e., 2 Y,(-)) with unique softness parameter s.. The comparison of these two smoothed processes yields relaxed loss function for the soft learning of the location that deals on its own with the temporal uncertainty of the labels. Indeed, in such setting, the model is given input sequences of point-like events and directly infers point predictions without having to resort to distributions or heatmaps (see Figure (right)))) (righ()); it is only the loss function that views these point labels and predictions as smoothed processes. In discrete time settings, the loss can be written as
(s) = E(000 * Vr,0(,,, $sp y:()), with $sp s -auasssn filter,
(2)
where (cs)) can be any measure of distance. The learning is characterized as soft since the loss is not strictly constraining in terms of precision mass concentration. Indeed, the mass of each event can be both scattered over numerous time-steps and slightly shifted temporally without any abrupt increase in loss. Thus, the model's reliance exact label locations is relaxed.
Measure In noise-free settings, the average stepwise cross-entropy a common choice of loss function for state-of-the-ar models (Lea et al. 2017 Wu et al., (0188) Hawthorne et (019). While potentially unbounded penalization of false predictions might be ideal when training on clean datasets, such behavior can be highly detrimental when labels are subject to temporal misalignment. Therefore, for all experiments in Section 5 C set to the (bounded) average local mean-squared error.
Properties Symmetrically smoothing both the labels and predictions solves several of the issues highlighted in the previous section (see Figure First, in a noise-free setting, the optimal predictions with respect to LSLL are the original annotations themselves. (Solves 1). Second, since the predictions are also smoothed over time, each trigger adds detection mass not only after, but also before the prediction time. Therefore, the model is not required the estimate the left-tail of the label distribution before the actual event occurrence (Solves 4). The prediction mass for a particular event is not necessarily dispersed over time anymore. For instance, in noise- free settings, the point-like targets themselves are the solution to the optimization problem. However, LELL does not strictly constrain the mass of each event be contained in single time-step (Partially Solves 2 & 3).
4

---

Under review as a conference paper at ICLR 2020
Solves 2
Smootied Predictions
Solves 3
Sescothed Predictions
Solves
Smocothed Precictions
Predietion
Predietion
Optimal Prediction
Goorrd
Groznd
GooLod
Truth
Truth
Teatl
Figure 4: The different issues arising from only smoothing the labels solved by our approach. Solves 2: the predictions are unambiguous as the model infers point predictions that are converged towards well-defined points in time. Solves no disentanglement required as the model di- rectly infers point predictions instead of distributions. Solves 4: point predictions span symmetric distributions through smoothing, thus solving the left-tail estimation issue.
4.2 SOFTLoC Loss
The potential dispersion of the prediction mass and its direct consequences on localization perfor- mance still need be addressed. To that end. we propose leverage the properties the weakly- supervised model defined in (Schroeter et al. 2019), which achieves precise temporal localization using only occurrence counts for training. Aside from exhibiting strong localization performance, the loss introduced in that work possesses an implicit mass convergence property, which concentrates the scattered prediction mass toward well-defined single points in time:
Lac = - log
EI (1 II (1- (n.o(1)) AEFİEA
(3)
JEAC
where is the set of all subsets of 21, Te} of size Ex (A)).
Full SoftLoc Model Incorporating this mass convergence loss as a regularizer to our soft localiza- tion learning loss LSLL allows the model directly achieve precise impulse-like like localization, without weakening its noise robustness properties. Thus, this eliminates prediction ambiguity, as only single point prediction is outputted per event occurrence (Solves 2 & 3). Overall, when trained with the SoftLoc loss
(4) the model simultaneously softly learns to mimic the localization annotation, while converging the scatter mass toward impulse-like predictions. this equation, a, regulates the predominance mass convergence against the soft learning (for training iteration r). From practical standpoint, starting with a moderate a, allows an initial relaxed localization learning, before performing stronger convergence (see Section for the specific settings used in this paper).
Lsoft (0) (1 a, Ls). + ar BMc(O)
End-to-end Learning of Localization One of the key factors of the predominance of the deep learning models over classical ones relies on their ability solve problems in an end-to-end fash- ion (Collobert et al.) 2011 Krizhevsky et al. 2012). without the need to resort to partial optimization classical approach (see Section 3) our proposed method is an end-to-end solution to the problem of temporal localization in the presence of misaligned labels (see Issue 2). This solution eliminates the need for hand-crafted components (c.g. peak picking) and is expected to better serve the task at hand.
or hand-crafted heuristics. contrast
Continuous Setting While experiments in Section 5 most state-of-the-art temporal local- ization models perform discretization of time, the loss definition can easily be adapted to suit continuous-time frameworks.
4.3
GENERALIZATION OF PAST WORKS
Our versatile SoftLoe model is generalization of several past works. Indeed, depending the softness parameter Su. the model encompasses a wide range training regimes from classical fully-supervised count-based weakly-supervised.
5

---

Under review as a conference paper at ICLR 2020
Softness - 0 By tending Sar toward zero, the model becomes similar to count-aware localization
RNN with soft localization learning loss. For instance, setting C (.) = logg1 - I-I) yields
lim LSLL (0)= - -- log (1 -Vvro(t) (s((t)) -0
f,t
(5)
(e(() log (0r, + (1- vs log - (c,((t), Σ 3,t
which corresponds to the sum of all stepwise cross-entropies. By further setting o, = 0 (i.e., dis- carding any count-awareness), our loss function becomes identical to the ones found in numerous temporal detection works (e.g., drum detection (Wu et al. 20188), piano onset detection (Hawthorne al. 2017). and video action segmentation (Lea 2017)).
Softness + oo Setting Sn + oo causes the gradient of LSLL (0) to vanish, discarding any prior information of localization, thus making the training weakly-supervised (Schroeter et al. 2019
lim LSoNLo (0)= = a+ BM)(()) x Buc(O). Sy0000
4.4 DEALING WITH UNCERTAINTIES
The introduced softness parameter can be leveraged to deal with different kinds of uncertainties. First, in contrast the traditional approach of aggregating the annotations of multiple individuals (thus trading off dataset richness for noise reduction), our model can be trained on all conflicting individual sequences, since it can cope with noisy annotations. Second, an annotator specific softness s? can further be implemented to model their respective reliability. Finally, an extract specific softness can be incorporated to capture the noise or annotation complexity of certain more challenging sequences.
Experiments conducted in the section below show that the performance is robust to variations in the softness parameter. Indeed, this hyperparameter only acts as a coarse indicator of temporal uncertainty and thus does not need to strictly match the underlying noise distribution.
EXPERIMENTS
In this section, we demonstrate the effectiveness and flexibility of our approach in a broad range of challenging experiments (music event detection, times series detection, video action segmentation).. Experiment and implementation details can be found on the paper's 's website])
5.1
MUSIC EXPERIMENTS
5.
PIANO ONSET EXPERIMENT
Piano transcription and more specifically piano onset detection is difficult problem as it requires precise and simultaneous detection of hits from 88 different polyphonic channels.
Dataset This experiment based on the MAPS database (Emiya et al. 2010). The dataset creation protocol strictly follows the one from Hawthore et al. (2017). (Only onsets are considered for the comparison.) To evaluate the robustness, the training labels are artificially perturbed according to a normal distribution c. (O, o²), while the test labels are kept intact for unbiased evaluation.
Benchmarks Three different benchmarks are considered. First, the state-of-the-a art model (on clean data) proposed by Hawthorne et al. (2017) is highly representative of models aiming for optimal performance with little regard for annotation noise (Hawthorne). Second, smoothed version of the first benchmark with extended onset length (i.e., over 96ms) illustrates the common practice used achieve robustness (Hawthorne (smoothed)). Finally, the first benchmark performs local classification using standard cross-entropy, the soft bootstrapping loss proposed by Reed et al. (2014) leveraged instead for increased robustness (Bootstrap (soft)).
Anonymous link: http://giiuuh.com/Sfflll ocICI R/submission

---

Under review as a conference paper at ICLR 2020
Architecture, Training and Evaluation Our network is comprised of six convolutional layers (representation learning) followed by 128-umit LSTM (temporal dependencies learning) and two fully-connected layers (prediction mapping). The network is trained using mel-spectrograms (Stevens) et a 1937) and their first derivatives stacked together as model input, while data augmentation in the form of sample rate variations is applied for increased robustness and performance. The loss (Equa- tion 4 with softness Su 100ms is optimized using the Adam algorithm (Kingma & Bal (2015). The models are evaluated on the noise-free test set using Fi -scores computed with the standard mir_eval library (Raffel et al.) and a 50ms tolerance (Hawthorne et al. 2017). (a, max max(min( --00
10 .9). 2)..
Results As depicted in Figure 5 our pro- posed SoftLoc approach displays strong ro- bustness against label misalignment: in con- trast to all benchmarks, the performance ap- pears almost invariant to the noise level. (See Appendix A.1 for discussion on the model's performance for > 200ms.) At a = 150ms, only 26% of training labels lie within the 50ms tolerance. In this context, the score achieved by our SoftLoc model (i.e., 75%) is unattainable for classical approaches, which do not take label uncertainty into account and attempt to strictly fit the noisy annota- tions. While standard tricks, such label smoothing, slightly improve noise robustness (e.g., Hawthorne (smoothed)), their effective- ness is limited in contrast to our proposed ap- proach. Finally, the parameters used through- out this experiment are fixed. However, as our loss is a strict generalization of the standard cross-entropy loss used by Hawthhrne et al. (017)), the small performance gap for small noise levels can be reduced by setting at = Sh + Oms and L = - log(11 H-)).
F5 Fy-worre PN
Our (SoftLec)
Hunthorne
Houtthrre (ummanthed
Bootstrop (sult)
30-
175
Noisc
[me]]
Figure 5: Fi piano onset detection performance of our approach (softness Su = 100ms) and the bench- mark models as function of label noise levels.
To assess the usefulness of the different components of Lsoft we repeat the above experiments keeping only individual parts of the loss function. T that LSL. is the main driver of performance in noise-free settings, while LMC ensures stability under increased label misaligmment. (A simple threshold-baad localization from the dispersed mass produced by LSLL.) Overall, while each loss individually produces reasonable predictions, only the combined LSSoLLwe yields both competitive scores in free settings and strong robustness to temporal misalignment.
Ablation Study
peak-picking algorithm was implemented to infer
5.1.2 DRUM DETECTION EXPERIMENT
The softness S, is a defining model hyperparameter. In this section. 210 independent runs for the same drum detection experiment are conducted with varying noise and softness levels in order to highlight the correlation between this key parameter, label noise and the final localization performance.
Dataset The experiment is based on the D-DTD Eval Random drum detection task (IDMT-SMT. Drums dataset (Dittmar& & Gärtner 2014)) performed by Wu et al. (2018). The goal the correct
Table I: Ablation Study. Piano onset detection performance of our model trained with loss functions LooLLoe (Sy = 100ms), ESLL and LMC respectively in various noise level settings.
Loss
Oms
50ms
100ms
150ms
200ms
LsL. (a:, 0)
76.06
76.00
75.10
66.88
46.91
LMc (a, =
71.59
73.04
68.69
70.33
67.26
LsorrLo
76.88
76.34
75.86
74.87
73.68
7

---

Under review as a conference paper at ICLR 2020
100
0.96
Noise a [ms]]
0.92
0.90
20
0.88
F1
40
60
80
100
120
140
Softness Sar [ms]
Figure 6: Drum detection performance with respect to model softness and label noise. Fi-scores are Gaussian Nadaraya-Wandd Watson estimates based on runs (white dots) sampled uniformly random.
temporal localization of three different classes of drum hits hi-hats (HH), kick drums (KD), and snare drums (SD) - within 50ms tolerance window. Normally distributed errors ~ (O, o?) are artificially introduced on all training and validation labels, while the test labels are kept intact for unbiased inference. The noise level a for each run is uniformly sampled from the range [Omss 100mns.
Architecture, Training and Evaluation The network is similar the in Section 5.1.1 except for the number of filters and nodes. The model softness Sn for each run is uniformly sampled from [0ms, 150ms). Training and evaluation are carried out in the same way as the piano experiment in
Section 5.1.1 (Learning rate: 10-1, batch size: 32, iterations: 1.5 x 10 sample length: 1.5s).
Results The results of the 210 runs are displayed in Figure 6 A Gaussian Madaraya-Watson kernel regression (Nadaraya. 1964] Watson 1964) is used to interpolate the Fi Ficsc, offering detailed view of the model's response varying label noise levels. This figure not only confirms the model's high robustness to label misalignments, but also reveals that these results are very robust to changes in the softness level. Indeed, wide range of softnesses yield optimal performance, long Sar 2 Obviously, extreme softness levels (c.g. Sh. - oc) would however induce partial or even total loss of the information conveyed by the localization prior, resulting in a decrease performance (see Table 2}. Robustness considerations aside, our SoftLoc model displays an outstanding overall performance with Fi Fisscores over 95% across all noise levels; the model even when trained on extremely noisy labels (e.g., 100mss outperforms several standard benchmarks Wu et al. (2018) which were trained on noise-free training samples (a 0ms).
Noise-free Comparison In clean settings (i.e., a Omn) the benchmark models have a clear advantage as they correctly assume noise-free labels. Despite this, our SoftLoe model achieves state- of-the-art performance on three different metrics (KD. HH, precision) demonstrating that robustness does not come at the expense of raw localization performance (see Table 2
5.2
TIME SERIES DETECTION
The timely detection of events in healthcare time series is a crucial challenge to improve medical decision making. The task tackled in this section consists in the precise temporal detection smoking episodes using wearable sensors features based the puffMarker dataset (Saleheen al. 2015). Once again, in order to conduct the robustness analysis, the original annotations are artificially misaligned. However,
each time-step in this dataset represents full respiration cycle, the noise distributions must be applied in a discrete fashion: namely, rounded normal distribution (i.e., E, ~ (O, o²)T) or binary constant length shifting of labels (Ō steps either to the left or the right with equal probability). denoted B(-5,
6, 8). This task is particularly challenging as detections have to be perfectly aligned with the ground-truth be considered correct.
Model and Benchmark As the focus is set on robustness rather than raw performance, the model architecture kept extremely simple: 14-node fully connected layer followed by 14-unit LSTM
8

---

Under review as a conference paper at ICLR 2020
Table Noise-free Drum Detection, Comparison of our SoftLoc model (sw 100ms) and state-of- the-art models evaluated in Wu et al. (2018) on the clean D-DTD Eval Random task (a 0ms). The Ficrcs per instrument (KD/SD/HH), the average precision, recall, and overall F1 are displayed.
METHOD
KD
SD
HH
PRE
REC
F1
RNN
97.2
92.9
97.3
95.7
96.9
95.8
TANHB
95.4
93.1
97.3
93.9
97.1
95.3
RELUTS
86.6
93.9
97.7
92.7
95.0
92.7
LSTMPB
98.4
96.7
97.4
97.7
97.6
97.5
GRUTS
91.4
93.2
96.2
91.8
97.2
93.6
S > 00
96.0
90.4
97.1
95.1
93.9
94.5
100ms
98.6
95.7
97.8
98.3
97.2
97.4
and a final fully connected layer with softmax activation. Both the standard cross-entropy (CE) and our LsotLLe loss function are evaluated. The LR-M model proposed by [Adams & Marlin (2017), which developed achieve strong robustness to temporal misalignment of labels this particular dataset, is also considered as benchmark.
Results The results, produced using 6-fold (leave-one (laav-one-pattent-ouut patient-out) cross validation. marized in Table 3 Not only does training with the proposed Loottoo loss function yield a strong improvement in robustness when compared the standard cross-entropy, but our simple recurrent model also significantly outperforms the robust LR-M model on all metrics. In addition, our approach displays low standard deviations, which underlines the consistency and robustness These observations hold for both noise distributions (A and B); hence, the normal smoothing filters do require the underlying noise be normally distributed in order for the model to be effective. Further testing with skew normal distribution of noise confirm these results even in non-symmetric settings.
the learning.
5.3 VIDEO ACTION SEGMENTATION
Video action segmentation dense classification problem where each time-step has be mapped to one action class differs substantially from music event localization or time series detection problems, where scattered events from multiple classes have be precisely localized. Nonetheless, the properties of the SoftLoc loss can still be leveraged on such task; in this context, while the role of LSL. unchanged, .MC acts as a count-based regularizer, rather than a means for mass convergence.
Experiments Several video segmentation experiments from Lea et al. 2017 are replicated using either the standard cross-entropy (original loss), LSLL or LSonLox as training loss for the ED- TCN model. As the ED-TCN model already exhibits strong robustness properties against label misalignment Lea et al. (2017), these experiments will allow to measure the additional marginal gain in performance and robustness when replacing the standard cross-entropy with our the proposed LSontLoe loss function. To assess robustness, each label sequence in the training set is either delayed or advanced by fixed constant ò. (8)=)8). 78).
Table 3: Smoking Puff Detection. Comparison of LR-M (Adams & Marlin 2017 and the deep model trained with CE or LsoftLo with respect to misalignment distributions [N o²) and B(-6, 8). Reported metrics are mean and standard deviation of ten 6-fold cross-validated Fiscores.
8 0
1
3
1
LR-M
93.0
80.6 (8.6)
65.9
64.0 (15.6)
55.0 (19.7)
N
CE
92.6 (2.)
(16.2)
36.0 (15.6)
28.9 (17.0)
25.8 (16.2)
SOFFLOO
93.1 (25)
90.6 (3.4)
87.8 (4.1)
83.6 (5.2)
79.0 (6.9)
LR-M
65.5 (14.5)
54.9 (20.4)
44.1 (19.7)
51.8 (19.8)
B
CE
41.7 (15.3)
28.3 (14.5)
26.6 (15.3)
22.8 (15.1)
LSarTi
90.8 (3.3)
87.0 (4.7)
81.7 (7.2)
72.4 (10.1)

---

Under review as a conference paper at ICLR 2020
Table 4: Video Action Segmentation. Comparison of various training losses (CE, LSLL and softloe) with respect to different label misalignment levels for the ED-TCN model on 50 Salads (mid). Metrics are mean and standard deviation Fj@10 (Lea al. 2017) of ten 5-fold cross- voossvaaliations.
Loss
8 = Os
5s
10s
15s
20s
CE
66.7 (1.6)
59.7 (1.2)
43.4 (0.9)
33.6 (1.1)
26.7 (0.8)
ØSLL
66.7 (1.0)
60.6 (0.9)
47.5 (1.2)
36.1 (0:8)
28.0 (1.2)
Lsortoo
67.2 (0.8)
61.5 (1.3)
48.0(1.0) (1.0)
38.0 (1.9)
29.5 (1.1)
Results As summarized in Table and Table (in Appendix B.2 replacing the standard cross- entropy loss with LSoiLoe does not only significantly increase the robustness of the ED-TCN model which was already shown to be robust label misalignment (Lea al. 2017) but also achieves competitive performance in woise-free settings. Further experiments with different softness parameters (see Figure Appendix [.I)) reveal that increasing the model softness Sy as the underlying noise levels increase produces optimal performance. For instance, in noisy settings, greater performance can be achieved (up to 25% overperformance) by simply choosing a large enough softness. Overall, the SoftLoc loss function displays strong results very different task (i.e., temporal segmentation as opposed to temporal localization), highlighting once again its versatility of application.
6 CONCLUSION
In this work, we have shown how relaxing annotation requirements (i.e., weakening the model's reliance on the exact location of events) only has the practical benefit of alleviating annotation efforts but, more importantly, leads to model that is robust to temporal noise without compromising performance on clean training data. This contrasts with traditional approaches which attempt to strictly mimic the annotations, leading to poor predictions when training with noisy labels. We have demonstrated these claims on a number of classical challenging tasks, in which our Softl Loc loss exhibits state-of-the-art performance.
The proposed loss function is agnostic to the underlying network and hence can be used as a loss replacement in almost any recurrent architecture. The versatility of the model can find applications in a wide array tasks, even beyond temporal localization.
REFERENCES
Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijayanarasimhan. YouTube-8m: A large-scale video classification benchmark. arXiv preprint arXiv:1609 08675, 2016.
Roy Adams and Ben Marlin. Learning Time Series Detection Models from Temporally Imprecise Labels. In Proceedings of the International Conference on Artificial Intelligence and Statistics, volume 54, pp. 157-165. PMLR, 2017.
Michele Banko and Eric Brill. Scaling to very very large corpora for natural language disambiguation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pp. 26-33. Association for Computational Linguistics, 2001.
Hakan Bilen and Andrea Vedaldi. Weakly supervised deep detection networks. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2846-2854. IEEF, 2016.
Sebastian Böck, Jan Schlüter, and Gerhard Widmer. Enhanced peak picking for onset detection with recurrent neural networks. In Proceedings of the 6th International Workshop Machine Learning and Music (MML), pp. 15-18, 2013.
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12 (Aug):2493-2537, 2011.
10

---

Under review as a conference paper at ICLR 2020
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248-255. IEEE, 2009.
Christian Dittmar and Daniel Gärtner. Real-time transeription and separation of drum recordings based on NMF decomposition. In Proceedings of International Conference Digital Audio Effects (DAFx), 2014.
Valentin Emiya, Roland Badeau, and Bertrand David. Multipitch estimation of piano sounds using new probabilistic spectral smoothness principle. IEEE Transactions on Audio, Speech, and Language Processing, 1600):66441154, 2010.
Robert Fergus, Pietro Perona, and Andrew Zisserman. Object class recognition by unsupervised scale- invariant learning. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), pp. 264-271. IEEE, 2003.
Alon Halevy, Peter Norvig, and Fernando Pereira. The unreasonable effectiveness of data. IEEE Intelligent Systems, (2):8-1) 12, 2009.
Curtis Hawthorne, Erich Elsen, Jialin Song, Adam Roberts, Ian Simon, Colin Raffel, Jesse Engel, Sageev Oore, and Douglas Eck. Onsets and frames: Dual-objective piano transcription. arXiv preprint arXiv:1710.11155: 2017.
Curtis Hawthorne, Andrew Stasyuk, Adam Roberts, Ian Simon, Anna Huang, Sander Dicleman, Erich Elsen, Jesse Engel, and Douglas Eck. Enabling factorized piano music modeling and generation with the maestro dataset. Proceedings of International Conference on Learning Representations (ICLR), 2019.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Proceedings of International Conference on Learning Representations (ICLR), 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. In Advances in Neural Information Processing Systems (NIPS), pp. 1097-1105, 2012.
Anurag Kumar and Bhiksha Raj. Audio event detection using weakly labeled data. Proceedings of International Conference Multimedia, pp. 1038-1047. ACM, 2016.
Colin Lea, Michael D Flynn, Rene Vidal, Austin Reiter, and Gregory D Hager. Temporal convolutional networks for action segmentation and detection. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), pp. 166-165, IEEE, 2017.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7533):436, 2015.
Daniel Merget, Matthias Rock, and Gerhard Rigoll. Robust facial landmark detection via a fully- convolutional local-global context network. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), pp. 781-790, 2018.
Elizbar A Nadaraya. On estimating regression. Theory of Probability & Its Applications, 9(1): 141-142, 1964.
Colin Raffel, Brian McFee, Eric J Humphrey, Justin Salamon, Oriol Nieto, Dawen Liang, and Daniel PW Ellis. mir_eval: A transparent implementation common MIR metrics.
Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks noisy labels with bootstrapping. arXiv preprint 1rXiv:1412.659 2014.
Nazir Saleheen, Amin Ahsan Ali, Syed Monowar Hossain, Hillol Sarker, Soujanya Chatterjee, Benjamin Marlin, Emre Ertin, Mustafa Al' Absi, and Santosh Kumar. puffmarker: a sensor approach for pinpointing the timing of first lapse in smoking cessation. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pp. 999-1010. ACM, 2015.
11

---

Under review as a conference paper at ICLR 2020
Jan Schlüter and Sebastian Böck. Improved musical onset detection with convolutional neural networks. Proceedings of International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6979 6983. IEEE. 2014.
Jürgen Schmidhuber. Deep learning in neural networks: An overview. Neural Networks, 61:85-117, 2015.
Julien Schroeter, Kirill Sidorov, and David Marshall. Weakly-supervised temporal localization via occurrence count learning. In Proceedings of International Conference on Machine Learning (ICML), pp. 5649-5659, 2019.
Stanley Smith Stevens, John Volkmann, and Edwin B Newman. A scale for the measurement of the psychological magnitude pitch. The Journal of the Acoustical Society of America, 8(3):185-100 1937.
Jonathan Tompson, Ross Goroshin, Arjun Jain, Yann LeCun, and Christoph Bregler. Efficient object localization using convolutional networks, In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), pp.
2015.
Jonathan J Toompso, Arjun Jain, Yann LeCun, and Christoph Bregler. Joint training of a convolutional network and graphical model for human pose estimation. In Advances in Neural Information Processing Systems (NIPS), pp. 1799-1807, 2014.
Limin Wang, Yuanjun Xiong, Dahua Lin, and Luc Van Gool. Untrimmednets for weakly supervised action recognition and detection. In Proceedings of Conference Computer Vision and Pattern Recognition (CVPR), pp. 4325-4334. IEEE, 2017.
Geoffrey S Watson. Smooth regression analysis. Sankhya: The Indian Journal of Statistics, Series A, pp. 372-372, 1964.
Chih-We Wei Wu, Christian Dittmar, Carl Southall, Richard Vogl. Gerhard Widmer, Jason Hockman, Meinard Muller, and Alexander Lerch. A review of automatic drum transcription. IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 26(9): 2018.
Karthik Yadati, Martha Larson, Cynthia CS Liem, and Alan Hanjalic. Detecting socially significant music events using temporally noisy labels. IEEE Transactions on Multimedia, 2009):252--2400, 2018.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In Proceedings of International Conference on Learning Representations (ICLR), 2017.
12

---

Under review as a conference paper at ICLR 2020
A PIANO ONSET DETECTION
A.1 EXTREME NOISE SETTINGS
Figure (in the main text) depicts the strong invari- ance of our SoftLoc model to label misalignment on a broad array of noise levels (i.e., up to a 200ms).
70
In this section, we evaluate the model's performance on an even wider range in order to fully assess its behavior in extreme settings. To that end additional piano onset detection experiments, with noise levels up to 1000ms, were conducted following the
Fy-scoe [N]
30
protocol described Section 5.1. The results are displayed in Figure 7
20
Overall, this figure confirms the remarkable robust- ness of SoftLoc model label misalignment. While the absolute performance unsurprisingly de- creases as the training data becomes less accurate, the detection capability of the model in noisy settings outshines any classical approach (see Figure the main text). Finally, these results could further be improved by increasing the model softness Sy (see Section 5.2).
* Oura (SoftLoc)
16
200
400
600
801
1000
Noise
[ms]
Figure 7: Fi piano onset detection perfor- mance of the SoftLoc model (s,, 100ms) as function of label misalignment.
(medinlan) Fatsa Regatiee
Pestiire
Bhe di 6- 8
Pouttree Fatie Aegattiv
jeradetioon
joound
Charrell
GO
(a) Noise-free training data (a 0ms)
(b) Noisy training data (a 50ms)
Fatae Postlive
(eoddimiss)
Fablo (Prdditioon Pooitice
(Creund-Trrutt)
[aroand muth)
eve e
Chantell
Timee
Tiea
Very noisy training data 100ms)
(d) Extremely noisy training data (a 200ms)
Figure 8: Out-of-sample predictions of our SoftLoc model trained data subject to various levels noise, ranging from (a) the noise-free case o = Oms to (d) the extremely noisy 200ms. (Schubert- Piano Sonata in minor, D 784, Opus 143, 3. Mov)
13

---

Under review as a conference paper at ICLR 2020
A.2 FURTHER ILLUSTRATIONS
Timeliness of SoftLoc predictions Figure [ illustrates how consistently precise and well-centered (i.e., neither too late nor early) the predictions are regardless of the noise setting. Indeed, there is almost no difference in prediction centering when comparing the results for a = Oms or a 200ms.
Noisy Labels and Ground-Truth Discrepancy To further illustrate the complexity of the localiza- tion task when annotations are subject to misalignment, we consider the training labels as predictions and then compare them to the clean ground-truth. Figure displays an example the quality of the training labels. Obviously, in the noise-free free setting (i.e., Oms), the localization is spotless as the training labels and the ground truthss are identical. However, as the noise level increases, the proportion of labels that stay within the 50ms tolerance window decreases significantly. More precisely, the performance (i.e., Fi-score) of the labels themselves is 68.2%, 39.8% and 23.7% for a equal 50ms, 100ms and 200ms respectively.
Trie Positiie
(Ground-
(Predittion
Eala Posttiun
iBrourd-Truthi
Charmel
8:60
8/688
@
$88
OBC
88
@0
26
Timee
(a) Noise-free labels (a Oms)
(b) Noisy labels 50ms)
Fabte Pasttive
(Dround Truth
1its voditive (Grsund-inutt)
(Prediction)
600
00 CMO
0
25
26
(c) Very noisy labels (a = 1 100ms)
(d) Extremely noisy labels (a 200ms)
Figure 9: In-sample performance of the noisy training labels themselves (as predictions) when compared to the clean ground-truth. (Lisz Hungarian Rhapsody No. 10)
14

---

Under review as a conference paper at ICLR 2020
B
VIDEO ACTION SEGMENTATION
B.1 IMPACT OF THE SOFTNESS PARAMETER
As depicted in Figure To training with the SoftLoc loss function instead of the standard cross-entropy yields improved performance (up to 25%) in all noise settings almost regardless, of the softness Si- The only exception occurs when selecting softness level that is too wide while training with noise-free (8 0) labels. As also observed in Section 5.1.2 the model achieves optimal performance when the softness level Sa slightly larger than noise level However, although the efficiency of the approach is bound decrease when the disparity between selected softness and noise level is becoming too large, performance close to the optimal one can be achieve with a wide range of softnesses Su.
=20s
=15s
Relative Performance [%]
11s
Os
-5
0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
Softness SM
Figure 10: Video Action Segmentation. Relative performance of the ED-TCN model trained with L - relative to CE - with respect to the softness level Sy for various noise levels 8.
15

---

Under review as a conference paper at ICLR 2020
B.2 ADDITIONAL RESULTS
Table 5: Video Action Segmentation. Performance comparison of different training losses (cross- entropy, LSLL and Lsoiloe for the ED-TCN model various datasets and measures. Metrics mean and standard deviation Fj@10 or F,050 (Lea et al. 2017) ten 5-fold cross-validation.
50 SALADS (MID)
LOss
5s
10s
15s
20s
Fi010
CF
51.8 (0.7)
38.5 (1.1)
19.7 (0.7)
10.7 (0.8)
6.9 (1.0)
LsLa.
50.7 (0.5)
38.8 (1.4)
22.3
12.3 (0.8)
7.8 (0.9)
Lorttor
49.8 (0.9)
39.5 (1.2)
23.4 (1.4)
13.7 (0.9)
7.5 (0.8)
50 SALADS (EVAL)
Loss
6 Os
5s
10s
15s
20s
Fj010
CE
74.9 (0.8)
72.7 (0.9)
59.6
48.1
41.8
PSLL
75.4
73.0 (0.9)
61.9 (0.8)
49.2 (0.9)
41.8 (1.2)
LSOPILOC
75.5 (1.2)
73.7 (1.3)
62.7 (1.5)
(0.8)
43.8 (1.7)
F,050
CE
(3.7)
52.7 (1.5)
34.4
21.9 (1.4)
15.3 (1.0)
LSLL
(3.0
55.3 (L.)
35.1 (1.0)
22.8 (1.2)
15.7 (1.2)
Lsorrtoc
(1.1)
55.7 (0.9)
36.8 (1.1)
23.6 (0.9)
16.2
GTEA DATASET
LOSs
8 Os
5s
10s
15s
20s
F
CE
74.7 (1.2)
64.3 (0.9)
37.1 (1.9)
27.6 (1.4)
23.8 (1.6)
LSLL
74.1 (1.3)
64.3 (2.4)
21.4 (2.0
22.8 (1.7)
LsuriLue
73.4 (1.2)
65.2 (1.2)
43.8 (2.7)
28.5 (1.6)
22.5 (0.8)
F,050
CE
(1.8)
(1.9)
(1.0
8.1 (0.9)
7.8 (1.2)
LSLL
54.5 (1.5)
33.7 (2.7)
(1.0
8.0 (0.8)
5.9 (1.0)
LSurrLoc
52.2
34.8 (1.2)
15.4 (1.3)
(1.4)
5.1 (0.5)
16


Reviews:
[{"review_id": "r1e8STrIqH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose SoftLoc, an interpolation of two temporal event prediction losses (one based on label & prediction smoothing, and one based on weakly-supervised count-based loss) that makes the predictor more robust to noisy training data. They demonstrate on various temporal alignment datasets from music to wearable sensors to video action segmentations, that their method is performs well both on noisy-free and noisy settings compared to prior approaches.\n\nStrengths:\n(1) relatively thorough experimental valuations: using 4 datasets comparing with sufficient number of prior approaches (One potential improvement could be to try noise distributions other than Gaussian)\n(2) simple objective and consistent improvements. It is encouraging that the simple modification enables such consistent empirical improvements.\n\nWeaknesses:\n(1) the novelty of the method appears limited. The weakly-supervised loss is borrowed from the prior work, so it seems like the main algorithmic novelty is to add noise to predictions (as opposed to just adding to labels as done in prior work). \n\nOther comments:\n(1) in 5.1.1, is there justification for \\alpha_\\tau expression? How did you pick that? \n(2) is Issue 1 an actual\n problem? Even with just label smoothing, I would expect predictor will find the mode and peak-picking can produce the correct predictions. Similarly, discussions of how the proposed method can solve the issues listed may benefit from some rewriting, or even simple toy experiments to demo that those are actually the concerns that are not addressed by prior work. Such toy experiments may complement the existing end-to-end experiments to demonstrate the precise properties of the proposed method. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SylPrLSRtr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new loss for training models that predict where events occur in a sequence when the training sequence has noisy labels. The central idea is to smooth the label sequence and prediction sequence and compare these rather than to force the model to treat all errors as equally serious.\n\nThe proposed problem seems sensible, and the method is a reasonable approach. The evaluations are carried out on a variety of different tasks (piano onset detection, drum detection, smoking detection, video action segmentation).\n\nSuggestions\n* While the authors do bring up lots of related work on learning from noisy labels, the insights from that work, and its relationship to this proposed technique could be more productively explored\n* The connections between the assumptions of the evaluation metric and the motivation for the smoothing methodology could be more productively elucidated\n* The task explored in this paper, and the task-specific problem, should be described more generally since ICLR has a generalist readership. For instance, the paper gets off on a rather strange footing discussing large data (indeed, but evident to the vast majority of ICLR readers), but little is said in terms of the specifics of the temporal localisation problem except via citations to other papers. In particular, the task set up, the problem posed by label misalignment could be described elegantly right in the introduction with a carefully designed figure. In section 2, the description is hard to follow, the mathematical notation is vague and hard to parse. For instance, the label sequence Y, is d-dimensional- the meaning of d should be clarified. Calling it “discretised” is also a bit strange, for most readers- it’s a label sequence.\n\nThe evaluation objective needs to be clarified, at least qualitatively; ideally in the introduction. Introducing a new objective is meaningful not only in light of noisy labels (always a problem), but in light of how the evaluation is carried out.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1lBTZiAKS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper deals with temporal label noise, or label misalignment problem in localization, which is the long-standing problem of robust event localization regarding the temporally misaligned labels in sequential data.  Some existing works are constrained to hardly fit the annotations of data during training. The work models the temporal label misalignment with perturbations to the ground-truth timestamps, and presents a soft localization learning scheme which relaxes the reliance on the exact position of labels. The idea introduced to temporal data is well-motivated since the label for temporal data is often dispersed. Technically speaking, the proposed SoftLoc loss comprises two terms: 1). $\\mathscr{L}_{SLL}$: a soft learning loss that relaxes the prediction mass concentration, by symmetrically filtering the labels and predictions. This term helps to relax the model’s reliance on exact label locations. 2). $\\mathscr{L}_{MC}$: a mass convergence loss that acts as a regularizer to facilitate the model with precise impulse-like localizations. With a trade of factor to balance two terms, the SoftLoc model can achieve precise impulse-like localization performance without weakening the model robustness. Various applications, such as PIANO ONSET, DRUM DETECTION and TIME SERIES DETECTION are performed to verify the effectiveness of the proposed method. And state-of-the-art performance is achieved.\n\nDespite the achievement indicated by the experiments, I still have some concerns about this paper.\n\n(1)\tThe authors propose the relaxed loss L_{SLL} for the soft learning of the location. And the label smoothing idea (applying a ˜S^2-Gaussian filter to the labels) has been introduced to increase the robustness to temporal misalignment of annotations [2]. Although the authors discuss the several inherent drawbacks of it in the 2nd paragraph of Related Work, they still follow the label smoothing idea in their model (Eq.2) in Section 4.1 in a two-side smoothed process. The reason should be clarified.\n\n(2)\tThe authors claim that their method has the advantage of generalizing some regimes of weakly-supervised learning. And the adopted L_{MC} is a weakly-supervised loss. Does the advantage come from L_{MC}?   \n\n(3)\tThe authors propose the two-side relaxed loss L_{SLL} for the soft learning of temporal localization problem. However, in the ablation study, the authors do not give the results with different level of label noise for one-side variant. Adding these results could help to demonstrate the necessity of the two-side relaxed loss L_{SLL}. \n\n\nReferences:\n[1] Improved musical onset detection with convolutional neural networks\n[2] Onsets and frames: Dual-objective piano transcription\n\n\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Main content:

Blind review #3 summarizes it well:

This paper proposes a new loss for training models that predict where events occur in a sequence when the training sequence has noisy labels. The central idea is to smooth the label sequence and prediction sequence and compare these rather than to force the model to treat all errors as equally serious.

The proposed problem seems sensible, and the method is a reasonable approach. The evaluations are carried out on a variety of different tasks (piano onset detection, drum detection, smoking detection, video action segmentation).

--

Discussion:

The reviewers were concerned about the relatively low level of novelty, simplicity of the proposed approach (which the authors argue could be seen as a feature rather than a flaw, given its good performance), and inadequate motivation.

--

Recommendation and justification:

After the authors' revision in response to the reviews, this paper could be a weak accept if not for the large number of stronger submissions.
Decision: Reject