Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The authors propose SoftLoc, an interpolation of two temporal event prediction losses (one based on label & prediction smoothing, and one based on weakly-supervised count-based loss) that makes the predictor more robust to noisy training data. They demonstrate on various temporal alignment datasets from music to wearable sensors to video action segmentations, that their method is performs well both on noisy-free and noisy settings compared to prior approaches.

Strengths:
(1) relatively thorough experimental valuations: using 4 datasets comparing with sufficient number of prior approaches (One potential improvement could be to try noise distributions other than Gaussian)
(2) simple objective and consistent improvements. It is encouraging that the simple modification enables such consistent empirical improvements.

Weaknesses:
(1) the novelty of the method appears limited. The weakly-supervised loss is borrowed from the prior work, so it seems like the main algorithmic novelty is to add noise to predictions (as opposed to just adding to labels as done in prior work). 

Other comments:
(1) in 5.1.1, is there justification for \alpha_\tau expression? How did you pick that? 
(2) is Issue 1 an actual
 problem? Even with just label smoothing, I would expect predictor will find the mode and peak-picking can produce the correct predictions. Similarly, discussions of how the proposed method can solve the issues listed may benefit from some rewriting, or even simple toy experiments to demo that those are actually the concerns that are not addressed by prior work. Such toy experiments may complement the existing end-to-end experiments to demonstrate the precise properties of the proposed method. 
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper deals with temporal label noise, or label misalignment problem in localization, which is the long-standing problem of robust event localization regarding the temporally misaligned labels in sequential data.  Some existing works are constrained to hardly fit the annotations of data during training. The work models the temporal label misalignment with perturbations to the ground-truth timestamps, and presents a soft localization learning scheme which relaxes the reliance on the exact position of labels. The idea introduced to temporal data is well-motivated since the label for temporal data is often dispersed. Technically speaking, the proposed SoftLoc loss comprises two terms: 1). $\mathscr{L}_{SLL}$: a soft learning loss that relaxes the prediction mass concentration, by symmetrically filtering the labels and predictions. This term helps to relax the model’s reliance on exact label locations. 2). $\mathscr{L}_{MC}$: a mass convergence loss that acts as a regularizer to facilitate the model with precise impulse-like localizations. With a trade of factor to balance two terms, the SoftLoc model can achieve precise impulse-like localization performance without weakening the model robustness. Various applications, such as PIANO ONSET, DRUM DETECTION and TIME SERIES DETECTION are performed to verify the effectiveness of the proposed method. And state-of-the-art performance is achieved.

Despite the achievement indicated by the experiments, I still have some concerns about this paper.

(1)	The authors propose the relaxed loss L_{SLL} for the soft learning of the location. And the label smoothing idea (applying a ˜S^2-Gaussian filter to the labels) has been introduced to increase the robustness to temporal misalignment of annotations [2]. Although the authors discuss the several inherent drawbacks of it in the 2nd paragraph of Related Work, they still follow the label smoothing idea in their model (Eq.2) in Section 4.1 in a two-side smoothed process. The reason should be clarified.

(2)	The authors claim that their method has the advantage of generalizing some regimes of weakly-supervised learning. And the adopted L_{MC} is a weakly-supervised loss. Does the advantage come from L_{MC}?   

(3)	The authors propose the two-side relaxed loss L_{SLL} for the soft learning of temporal localization problem. However, in the ablation study, the authors do not give the results with different level of label noise for one-side variant. Adding these results could help to demonstrate the necessity of the two-side relaxed loss L_{SLL}. 


References:
[1] Improved musical onset detection with convolutional neural networks
[2] Onsets and frames: Dual-objective piano transcription




review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper proposes a new loss for training models that predict where events occur in a sequence when the training sequence has noisy labels. The central idea is to smooth the label sequence and prediction sequence and compare these rather than to force the model to treat all errors as equally serious.

The proposed problem seems sensible, and the method is a reasonable approach. The evaluations are carried out on a variety of different tasks (piano onset detection, drum detection, smoking detection, video action segmentation).

Suggestions
* While the authors do bring up lots of related work on learning from noisy labels, the insights from that work, and its relationship to this proposed technique could be more productively explored
* The connections between the assumptions of the evaluation metric and the motivation for the smoothing methodology could be more productively elucidated
* The task explored in this paper, and the task-specific problem, should be described more generally since ICLR has a generalist readership. For instance, the paper gets off on a rather strange footing discussing large data (indeed, but evident to the vast majority of ICLR readers), but little is said in terms of the specifics of the temporal localisation problem except via citations to other papers. In particular, the task set up, the problem posed by label misalignment could be described elegantly right in the introduction with a carefully designed figure. In section 2, the description is hard to follow, the mathematical notation is vague and hard to parse. For instance, the label sequence Y, is d-dimensional- the meaning of d should be clarified. Calling it “discretised” is also a bit strange, for most readers- it’s a label sequence.

The evaluation objective needs to be clarified, at least qualitatively; ideally in the introduction. Introducing a new objective is meaningful not only in light of noisy labels (always a problem), but in light of how the evaluation is carried out.


review_title: Official Blind Review #3

Meta-review: Main content:

Blind review #3 summarizes it well:

This paper proposes a new loss for training models that predict where events occur in a sequence when the training sequence has noisy labels. The central idea is to smooth the label sequence and prediction sequence and compare these rather than to force the model to treat all errors as equally serious.

The proposed problem seems sensible, and the method is a reasonable approach. The evaluations are carried out on a variety of different tasks (piano onset detection, drum detection, smoking detection, video action segmentation).

--

Discussion:

The reviewers were concerned about the relatively low level of novelty, simplicity of the proposed approach (which the authors argue could be seen as a feature rather than a flaw, given its good performance), and inadequate motivation.

--

Recommendation and justification:

After the authors' revision in response to the reviews, this paper could be a weak accept if not for the large number of stronger submissions.
Decision: Reject