Learning under Temporal Label Noise

Authors: Sujay Nagaraj, Walter Gerych, Sana Tonekaboni, Anna Goldenberg, Berk Ustun, Thomas Hartvigsen
Primary area: learning on time series and dynamical systems
Keywords: label noise; time series; healthcare; classification

Abstract:
Many time series classification tasks, where labels vary over time, are affected by label noise that also varies over time. Such noise can cause label quality to improve, worsen, or periodically change over time. We first propose and formalize temporal label noise, an unstudied problem for sequential classification of time series. In this setting, multiple labels are recorded over time while being corrupted by a time-dependent noise function. We first demonstrate the importance of modeling the temporal nature of the label noise function and how existing methods will consistently underperform. We then propose methods to train noise-tolerant classifiers by estimating the temporal label noise function directly from data. We show that our methods lead to state-of-the-art performance under diverse types of temporal label noise on real-world datasets.

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 5

Reviews and discussion:
<thread>
creation date:
1730039792556

thread_type:
official_review

<thread_object>
creator: reviewer_eoq6

<content>
summary:
The paper introduces the problem of temporal label noise, where label quality fluctuates over time due to time-dependent noise. This type of noise may improve, deteriorate, or change periodically, impacting the accuracy of predictions. The authors define this problem and highlight that existing methods fail to handle the temporal aspect of label noise effectively. They propose methods to estimate the time-varying noise function directly from data, allowing the development of noise-tolerant classifiers. Experiments on real-world datasets validate their proposal, demonstrating its effectiveness under various types of temporal label noise.

strengths:
1. The problem of Temporal Label Noise is both interesting and important, and this paper is the first to formally define it.

2. The paper is well-organized, clearly written, and easy to follow.

3. The mathematical symbols and equations are well-defined and easy to understand.

weaknesses:
1. My primary concern is with the problem formulation, or more specifically, the goal of the problem. In Section 2.1, the paper aims to "estimate parameters $\hat{\theta}$ for a model robust to noise," which implicitly assumes the existence of a **fixed and static optimal model** $\theta^\star$—i.e., that p(y_t|x_t) remains consistent across all timestamps. However, in time-series problems, the environment is constantly changing, and the optimal model for different timestamps should also be changing.

2. The other major concern relates to the methodology. The paper appears to assume that the function Q is known to the learner, and focuses only on optimizing the parameter w. However, I think that **obtaining an appropriate function Q is the most challenging aspect of this problem**. Although in Section 3.2, the authors select Q as a fully connected neural network as a general form for the practical scenarios, it seems to be inconsistent with the earlier discussions in Table 1.

3. Another concern is about the experiments. The authors seem to only conduct experiments on low-dimensional datasets. I think that including experiments on larger-sized datasets would strengthen the results and make the findings more convincing.

Please let me know if I have misunderstood any part of the paper.

questions:
See Weaknesses above
</content>
</thread_object>

</thread>
<thread>
creation date:
1730607124683

thread_type:
official_review

<thread_object>
creator: reviewer_qqux

<content>
summary:
The paper proposes a model to incorporate varying noise rates for time-series classification. The method specifies a matrix-valued function indicating label-noise distribution. This noise function is incorporated into a forward temporal loss over noisy labels that is used to learn a classifier (using a neural net) minimizing the loss. The authors describe three approaches: continuous, discontinuous, and plug-in. They empirically evaluate these approaches in various benchmark datasets. They also demonstrate the approach in a real-world dataset within the healthcare domain in which the authors claim that the problem of temporal noisy labels is prevalent in time-series classification. The empirical results clearly indicate that the proposed approach works well compared to SOTA static approaches that treat noisy labels as static rather than temporal.

strengths:
1) The formulation of the proposed methodology is clear and concise.
2) The motivation and subsequent real-world example is well demonstrated.

weaknesses:
(1) It is not clear how is learning achieved when weights are initialized for each time step in discontinuous estimation. It looks like the dataset is small for the learning models to converge as all the datasets largely seem to have less than 1000 samples.
(2) Can the estimator in the discontinuous case be replaced by a simpler model (with lesser number of parameters)? If so, can such an alternative be used as baseline mechanisms to compare for temporal noise (as all alternative mechanisms shown in the paper are for static cases)? 
(3) It is interesting that both continuous and discontinuous estimations perform quite similarly on the real-world stress detection example. Does this mean that the proposed approach works best when there is large variance in noise rates?

questions:
(1) Is there an assumption that all data instances have equal number of time steps?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730660870810

thread_type:
official_review

<thread_object>
creator: reviewer_nmrx

<content>
summary:
The paper discusses the problem of learning from time series classification task, where the label noise can vary over time. Each instance is a sequence over T time steps, where at iteration $t$ we have access to the features at time $t$, and a noisy label $\tilde y_t$ obtained from the true label $y_t$. Each of this instance is i.i.d., and the assumption is that the true label $y_t$ at time $t$ only depends on the past features $x_{1:t}$, and the noisy labels are conditionally independent to the features given by the true labels. In particular, the authors assume there exists a noise mapping $Q_t$ that describes the noise process at time $t$. They propose a novel method to learn a classifier that maps sequences $x_{1:t}$ to a label $y$. This model simultaneously approximates the noisy mapping $Q_t$ (Eq 3), that is used to approximate the true labels, which are used during training

strengths:
Learning from time series data is an interesting problem. The paper addresses a challenging problem, where the label noise can change over time, and it is well-motivated. Overall, the introduction of the paper and the experimental sections are well-written and easy to follow. I found the use of the minimum-volume simplex assumption as an objective to solve the problem to be intriguing.

weaknesses:
(**important**). The appendix of the paper is only a draft and it looks like it was not finished to be written. It also seems to be dissociated to the main paper.
- Appendix A is self contained and not referenced in the main paper. It is not discussed what the content of this section adds as a contribution.
- Appendix B looks dissociated to the main paper. First of all, three assumptions are introduced (they seems related to the 2 assumptions used in the main paper. In this case, why re-introduce them, and why do we use 3 assumptions rather than 2?). Most of the proofs are just a sequence of mathematical equations with text. Lines 810 to 840 are just a sequence of equations without text (maybe do a table?). The proof of Proposition 1 does not appear, which is only the theoretical result in the main paper (is Theorem 3 the proof of Proposition 1? Why does it have  different notation?).
- Section G.2 is empty. Page 25 to 47 include a sequence of a lot of figure without almost no text. I recommend the authors to only include the figures that are used to “say” something, together with a text explaination.

 The technical section is also sometimes unclear.  

Notations is sometimes unclear. What is q_t in line 140? According to the notation defined in the paper $\mathcal{X} \subseteq \mathbb{R}^{d \times T}$, but the function $h_{\theta}$ that has domain $\mathcal{X}$ can also have an input matrix $d \times t$. 
 
There is no comment on the assumptions 1 and 2 (except that they are two standard assumptions). I believe a few lines commenting on those assumptions would be helpful.

In lines 259-261, why is the Frobenius norm a convex surrogate for the volume? (A citation is also probably needed).  

In Line 260, should $R_t$ be defined over $\tilde{y}$ rather than $y$?

 In lines 262-264, my understanding is that $\lambda$ is the Lagrange multiplier of the constraint of Equation~2. I would be clearer on this, since $\lambda$ does not appear in Eq 2.

questions:
See also weakness.

The model does not use the noisy label in the prediction (the predictor h_{\theta} only depends on x_{1:t}).  I believe it would be interesting to include the noisy label in the prediction of the true label (i.e., if the noisy label is always accurate, can we use it for prediction? I understand this is a slightly different setting, as the goal of the paper is to learn a predictor).   

The fact that we learn a noisy label structure Q_w(t) that is “continual” over time seems implicit in Eq. (3). However, it is not clear what the difference is between Eq(3) and the model discussed in “discontinuous estimation”. It seems to me that $t$ is an integer, and $Q_w(t)$ could be completely different than $Q_w(t+1)$ in principle (what is the temporal relationship discussed in line 274)

It is a bit unclear how Section 2, in particular section 2.2, is related to the proposed method in Section 3. It seems to me, that the “loss of the classifier” is embedded in the constraint of Equation 2. Lines 233-234 says that Eq(2) minimizes the forward temporal loss as in Def 2, but this loss does not actually appear on Equation 2.   

Typos: 277 continuuity
</content>
</thread_object>

</thread>
<thread>
creation date:
1730790684502

thread_type:
official_review

<thread_object>
creator: reviewer_gnk9

<content>
summary:
The manuscript introduced temporal label noise in time series classification tasks and proposed a novel framework that are robust to it. Experiments were conducted on 4 datasets with synthetic noise injection and a real-world noisy labelled dataset, in which the method proposed in the manuscript is superior to other methods. The authors also visualized the dynamic evolution of label noise over time.

strengths:
1.	The problem proposed by the authors is novel and interesting, and they evaluate it on real-world datasets.
2.	The method introduced in the manuscript is cleverly crafted and has a theoretical foundation.

weaknesses:
1.	In many settings, the performance of Plug-In method is inferior to that of static methods.
2.	Only one of the five datasets in the manuscript contains both clean and noisy labels. In the remaining real-world scenarios, there is no experimental evidence to confirm that “the label noise evolves over time” actually exists.
3.	The experimental scenarios are focused on healthcare; validating the work in more diverse scenarios would broaden its applicability.

questions:
1.	What is the meaning of variable d in line 126? Is that mean a multivariate time series with d variables?
2.	When performing a train-test split on the datasets, were individuals also split?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732244439494

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Response

comment:
We thank all reviewers for their time and feedback!

We are pleased that reviewers recognized the **“novel”**, **“interesting”**, and **“challenging”** [gNk9, EoQ6, Nmrx] problem of temporal label noise introduced in our work.  Reviewers appreciated our **“cleverly crafted”** [gNk9] framework, which leverages a **“theoretical foundation”**[gNk9] and **“clear and concise methodology”**[qqUX]. Overall, reviewers found our paper **“well-organized and clearly written”**[EoQ6] with experiments **“demonstrat[ing] practical utility”**[qqUX] in real-world datasets. Additionally, reviewers noted the **“valuable empirical results”**[qqUX] that illustrate our approach’s robustness across various temporal noise settings and datasets.

We have responded to each reviewer’s comments individually and have also uploaded an updated version of the paper incorporating feedback from all reviewers. A brief summary of the changes is as follows:
* We have made changes to the paper based on reviewer feedback for clarity. In particular:
  * Discussed the role of Assumptions 1 and 2 and their intuitive meaning
  * Sign-posting in Section 3 to motivate the procedures we construct to learn the temporal noise function from noisy data
  * Further clarification on the difference between Continuous and Discontinuous methods in Section 3.4
* We have polished the Appendix based on reviewer Nmrx’s suggestions. This includes moving extra supplemental figures not referenced in the text to our anonymous repo, and making the notation/proofs easier to follow.
* We have added a Limitations and Future Work section, which addresses questions from reviewers and also outlines future research directions that our work can inspire.

Overall, we believe incorporating this feedback has improved the paper. We thank the reviewers for helping us further refine the paper and look forward to answering any remaining questions over the coming days.

– 

If you feel we have not sufficiently addressed your concerns to motivate increasing your score, we would love to hear from you further on what points of concern remain and how we can improve the work in your eyes. Thank you again!
</content>
</thread_object>

</thread>


Metareview:
metareview:
A fairly good paper that should be accepted for publication at ICLR. I hope this paper can advance the research area label-noise learning.

My only comment is about related work. The standard non-temporal label noise can be regarded as a type of distribution shift, where the test distribution is clean and the training distribution changes to some noisy version. Then, the temporal label noise is actually a type of continuous distribution shift, where the change from test to training is also time-dependent. Even though continuous distribution shift focused on continuous covariate shift rather than continuous class-posterior shift (i.e., label noise), it is still related to temporal label noise as a bigger topic covering the problem under consideration. However, the term distribution shift didn't appear at all. I hope you can acknowledge that your temporal label noise is a special case of continuous distribution shift, so that you have also contributed to distribution shift research in addition to label-noise research.