Learning under Temporal Label Noise
Authors: Sujay Nagaraj
Walter Gerych
Sana Tonekaboni
Anna Goldenberg
Berk Ustun
Thomas Hartvigsen
Venue: ICLR 2025 Poster (2025)
Keywords: label noise; time series; healthcare; classification
Primary area: learning on time series and dynamical systems

Abstract: Many time series classification tasks, where labels vary over time, are affected by label noise that also varies over time. Such noise can cause label quality to improve, worsen, or periodically change over time. We first propose and formalize temporal label noise, an unstudied problem for sequential classification of time series. In this setting, multiple labels are recorded over time while being corrupted by a time-dependent noise function. We first demonstrate the importance of modeling the temporal nature of the label noise function and how existing methods will consistently underperform. We then propose methods to train noise-tolerant classifiers by estimating the temporal label noise function directly from data. We show that our methods lead to state-of-the-art performance under diverse types of temporal label noise on real-world datasets.

Published as a conference paper at ICLR 2025
LEARNING UNDER TEMPORAL LABEL NOISE
Sujay Nagaraj' Walter Gerych²
Sana Tonekaboni
Anna Goldenberg'
Berk Ustun"
Thomas Hartvigsen
Universiiy of Toronto
2MIT Broad Institute
+UCSD
University of Virginia
Corresponding authors: s s.nagarajjmaa wtoronto. ca, hartviggenévirginia... edu
ABSTRACT
Many time series classification tasks, where labels vary over time, are affected by label noise that also varies over time. Such noise can cause label quality to improve, worsen, or periodically change over time. We first propose and formalize temporal label noise, unstudied problem for sequential classification of time series. In this setting, multiple labels are recorded over time while being corrupted by time-dependent noise function. We first demonstrate the importance of modeling the temporal nature the label noise function and how existing methods will consistently underperform. We then propose methods to train noise-tolerant classifiers by estimating the temporal label noise function directly from data. We show that methods lead state-of-the ahr performance under diverse types of temporal label noise on real-world datasets.'
INTRODUCTION
Supervised learning datasets often contain incorrect, or noisy, observations of ground truth labels. Such label noise commonly arises during human annotation [1, 28] when labelers lack expertise [31, 77]. labeling is hard [13, 30, 77], labels are subjective [49, 57, 62], or during automatic annotation where there are systematic issues like measurement error [32, 52). These factors, among others, make label noise prevalent in supervised learning. Label noise is key vulnerability in supervised learning [17, 25, 75J. Intuitively, models trained with noisy labels may learn to predict noise- such models then underperform at test time when attempting to predict the ground truth [26, 42].
Label noise has been studied extensively, producing a stream of works on static prediction tasks where we predict single outcomes [3, 43, 45, 67]. Many non-static prediction tasks (i.c., time series) have features and labels that evolve over time. these cases, it is only natural for label noise to also evolve over time label noise can be temporal. This notion of time-varying noise has yet to be formalized, yet it is found in numerous real-world tasks. For example:
Human Activity Recognition. Wearable device studies often ask participants to annotate their activities (e.g., exercise) over time. Participants often mislabel their activities due recall bias, time of day, or labeling-at-random for monetized studies [27, 61].
Reff-Repotted Outcomes for Mental Health: Mental health studies often collect self-reported survey data (e.g., depression) over long periods of time. Such self-reporting is known to be biased [5, 53, 55, 72, 46), as participants are more less likely report certain outcomes. For example, the accuracy of self-reported alcohol consumption is often seasonal [8].
Clinical Measurement Error: Clinical prediction models often predict outcomes (c.g., mortality) de- rived from clinician notes in electronic health records. These labels may capture noisier annotations during busier times e.g., when a patient is deteriorating [76].
Addressing the problem of temporal label noise is challenging from a technical perspective. Existing methods for learning under static label noise [48, 50] are not flexible enough to account for noise rates varying over time these approaches have no mechanism to capture temporal label noise and therefore implicitly misspecify the noise model. Additionally, the label noise function, temporal or otherwise, is often unknown; most datasets lack an indication for which instances or time steps are
'lll of our code is available https:/gginubbcco//saayyggggggg
abelNoise

---

Published as a conference paper at ICLR 2025
Standerrd supervized leamning igmores label noise. Existing mothods assume sgnoring temporal nature
Accurataly modtilnng
function,
nolse-tolerant
50
True Noise
Static Mathods
Reconstruction Eror %
30
Ignore Nolee
Temporal Methods
Test Errrr %
120
Noise
15
13
10-
10
5-
Iyrore Nolse
Stallc
Temporal
0
Noiew
Sali:
Temporn
Time
* Best Performing
Figure 1: In time series tasks, label quality can vary over time due temporal label noise. Existing methods assume noise can perform poorly they modeling temporal label noise can improve performance. improvements on an activity recognition task where we compare reconstruction error and accuracy between static and temporal methods subject to 30% temporal label noise across 10 draws (see results for moving dataset in Appendix D.4 for details).
static noise model time. We propose that accurate demonstrate this by showing performance
more likely to be accurate. Another practical limitation is the lack of ground truth data in real- world noisy datasets.
To address these challenges, we propose objective functions for sequence- vo-sequence classification tasks in the context of time series data. These objectives learn the temporal noise function directly from time series data with temporally noisy labels. Our methods leverage temporal loss function that robust temporal label noise. Our algorithms can be used out of the box and allow practitioners to learn noise-tolerant time series classifiers, even with unknown, temporal label noise. Our main contributions
1. We formalize the problem of learning from noisy labels in temporal settings.
We propose a novel loss function for training models that are robust to temporal label noise. On its own, show this can already be used improve prior methods.
3. We develop suite of methods to learn time series classifiers under temporal label noise. Our methods can overcome the practical challenges of this regime - i.e. by estimating an unknown noise function from time series data.
We show the consequences of accounting for temporal label noise by simply using existing methods for noisy labels these methods perform poorly under temporal noise. Our results highlight the necessity of accounting for temporal noise, our methods lead more noise-robust classifiers temporal or otherwise.
2 RELATED WORK
Our work is motivated by a series problems that arise in machine learning in healthcare [20, 40, 46]. Existing approaches primarily rely on supervised learning model changes in latent states over time [14, 35, 64, 68, 73, 85]. In practice, these underlying latent states can reflect clinical states that patients can transition in and out of (e.g., sick vs healthy). Existing approaches for this problem typically rely modeling sequences with deep neural networks [11, 65] and using attention- based mechanisms prioritize relevant segments of data [71]. An understudied problem this domain is label noise what happens when inaccurate labels arise in real-world time series datasets?
The vast majority of work on noisy labels studies label noise static prediction tasks where we predict single outcome and where label noise cannot change over time [3, 43, 45, 67, 47] We consider how label noise can arise time series a prediction task where the noise rates can change over time. Recent work in this setting focuses identifying noisy instances in time series, specifically by exploiting the notion that labels at neighboring time steps are unlikely to be corrupted together [see e.g., 2, 12]. In contrast, robust to temporal label noise. With respect to existing work, our approach can account for label noise in tasks where labels at neighboring time steps more likely to be corrupted together (e.g., due
focus developing algorithms to train models that
2

---

Published as a conference paper at ICLR 2025
seasonal fluctuations in annotator error). We develop methods for empirical risk minimization with noise-robust loss functions [c.f., 21, 38, 43, 48, 74]. Our work establishes the potential to learn from noisy labels in this setting when we have knowledge of the underlying noise process [c.f., 48, 50], and when we are able to learn the noise model from noisy data [see e.g., 37, 38, 50, 79, 82, 86).
3 FRAMEWORK
Preliminaries We consider temporal classification task over C classes and T time steps. Each instance is characterized by triplet of sequences over T time steps (1:T. Vi.r) E X x y x Y' where X' C RdxT represents multivariate time-series and y (1...., C) represents a sequence of labels. Here, XI.T. VI.7, and Y1.7 are sequences of instances, clean labels and noisy labels, respectively. For example, we can capture settings where XI.T. YI.7 are recordings from an accelerometer with true activity labels, and V1:7 represent noisy annotations of activity.
Under temporal label noise, the true label sequence Y1:7 is unobserved, and we only have access to a set of noisy instances D {(x1:I7 (1:r):) We assume that each sequence in D is generated i.i.d. from a joint distribution Px Mumr-rr where the label noise process can vary over time. This distribution obeys standard assumptions in temporal modeling and label noise [4, 10, 69]:
Assumption 1 (Future Independence), label at time depends only on the past sequence of feature vectors up to t: p(y1:r X1:r) = I p(y,(x1a)
Assumption 2 (Feature Independence). The sequence of noisy labels conditionally independent of the features given the true labels: y1:t 1. X1:t Y1:t 1.....,,
These assumptions are relatively straightforward, as they require that the current observation is independent of future observations and assume feature-independen noise regime. Assumption 1 allows the joint sequence distribution to factorize as: p(ý1.r x1.1) I 9e (f Here, introduce qt, a slight abuse of notation, to denote a probability that is time dinendepnn.nnt. Assumption allows for the noisy label distribution / to be further decomposed as:
(x(F,((1.))
Σ 9e(3, y, --)pyy, x1:))
BEy
3.1 LEARNING FROM TEMPORAL LABEL NOISE
Our goal is to learn temporal classification model hg Rdxt RC with model parameters @ E θ and < T. Here, ha (X1:e) returns an estimate of (xt)) ). To infer the label at time step t, he takes input a sequence of feature vectors up to I, and outputs sequence of labels by taking the arg max of the predicted distribution for each time step (see e.g., [4, 69]). We estimate parameters for model robust to noise, by maximizing the expected accuracy as measured in terms of the clean labels:
argmaxx E. Eysrbbrr 9E0
II ply, he(x1:t) x1:t)
However, during training time we only have access to sequences of noisy labels. To demonstrate how we can sidestep this limitation, we first need to introduce flexible way to model noise rates varying over time. Existing methods assume that noise is time-invariant (Fig. 1). To relax this assumption, we capture the temporal nature of noisy labels using a temporal label noise function, in Def. 1.
Definition Given a temporal classification task with C classes and noisy labels, the temporal label noise function is matrix-valuee function 12 + [0, C| that specifies the label noise distribution at any time > 0.
We denote the output of the temporal noise function at time t Q, == Q(t). This is a C' x C matrix whose i, 5 entry encodes the flipping probability of observing a noisy label j given clean label at time t: qe (5, ensuring that Q encodes a valid probability distribution [38, 50].
y. = i). We observe that Q, is positive, row-stochastic, and diagonally dominant
Qu denotes temporal noise function parameterized by function with parameters This parame- terization can be constructed to encode essentially any temporal noise function. As shown Table 1, can capture wide variety of temporal noise using this representation.
3

---

Published as a conference paper at ICLR 2025
Pattern
Depiction
Noise Model Qu
Parameters
Applications
My (ogy)dy) controls frequency
Annotation reliability varies over time of day egg. due to changes annotator attentiveness day [16].
Periodic
2
sin(nyy dg)
d controls shift
Annotation reliability improves rapidly (e-g.. agnostic sccuracy COVID-19 rapilly improved at the onset of the pandemic [SI or improves and plateaus (e.g., ireducible uncertainty in bels [54])
(u) controls inirial noise
Decay
Ony expl Agg)
B controls decay rate
Wi (mey Piy Niy))
Annotation reliability decreases ahruptly e-g- due to adoption improved clinical guide- lines [66]. Or, decrenses gradually egg. due to underdiagnosis during the pandemic due bealth- care disruptions [18]
controls limit
Growth
B controls growth rate
controls inflection point
Table 1: Overview of temporal noise functions for time series classification tasks (see Appendix D for other examples). We show the noise model (t)) and parameters w to model g(), j Y, = i). Practitioners can account for temporal variation by choosing parametric class capture effects and fit parameters from data.
3.2 Loss CORRECTION
Modeling temporal label noise is the first piece of the puzzle in training time series classifiers robust to temporal label noise. However, we still need to consider how to leverage these noise models during empirical risk minimization. It remains unclear if and how existing loss correction techniques work for time series. Here, we present theoretical results showing that learning is possible in our setting when we know the true temporal noise function (t). We include proof's Appendix A.
We begin by treating the noisy posterior as the matrix-vector product of a noise transition matrix and clean class posterior Eq. (1). To this effect, we define the forward temporal loss:
Definition Given a temporal classification task over T time steps, a noise function Q(t., and proper composite loss function le [58), the forward temporal loss of model ha on an instance (h:T,aa1:7) is:
7 7sn (Drr, ho)
Σ le(Te, Q. ho(m))))
An intriguing property of the forward temporal loss is that its minimizer over the noisy labels maximizes the likelihood of the data over the clean labels. This suggests that the forward temporal loss is robust to label noise:
Propusition 1. A classifier that minimizes the empirical forward temporal loss over the noisy labels maximizes the empirical likelihood of the data over the clean labels.
argmin Ey N Ni.r PEB
(e X1.T, he) = argmin 6EB
Biy (x(1x)))
1=1
Proposition I implies that train on the noisy distribution and learn noise-tolerant classifier in expectation.
METHODOLOGY
Our results the previous section show that we can train models that are robust to label noise. However, the temporal noise function Q is not available to the practitioner and must be learned from data. Our algorithm seeks to first learn this function from noisy data then account for noise using Def. 2. This strategy can be applied in multiple ways. In what follows, we propose method where we jointly learn the temporal noise function and the model. We also present extensions for different use-cases in Section 4.3 and discuss how to choose between methods in Section 4.4
4

---

Published as a conference paper at ICLR 2025
4.1 FORMULATION
We start with method that simultaneously learns a time series classifier and temporal noise function. Given noisy dataset, learn these elements by solving the following optimization problem Vt E []:
min Vol(Qu(t)) w,
ha = p(ie (1:e))
S.t.
Eq. (2) is designed to return faithful representation of the noise function by imposing the minimum- volume simplex assumption [38], and a noise-tolerant temporal classifier 0 by minimizing the forward temporal loss in Def.
Here, the objective minimizes the volume of the noise matrix, denoted as Vol(Q.). This returns a matrix Qt, at each time step, that oheys the minimum-volume simplex assumption, which is standard condition used to ensure identifiability of label noise in static tasks [see c.g., 38, 83]. In practice, this assumption ensures that Q, encloses the noisy conditional data distribution time t: p( x1:)). Containment ensures that the estimated noise matrix could have generated each point in the noisy dataset i.e. so that the corresponding noisy probabilities ply, X1:1) obey Eq. (1). Our use of this assumption guarantees the identifiability of Q, when the posterior distribution is sufficiently-scatteeeed over the unit simplex [see also 38, for details].
4.2 ESTIMATION PROCEDURE
The formulation above applies to any generic matrix-valued function according Def. with parameters w. Because time series classification tasks can admit many types of temporal label noise functions (Table 1), we must ensure w has sufficient representational capacity to handle many noise functions. Therefore in practice, we instantiate our solution as a fully connected neural network with parameters w, (-) : + [0, 11 adjusted to meet Def. (see Appendix D.3 for implementation details). We can now model any temporal label noise function owing to the universal approximation properties of neural networks.
Provided the function space @ defines autoregressive models of the form p(ye(a1:)) (e.g.. RNNs, Transformers, etc.), we can solve Eq. (2) using an augmented Lagrangian method for equality- constrained optimization problems [6]:
[(0,w)= = +
T Σ [10,00), (R)(O,w) + +RR((w))
( SIR((O,w)P)
Here: (()ll denotes the Frobenius norm of Qw which acts convex surrogate for (t) [7]. Likewise, R0, w) = - (T (t) he(#1:.)) denotes the violation of the equality constraint for each t 1...T. X E R. is the Lagrange multiplier and e > 0 is penalty parameter. Both these terms are initially set default value of 1, we gradually increase the penalty parameter until the constraint holds and X converges to the Lagrangian multiplier for the optimization problem Eq. (2) [6]. This approach recovers the best-fit parameters to the optimization problem in Eq. (2), We call this approach Continuous Estimation and provide additional details on our implementation in Appendix B.
4.3 ALTERNATIVE APPROACHES
We describe two alternative approaches that extend existing label noise methods temporal settings and may be valuable to use in tasks where we cannot assume continuity or have access to additional information.
Discontinuous Estimation Another approach is to assume that there is no temporal relationship between each Q: across time and treat each time step independently. This approach is well suited for tasks where time steps unevenly spaced (e.g., some clinical data involves labels collected over years or at different frequencies [56]). We can address such tasks through an estimation procedure where learn model that achieves the same objective without assuming continuity. In contrast to
5

---

Published as a conference paper at ICLR 2025
Eq. (3), this approach Q is parameterized with separate set of trainable real-valued weights, fitting the parameters at each time step using the data from that time step. This approach can generalized to extend any sfate-of-the-at technique for noise transition matrix estimation in the static setting (e.g.. Li et al. [38], Yong et al. [83], etc.).
also advantageous have simple, plug-in estimator of the temporal noise function. Plug-in estimators are model-agnostic, can flexibly be deployed to other models, and can be efficient estimate. We can construct plug-in estimator of temporal label noise using anchor points, instances whose labels are known to be correct. Empirical estimates of the class probabilities of anchor points can be used estimate the noise function. This estimate can then be plugged into Def. 2 to train noise-tolerant time series classifier. Formally, in temporal setting, anchor points [41, 50, 79] are instances that maximize the probability of belonging to class at time step
Plug-In Estimation
arg max p(), = #).;))
Since ply, = xi.)) R for the clean label, we can express each entry of the label noise matrix as:
Q(t),, = p(x, = xi.))
(5)
We construct a plug-in estimate of (t) using two-step approach: we identify anchor points for each class and time step / = 1...., T and replace each entry of Q()) with the empirical estimate defined in Eq. (5) This is generalization of the approach of Patrini et al. [50] developed for static prediction tasks.
4.4 DISCUSSION
All three methods in Section 4.2 and 4.3 improve performance in temporal classification tasks by accounting for temporal label noise (see e.g.. Fig. and Section 5). However, they each have their strengths and limitations. Here we provide guidance for users to choose between methods:
Continuous estimation imposes continuity across time steps assuming that nearby points likely have similar noise levels i.e., (() = (t + 1). This assumption can improve reconstruction in tasks with multiple time steps by reducing the effective number of model parameters. Conversely, it may also lead to misspecification in settings that exhibit discontinuity.
Discontinuous estimation noise levels of nearby points are independent of one another. This approach requires fitting more parameters, which scales according T. This can lead to computational challenges and overfitting. especially for long sequences.
handle discontinuous temporal noise processes assuming the
Plug-In In estimation has simpler optimization problem. This is useful in that use separate datasets to estimate the noise model and to train the classifier. practice, the main challenges in this approach stem from verifying the existence of anchor points [79].
EXPERIMENTS
We benchmark methods collection of temporal classification tasks from real-world applica- tions. Our goal is to evaluate methods in terms of robustness to temporal label noise, and characterize when it is important consider temporal variation in label noise. We include additional details setup and results in Appendix D and code to reproduce our results on GitHub.
5.1 SETUP
We work with four real-world datasets from healthcare. Each dataset reflects binary classification tasks over complex feature space, with labeled examples across multiple time steps, where the labels are likely to exhibit label noise. The tasks include:
1. moving: human activity recognition task where we detect movement states (e.g., walking vs. sitting) using temporal accelerometer data adults [59].
2. seni or: similar human activity recognition task above but in senior citizens [44]

---

Published as a conference paper at ICLR 2025
3. sleeping: sleep state detection (e.g.. light sleep REM) task using continuous EEG data [22]
4. blinking: eye movement (open vs. closed) detection task using continuous EEG data [60] from continuous EEG data.
We consider semi-synthetic setup where we treat labels in the training sample ground truth labels and create noisy labels by corrupting them with a noise function for different noise rates. This setup reflects standard approach used to benchmark algorithms for learning under label noise [see e.g..
48, 50J. We consider six different label noise functions:
Linear: noise rates linearly fall over time.
Decay: noise rates decay exponentially over time.
Growth: noise rates rapidly rise over time.
Periodic: noise rates vary seasonal fashion.
Mixed: noise rates for Y: grow and noise rates for yr = fall over time.
Static: noise rates are constant over time.
We present two sample noise functions Fig. 2, and a complete list in Fig. 4 in Appendix D.
We split each dataset into noisy training sample (80% used to train the models and correct for label noise) and a clean test sample (20%, used to compute unbiased estimates of out-of-sample performance).
We train time series classifiers on the aforementioned datasets using the three techniques described in Section 4: Continuous, Discontinuous, and Plug-In. We compare these models to baseline models that ignore label noise (Ignore) to loss-tolerant methods that assume static noise model across time steps: Anchor [41, 50, 79] and VolMinNet [38].
We evaluate each model in terms of the following metrics:
Test Accuracy measures how well each method performs on held-out test data with clean labels. Approximation Error characterizes how well each method learns the temporal noise function, using the Error( (Qt, Q:) E Q1Q: Qrll between the true Q: and estimated Q, for all t.
5.2 RESULTS
We summarize our results for two temporal noise functions in Table and Table 3. We evaluate the performance of methods across different noise models and attribute the gains to ability to capture the underlying temporal noise function in Fig. 2. We include additional results for other noise models and for multi-class classification tasks Appendix D.4. In what follows discuss key results.
On the Performance Gains of Modeling Temporal Noise First, we show clear value in accounting for temporal label noise. Table shows the performance of each method on all four datasets. We
Perlodic
Mixed
50-
50
Noise Rate %
Noise Rate %
40
30:
30:
Continuous
20
20
Discontinuous
Discontinuous
Continuous
Plug-In
True
Plug-In
0
40
60
80
100
20
40
80
100
Time
Time
Figure Comparison of the ground unseen temporal noise function Q()) and its estimate (()) from each Temporal method on the moving data. We show the noise rate for the negative class only for clarity. We show results for Periodic and Mixed noise regimes. As shown, our Temporal methods can learn the true label noise function across different noise patterns. The resulting noise models have lower reconstruction error and superior to static approaches.
7

---

Published as a conference paper at ICLR 2025
Stotie
Temporol
Dutasel
Metric
Ignore
Anchor
VeiMinNet
Plug-In
Discontinuous
Continuous
moving [59]
Test Error
29.4 1.7%
20.9
2.6%
14.9 + 2.7%
20.0 + 1.8%
15.9 2.7%
4.2 + 2.2%
192, d' 14. 50
Apprux. Error
42.4 3,4%
35.3 L. 0.8%
36.8 1.7%
32.6 0.5%
10.3 1. 3.9%
sentor [44]
Test Frrr
22.7 + 1.7%
20.7 .01%
19.0 0.7%
18.8 + 1.1%
13.6 1.2%
11.0 + 0.3%
444, d' 6. 100
Approx. Error
35.9 2.5%
36.3 4: 0.4%
26.9 4: 1.2%
21.7 0.2%
6.4 :: 0.8%
blinking [60]
Test Error
34.1 2.0%
34.1 2.3%
29.6 2.2%
29.6 =: 2.8%
29.9 = 3.0%
29.6 :: 2.3%
290, d' 14, 50
Approx. Error
35.3 0.8%
35.2 + 0.7%
19.6 + 1,1%
26.6 = 0.9%
14.9 + 2.3%
sleeping [23]
Tost Error
28.7 = 0.8%
24.9 1.1%
26.8 = 1,4%
20.4 # 1.8%
19.6 0.8%
16.3 = 0.4%
964, d' 7, 7 100
Approx. Frm
34.3 1.8%
41.8 + 0.1%
19.1 3.5%
22.4 0.25%
4.9 + 0.5%
Table 2: Model performance and approximation error for all methods and datasets when labels are corrupted using a Mixed noise function (average 30% label noise across all time steps, one class with decreasing noise rate, one class with increasing noise rate). We report the clean test error (%) and mean approximation error of (()) + st.dev over 10 runs. The best pessforeformin methods are highlighted in Green. Continuous outperforms all baselines.
Static
Temporal
Dataset
Metric
Ignore
Anchor
VolMinNet
Plug-In
Discontinuous
Continuous
moving [59]
Test Error
24.0 1: 5.1%
18.0 =: 3.6%
13.5 1. 6.0%
18.5 1. 4.3%
14.0 =: 5.7%
1.7 1 0.6%
a 192, d 14. 50
Approx. Error
49.9 4.7%
43.5 # 3.0%
48.1 + 4.4
33.5 = 5.0%
9.1 + 1.6%
senier [44]
Test Erroe
22.5 +: 2.2%
20.1 1.5%
16.4 :: 2.9%
19.9 :: 1,38
14.1 = 2.6%
10.6 1: 0.7%
a= 4444 d= 100
Approx. Error
47.3 4.0%
42.4 + 3,9%
46.9 + 3,9%
20.5 = 2.1%
9.7 +: 2.7%
blinking [60]
Test Frmr
37.4++.119 3.1%
35.8 - 2.7%
33.2 + 2.8%
35.8 + 2.3%
32.0 - 2.9%
30.4 + 3.1%
290, d 14, T 50
Approx. Errr
49.3 4.2%
42.4 + 3.9%
46.4 + 4.4%
24.7 = 3.3%
10.5 + 3.3%
a aleeping ng [22]
Test Frmr
23.3 + 3.1%
22.8
2.7%
22.4 + 3.0%
22.1 + 2.0%
20.1 + 3.5%6
15.2 + 0.8%
161 d' = 7,1 100
Approx. Error
44.9
4.0%
42.6 1: 4.1%
43.9 1 4.3%
19.1 = 1.8%
8.8 :1: 2.6%
Table 3: Model performance and approximation error for all methods and datasets when labels are corrupted using the Periodic noise function (average 30% label noise across all time steps). We report the clean test error (%) and mean approximation error of (t) + st.dev over 10 runs. The hest-performing methods are highlighted in green. Continuous outperforms all basclines.
find that the temporal methods are consistently more accurate than their non-temporal counterparts, highlighting the importance of modeling temporal noise. For example, Plug-In is a temporal extension of Anchor, and we see that Plug-In outperforms Anchor in most settings, For example, on moving, the temporal method reduces the test error by over 10% compared to the nearest- peerresstpeforomngg static counterpart. This shows that our methods are robust to label noise despite being trained on data that is 30% corrupted with noisy labels, with no prior knowledge of the noise. This can have important consequences - e.g., in healthcare, where the noise unknown and accurate models support life-or-death decisions.
Among the temporal methods, Continuous achieves the best performance in comparison to Plug-In and Discontinuous. Continuous' superiority is even clearer when compared to the static methods. Comparing the results for Table 2 (which assumes a Mixed noise model) and Table 3 (which assumes Periodic noise model), we can see that these results hold across multiple types of temporal label noise. Fig. 2 demonstrates the ability of our methods to accurately reconstruct the underlying label noise. More importantly, the benefit of Continuous becomes more evident as the amount of noise increases in the data. In all these cases, we observe that temporal methods are consistently more robust to both temporal and static label noise. Overall, these findings suggest that we can improve performance by explicitly modeling how noise varies across time instead of assuming it is distributed uniformly in time.
On Learning the Temporal Noise Function Our results show that the performance increases hand in hand with our ability to estimate the noise model. In particular, we observe that low values of reconstruction consistently lead to low values of error rates.
We note that Continuous has the best reconstruction error (Approx. Error) on all datasets (Table 2). We qualitatively compare estimated noise functions and the ground truth for Continuous, Discontinuous,
8

---

Published as a conference paper at ICLR 2025
and other baselines in Fig. 2. We DNN to model Q. which consistently estimates the noise function with lower mean absolute error across different families noise functions. Appendix D.4 has extended results all other datasets (including multi-class).
On the Risk of Misspecification As shown in Section 3.2, our ability to learn under temporal label noise depends on our ability correctly specify the temporal noise process. However, existing Q Q-estimators assume static, time-invariant label noise which means that they will always fail to capture the correct noise function when temporal. In Fig. (Appendix D), we validate this experimentally by showing how model that assumes static noise compares to the forward temporal loss model learned using the true temporal noise function (sce Appendix D.4 for other results). Our results highlight that there is downside to modeling temporal effects. As shown, performance does not change when the temporal noise process is truly static. In contrast, performance drops when temporal effects exist. Using our temporal approaches they relax this assumption to admit different types of temporal label noise. There is no cost to doing so, as our methods are competitive even if the noise model is truly static.
DEMONSTRATION ON STRESS DETECTION
We now demonstrate the viability of our approach on real-world time series application where we have both clean and noisy labels. We focus on the task of stress detection, which is common feature modern smartwatches [9, 19] and is increasingly used to support interventions to mitigate clinical burnout [23, 29, 15]. Although many models self-reported stress labels, these labels are subject to temporal label noise which arises from subjectivity, forgetfulness, and seasonal patterns in self-reported outcomes [53]. Given these effects, objective physiological measures (e.g., heart rate variability, temperature, etc.) are the standard for measuring stress and guiding interventions.
Setup Our goal is learn a time series classifier using the noisy labels (i.e., self-reported stress labels) that will generalize the clean labels (i.e., physiological indicator of stress). We use dataset from Goodday [24] monitoring stress in healthcare workers using physiological measures and self-reported measures. The dataset contains sequence has 9 features. Here, the noisy label is: Di.t = if person self-reports stress on day t. The clean label is ys = 1 if the objective physiological measure of stress for person indicates stress on day t. We train sequential classifiers to predict stress over time. Our setup is identical to the one in Section 5, except that training labels reflect real-world noisy labels and we have clean labels representing the ground truth stress.
289 unique individuals over T = 50 days. Each
50%-
3iitt
Vi.t = 0
40%
Continuous
Nolse Rate
30%
True Discontinuous
20%
Continuous
Discontinuous
10%
Static
Static
30
0
10
20
30
40
50
Time Step
Time Step
Figure 3: Temporal effects in label noise in real-world stress detection task. We show noise rates when individuals stressed (left) and not stressed (right). True noise rate is the average disagreement rates between clean and noisy labels over time. We can see clear, temporal label noise patterns our temporal label noise methods do superior job of approximating it.
We report our results in Fig. 3 and Table 4. In Fig. 3, the black line indicates the average disagreement rates between the noisy self-reported label of stress and the clean physiological label over time. We can see clear temporal patterns in label noise, where participants under-report stress in seasonal manner.
Results
Our results are consistent with those in Section 5.2. In particular, we find that accounting for temporal label noise improves both training and test accuracy (Table 4). These results also highlight

---

Published as a conference paper at ICLR 2025
Noise Estimation
Model Performance
Method
What It Represents
Appros. Error
Train Error
Test Error
Ignore
ignoring label noise completely
17.6 0.0%
31.0 0.3%
Static
static label noise correction per time step
16.4 = 1.1%
31.0 + 0.3%
29.5 1: 4.0%
Discontinuous
temporal label noise correction with discontinuity
11.4 0.8
27.1
0.7%
26.0 2.5%
Continuous
temporal label noise correction imposing continuity
9.7 1.3%
21.1
0.5%
25.5 1.7%
Table 4: Noise rate estimation emror and model performance for stress detection. We train sequential classifiers predict stress using the training setup in Section 5. We report results for four methods highlight the mechanism driving performance improvements through an ablation study. We show the approximation error noise rates (left); and the clear label error rate on the training sample (middle) and test sample (right) All values correspond to the mean 10-CV estimates 1 st.dev.
the mechanism for these effects. Specifically, we see that lower approximation error leads to lower training error (via noise correction) and lower training error leads to lower test error (via generalization). this case, all model classes are the same and the gains in Continuous are driven by its ability learn the underlying temporal noise function in joint optimization procedure.
Discussion This demonstration shows the effectiveness of our methods in regimes without synthetic noise injection. We identified clean and noisy labels this dataset using physiological measures and self-reports, respectively. In this scenario, temporal label noise represents the time-dependent process that models the discrepancy between these labels.
CONCLUDING REMARKS
Temporal label noise is an under-explored problem facing many real-world time series tasks. In such tasks, we must predict sequence of labels, each of which can be subject to label noise whose distribution can change over time. Our work shows that existing methods for label noise substantially underperform when the distribution of label noise changes over time. As we show, we can achieve improved performance by accounting for temporal label noise. Given that this noise function is unknown, also highlight the viability of learning the noise model from data. Finally, perform a demonstration on a real-world source of temporal label noise self-reported labels in digital health studies. In this setting, methods continue to outperform methods that ignore such noise. Overall, temporal label noise presents a challenge in modeling time series our paper underscores the necessity of explicitly modeling and accounting for temporal noise achieve reliable and robust time series classification.
Limitations We assume that all time series in dataset have the same underlying label noise function. This may not be true different annotators can introduce differing noise patterns. We can potentially relax this assumption by considering models that represent differences across annota- tors [see e.g., 63, 36, 70]. Additionally, we work with stationary time series, where the predictive distribution is stable over time. Though this a standard assumption in time series work [see e.g.. 80, 81, 78, 84], it is limitation that can be addressed. For example, our loss loss-correction technique can be extended with learning theory from the study of non-stationary time series [33, 34). Finally, the lack of time series datasets with both noisy and clean labels few and far between. Without clean labels, it is difficult to verify that the temporal noise function learned is accurate or verify that model trained on noisy labels generalizes clean labels. We hope this paper will inspire the curation of more time series datasets containing both sets of labels.
10

---

Published as a conference paper at ICLR 2025
Acknowledgements This work was supported by funding from the National Science Foundation IIS 2040880, IIS 2313105, and the NIH Bridge2A1 Center Grant U54HG012510. SN was supported by CIHR Vanier Scholarship. AG is CIFAR AI Chair. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute. We also thank Abhishek Moturu and Vijay Giri for valuable discussions about the project.
REFERENCES
[1] Lora Aroyo and Chris Welty. Truth is a lic: Crowd truth and the seven myths of human annotation. Al Magazine, 36(1):15-24, 2015.
[2] Gentry Atkinson and Vangelis Metsis. Identifying label noise in time-series datasets. In Adjunct Pro- ceedings of the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of 2020 ACM International Symposium on Wearable Computers, pp. 238-243 2020.
[3] Eyal Beigman and Beata Beigman Klebanov. Learning with annotation noise. In Proceedings of the Joint Conference of the 47th Anmual Meeting of ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pp. 280 287, 2009.
[4] Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. neural probabilistic language model. Advances in Neural Information Processing Systems, 13, 2000.
Marianne Bertrand and Sendhil Mullainathan. Do people mean what they say? implications for subjective survey data. American Economic Review, 91(2)67777, 2001.
[6] Dimitri P Bertsekas. Constrained optimization and Lagrange multiplier methods. Academic press, 2014.
[7] Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[8] Christopher Carpenter. Seasonal variation in self-reports of recent alcohol consumption: racial and ethnic differences. Journal of Studies on Alcohol, 64(3):415-418, 2003.
[9] Jerry et al. Chen. Pain and Stress Detection Using Wearable Sensors and Devices-A Review. Sensors, 2021.
[10] Yo Joong Choe, Jaehyeok Shin, and Neil Spencer. Probabilistic interpretations of recurrent neural networks. P Probabilistic Graphicul Models, 2017.
[11] Edward Choi, Andy Schuetz, Walter F Stewart, and Jimeng Sun. Using recurrent neural network models for early detection of heart failure onset. Journal of the American Medical Informatics Association, 24(2): (61-370,220 2017.
[12] Beilci Cui, Minqing Zhang, Mengya Xu, An Wang, Wu Yuan, and Hongliang Ren. Rectifying noisy labels with sequential prior: Multi-scale temporal feature affinity learning for robust video segmentation. arXiv preprint arXiv:2307.05886 2023.
[13] Thomas G Day, John M Simpson, Reza Razavi, and Bernhard Kainz. Improving image labelling quality. Nature Machine Intelligence, 5(4):335-336, 2023.
[14] Huilong Duan, Zhoujian Sun, Wei Dong, Kunlun He, and Zhengxing Huang. On clinical event prediction in patient treatment trajectory using longitudinal electronic health records. IEEE Journal of Biomedical and Health Informatics, 24(7):2053 -0662333,, 2019.
[15] Daniel Ehrmann, Sara N Gallant, Sujay Nagaraj. Sebastian D Goodfellow, Danny Eytan, Anna Gold- enberg, Mjaye Mazwi. Evaluating and reducing cognitive load should be a priority for machine Icarning in healthcare. Nature medicine, 28(7):1331-13 1333, 2022.
[16] Simon Folkard. Diurnal variation logical reasoning. British Journal of Psychology. 66(1):1-8, 1975.
[17] Benoît Frénay and Michel Verleysen. Classification in the presence of label noise: a survey, IEEE Transactions on Neural Networks and Learning Systems, 25(5):845 869, 2013.
[18] Tejal K Gandhi and Hardeep Singh. Reducing the risk of diagnostic error in the covid- 19 era. Journal of Hospital Medicine, 15(6):363, 2020.
[19] Shruti et al. Gedam. A Review on Mental Stress Detection Using Wearable Sensors and Machine Learning Techniques. IEFF Access, 2021.
11

---

Published as a conference paper at ICLR 2025
[20] Marzyeh Ghassemi, Mike Wu, Michael C Hughes. Peter Szolovits, and Finale Doshi-Veler Pre- dicting intervention onset in the ICU with switching state space models. AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science, 2017:82 91, 2017. ISSN 2153-4063, URL http: /ttp:://wwwww wwww nebi www.ciinnlm......mmmmmm.mm pubmedcentral. .nh. wtt.g/ottileeeeegeeeeeeeeeeeeeee Feg1/3rti.d-9MM555533
[21] Aritra Ghosh, Himanshu Kumar, and Shanti Sastry. Robust loss functions under label noise for deep neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
[22] Any L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanow, Roger G Mark, Joseph E Mietus, George B Moody, Chung-Kant Peng, and H Eugene Stanley. Physiobank, physiotoolkit, and physionet: components of new research resource for complex physiologic signals. Circulation, 101 (23):e215 c220, 2000.
[23] Sarah M. et al. Goodday. Unlocking stress and forecasting its consequences with digital technology. npj Digital Medicine, 2019.
[24] Sarah M. et al. Goodday, An Alternativ the Light Touch Digital Health Remote Study: The Stress and Recovery in Frontline COVID-19 Hcalth Care Workers Study. JMIR Formative Research, 2021.
[25] Bo Han, Quanming Yao, Tongliang Liu, Gang Niu, Ivor W Tsang, James Kwok, and Masashi Sugiyama. A survey of label-noise representation learning: Past, present and future. arXiv preprint arXiv:201 1.04406, 2020.
[26] Hrayr Harutyunyan, Kyle Reing, Greg Ver Steeg, and Aram Galstyan. Improving generalization by controlling label-noise information neural network weights. In International Conference on Machine Learning, pp.
PMLR, 2020.
[27] Gary Hsieh and Rafal Kocielnik. You get who you pay for: The impact of incentives on participation bias. In Proceedings the 19th ACM Conference Computer-supporte Cooperative work & Social Computing, pp. 823 823-835, 835, 2016.
[28] Oana Inel, Khalid Khamkham, Tatiana Cristea, Anea Dumitrache, Ame Rutjes, Jelle van der Ploeg, Lukasz Romaszko, Lora Aroyo, and Robert-Jan Sips. Crowdtruth: Machine human computation framework for harnessing disagreement in gathering annotated data. In The Semantic Web-ISWC 2014: 13th International Semantic Web Conference, Riva del Garda, Italy, October 19-23, 2014. Proceedings. Part 11 13, pp- 486-504. Springer, 2014.
[29] Talha et al. lqbal. A Review of Biophysiological and Biochemical Indicators of Stress for Connected and Preventive Healthcare. Diagnostics, 2021.
[30] Neetha Jambigi. Tirtha Chanda, Vishnu Unnikrishnan, and Myra Spiliopoulou. Assessing the difficulty of labelling an instance crowdworking. Workshops of the European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 363-373. Springer, 2020.
[31] Hyun Jung. Yubin Park, and Matthew Lease. Predicting next label quality: A time-series model of crowdwork. In Proceedings of the AAAI Conference Human Computation and Crowdsourcing, volume 2, pp. 87-95, 2014.
[32] Takuya Kiyokawa, Keita Tomochika, Jun Takamatsu, and Tsukasa Ogasawara. Fully automated annotation with noise-masked visual markers for deep-learning-basee object detection. IEEE Robotics and Automation Letters, 4(2): 2019.
[33] Vitaly Kuznetsov and Mehryar Mohri. Generalization hounds for time series prediction with non-stationary processes. Algorithmic Learning Theory: 25th International Conference, ALT 2014, Bled, Slovenia, October 8-10, 2014. Proceedings 25, pp. 260-274. Springer, 2014.
[34] Vitaly Kuznetsov and Mehryar Mohri. Learning theory and algorithms for forecasting non-stationary time series. Advances in neural information processing systems, 28, 2015.
[35] Jeong Min Lee and Milos Hauskrecht. Modeling multivariate clinical event time-series with recurrent temporal mechanisms. Artificial intelligence in medicine, 112: 10201, 2021.
[36] Shikun Li, Shiming Ge. Yingying Hua. Chunhui Zhang, Hao Wen, Tengfei Liu, and Weigiang Wang. Coupled-view deep classifier learning from multiple noisy annotators. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 4667-4674, 2020.
12

---

Published as a conference paper at ICLR 2025
[37] Shikun Li, Xiaobo Xia, Hansong Zhang, Yibing Zhan, Shiming Ge. and Tongliang Liu, Estimating noise transition matrix with label correlations for noisy multi-labo learning, Advances Neural Information Processing Systems, 35:24184 0224418-24498,, 2022.
[38] Xuefeng Li, Tongliang Liu, Bo Han, Gang Niu, and Masashi Sugiyama. Provably end-to-end lahel-noise learning without anchor points. International Conference on Machine Learning, pp. 6403-6413. PMLR, 2021.
[39] Zachary Lipton, David Kale, Charles Elkan, and Randall Wetzel. Learning to diagnose with Istm recurrent neural networks. arXiv preprint arXiv:1511.0307 2015.
[40] Zachary Lipton, David Kale, and Randall Wetzel. Directly modeling missing data sequences with mns: Improved classification of clinical time series. In Machine learning for healthcare conference, pp. 253-270. PMLR, 2016.
[41] Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. JEEE Transactions on Pattern Analysis and Machine Intelligence, 38(3):447-461, 2015.
[42] Yang Liu. Understanding instance-level label noise: Disparate impacts and treatments. In International Conference on Machine Learning, pp- 6725-6735. PMLR, 2021.
[43] Yang Liu and Hongyi Guo. Peer loss functions: Learning from noisy labels without knowing noise rates. In International Conference on Machine Learning, pp. 6226-6236. PMLR, 2020.
[44] Aleksej Logacjov and Astrid Ustad. HAR70+ UCI Machine Learning Repository, 2023. DOL:
[45] Philip M Long and Rocco A Servedio. Random classification noise defeats convex potential boosters. In International Conference on Machine Learning, pp. 608 615,61152222 2008
[46] Sujay Nagaraj, Sarah Goodday, Thomas Hartvigsen, Adrien Boch, Kopal Garg, Sindhu Gowda, Luca Foschini, Marzych Ghassemi, Stephen Friend, and Anna Goldenberg. Dissecting the heterogeneity of "in the wild" stress from multimodal sensor data. NPJ Digital Medicine, 6 6(1):237, 2023.
[47] Sujay Nagaraj, Yang Liu, Flavio Calmon, and Berk Ustun. Regretful decisions under label noise. In The Thirleenth International Conference on Learning Representations, 2025. URL https:://oppnttt /pppnreviex.. net/ / forum? idd dd TB
[48] Nagarajan Natarajan, Inderjit Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. Advances in Neural Information Processing Systems, 26, 2013.
[49] Matthias Norden, Oliver Wolf, Lennart Lehmann, Katja Langer, Christoph Lippert, and Hanna Drimalla. Automatic detection of subjective, annotated and physiological stress responses from video data. In 10th International Conference on Affective Computing and Intelligent Interaction (ACII), pp. 1-8, IEEE, 2022.
[50] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition. pp. 1954-1952. 2017.
[51] Hanna M Peacock, Cindy De Gendt, Geert Silversmit. Sandra Nuyts. Jan Casselman, lean-Pascal Machiels, Francesco Giusti. Bart Van Gool, Vincent Vander Poorten, and Liesbet Van Eycken. Stage shift and relative survival for head and neck cancer during the 2020 covid-19 pandemic: a population based study of temporal trends. Frontiers in Oncology, 13, 2023.
[52] Kenneth A Philbrick, Alexander D Weston, Zeynettin Akkus, Timothy L Kline, Panagiotis Korfiatis, Tomas Sakinis, Petro Kostandy, Arunnit Boonrod, Atefeh Zeinoddini, Naoki Takahashi, et al. Ril-contour: a medical imaging dataset annotation tool for and with deep learning. Journal of Digital Imaging, 32: 571-581 2019.
[53] Emma et al. Pierson. Daily, weekly, seasonal and menstrual cycles in women's 's mood, behaviour and vital signs. Nature Human Behaviour, 2021.
[54] Martin V Pusic, Amy Rapkiewicz, Tenko Raykov, and Jonathan Melamed. Estimating the irreducible uncertainty in visual diagnosis: Statistical modeling of skill using response models. Medical Decision Making, 43(6):680-66 661, 2023.
[55] Tom Quisel, Luca Foschini, Alessio Signorini, and David C Kale. Collecting and analyzing millions of mhealth data streams. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1971-1980, 2017.
13

---

Published as a conference paper at ICLR 2025
[56] Andrea H Ramirez, Lina Sulieman, David J Schlueter. Alese Halvorson, Jun Qian, Francis Ratsinbazafy, Roxana Loperena, Kelsey Mayo, Mclissa Basford, Nicole Deflaux, al. The all rescarch program: data quality, utility, and diversity. Patterns, 3(8), 2022.
[57] Vikas C Raykar, Shipeng Yu, Linda Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni. and Linda Moy. Learning from crowds. Journal of Machine Learning Research, 11(4), 2010.
[58] Mark D. Reid and Robert C. Williamson, Composite binary losses, Journal of Machine Learning Research, (1833)2238722422 2010. URL /estt::/mmlls htg.g/rg//rr..///////////oo/ooo/ http//pppess////eeee reidl0e Da html.
[59] Jorge Ortiz, Davide Anguita. Alessandro Ghio, Luca Oneto, and Xavier Parra.
Hu- man Activity Recognition Using Smartphones. UCI Machine Learning Repository, 2012. DOI:
https//bbb.org/102444///CCCC
[60] Oliver Roesler. EEG Eye State.
UCI
Machine Learning Repository, 2013.
DOE:
http://bbb.org/1002444//////
[61] Akane Sano and Rosalind W Picard. Stress recognition using wearable sensors and mobile phones. In 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction, pp. 671-676. IEEE, 2013.
[62] Mike Schaekemaann, Edith Law, Kate Larson, and Andrew Lim. Expert disagreement in sequential labeling: A case study on adjudication in medical time series analysis. In SAD/CrowdBias @ HCOMP, pp. 55-66, 2018.
[63] Lars Schmarje, Vasco Grossmann, Claudius Zelenka, Sabine Dippel, Rainer Kiko, Mariusz Oszust, Matti Pastell, Jenny Stracke, Anna Valros, Nina Volkmann, et al. Is one annotation enough?-a a image classification benchmark for noisy and ambiguous label estimation. Advances Neural Information Processing Systems, 35:33215 33232, 2022.
[64] Ying Sha and May Wang. Interpretable predictions of clinical outcomes with attention-based recurrent neural network. Proceedings of the 8th ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics, pp. 233-240, 2017.
[65] Shahah Shamshirband, Mahdis Fathi, Abdollah Dehzangi, Anthony Theodore Chronopoulos, and Hamid UlinejadRRokny. A review on deep learning approaches in healthcare systems: Taxonomies, challenges, and open issues. Journal of Biomedical Informatics, 13:1003627 2021.
[66] Paul Shekelle, Martin P Eccles, Jeremy M Grimshaw, and Steven H Woolf. When should clinical guidelines be updated? Bmj, 323(7305):155- 157, 2001.
[67] Masashi Sugiyama, Tongliang Liu, Bo Han, Yang Liu. and Gang Niu. Learning and mining with noisy labels. In Proceedings of the 31st ACM International Conference Information & Knowledge Management, pp. 5152-5155, 2022.
[68] Qiuling Suo, Fenglong Ma, Giovanni Canino, Jing Gao, Aidong Zhang, Pierangelo Veltri, and Gnasso Agostino. multi-task framework for monitoring health conditions via attention-based recurrent neural networks. In AMIA annual symposium proceedings, volume 2017, pp. 1665. American Medical Informatics Association, 2017.
[69] Ilya Sutskever, Oriol Vinyals, and Quoe V Lc. Sequence to sequence learning with neural networks. Advances in neural information processing systems, 27, 2014.
[70] Ryutaro Tanno, Ardavan Saeedi, Swami Sankaranaray Daniel C Alexander, and Nathan Silberman. Learning from noisy labels by regularized estimation of annotator confusion. In Proceedings of the IEFF/CV conference on computer vision and pattern recognition, pp. 11244-11253, 2019.
[71] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
[72] Shaun Wallace, Tianyuan Cai, Brendan Le, and Luis A Leiva. Debiased label aggregation for subjective crowdsourcing tasks. In CHI Conference on Human Factors in Computing Systems Extended Abstructs, pp. 1-8, 2022.
[73] Lu Wang, Wei Zhang, Xiaofeng He, and Hongyuan Zha. Supervised reinforcement learning with recurrent neural network for dynamic treatment recommendation. In Proceedings of the 24th ACM SIGKDD international conference knowledge discovery & data mining, pp. 2447-2456, 2018
14

---

Published as a conference paper at ICLR 2025
[74] Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 322-330 330, 2019.
[75] Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations, arXiv preprint arXiv:21 10. /2088, 2021.
[76] Johanna I Westbrook, Enrico Coiera, William TM Dunsmuir, Bruce Brown, Norm Kelk, Richard Paoloni, and Cuong Tran. The impact of interruptions on clinical task completion. BMJ Quality & Safety, 19(4):284 289, 2010.
[77] Jacob Whitchill, Ting-fan Wu, Jacob Bergsma, Javier Movellan, and Paul Ruvolo. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. Advances in Neural Information Processing Systems, 22, 2009.
[78] Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong. Silvio Savarese. and Doyen Sahoo. Unified training of universal time series forecasting transformers. arXiv preprint 2rXiv:2402.02552 2024.
[79] Xiaobo Xia, Tongliang Liu, Nannan Wang. Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? Advances in Neural Information Processing Systems, 32, 2019.
[80] Zhijian Xu, Ailing Zeng, and Qiang Xu. Fits: Modeling time series with 10k parameters. arXiv preprint arXiv:2307.03755 2023.
[81] Xinyu Yang. Yu Sun. Xiaojie Yuan, and Xinyang Chen. Frequency aware generative models for multivariate time series imputation. In The eighty- Annuol Conference on Neural Information Processing Systems.
[82] Yu Yao, Tongliang Liu,, Bo Han, Mingming Gong, Jiankang Deng, Gang Niu, and Masashi Sugiyama. Dual Reducing estimation error for transition matrix in label-noise learning. Advances in Neural Information Processing Systems, 33:7260 12771, 2020.
[83] LIN Yong, Renjie Pi, Weizhong Zhang, Xiaobo Xia, Jiahui Gao, Xiao Zhou, Tongliang Liu, and Bo Han. A holistic view label noise transition matrix deep learning and beyond. In International Conference on Learning Representations, 2023.
[84] Jiawen Zhang, Shun Zheng, Xumeng Wen, Xiaofang Zhou, Jiang Bian, and Jia Li. Flastst: Towards robust. varied-horizon forecasting with clastic time-series transformer. arXiv preprint 2rXiv:2411.01842 2024.
[85] Shao Zhang, Jianing Yu, Xuhai Xu, Changehang Yin, Yuxuan Lu, Bingsheng Yao, Mclanic Tory, Lace M Padilla, Jeffrey Caterino, Ping Zhang, et al. Rethinking human-ai al collaboration in complex medical decision making: A case study in sepsis diagnosis. arXiv preprint 2rXiv: /2368, 2023.
[86] Yivan Zhang, Gang Niu, and Masashi Sugiyama. Learning noise transition matrix from only noisy labels via total variation regularization. In International Conference on Machine Learning, pp. 12501-12512. PMLR, 2021.
15

---

Published as a conference paper at ICLR 2025
Supplementary Materials
A PROOFS
In what follows, we use vector notation for completeness and clarity of exposition. We describe key quantities and notation that we use in Table 5. We will also use Assumption I and Assumption 2 to factor the noisy label distribution as follows:
p x1.T)
II 9o( x1:)).
(6)
Notation
Description
p(y, xix) == [ply, = cl x
Vector of prohabilities for each label value, for the clean label distribution ply, X1se) Rex1
p(S, X1a) = [ply, = Komllom
Vector of probabilities for each possible label value, for the noisy label distribution p(š, X1:s) RCx1
hola1:) Poly; X1:t mnd)
Classifier that predicts label distribution é given preced- ing observations Rxxt
R
ha(d)a)) (g0
When ha is a deep network, is the final logits, and * A - RC represents an invertible link function (e.g., softmax)
Q - ( - Iy, - kM
The temporal noise matrix at time t: Q e RCxe
hly ho(m)u)) log py, x1x mss))
Loss at y x R
Eoalur ho(o)))) Cere 2b no(or)))
A composite loss function using link function sb
(chho(z1:)) - (o(o
Vector of NLL. losses for each possible value of the ground truth RC > EC
Le ho(sra)) lelc, QT a (gu))
Forward loss for class
(yter, ho(wrr)) Eim, ? Liole, ho(raa) Sequence forward loss
Table 5: Quantitics and definitions used for the following proof
Proof of Proposition 1
Proof. Our goal is to show:
argmni Es 7 sea (o.. argmin Σ
First, note that:
Z Colly, = 1yyy, Q. " (go((x)))
(7)
Ls (y,, goxx)))))))
(8)
where ,,' - Q. Thus, 0, A--1 C is invertible, and is thus a proper composite loss [58]. Thus, shown in Patrini et al. [50]:
16

---

Published as a conference paper at ICLR 2025
argmin xi (o..e. argmin Ix (. (x(x)))))))
(9)
<p(8,, x1+))
(property of proper composite losses)
(b((()))) " "(i,, Xi:)))
(10)
= <b((() x1:t))
(11)
The above holds for the minimizer at a single time step, not the sequence as whole. To find the minimizer of the loss over the entire sequence:
argmin Exr Sur 7 oe (f1.T ge(x)))) argmin By byyr 7 ee 90
(12)
8
0
argmin By Ix Σ (0
(13)
argmin
Egirivir (5:
(14)
Eg, Irsa Z, (f..
argmin
(15)
argmin
Egg Imn (F go (x1:)))
(16)
0
As the minimizer of the sum will be the function that minimizes each element of the sum, then argming Eyrr seq (oi...yV.oosssa.. (x(x)))) *pp(yIT Xirr)). Note that the argming Σi Eynax (xxxx)r,,0(xxx)))) of the NLL is the data distribution. Thus, argming By o (Vi.r. gx argming Σl= Eynax (yrr (xxxx)))
40p(y1:T X1:T)), because the minimizer
B CONTINUOUS LEARNING ALGORITHM
We summarize the augmented lagrangian approach solving the continuous objective in Algorithm
Algorithm Continuous Learning Algorithm
Input: Noisy Training Dataset D, hyperparameters y and n
Output: Model 0, Temporal Noise Function w
C f- 1 and À
for A 1, 2, do gh - = arg orgming... C(B,w) w)
> Computed with SGD using the Adam optimizer
+ A c* (Ok w
b Update Lagrange multiplier
if k > and (ak wi) 7 (Bk ((kk-1,w-1)) then
et
else
ctc
end
if (8% w²)== == 0 then
break
end if
end for
17

---

Published as a conference paper at ICLR 2025
For all experiments set À 1,c 1, 2, and 1 2. k and the maximum number of SGD iterations are set to 15 and 10, respectively. This is to ensure that the total number of epochs is 150, which is the max number of epochs used for all experiments.
C PLUG-IN PROCEDURE
1. Fit probabilistic classifier predict noisy lahels from the observed data.
2. For each class y E Y' and time i [T.
Identify anchor points for class o arg maxz, p(Se - (1:r).
ii Set (()) as the probability of classifier predicting class y' time given z.
D SUPPORTING MATERIAL FOR EXPERIMENTAL RESULTS
In what follows, we descrihe details for reproducing the experiments shown in the paper. Our code available GitHub.
D.1 DATASET DETAILS
Dataset
Classification Task
blinking [60]
Eye Open vs Eye Closed
sleeping [22]
Sleep vs Awake
964
100
moving [59] Walking Not Walking
192
senior [44]
Walking vs Not Walking
444
100
Table 6: Datasets used in experiments. Classification tasks, number samples (n), dimensionality at each time step (c), and sequence length (I') are shown.
moving from UC Irvine [59] consists of inertial sensor readings of 30 adult subjects performing activities of daily living. The sensor signals are already preprocessed and a vector of features at each time step are provided. We apply z-score normalization the participant-level, then split the dataset into subsequences of fixed size 50. We use a batchsize of 64.
senior from UC Irvine [44] consists of inertial sensor readings 18 elderly subjects performing activities of daily living. The sensor signals are already preprocessed and a vector of features at each time step are provided. We apply z- z-soore normalization at the participant-level, then split the dataset into subsequences of fixed size 100. We use a batchsize of 256.
sleeping from Physionet [22] consists of EEG data measured from 197 different whole nights of sleep observation, including awake periods at the start, end, and intermittently. We apply z-score normalization at the whole night-level. Then downsample the data to have features and labels each minute, as EEG data sampled at 100Hz and labels are sampled at 1Hz. We then split the data into subsequences of a fixed size 100. We use a batchsize of 512.
blinking from UC Irvine [60] consists of data measured from one continuous participant tasked with opening and closing their eyes while wearing headset measure their EEG data We apply z-score normalization for the entire sequence, remove outliers (>5 SD away from mean), and split into subsequences of fixed size 50. We use batchsize of 128.
synthetic We generate data for binary and multiclass classification with 1 000 samples and = 50 features over T = 100 time steps. We generate the class labels and obvservations for each time step using Hidden Markov Model (HMM). The transition matrix generating the markov chain uniform ensuring an equal likelihood of any state at any given time. We corrupted them using multidimensional (50) Gaussian emissions. The mean of the gaussian for state/class is set to
18

---

Published as a conference paper at ICLR 2025
with variance 1.5 (i.e. class 1 has mean and variance 1.5). The high-dimensionality and overlap in feature-space between classes makes this a sufficiently difficult task, especially under label noise. We batchsize of 256
D.2 NOISE INJECTION AND NOISE REGIMES
We create noise labels for each dataset by applying one of the six temporal noise functions shown in Fig. We choose functions to capture different functional forms, assumptions on time-variance, and class-dependence.
Static
Decay
Flipping Probabllity
Flipping Proba bibbabllity
45
35
35
20
40
60
80
100
20
40
60
100
Time
Time Periodic
Growth
50
Flipping Probability
Flipping Probability
40
40-
25
26
160
25
Time Linear
Time Mixed
Flipping Probability
45-
Flipping Probabilty
35
30
70
40
60
60
100
20
80
100
Time
Time
PBe yy-1) 11
Figure 4: Temporal label noise function (tt) used the experiments. We present six examples for binary classification task (from top-left clockwise): static, decay periodic, mixed, linear, growth. Each plot shows the off- off-diagonal entries of various parameterized forms of Q(t).
19

---

Published as a conference paper at ICLR 2025
D.3 DETAILS ON TRAINING ALGORITHMS
GRU the GRU r Rd x + RC x 2 produces an outpul vector such that the output of r( 21-1) our model for (2))) and hidden stale 2, E & that summarizes 11s-- We use softmax activation on the output vector of the GRU to make it valid parameterization of pa (Ye #14)) The GRU has single hidden layer with 32 dimension hidden state.
Continuous Continuous uses additional fully-connected neural network with 10 hidden layers that outputs C' C-dimensional vector to represent each entry of flattened Qz To ensure the output of this network is valid for Def. 1, we reshape the prediction to be C x C, apply a row-wise softmax function, add this to the identity matrix ensure diagonal dominance, then rescale the rows to be row-stochastic. These operations are all differentiable, ensuring we can optimize this network with standard backpropagation.
VolMinNet and Discontinuous We do a similar paramcterization for VolMinNet and Discontinuous, using a set of differentiable weights represent the entries of Q rather than a neural network.
Anchor and Plug-In Patrini et al. [50] show that in practice taking the 97th percentile anchor points rather than the maximum yield better results, so we use that same approach in our experiments. They also describe two-stage approach: 1) estimate the anchor points after a warmup period 2) use the anchor points to train the classifier with forward corrected loss. We set the warmup period to 25 epochs.
Hyperparameters We train each model for 150 epochs using the adam optimizer with default parameters and learning rate of 0.01. We train all models using the same set of hyperparameters for experiment, set the batch size manually for each dataset, and avoid hyperparameter tuning to avoid label leakages. For VolMinNet, Discontinuous, and Continuous we use adam optimizer with default parameters and a learning rate 0.01 to optimize each respective -esstmmaato technique. was set to 1e for VolMinNet and Discontinuous for all experiments following the setup of Li et al. [38].
20

---

Published as a conference paper at ICLR 2025
D.4 ADDITIONAL RESULTS
D.4.1 STATIC APPROXIMATION
Here we study the performance of forward temporal loss where we know the noise function Q(t) that is, even we could perfectly estimate the noise process and where we have static estimate of (1) (the average over time). We find that even if the noise process is perfectly estimated, accounting temporal noise outperforms a static estimate.
Static
Decay
100
100
Accuracy %
00
Accuracy %
90-
80
80
70
% of Noisy Labels Growth
% of Noisy Labels Periodic
1.00
100
Accuracy %
Accuracy
60
5
10
15
20
25
30
35
40
10
15
20
25
30
35
40
% of Noisy Labels
% of Noisy Labels
Accuracy %
Accuracy 9
80
70
70
60
60
15
of Noisy Labels
% Noisy Labels
Method
Uncomrected
Fonward
Time
Tamporal
Static
Figure 5: Comparing performance of models trained with forward temporal loss no noise correction on synth with varying degrees of temporal label noise using either the true temporal noise function (Temporal) the average temporal noise function (Static). Error bars are dev. over 10 runs.
21

---

Published as a conference paper at ICLR 2025
D.4.2 MULTICLASS CLASSIFICATION
Static
Decay
Accuracy %
Accuracy %
65
85
75
10
20
25
30
35
40
10
20
30
35
40
% of Noisy Labels Growth
% of Noisy Labels Periodic
Accuracy %
Accuracy %
85
85
75
10
20
25
30
35
10
20
30
40
% of Noisy Labels
% of Noisy Labels
Linear
Mixed
95
Accuracy %
Accuracy %
65
85
75
75
10
20
25
30
35
40
10
20
25
30
35
% of Noisy Labels
% of Noisy Labels
Ignore
Anchor
Plug-In
VolMinNet
Discontinuous
Continuous
Figure 6: Comparison clean test set Accuracy (%) for synth across varying degrees temporal label noise comparing all methods for 3-class classification. Error bars are st. dev. over 10 runs.
22

---

Published as a conference paper at ICLR 2025
Static
Decay
0.3
0.3
Istimation
0.2
Extimation
0.2
0.1
0.1
0.0
0.0
% of Noisy Labels
% of Noisy Labels
Growth
Periodic
Estimation (MAE)
0.3
0.3
Estimation
0.1
0.0
20
0.0
20
% of Noisy Labels
% of Noisy Labels
Linear
Mixee
0.3
0.3
Estimationn Error
Estimation
0.1
0.1
0.0
20
% of Noisy Labels
% of Noisy Labels
Ignore
Plug-In
VolMinNet
Discontinuous
Continuous
Figure 7: Comparison of noisy function reconstruction Mean Absolute Error (MAF) for synth varying degrees of temporal label noise comparing methods for 3-class classification. Error bars are st, dev. over 10 runs.
23

---

Published as a conference paper at ICLR 2025
Static
Decay
80
Accuracy
Accuracy
10
15
20
25
30
35
40
10
15
20
30
35
40
% of Noisy Labels
%
of Noisy Labels
Growth
Periodic
80
80
Accuracy %
Accuracy
70
70
10
15
20
25
30
35
40
10
15
20
25
30
35
40
% of Noisy Labels
% of Noisy Labels
Linear
Mixed
80
80
Accuracy
Accuracy
70.
10
15
20
25
30
35
10
20
30
35
% of Noisy Labels
% of Noisy Labels
ignore
Anchor
Plug In
VolMinNet
Discortinuous
Continuous
Figure 8: Comparison of clean test set Accuracy (%) for s slepping across varying degrees of temporal label noise comparing all methods for 3-class classification. Error bars st. dev. over 10 runs.
24

---

Published as a conference paper at ICLR 2025
Static
Decay
0.3
0.3
Istimation Enor
0.2
0.2
Estimation
0.1
0.0
% of Noisy Labels
% of Noisy Labels
Growth
Periodic
Edtimation (MAE)
0.3
0.3
Estimation
0.2
0.0
20
30
35
10
20
% of Noisy Labels Linear
% of Noisy Labels
Mixed
0.3
0.3
Estimation Error
Estimattion Error
0.2
0.2
0.1
0.0
% of Noisy Labels
% of Noisy Labels
Ignore
Plug-In
Discontinuous
Continuous
Figure 9: Comparison noisy function rerunttruction Mean Absolute Error (MAE) for eepi ng across varying degrees temporal label noise comparing methods for 3-class classification. Error bars are st. dev. over 10 runs.
25

---

Published as a conference paper at ICLR 2025
Static
Decay
90
90
Accuracy %
Accuracy
70
70
60
10
15
20
25
30
35
40
10
15
20
25
30
35
40
% of Noisy Labels
% of Noisy Labels
Growth
Periodic
90
90
Accuracy %
Accuracy
60
10
15
20
25
30
35
40
10
15
20
25
30
35
40
% of Noisy Labels
% of Noisy Labels
Linear
Mixed
90
90
Ac Accuracy
Accuracy
70
60
10
25 % of Noisy Labels
15
20
30
35
10 15
20
30
35
40
% of Noisy Labels
Ignore
Anchor
Plug In
VolMinNet
Discontinuous
Continuous
Figure 10: Comparison of clean test set Accuracy (%) for moving across varying degrees of temporal label noise comparing all methods for 4-class classification. Error bars st. dev. over 10 runs.
26

---

Published as a conference paper at ICLR 2025
Static
Decay
0.3
0.3
Istimation Eror
0.2
0.2
Estimation
0.1
0.0
0.0
% of Noisy Labels
% of Noisy Labels
Growth
Periodic
Edtimation (MAE)
0.3
0.3
Estimation
0.1
0.0
20
0.0
20
% of Noisy Labels
% of Noisy Labels
Linear
Mixee
0.3
0.3
Estimation Error
Estimattion Error
0.2
0.1
0.0
% of Noisy Labels
% of Noisy Labels
Ignore
Plug-In
VolMinNet
Discontinuous
Continuous
Figure 11: Comparison of noisy function reconstruction Mean Absolute Error (MAE) for across varying degrees of temporal label noise comparing all methods for 4-class classification. Error bars are st, dev. over 10 runs.
27


Reviews:
[{"review_id": "b81m3MeHNH", "reviewer": "Reviewer_gNk9", "summary": "The manuscript introduced temporal label noise in time series classification tasks and proposed a novel framework that are robust to it. Experiments were conducted on 4 datasets with synthetic noise injection and a real-world noisy labelled dataset, in which the method proposed in the manuscript is superior to other methods. The authors also visualized the dynamic evolution of label noise over time.", "questions": "1.\tWhat is the meaning of variable d in line 126? Is that mean a multivariate time series with d variables?\n2.\tWhen performing a train-test split on the datasets, were individuals also split?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.\tThe problem proposed by the authors is novel and interesting, and they evaluate it on real-world datasets.\n2.\tThe method introduced in the manuscript is cleverly crafted and has a theoretical foundation.", "weaknesses": "1.\tIn many settings, the performance of Plug-In method is inferior to that of static methods.\n2.\tOnly one of the five datasets in the manuscript contains both clean and noisy labels. In the remaining real-world scenarios, there is no experimental evidence to confirm that “the label noise evolves over time” actually exists.\n3.\tThe experimental scenarios are focused on healthcare; validating the work in more diverse scenarios would broaden its applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IBlWWJGPko", "reviewer": "Reviewer_Nmrx", "summary": "The paper discusses the problem of learning from time series classification task, where the label noise can vary over time. Each instance is a sequence over T time steps, where at iteration $t$ we have access to the features at time $t$, and a noisy label $\\tilde y_t$ obtained from the true label $y_t$. Each of this instance is i.i.d., and the assumption is that the true label $y_t$ at time $t$ only depends on the past features $x_{1:t}$, and the noisy labels are conditionally independent to the features given by the true labels. In particular, the authors assume there exists a noise mapping $Q_t$ that describes the noise process at time $t$. They propose a novel method to learn a classifier that maps sequences $x_{1:t}$ to a label $y$. This model simultaneously approximates the noisy mapping $Q_t$ (Eq 3), that is used to approximate the true labels, which are used during training", "questions": "See also weakness.\n\nThe model does not use the noisy label in the prediction (the predictor h_{\\theta} only depends on x_{1:t}).  I believe it would be interesting to include the noisy label in the prediction of the true label (i.e., if the noisy label is always accurate, can we use it for prediction? I understand this is a slightly different setting, as the goal of the paper is to learn a predictor).   \n\nThe fact that we learn a noisy label structure Q_w(t) that is “continual” over time seems implicit in Eq. (3). However, it is not clear what the difference is between Eq(3) and the model discussed in “discontinuous estimation”. It seems to me that $t$ is an integer, and $Q_w(t)$ could be completely different than $Q_w(t+1)$ in principle (what is the temporal relationship discussed in line 274)\n\nIt is a bit unclear how Section 2, in particular section 2.2, is related to the proposed method in Section 3. It seems to me, that the “loss of the classifier” is embedded in the constraint of Equation 2. Lines 233-234 says that Eq(2) minimizes the forward temporal loss as in Def 2, but this loss does not actually appear on Equation 2.   \n\nTypos: 277 continuuity", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Learning from time series data is an interesting problem. The paper addresses a challenging problem, where the label noise can change over time, and it is well-motivated. Overall, the introduction of the paper and the experimental sections are well-written and easy to follow. I found the use of the minimum-volume simplex assumption as an objective to solve the problem to be intriguing.", "weaknesses": "(**important**). The appendix of the paper is only a draft and it looks like it was not finished to be written. It also seems to be dissociated to the main paper.\n- Appendix A is self contained and not referenced in the main paper. It is not discussed what the content of this section adds as a contribution.\n- Appendix B looks dissociated to the main paper. First of all, three assumptions are introduced (they seems related to the 2 assumptions used in the main paper. In this case, why re-introduce them, and why do we use 3 assumptions rather than 2?). Most of the proofs are just a sequence of mathematical equations with text. Lines 810 to 840 are just a sequence of equations without text (maybe do a table?). The proof of Proposition 1 does not appear, which is only the theoretical result in the main paper (is Theorem 3 the proof of Proposition 1? Why does it have  different notation?).\n- Section G.2 is empty. Page 25 to 47 include a sequence of a lot of figure without almost no text. I recommend the authors to only include the figures that are used to “say” something, together with a text explaination.\n\n The technical section is also sometimes unclear.  \n\nNotations is sometimes unclear. What is q_t in line 140? According to the notation defined in the paper $\\mathcal{X} \\subseteq \\mathbb{R}^{d \\times T}$, but the function $h_{\\theta}$ that has domain $\\mathcal{X}$ can also have an input matrix $d \\times t$. \n \nThere is no comment on the assumptions 1 and 2 (except that they are two standard assumptions). I believe a few lines commenting on those assumptions would be helpful.\n\nIn lines 259-261, why is the Frobenius norm a convex surrogate for the volume? (A citation is also probably needed).  \n\nIn Line 260, should $R_t$ be defined over $\\tilde{y}$ rather than $y$?\n\n In lines 262-264, my understanding is that $\\lambda$ is the Lagrange multiplier of the constraint of Equation~2. I would be clearer on this, since $\\lambda$ does not appear in Eq 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5uMYBUUw68", "reviewer": "Reviewer_qqUX", "summary": "The paper proposes a model to incorporate varying noise rates for time-series classification. The method specifies a matrix-valued function indicating label-noise distribution. This noise function is incorporated into a forward temporal loss over noisy labels that is used to learn a classifier (using a neural net) minimizing the loss. The authors describe three approaches: continuous, discontinuous, and plug-in. They empirically evaluate these approaches in various benchmark datasets. They also demonstrate the approach in a real-world dataset within the healthcare domain in which the authors claim that the problem of temporal noisy labels is prevalent in time-series classification. The empirical results clearly indicate that the proposed approach works well compared to SOTA static approaches that treat noisy labels as static rather than temporal.", "questions": "(1) Is there an assumption that all data instances have equal number of time steps?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1) The formulation of the proposed methodology is clear and concise.\n2) The motivation and subsequent real-world example is well demonstrated.", "weaknesses": "(1) It is not clear how is learning achieved when weights are initialized for each time step in discontinuous estimation. It looks like the dataset is small for the learning models to converge as all the datasets largely seem to have less than 1000 samples.\n(2) Can the estimator in the discontinuous case be replaced by a simpler model (with lesser number of parameters)? If so, can such an alternative be used as baseline mechanisms to compare for temporal noise (as all alternative mechanisms shown in the paper are for static cases)? \n(3) It is interesting that both continuous and discontinuous estimations perform quite similarly on the real-world stress detection example. Does this mean that the proposed approach works best when there is large variance in noise rates?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VnvAACL5Iq", "reviewer": "Reviewer_EoQ6", "summary": "The paper introduces the problem of temporal label noise, where label quality fluctuates over time due to time-dependent noise. This type of noise may improve, deteriorate, or change periodically, impacting the accuracy of predictions. The authors define this problem and highlight that existing methods fail to handle the temporal aspect of label noise effectively. They propose methods to estimate the time-varying noise function directly from data, allowing the development of noise-tolerant classifiers. Experiments on real-world datasets validate their proposal, demonstrating its effectiveness under various types of temporal label noise.", "questions": "See Weaknesses above", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The problem of Temporal Label Noise is both interesting and important, and this paper is the first to formally define it.\n\n2. The paper is well-organized, clearly written, and easy to follow.\n\n3. The mathematical symbols and equations are well-defined and easy to understand.", "weaknesses": "1. My primary concern is with the problem formulation, or more specifically, the goal of the problem. In Section 2.1, the paper aims to \"estimate parameters $\\hat{\\theta}$ for a model robust to noise,\" which implicitly assumes the existence of a **fixed and static optimal model** $\\theta^\\star$—i.e., that p(y_t|x_t) remains consistent across all timestamps. However, in time-series problems, the environment is constantly changing, and the optimal model for different timestamps should also be changing.\n\n2. The other major concern relates to the methodology. The paper appears to assume that the function Q is known to the learner, and focuses only on optimizing the parameter w. However, I think that **obtaining an appropriate function Q is the most challenging aspect of this problem**. Although in Section 3.2, the authors select Q as a fully connected neural network as a general form for the practical scenarios, it seems to be inconsistent with the earlier discussions in Table 1.\n\n3. Another concern is about the experiments. The authors seem to only conduct experiments on low-dimensional datasets. I think that including experiments on larger-sized datasets would strengthen the results and make the findings more convincing.\n\nPlease let me know if I have misunderstood any part of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)