Title: Scale-teaching: Robust Multi-scale Training for Time Series Classification with Noisy Labels
Venue: NeurIPS 2023
Accepted: yes
Keywords: time series classification

deep neural networks

noisy labels
Abstract: Deep Neural Networks (DNNs) have been criticized because they easily overfit noisy (incorrect) labels. To improve the robustness of DNNs, existing methods for image data regard samples with small training losses as correctly labeled data (small-loss criterion). Nevertheless, time series' discriminative patterns are easily distorted by external noises (i.e., frequency perturbations) during the recording process. This results in training losses of some time series samples that do not meet the small-loss criterion. Therefore, this paper proposes a deep learning paradigm called Scale-teaching to cope with time series noisy labels. Specifically, we design a fine-to-coarse cross-scale fusion mechanism for learning discriminative patterns by utilizing time series at different scales to train multiple DNNs simultaneously. Meanwhile, each network is trained in a cross-teaching manner by using complementary information from different scales to select small-loss samples as clean labels. For unselected large-loss samples, we introduce multi-scale embedding graph learning via label propagation to correct their labels by using selected clean samples. Experiments on multiple benchmark time series datasets demonstrate the superiority of the proposed Scale-teaching paradigm over state-of-the-art methods in terms of effectiveness and robustness.

Scale-teaching: Robust Multi-scale Training for Time Series Classification with Noisy Labels

Abstract

Deep Neural Networks (DNNs) have been criticized because they easily overfit noisy (incorrect) labels. To improve the robustness of DNNs, existing methods for image data regard samples with small training losses as correctly labeled data (small-loss criterion). Nevertheless, time series' discriminative patterns are easily distorted by external noises (i.e., frequency perturbations) during the recording process. This results in training losses of some time series samples that do not meet the small-loss criterion. Therefore, this paper proposes a deep learning paradigm called Scale-teaching to cope with time series noisy labels. Specifically, we design a fine-to-coarse cross-scale fusion mechanism for learning discriminative patterns by utilizing time series at different scales to train multiple DNNs simultaneously. Meanwhile, each network is trained in a cross-teaching manner by using complementary information from different scales to select small-loss samples as clean labels. For unselected large-loss samples, we introduce multi-scale embedding graph learning via label propagation to correct their labels by using selected clean samples. Experiments on multiple benchmark time series datasets demonstrate the superiority of the proposed Scale-teaching paradigm over state-of-the-art methods in terms of effectiveness and robustness.

1 Introduction

Time series classification has recently received much attention in deep learning [1, 2]. Essentially, the success of Deep Neural Networks (DNNs) is driven by a large amount of well-labeled data. However, human errors [3] and sensor failures [4] produce noisy (incorrect) labels in time series datasets. For example, in electrocardiogram diagnosis [5], physicians with different experiences tend to make inconsistent category judgments. In recent studies [6, 7], DNNs have shown their powerful learning ability, which, however, makes it relatively easier to overfit noisy labels and inevitably degenerate

(b) Lightning7

(a) PLAID

(c) ElectricDevices

(d) MedicalImages

0       400       800       1200    1400

0         200        400        600    700

0         100        200        300    350

100

0

100

0

100

0

Time

Value

Value

Value

0         100         200         300

0        40       80       120       160

0        20       40        60         80

5

0

5

0

2.5

0

Time

Value

Value

Value

0      20      40      60      80      100

5

0

1

0

-1

Time

Value

Value

Value

0      10      20      30      40      50

0       5       10       15      20      25

5

0

0      20      40      60      80      100

5

0

1

0

-1

Time

Value

Value

Value

0      10      20      30      40      50

0       5       10       15      20      25

5

0

Finer

Coarser

the robustness of models. Moreover, time series data has complex temporal dynamics that make it challenging to manually correct noisy labels [8].

To cope with noisy labels, existing studies on label-noise learning [9, 10] use the memory effect of DNNs to select samples with small losses for training. DNNs memorize the data with clean labels first, and then those with noisy labels in classification training (small-loss criterion) [11]. It is worth noting that the small-loss criterion is not affected by the choice of training optimizations and network structures [12], and is widely utilized for label-noise learning in computer vision [13, 14]. However, the small loss criterion cannot always be applied to time series because the discriminative patterns of time series data are easily distorted by external noises [15, 16]. For example, in a smart grid, distortions may occur due to sampling frequency perturbations, imprecise sensors, or random differences in energy consumption [17]. Such distortions can make it difficult for DNNs to learn the appropriate discriminant patterns of time series, resulting in large training losses for some clean labeled samples. In addition, the small-loss criterion only utilizes the data's label information and does not consider the inherent properties of time series features (i.e., multi-scale information).

Multi-scale properties are crucial in time series classification tasks. In recent years, multi-scale convolution [16], dynamic skip connections [18, 19] and adaptive convolution kernel size [20] have been utilized to learn discriminative patterns of time series. Furthermore, according to related studies [2, 20, 21], the selection of appropriate time scales for time series data can facilitate DNNs to learn class-characteristic patterns. With correct labels, the above studies indicate that the multi-scale properties of time series data can help DNNs learn appropriate discriminative patterns for mitigating the negative effects of time series recording noises. Nevertheless, it remains an open challenge as to how the multi-scale properties of time series can be used for label-noise learning.

To this end, we propose a deep learning paradigm, named Scale-teaching, for time-series classification with noisy labels. In particular, we design a fine-to-coarse cross-scale fusion mechanism for obtaining robust time series embeddings in the presence of noisy labels. We select four time series datasets from the UCR archive [22] to explain our motivation. As shown in Figure 1, in the single scale case (top row), the red and blue samples from the same category have large differences in certain local regions (the green rectangle in Figure 1). By downsampling the time series from fine to coarse, some local regions between the red and blue samples did become similar. Meanwhile, existing studies [12, 23] show that multiple DNNs with random initialization have classification divergence for noisy labeled samples, but are consistent for clean labeled samples. The above findings inspire us to utilize multiple DNNs to combine robust embeddings at different scales to deal with noisy labels. Nonetheless, the coarse scale discards many local regions in the fine scale (as in Figure 1 (c)), which may degenerate the classification performance. Hence, we propose the Scale-teaching paradigm, which can better preserve the local discriminative patterns of fine scale while dealing with distortions.

More specifically, the proposed Scale-teaching paradigm performs the cross-scale embedding fusion in the finer-to-coarser direction by utilizing time series at different scales to train multiple DNNs simultaneously. The cross-scale embedding fusion exploits complementary information from different scales to learn discriminative patterns. This enables the learned embeddings to be more robust to distortions and noisy labels. During training, clean labels are selected through cross-teaching on those networks with the learned embeddings. The small-loss samples in training are used as (clean) labeled data, and the unselected large-loss samples are used as (noisy) unlabeled data. Moreover,

multi-scale embedding graph learning is introduced to establish relationships between labeled and unlabeled samples for noisy label correction. Based on the multi-scale embedding graph, the label propagation theory [24] is employed to correct noisy labels. This drives the model to better fit time series category distribution. The contributions are summarized as follows:

We propose a deep learning paradigm, called Scale-teaching, for time-series label-noise learning. In particular, a cross-scale fusion mechanism is designed to help the model select more reliable clean labels by exploiting complementary information from different scales.

We further introduce multi-scale embedding graph learning for noisy label correction using the selected clean labels based on the label propagation theory. Unlike conventional image label-noise learning methods focused on sample loss levels, our approach uses well-learned multi-scale time series embeddings for noise label correction at sample feature levels.

Extensive experiments on multiple benchmark time series datasets show that the proposed Scale-teaching paradigm achieves a state-of-the-art classification performance. In addition, multi-scale analyses and ablation studies indicate that the use of multi-scale information can effectively improve the robustness of Scale-teaching against noisy labels.

2 Related Work

Label-noise Learning. Existing label-noise learning studies focus mainly on image data [10]. These studies can be broadly classified into three categories: (1) designing noise-robust objective functions [25, 26] or regularization strategies [27, 28]; (2) detecting and correcting noisy labels [13, 29, 30]; (3) transition-matrix-based [31, 32] and semi-supervised-based [14, 33] methods. In contrast to the methodologies in the first and third categories, approaches categorized under the second category have received considerable attention in recent years [7, 34]. Methods of the second category can be further divided into sample selection and label correction. The common methods of sample selection are the Co-teaching family [12, 13, 23] and FINE [35]. Label correction [36, 37] attempts to correct noisy labels by either using prediction results of classifiers or pseudo-labeling techniques. Recently, SREA [4] utilizes pseudo-labels generated based on a clustering task to correct time-series noisy labels. Although the above methods can improve the robustness of DNNs, how the multi-scale properties of time series are exploited for label-noise learning has not been explored.

Multi-scale Time Series Modeling. In recent years, multi-scale properties have gradually gained attention in various time series downstream tasks [18, 38], such as time series classification, prediction, and anomaly detection [39]. For example, Cui et al. [16] employ multiple convolutional network channels of different scales to learn temporal patterns that facilitate time series classification. Chen et al. [19] design a time-aware multi-scale RNN model for human action prediction. Wang et al. [40] introduce a multi-scale one-class RNN for time series anomaly detection. Also, recent studies [41, 42, 43, 44] indicate that multi-scale properties can effectively improve the performance of long-term time series prediction. Unlike prior work, we utilize multiple DNNs with identical architectures to separately capture discriminative temporal patterns across various scales. This enables us to acquire robust embeddings for handling noisy labels via a cross-scale fusion strategy.

Label Propagation. Label propagation (LP) is a graph-based inductive inference method [24, 45] that can propagate pseudo-labels to unlabeled graph nodes using labeled graph nodes. Since LP can utilize the feature information of data to obtain pseudo-labels of unlabeled samples, related works employ LP in few-shot learning [46] and semi-supervised learning [47, 48]. Generally speaking, DNNs have the powerful capability for feature extraction, and the learned embeddings tend to be similar within classes and different between classes. Each sample contains feature and label information. Intuitively, the embeddings of samples with noisy labels obtained by DNNs closely align with the true class distribution when the DNNs do not fit noisy labels in the early training stages. Naturally, we create a nearest-neighbor graph based on well-learned multi-scale time series embeddings at the feature level. Subsequently, we employ LP theory to correct the labels of unselected noisy samples using the labels of clean samples chosen by the DNNs. This approach leverages robust multi-scale embeddings to address the issue of noisy labels.

Fine Scale Embeddings A

Medium Scale

Embeddings B

Cross-Scale Fusion

Cross-Scale Fusion

Clean Labels

Clean Labels

Finer

Coarser

Embeddings A

Embeddings (A

⨀

B)

Embeddings (A

⨀

B

⨀

C)

Corrected Labeled

Samples

Clean Labeled

Samples

Incorrect

/

Unlabeled

Samples

Embedding Fusion

Embeddings C

Encoder

Encoder

Encoder

Clean

Label

Propagation

Complementary

Information

+

+

+

3 Proposed Approach

3.1 Problem Definition

Given a noisy labeled time series dataset D = { ( X i , ̂ y i ) } N i =1 , it contains N time series, where X i ∈ R L × T , L denotes the number of variables, and T is the length of variable. ̂ y i ∈ { 1 , . . . , C } is the observed label of X i with η probability of being a noisy label. Our goal is to enable the DNNs trained on the noisy labeled training dataset D train to correctly predict the ground-truth labels of the given time series in the test set. Specifically, the problem to be addressed in this paper consists of two steps. The first is to select clean labeled time series from D train , and the second is to perform noisy label correction for time series in D train that have not been selected as clean labels.

3.2 Model Architecture

The overall architecture of Scale-teaching is shown in Figure 2. While this figure illustrates Scaleteaching with input time series at three scales, it can be extended to models with more scales, exceeding three. We utilize a consistent structural encoder to learn embeddings for each input scale sequence. Each encoder undergoes training at two levels: embedding learning for clean sample selection at the feature level and label correction with the multi-scale embeddings. For embedding learning, we propose a cross-scale fusion (Section 3.3) mechanism from fine to coarse to obtain robust embeddings. This approach enables the selection of more dependable clean labels through the small-loss criterion. Specifically, embeddings ( A ⊙ B ⊙ C ) encompass multi-scale information from fine, medium, and coarse scale sequences derived from the same time series. Regarding noisy label correction, we introduce multi-scale embedding graph learning (Section 3.4) based on label propagation, utilizing the selected clean samples to correct the labels of unselected large-loss samples.

3.3 Cross-scale Fusion for Clean Label Selection

After downsampling the original time series at different scales, it eliminates some of the differences in local regions between samples of the same category (as in Figure 1). However, the downsampled sequences (i.e., coarse scale) discard many local regions of the original time series. This tends to degrade the model's classification performance if the downsampled sequences are used directly for classification (please refer to Table 2 in the Experiments section). Meanwhile, existing studies [12, 23] on label-noise learning show that DNNs with different random initializations have high consistency in classification results for clean labeled samples in the early training period, while there is disagreement in the classification of noisy labeled samples. Based on the above findings, we utilize multiple DNNs (or encoders) with different random initializations to learn embeddings of different downsampled scale sequences separately, and perform cross-scale fusion. On the one hand, we exploit complementary

information between adjacent scale embeddings to promote learned embeddings to be more robust for classification. On the other hand, we leverage the divergence in the classification of noisy labeled samples by different DNNs to mitigate the negative impact of noisy labels in training. In this way, we can utilize the cross-scale fusion embeddings for classification, thus better using the small loss criterion [11, 29] for clean label selection. Specifically, downsampling is employed to generate different scale sequences from the same time series. Given a time series X i = { x 1 , x 2 , . . . , x T } , supposing the downsampling ratio is k . Then, we only keep data points in X i as follows:

where k ∈ [1 , T/ 2] , and a larger k indicates that X k i is coarser. As shown in Figure 2, time series with multiple downsampling intervals (i.e., k = 1, 2, 4) is treated as the input data for training. To better utilize the small-loss criterion for clean label selection, each time series sample performs cross-scale fusion from fine to coarse (i.e., A → B, B → C) in the embedding space, which is mathematically defined as:

where r k i represents the single-scale embedding acquired by learning X k i through an encoder. Meanwhile, v k i (or v k -t i ) denotes the embedding of the time series X k i (or X k -t i ) after performing cross-scale fusion. Here, t denotes the interval between adjacent downsampling ratios, and ∥ signifies the concatenation of two vectors to form a new vector. Notably, when k = 1 , we employ the singlescale for classification training, resulting in v k i = r k i . By combining ( r k i -v k -t i ) and ( r k i · v k -t i ) for vector concatenation, v k i can capture more nuanced discriminative information between r k i and v k -t i than that of simply concatenating r k i with v k -t i . The function f ( · ) represents a two-layer nonlinear network mapping function for fusing information of r k i and v k -t i . Additionally, v k i has the same dimension as r k i and serves as the input data for the multi-scale embedding graph learning process.

3.4 Multi-scale Embedding Graph Learning for Noisy Label Correction

We now present the multi-scale embedding graph learning module for correcting noisy labels. This module incorporates selected clean labels using label propagation theory. The process consists of two stages: graph construction and noisy label correction.

Graph Construction. It is assumed that the set of cross-fusion embeddings obtained from a batch of time series is defined as V = { v k 1 , v k 2 , . . . , v k M } , where M is the batch size. Intuitively, samples close to each other in the feature space have a high probability of belonging to the same class. However, in label-noise learning, v k i obtained from the current iterative training of the model may have unstable information, resulting in large deviations in the information of the nearest-neighbor samples of v k i . To address this issue, the proposed approach performs a momentum update [49] on v k i during training, which is defined as:

where e is the current training epoch and α denotes the momentum update parameter.

The multi-scale embeddings nearest-neighbor graph can be created by using Euclidean distance among different ¯ v k i . A common approach is the use of the Gaussian similarity function [45] to obtain the nearest-neighbor graph edge weight, which is defined as:

where d ( · ) is the Euclidean distance function and σ is a fixed parameter. W ∈ R M × M is a symmetric adjacency matrix, and the element W ij denotes the nearest-neighbor edge weight between the embedding v k i and v k j (note that larger values indicate closer proximity). Then, W is normalized based on the graph laplacians [50] to obtain Q = D -1 / 2 WD -1 / 2 , where D = diag ( W 1 n ) is a diagonal matrix. Specifically, the K neighbors with the largest values in each row of Q are employed to create the nearest-neighbor graph. It is noteworthy that the embeddings in each mini-batch are utilized to generate the nearest-neighbor graph, thus obtaining Q within short computational time.

Noisy Label Correction. Specifically, small training loss samples acquired by DNNs in the early training period can be considered as clean samples, while samples with large training losses are considered as noisy ones [12, 14]. The above learning pattern of DNNs has been mathematically validated [11] (see Appendix A for details). Under this criterion, prior studies [12, 13, 23] have typically employed samples with small losses after a e warm warm-up training as clean labels. Following [29], we extend the small-loss sample selection process to operate within each class, thereby enhancing the overall quality of the chosen clean labels. In our method, samples chosen with clean labels are considered labeled data, whereas unselected samples are treated as unlabeled data.

We utilize clean samples selected from time series at different scales in a cross-teaching manner (as in Figure 2). This could explore complementary information from different scale fusion embeddings to deal with noisy labels. It is supposed that there is a corresponding one-hot encoding matrix Y ∈ R M × C ( Y ij ∈ { 0 , 1 } ) for the cross-fusion embeddings V . If y i is identified as a clean label, we employ y i to set Y i as a one-hot encoded label. Otherwise, all the elements in Y i are identified as zero. Through Y , the pseudo-label of each node in the nearest-neighbor graph Q can be obtained in an iterative way based on the label propagation theory. The specific solution formula is defined as:

where F t ∈ R M × C denotes the predicted pseudo-label of the t -th iteration and β ∈ (0 , 1) is a hyperparameter. Naturally, F t has a closed-form solution [24] defined as follows:

where F ∈ R M × C is the final pseudo-labels and I denotes the identity matrix. Finally, the corrected label obtained for an unselected large-loss sample X i is defined as:

However, F is the estimated pseudo-labels, which inevitably contain some incorrect labels. To address this issue, two strategies are used to improve the quality of pseudo-labels in F . For the first strategy, the model continues training e update epochs by using small-loss samples after e warm epochs warm-up training to improve the robustness of the multi-scale embeddings. Then, the noisy label correction is performed after ( e warm + e update ) epoch. For the second strategy, a dynamic threshold φ e ( c ) = δ e ( c ) max( δ e ) γ is utilized for each class [51] to select the pseudo-labels with a high confidence for noisy label correction, where δ e ( c ) is the number of labeled samples contained in class c in the e -th epoch, and γ is a constant threshold.

Overall Training. Finally, each encoder utilizes the selected clean samples in combination with multi-scale embedding graph learning to perform noisy label correction for unselected large-loss samples. Combining the training data of the selected clean labels and those of corrected labels, the proposed Scale-teaching paradigm utilizes cross-entropy for time-series label-noise learning. Please refer to Algorithm 1 in the Appendix for the specific pseudo-code of Scale-teaching.

4 Experiments

4.1 Experiment Setup

Datasets. We use three time series benchmarks (four individual large datasets [3, 52, 53], UCR 128 archive [22], and UEA 30 archive [54]) for experiments. Among the four individual large datasets, HAR [52] and UniMiB-SHAR [3] are human activity recognition scenarios; FD-A [53] is the mechanical fault diagnosis scenario; Sleep-EDF [52] belongs to the sleep stage classification scenario. The UCR archive [22] contains 128 univariate time series datasets from different real-world scenarios. The UEA archive [54] contains 30 multivariate time series datasets from real-world scenarios. For details on the above datasets, please refer to Appendix B. Since all the datasets in three time series benchmarks are correctly labeled, we utilize a label transformation matrix T to add noises to the original correct labels [4], where T ij is the probability of label i being flipped to j . We use three types of noisy labels for evaluations, namely Symmetric (Sym) noise, Asymmetric (Asym) noise, and Instance-dependent (Ins) noise. Symmetric (Asymmetric) noise randomly replaces a true label with other labels with an equal (unequal) probability. Instance noise [55] means that the noisy label is instance-dependent. Like [4, 12, 23], we use the test set with correct labels for evaluations.

Baselines. We select seven methods for comparative analyses, namely 1) Standard: direct training of the model using cross-entropy with all noisy labels; 2) Mixup [56]; 3) Co-teaching [12]; 4) FINE [35]; 5) SREA [4]; 6) SELC [37]; and 7) CULCU [23]. Among them, Standard, Mixup, and Co-teaching are the benchmark methods for label-noise learning. FINE, SELC, and CULCU are the state-of-the-art methods that do not need to focus on data types, and SREA is the state-of-the-art method in time series domain. In addition, for fair comparisons, all the baselines and the proposed Scale-teaching paradigm use the same encoder and classifier. We focus on the ability of different label-noise learning paradigms to cope with time series noise labels, rather than the classification performance achieved by using fully correct labels. Hence, considering the trade-off between the running time and classification performance, we choose FCN [57] as the encoder of Scale-teaching. For more details of baselines, please refer to Appendix C.

Implementation Details. Based on the experience [19, 44] in time series modeling, we utilize three different sampling intervals 1, 2, 4 as the input muti-scale series data for Scale-teaching. We use Adam as the optimizer. The learning rate is set to 1e-3, the maximum batch size is set to 256, and the maximum epoch is set to 200. e warm is set to 30 and e update is set to 90. α in Eq. 3 is set to 0.9, σ in Eq. 4 is set to 0.25, β in Eq. 5 is set to 0.99, the largest neighbor K is set to 10, and γ is set to 0.99. In addition, following the parameter settings suggested in [23], we linearly decay the learning rate to zero from the 80 -th epoch to 200 -th epoch. For a comprehensive understanding of the hyperparameter selection and the implementation of the small-loss criterion applied to Scale-teaching, please consult Appendix C. To reduce random errors, we utilize the mean test classification accuracy of the last five epochs of the model on the test set as experimental results. All the experiments are independently conducted five times with five different seeds, and the average classification accuracy and rank are reported. Finally, we build our model using PyTorch 1.10 platform with 2 NVIDIA GeForce RTX 3090 GPUs. Our implementation of Scale-teaching is available at https://github.com/qianlima-lab/Scale-teaching .

4.2 Main Results

We evaluate each time series benchmack using four noise ratios, Sym 20%, Sym 50%, Asym 40%, and Ins 40%. Due to space constraints, we only give the average ranking of all the methods on each benchmark in Table 1. Please refer to Appendix D for the specific test classification accuracies. Besides, for UCR 128 and UEA 30 archives, we use the Wilcoxon signed rank test (P-value) [58] to analyze the classification performance of baselines. As shown in Table 1, the proposed Scale-teaching paradigm achieves the best Avg Rank in all the cases. It is found that Mixup [56] and FINE [35] perform worse than the Standard method in most cases. For Mixup, the complex dynamic properties

30

105

17

55

87

(a) Fine

→

Medium

(b) Medium

→

Coarse

(c) Coarse

→

Fine

Scale-teaching

w/o Cross-scale fusion

Fine scale

Medium scale

Coarse scale

VS.

VS.

VS.

375

196

56

71

532

116

147

585

26

207

364

523

39

w/o Cross-scale fusion

w/o Cross-scale fusion

Scale-teaching

Scale-teaching

(Fine

⨀

Medium

⨀

Coarse) scale

(Fine

⨀

Medium) scale

of the original time series are destroyed probably due to the mixture of two different time series mechanisms. FINE uses embeddings of the input data to select clean labels. Although FINE achieves advanced classification performance for image data, it is difficult to be used directly for time series data because its discriminative patterns are easily distorted by external noises. SREA [4] has a good performance on the UEA 30 archive, while it performs poorly on the other benchmarks. Meanwhile, Co-teaching [12], SELC [37], and CULCU [23] are more robust against time series noisy labels in different cases, further indicating that the small-loss criterion is also applicable to time series.

4.3 Multi-scale Analysis

To explain the multi-scale mechanism in the Scale-teaching paradigm, we add an ablation study based on Scale-teaching (w/o cross-scale fusion). We select the UCR 128 archive to analyze the classification results obtained by the fine, medium, and coarse scale classifiers. As shown in Figure 3, the classification results of different scale sequences have evident complementary information. Scaleteaching can effectively use complementary information between cross-scale to obtain more robust embeddings and clean labels. In response to the tendency of the coarse scale to ignore discriminative patterns in fine scale (please see Table 2), our proposed cross-scale fusion mechanism can effectively improve the classification performance of medium and coarse scales while retaining complementarity. Please refer to Appendix E for the specific classification results of Figure 3 and Table 2. In Appendix E, we also analyze the order and size of the downsampled input scale sequence for Scale-teaching.

Scale-teaching utilizes multi-scale embeddings to generate the nearest-neighbor graph, and uses clean labels selected for noisy label correction. To explore the distribution of different classes of embeddings, we employ t-SNE [59] for dimensionality reduction visualization. Specifically, we utilize the UniMiB-SHAR dataset containing Sym 20% noisy labels for visualization. As shown in Figure 4, we find that the embeddings learned by Scale-teaching are more discriminative across classes than the Standard and CULCU methods that use a single scale series for training. The above results suggest that Scale-teaching can effectively exploit the complementary information between different scales, prompting the learned embeddings to be more discriminative between classes. In addition, we choose the FD-A dataset for t-SNE visualization, and please refer to Appendix E.

(c) Scale-teaching (91.32%)

(a) Standard (86.43%)

(b) CULCU (83.39%)

-60      -40       -20      0      20      40      60

80

40

0

-40

-60      -40       -20      0      20      40      60

60

20

-20

-40

-60     -40      -20     0     20     40     60     80

60

20

-20

-60

T-SNE_1

T-SNE_2

T-SNE_1

T-SNE_1

10

10

14

15

16

2111

16

(b) ECG5000

Standard (Cross Entropy)

Scale-teaching (Ours)

0            50           150           200

1.7

1.6

1.4

1.2

1.0

Epoch

Loss value

0            50           150           200

1.8

1.6

1.4

1.2

1.0

Epoch

0            50           150           200

3.75

3.70

3.65

3.60

3.50

3.45

Epoch

Loss value

0            50           150           200

3.60

3.50

3.40

3.30

3.20

3.10

Epoch

Standard (Cross Entropy)

Scale-teaching (Ours)

clean

clean

noisy

noisy

noisy

clean

noisy

clean

4.4 Small-loss Analysis

To analyze the application of the small-loss criterion to time series data, we visualize the change of loss values for ground-truth clean and noisy time series samples during training. Specifically, Figure 5 shows the change in loss values of the models trained by the Standard method and Scale-teaching on two UCR time series datasets. When the model is trained with the Standard method, differences can be found in the loss values of clean and noisy samples in the network early training, especially in Figure 5 (b). The Standard method makes the model gradually fit the noisy samples as the training proceeds, while Scale-teaching improves the ability of the model to handle noisy labels. To further prove its effectiveness, we selected two other UCR datasets for the loss value change analysis, which have the same pattern as Figure 5. Also, we report the HAR and UniMiB-SHAR dataset's loss value probability distributions of clean and noisy samples. For more details, please refer to Appendix F.

4.5 Ablation Study

To verify the robustness of each module in Scale-teaching, the ablation experiments have been conducted in the HAR and UniMiB-SHAR datasets, and the results are shown in Table 3. Specifically, (1) w/o cross-scale fusion : the cross-scale embedding fusion from fine to coarse mechanism is ablated; (2) only single scale : only the original time series is used for training; (3) w/o graph learning : the multi-scale embedding graph learning module for noisy label correction is ablated; (4) w/o moment : the embedding momentum update mechanism (Eq. 3) is ablated; (5) w/o dynamic threshold : using a dynamic threshold to select high-quality propagation pseudo-labels is ablated.

As shown in Table 3, the cross-scale fusion strategy (w/o cross-scale fusion) and the clean labels crossteaching mechanism (only single scale) can effectively improve the classification performance of Scale-teaching, especially on the UniMiB-SHAR dataset with a large number of classes. Meanwhile, in terms of label correction based on multi-scale embedding graph learning, the results of the corresponding ablation module show that improving the stability of embedding (w/o moment) and

selecting high-quality pseudo-labels (w/o dynamic threshold) can effectively improve the performance of label correction based on graph learning.

Furthermore, we select the four individual large datasets without noisy labels for evaluation. As shown in Table 4, Scale-teaching's classification performance is still better than most baselines. It's worth mentioning that SREA [4] employs an unsupervised time series reconstruction loss as an auxiliary task, which reduces the model's classification performance without noisy labels. We also provide the corresponding test classification results for Tables 3 and 4 under the F1-score metric in Appendix G. Additionally, we find the running time of Scale-teaching, which is faster than FINE, SREA and CULCU for datasets with a larger number of samples or longer length of the sequence. We further analyze the classification performance of the proposed Scale-teaching paradigm and time series classification methods [15, 60] in Appendix G.

5 Conclusions

Limitations. The input scales of our proposed Scale-teaching paradigm can only select a fixed number of scales for training, and the running time will increase as the number of scales increases.

Conclusion. In this paper, we propose a deep learning paradigm for time-series classification with noisy labels called Scale-teaching. Experiments on the three time series benchmarks show that the Scale-teaching paradigm can utilize the multi-scale properties of time series to effectively handle noisy labels. Comprehensive analyses on multi-scale and ablation studies demonstrate the robustness of the Scale-teaching paradigm. In the future, we will explore the design of scale-adaptive time-series label-noise learning models.

Acknowledgments

We thank the anonymous reviewers for their helpful feedbacks. We thank Professor Eamonn Keogh and all the people who have contributed to the UCR 128 archive, UEA 30 archive, and the four large individual time series classification datasets. The work described in this paper was partially funded by the National Natural Science Foundation of China (Grant Nos. 62272173, 61872148, 62206041), the Natural Science Foundation of Guangdong Province (Grant Nos. 2022A1515010179, 2019A1515010768), the Science and Technology Planning Project of Guangdong Province (Grant No. 2023A0505050106), the Fundamental Research Funds for the Central Universities under grants DUT22RC(3)015. The authors would like to thank Siying Zhu, Huawen Feng, Yu Chen, and Junlong Liu from SCUT for their review and helpful suggestions.

References

Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and PierreAlain Muller. Deep learning for time series classification: a review. Data mining and knowledge discovery , 33(4):917-963, 2019.

Mingyue Cheng, Qi Liu, Zhiding Liu, Zhi Li, Yucong Luo, and Enhong Chen. Formertime: Hierarchical multi-scale representations for multivariate time series classification. arXiv preprint arXiv:2302.09818 , 2023.

Gentry Atkinson and Vangelis Metsis. Tsar: a time series assisted relabeling tool for reducing label noise. In The 14th PErvasive Technologies Related to Assistive Environments Conference , pages 203-209, 2021.

Andrea Castellani, Sebastian Schmitt, and Barbara Hammer. Estimating the electrical power output of industrial devices with end-to-end time-series classification in the presence of label noise. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages 469-484. Springer, 2021.

Benoît Frénay, Gaël de Lannoy, and Michel Verleysen. Label noise-tolerant hidden markov models for segmentation: application to ecgs. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages 455-470. Springer, 2011.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM , 64(3):107115, 2021.

Sheng Liu, Zhihui Zhu, Qing Qu, and Chong You. Robust training under label noise by over-parameterization. arXiv preprint arXiv:2202.14026 , 2022.

Gentry Atkinson and Vangelis Metsis. A survey of methods for detection and correction of noisy labels in time series data. In IFIP International Conference on Artificial Intelligence Applications and Innovations , pages 479-493. Springer, 2021.

Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International conference on machine learning , pages 2304-2313. PMLR, 2018.

Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions on Neural Networks and Learning Systems , 2022.

Xian-Jin Gui, Wei Wang, and Zhang-Hao Tian. Towards understanding deep learning from noisy labels with small-loss criterion. arXiv preprint arXiv:2106.09291 , 2021.

Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems , 31, 2018.

Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? In International Conference on Machine Learning , pages 7164-7173. PMLR, 2019.

Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. arXiv preprint arXiv:2002.07394 , 2020.

Patrick Schäfer. The boss is concerned with time series classification in the presence of noise. Data Mining and Knowledge Discovery , 29:1505-1530, 2015.

Zhicheng Cui, Wenlin Chen, and Yixin Chen. Multi-scale convolutional neural networks for time series classification. arXiv preprint arXiv:1603.06995 , 2016.

Patrick Schäfer and Ulf Leser. Fast and accurate time series classification with weasel. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management , pages 637-646, 2017.

Shiyu Chang, Yang Zhang, Wei Han, Mo Yu, Xiaoxiao Guo, Wei Tan, Xiaodong Cui, Michael Witbrock, Mark A Hasegawa-Johnson, and Thomas S Huang. Dilated recurrent neural networks. Advances in neural information processing systems , 30, 2017.

Zipeng Chen, Qianli Ma, and Zhenxi Lin. Time-aware multi-scale rnns for time series modeling. In IJCAI , pages 2285-2291, 2021.

Wensi Tang, Guodong Long, Lu Liu, Tianyi Zhou, Michael Blumenstein, and Jing Jiang. Omniscale cnns: a simple and effective kernel size configuration for time series classification. In International Conference on Learning Representations , 2022.

Víctor Campos, Brendan Jou, Xavier Giró-i Nieto, Jordi Torres, and Shih-Fu Chang. Skip rnn: Learning to skip state updates in recurrent neural networks. arXiv preprint arXiv:1708.06834 , 2017.

Hoang Anh Dau, Anthony Bagnall, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, and Eamonn Keogh. The ucr time series archive. IEEE/CAA Journal of Automatica Sinica , 6(6):1293-1305, 2019.

Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Jun Yu, Gang Niu, and Masashi Sugiyama. Sample selection with uncertainty of losses for learning with noisy labels. In International Conference on Learning Representations , 2022.

Dengyong Zhou, Olivier Bousquet, Thomas Lal, Jason Weston, and Bernhard Schölkopf. Learning with local and global consistency. Advances in neural information processing systems , 16, 2003.

Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 322-330, 2019.

Xiong Zhou, Xianming Liu, Junjun Jiang, Xin Gao, and Xiangyang Ji. Asymmetric loss functions for learning with noisy labels. In International conference on machine learning , pages 12846-12856. PMLR, 2021.

De Cheng, Yixiong Ning, Nannan Wang, Xinbo Gao, Heng Yang, Yuxuan Du, Bo Han, and Tongliang Liu. Class-dependent label-noise learning with cycle-consistency regularization. Advances in Neural Information Processing Systems , 35:11104-11116, 2022.

Kilian Fatras, Bharath Bhushan Damodaran, Sylvain Lobry, Remi Flamary, Devis Tuia, and Nicolas Courty. Wasserstein adversarial regularization for learning with label noise. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2021.

Nazmul Karim, Mamshad Nayeem Rizve, Nazanin Rahnavard, Ajmal Mian, and Mubarak Shah. Unicon: Combating label noise through uniform selection and contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9676-9686, 2022.

Deep Patel and PS Sastry. Adaptive sample selection for robust learning under label noise. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 3932-3942, 2023.

Bo Han, Jiangchao Yao, Gang Niu, Mingyuan Zhou, Ivor Tsang, Ya Zhang, and Masashi Sugiyama. Masking: A new perspective of noisy supervision. Advances in neural information processing systems , 31, 2018.

Zhaowei Zhu, Jialu Wang, and Yang Liu. Beyond images: Label noise transition matrix estimation for tasks with lower-quality features. In International Conference on Machine Learning , pages 27633-27653. PMLR, 2022.

Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Fernandez-Granda. Earlylearning regularization prevents memorization of noisy labels. Advances in neural information processing systems , 33:20331-20342, 2020.

Mingchen Li, Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with early stopping is provably robust to label noise for overparameterized neural networks. In International conference on artificial intelligence and statistics , pages 4313-4324. PMLR, 2020.

Taehyeon Kim, Jongwoo Ko, JinHwan Choi, Se-Young Yun, et al. Fine samples for learning with noisy labels. Advances in Neural Information Processing Systems , 34:24137-24149, 2021.

Eric Arazo, Diego Ortego, Paul Albert, Noel O'Connor, and Kevin McGuinness. Unsupervised label noise modeling and loss correction. In International conference on machine learning , pages 312-321. PMLR, 2019.

Yangdi Lu and Wenbo He. Selc: Self-ensemble label correction improves learning with noisy labels. arXiv preprint arXiv:2205.01156 , 2022.

Lifeng Shen, Zhuocong Li, and James Kwok. Timeseries anomaly detection using temporal hierarchical one-class network. Advances in Neural Information Processing Systems , 33:1301613026, 2020.

Qianli Ma, Zhen Liu, Zhenjing Zheng, Ziyang Huang, Siying Zhu, Zhongzhong Yu, and James T Kwok. A survey on time-series pre-trained models. arXiv preprint arXiv:2305.10716 , 2023.

Zhiwei Wang, Zhengzhang Chen, Jingchao Ni, Hui Liu, Haifeng Chen, and Jiliang Tang. Multiscale one-class recurrent neural networks for discrete event sequence anomaly detection. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining , pages 3726-3734, 2021.

Junchen Ye, Zihan Liu, Bowen Du, Leilei Sun, Weimiao Li, Yanjie Fu, and Hui Xiong. Learning the evolutionary and multi-scale graph structure for multivariate time series forecasting. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 2296-2306, 2022.

Amin Shabani, Amir Abdi, Lili Meng, and Tristan Sylvain. Scaleformer: iterative multi-scale refining transformers for time series forecasting. arXiv preprint arXiv:2206.04038 , 2022.

Huiqiang Wang, Jian Peng, Feihu Huang, Jince Wang, Junhui Chen, and Yifei Xiao. Micn: Multi-scale local and global context modeling for long-term series forecasting. In The Eleventh International Conference on Learning Representations , 2023.

Tian Zhou, Ziqing Ma, Qingsong Wen, Liang Sun, Tao Yao, Rong Jin, et al. Film: Frequency improved legendre memory model for long-term time series forecasting. arXiv preprint arXiv:2205.08897 , 2022.

Fei Wang and Changshui Zhang. Label propagation through linear neighborhoods. In Proceedings of the 23rd international conference on Machine learning , pages 985-992, 2006.

Yanbin Liu, Juho Lee, Minseop Park, Saehoon Kim, Eunho Yang, Sung Ju Hwang, and Yi Yang. Learning to propagate labels: Transductive propagation network for few-shot learning. arXiv preprint arXiv:1805.10002 , 2018.

Konstantinos Kamnitsas, Daniel Castro, Loic Le Folgoc, Ian Walker, Ryutaro Tanno, Daniel Rueckert, Ben Glocker, Antonio Criminisi, and Aditya Nori. Semi-supervised learning via compact latent space clustering. In International conference on machine learning , pages 2459-2468. PMLR, 2018.

Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. Label propagation for deep semi-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5070-5079, 2019.

Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. arXiv preprint arXiv:1610.02242 , 2016.

Fan RK Chung. Spectral graph theory , volume 92. American Mathematical Soc., 1997.

Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems , 34:18408-18419, 2021.

Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting. arXiv preprint arXiv:2106.14112 , 2021.

Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik. Self-supervised contrastive pre-training for time series via time-frequency consistency. arXiv preprint arXiv:2206.08496 , 2022.

Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The uea multivariate time series classification archive, 2018. arXiv preprint arXiv:1811.00075 , 2018.

Jiacheng Cheng, Tongliang Liu, Kotagiri Ramamohanarao, and Dacheng Tao. Learning with bounded instance and label-dependent label noise. In International Conference on Machine Learning , pages 1789-1799. PMLR, 2020.

Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 , 2017.

Zhiguang Wang, Weizhong Yan, and Tim Oates. Time series classification from scratch with deep neural networks: A strong baseline. In 2017 International joint conference on neural networks (IJCNN) , pages 1578-1585. IEEE, 2017.

Janez Demšar. Statistical comparisons of classifiers over multiple data sets. The Journal of Machine learning research , 7:1-30, 2006.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008.

Angus Dempster, François Petitjean, and Geoffrey I Webb. Rocket: exceptionally fast and accurate time series classification using random convolutional kernels. Data Mining and Knowledge Discovery , 34(5):1454-1495, 2020.

Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra Perez, and Jorge Luis Reyes Ortiz. A public domain dataset for human activity recognition using smartphones. In Proceedings of the 21th international European symposium on artificial neural networks, computational intelligence and machine learning , pages 437-442, 2013.

Daniela Micucci, Marco Mobilio, and Paolo Napoletano. Unimib shar: A dataset for human activity recognition using acceleration data from smartphones. Applied Sciences , 7(10):1101, 2017.

Christian Lessmeier, James Kuria Kimotho, Detmar Zimmer, and Walter Sextro. Condition monitoring of bearing damage in electromechanical drive systems by using motor current signals of electric motors: A benchmark data set for data-driven classification. In PHM Society European Conference , volume 3, 2016.

Ralph G Andrzejak, Klaus Lehnertz, Florian Mormann, Christoph Rieke, Peter David, and Christian E Elger. Indications of nonlinear deterministic and finite-dimensional structures in time series of brain electrical activity: Dependence on recording region and brain state. Physical Review E , 64(6):061907, 2001.

Hassan Ismail Fawaz, Benjamin Lucas, Germain Forestier, Charlotte Pelletier, Daniel F Schmidt, Jonathan Weber, Geoffrey I Webb, Lhassane Idoumghar, Pierre-Alain Muller, and François Petitjean. Inceptiontime: Finding alexnet for time series classification. Data Mining and Knowledge Discovery , 34(6):1936-1962, 2020.

Supplementary Material: Scale-teaching: Robust Multi-scale Training for Time Series Classification with Noisy Labels

A Small-loss Criterion

DNNs have been widely known to first learn simple and generalized patterns, which is achieved by learning clean data. After that, the networks gradually overfit noisy ones. In other words, when we train a model with a dataset containing incorrectly labeled samples, we can consider the samples with small training losses as clean ones and use them to update the model. Formally, let f ∗ be the target concept which determines the true label of x and model g ∗ = g ( x ; Θ ∗ ) minimizing the expected loss, i.e.,

Then, the small-loss criterion can be stated as follows[1]:

̸

Theorem 1. Suppose g is ϵ -close to g ∗ , i.e., ∥ g -g ∗ ∥ ∞ = ϵ , for two examples ( x 1 , ˜ y ) and ( x 2 , ˜ y ) , assume f ∗ ( x 1 ) = ˜ y and f ∗ ( x 2 ) = ˜ y , if T satisfies the diagonally-dominant condition T ii > max { max j = i T ij , max j = i T ji } , ∀ i , and ϵ < 1 2 · ( T ˜ y ˜ y -T f ∗ ( x 2 )˜ y ) , then ℓ CE ( g ( x 1 ) , ˜ y ) < ℓ CE ( g ( x 2 ) , ˜ y ) .

̸

̸

The work [11] provides the proof of this theorem. It shows that during training, the model can select clean samples according to the loss values. The reason is that the loss values of clean samples among the samples with the same observed labels are smaller. It is worth noting that the theorem is under the assumption of the class-dependent noise type and requires the transition matrix to satisfy the diagonally-dominant condition. Additionally, the finite data may also make the conditions of the theorem difficult to hold because the model g may be far away from g ∗ .

B Dataset Information

To evaluate the robustness of our proposed Scale-teaching and baselines on the time-series label-noise learning task, we selected three benchmark time-series datasets for experimental analysis.

B.1 Four individual large datasets

The statistical information of the four individual time series datasets is shown in Table 5. And the specific dataset information is as follows:

Human Activity Recognition (HAR)

The HAR dataset [52, 61] is collected from 30 students performing six human actions (i.e., walking, walking upstairs, downstairs, standing, sitting, and lying down) by wearing sensors.

University of Milano Bicocca Smartphone-based Human Activity Recognition (UniMiB SHAR)

The UniMiB SHAR dataset [3, 62] is human activity information collected at a sampling rate of 50 Hz from volunteers with a smartphone with an accelerometer sensor in the front pocket of their pants. Specifically, each accelerometer entry is labeled by specifying the type of ADL (e.g., walking, sitting, or standing) or the type of fall (e.g., forward, fainting, or backward).

Faulty Detection Condition A (FD-A)

The FD-A dataset [52, 63] is generated by an electromechanical drive system that monitors the condition of rolling bearings and detects their failure. Each rolling bearing can be classified into three categories: undamaged, inner damaged, and externally damaged.

Sleep Stage EEG Signal Classification (Sleep-EDF)

The Sleep-EDF dataset [52, 64] includes the whole night PSG sleep recordings, which contain five EEG sleep signal recordings: Wake (W), Non-rapid eye movement (N1, N2, N3), and Rapid Eye Movement (REM).

B.2 UCR 128 Archive

The UCR time series archive [22] contains 128 univariate datasets and is widely used for classification in the time series mining community. Each UCR dataset includes a single training set and a single test set, and each time series sample has been z-normalized. In addition, we uniformly use the meanimputation method to preprocess the datasets that contain missing values. For detailed information about UCR datasets, please refer to https://www.cs.ucr.edu/~eamonn/time_series_data_ 2018/ .

B.3 UEA 30 Archive

The UEA time series archive [54] contains 30 multivariate datasets, mainly derived from Human Activity Recognition, Motion classification, ECG classification, EEG/MEG classification, Audio Spectra Classification, and other realistic scenarios. Each dataset contains a partitioned training set and a test set. In addition, we use the mean-imputation method to deal with datasets with missing values. For detailed information about UEA datasets, please refer to https: //www.timeseriesclassification.com/dataset.php .

C Baselines

To analyze the performance and effectiveness of Scale-teaching on time-series label-noise learning, we selected seven baselines for comparative analysis. The specific information is as follows.

Standard directly employs all samples in the training set containing noisy labels and performs supervised classification training using cross-entropy loss. Then, the trained model is used to make predictions on the test set.

Mixup [56] trains a neural network on convex combinations of pairs of time series samples and their labels (whatever is clean or noisy). For the specific open source code, please refer to https://github.com/facebookresearch/mixup-cifar10 .

Co-teaching [12] trains two deep neural networks simultaneously, and lets them teach each other given every mini-batch with selected clean labels based on a small-loss criterion. For the specific open source code, please refer to https://github.com/bhanML/ Co-teaching .

FINE [35] utilizes a novel detector for clean label selection. Especially, FINE focus on each data point's latent representation dynamics and measures the alignment between the latent distribution and each representation using the eigen decomposition of the data gram matrix. For the specific open source code, please refer to https://github.com/Kthyeon/FINE_ official .

SREA [4] employs a novel multi-task deep learning approach for time series noisy label correction that jointly trains a classifier and an autoencoder with a shared embedding representation. For the specific open source code, please refer to https://github.com/ Castel44/SREA .

SELC [37] utilizes a simple and effective method self-ensemble label correction (SELC) to progressively correct noisy labels and refine the model. For the specific open source code, please refer to https://github.com/MacLLL/SELC .

CULCU [23] incorporates the uncertainty of losses by adopting interval estimation instead of point estimation of losses to select clean labels based on Co-teaching. CULCU has two

versions: CNLCU-S and CNLCU-H, where CNLCU-S uses soft labels for training and CNLCU-H uses hard labels for training. According to the original paper's [23] experimental results, CNLCU-S has a better performance. Hence, we use CNLCU-S as a baseline. For the specific open source code, please refer to https://github.com/xiaoboxia/CNLCU .

Finally, based on the source code of the above baselines, we provide the reproduction source code of all baselines, as well as the source code of our proposed Scale-teaching (refer to Algorithm 1). For the specific open-source code, please refer to our GitHub repository https://github.com/ qianlima-lab/Scale-teaching .

Our experiment contains 162 datasets. It would be time-consuming to perform hyperparameter selection for each dataset. Therefore, the hyperparameters of Scale-teaching are not carefully tuned for each dataset, and most of the hyperparameters are set based on the default hyperparameters of related works. The learning rate and maximum epoch are set based on the parameters of existing noise-label learning methods, such as FINE and CULCU. α in Eq. 3, σ in Eq. 4 and β in Eq. 5 are set based on the default hyperparameters of related label propagation works. e warm is based on FINE settings. e update , γ and batch size are based on manual empirical settings without specific hyperparameter analysis. The largest neighbor K is set based on human experience, and we had a simple test on several datasets, and found that a larger value of does not improve the classification performance, but instead increases the running time of the model.

For the implementation of small-loss criterion in Scale-teaching, we select small-loss samples within each class from the mini-batch data as clean labeled data. For stduies [12, 13], they use warm-up training to decrease λ ( e ) from 1 to 1 -η . λ ( e ) denotes the selection ratio of small-loss samples within the mini-batch data without considering the difference of class, and η is the ratio of noise labels in the training set. Based on the above criterion, the current work [29] uses the Jensen-Shannon divergence to calculate difference d between the classification result p i of sample X c i and the observation label ̂ y i . Following [29], for each class c , we consider the observed label of X c i as a clean label when the d of the training sample X c i is less than d e avg after a e warm warm-up training. d e avg denotes the average of d s of all the training samples when the epoch is e . We observed that using the Jensen-Shannon divergence method [29] and directly employing stduies [12, 13] for clean sample selection within each class have distinct strengths and weaknesses when applied to various time series datasets. In our study, we implemented the strategy of stduies [12, 13] for clean sample selection within each class on four individual large datasets and the UCR 128 archive. Meanwhile, the Jensen-Shannon divergence method [29] was applied to the UEA 30 archive for clean sample selection within each class.

D Details of Main Results

For the four individual large time series datasets, the specific classification results of our proposed Scale-teaching paradigm and baselines are shown in Table 6. For the UCR 128 archive, the specific classification results for all methods with different noise ratios are shown in Table 11 (Sym 20%), 12 (Sym 50%), 13 (Asym 40%), and 14 (Ins 40%). For the UEA 30 archive, the specific classification results for all methods at different noise ratios are shown in Tables 15 (Sym 20%), 16 (Sym 50%), 17 (Asym 40%), and 18 (Ins 40%). For layout and reading convenience, we only give the average classification accuracy for multiple runs of all methods without standard deviation on the UCR 128 archive and UEA 30 archive.

E Details of Multi-scale Results

To analyze the multi-scale mechanism in the Scale-teaching paradigm, we provide the classification performance of classifiers corresponding to fine, medium and coarse scales, as shown in Tables 19 and 21. And the classification results by ablation cross-scale fusion mechanism based on the Scaleteaching are shown in Tables 20 and 22. For the abbreviations in Tables 19, 20, 21 and 22, such as a_t_b_f, b_f_c_t, and c_t_a_f, where a denotes fine classifier, b denotes medium classifier, and c denotes coarse classifier, and t and f represent correct and incorrect classification results, respectively. For example, a_t_b_f indicates the number of samples correctly predicted by the fine classifier and incorrectly predicted by the medium classifier. In addition, we provide t-SNE [59] visualization on the FD-A dataset with Sym 50% noisy labels (as in Figure 6) to explore the distribution of different classes of embeddings. Figure 6 shows that the cross-scale fusion mechanism in Scale-teaching for

Algorithm 1 The proposed Scale-teaching paradigm.

Input: encoders [ w A , w B , w C ] , classifiers [ c A , c B , c C ] , fine-scale series x A , medium-scale series x B , and coarse-scale series x C

Output: [ w A , w B , w C ] and [ c A , c B , c C ]

Note: For clarity, our analysis utilizes three distinct scales for training, but this approach can be extended to incorporate multiple scales.

1: Step one: Obtain single-scale embeddings r A , r B , r C ;

r A = w A ( x A ) ;

r B = w B ( x B ) ;

r C = w C ( x C ) ;

2: Step two: Obtain cross-scale embeddings v A , v B , v C ;

v A = r A ;

v B = Eq. 2 ( r B , v A ) ;

v

C

= Eq. 2

(

r

C

, v

B

)

;

3: Step three: Obtain clean labels y A , y B , y C for cross-teaching training;

y A = c C ( v C ) via small loss criterion;

y B = c A ( v A ) via small loss criterion;

y C = c B ( v B ) via small loss criterion;

4: Step four: Obtain corrected labels yc A , yc B , yc C for classification training;

yc A = Eq. 6 ( v A , y A ) via label propagation;

yc B = Eq. 6 ( v B , y B ) via label propagation;

yc C = Eq. 6 ( v C , y C ) via label propagation;

5: Step five: Overall training;

Update encoder w A and classifier c A via cross-entropy loss( v A , y A & yc A );

Update encoder w B and classifier c B via cross-entropy loss( v B , y B & yc B );

Update encoder w C and classifier c C via cross-entropy loss( v C , y C & yc C ).

(c) Scale-teaching (99.79%)

(a) Standard (96.73%)

(b) CULCU (98.77%)

-80         -40       -20      0      20      40

80

-60      -40       -20      0      20      40      60      80

60

20

-20

-60

-60

-40

-20

0

20         40

80

60

20

-20

-60

T-SNE_1

T-SNE_2

T-SNE_1

T-SNE_1

60

20

-20

-60

0.88

0.885

0.89

0.895

0.9

0.905

0.91

0.915

0.92

0.925

2

3

4

5

[1,2,3,4,5]

[1,2,4,8,16]

0.74

0.75

0.76

0.77

0.78

0.79

0.8

0.81

2

3

4

5

[1,2,3,4,5]

[1,2,4,8,16]

(a) HAR Dataset

(b) UniMiB-SHAR Dataset

time-series label-noise learning can make the embeddings of different classes more discriminative, thus facilitating clean sample selection and noisy label correction.

Impact of downsampling scale sequence list. Scale-teaching can be performed using a variety of different downsampling scales for label-noise learning. Based on the experience of [19, 44] on time series classification and prediction tasks, we utilize the downsampling scales of [1,2,4] for the experimental analyses of Scale-teaching. However, for real-world scenarios that actually contain noisy labels, it is generally not possible to perform hyperparametric analyses using a clean-labeled validation set. To facilitate the analysis, in this paper, we use the classification performance of the test set for multi-scale hyperparameter analyses. However, to avoid test set information leakage, we do not use the hyperparameter analysis result for Scale-teaching in our experiments. We use two multi-scale sampling strategies for analyses, which are (1) {[1,2], [1,2,3], [1,2,3,4], [1,2,3,4,5]}; (2) {[1,3], [1,2,4], [1,2,4,8], [1,2,4,8,16]}. From Figure 7, we find that Scale-teaching using four different scales for training has the highest classification accuracy, which indicates that more input scales do not necessarily make the classification performance better. In addition, using three or four scales of sequences can effectively improve the classification performance of Scale-teaching compared with using two different scales.

Impact of input scales of sequences order. Scale-teaching employs a finer-to-coarser strategy for cross-scale embedding fusion. Intuitively, when a single scale is used for classification, the original single scale (finer) time series is better overall because it does not discard the original sequence information compared to coarser scale time series. Therefore, Scale-teaching is trained using the finer-to-coarser cross-scale fusion strategy. To analyze the difference in classification performance between different fusion directions, we subtract the classification accuracy using the finer-to-coarser and coarser-to-finer training approaches, and the specific results are shown in Figure 8. We can find that the classification performance of finer-to-coarser is better overall, which is due to its ability to use a single fine-scale sequence with an excellent classification performance from the beginning to gradually promote the classification performance of multiscale fusion embeddings.

0.007

-0.0084

0.0219

0.0095

-0.01

-0.005

0

0.005

0.01

0.015

0.02

0.025

HAR

FD-A

Sleep-EDF

Dataset

(b) CricketZ

(a) Adiac

Standard (Cross Entropy)

Scale-teaching (Ours)

0            50           150           200

2.5

2.3

2.1

1.9

Epoch

Loss value

0            50           150           200

2.5

2.3

2.1

1.9

1.7

Epoch

0            50           150           200

3.60

3.55

3.50

3.45

3.40

Epoch

Loss value

0            50           150           200

3.60

3.50

3.40

3.30

3.20

3.10

Epoch

Standard (Cross Entropy)

Scale-teaching (Ours)

clean

clean

noisy

noisy

clean

noisy

noisy

clean

F Small-loss Visualization

The small-loss criterion has been extensively validated for clean label selection in label-noise learning for computer vision. To further analyze the application of the small-loss criterion in time series data, we provide the change of loss values of the models trained by the Standard method and Scale-teaching on Adiac and CricketZ UCR datasets (as in Figure 9). Also, we visualize the probability distributions of the ground-truth clean and noisy (corrupted) sample loss values on the test set with different training strategies. Specifically, Figures 10 and 11 show the loss probability distributions of the models trained by different strategies on the HAR dataset and UniMiB-SHAR with Aysm 40% noisy labels. Both red (clean) and blue (corrupted) in Figure 10 and Figure 11 contain two peaks, which indicate that some correctly labeled samples are still difficult to learn (large loss) and some incorrectly labeled samples are also easy to learn (small loss). Compared with the Standard method (Figure 10 (a) and Figure 11 (a)), Scale-teaching (Figure 10 (b) and Figure 11 (b)) can clearly distinguish clean and noisy samples by the loss value distribution, further validating the robustness of the multi-scale embeddings to cope with time-series noisy labels.

G Other Analysis

The test F1-score results of ablation study. Following [4], we select the averaged F1-score on the test set as a new metric for ablation analysis in Section 4.5. Hence, we give the corresponding test classification F1-score (%) in Tables 7 and 8.

Running time analysis. We select two datasets for running the time-consuming analysis, the FD-A dataset with the largest sequence length and the Sleep-EDF dataset with the largest samples. We performed the running time statistics on the NVIDIA GeForce RTX 3090 GPU using all baselines, and the results are shown in Table 9. On the FD-A dataset with the longest sequence length, Coteaching and CULCU take essentially twice as long to run as the Stanard method because they use

(a) Standard

(b) Scale-teaching

cleandensity

clean density

3.0

corrupted density

corrupted density

sample density

2.5

sample density

3

2.0

1.5

2

1.0

1

0.5

0.0

1.5

1.0

-0.5

0.0

0.5

1.0

1.5

2.0

2.5

0

1.0

0.5

0.0

0.5

1.0

1.5

loss

loss

(a) Standard

(b) Scale-teaching

1.6

cleandensity

cleandensity

1.4

corrupteddensity

2.0

corrupted density

sample density

1.2

sample density

1.5

1.0

0.8

1.0

0.6

0.4

0.5

0.2

0.0

0.0

1.5

1.0

0.5

0.0

0.5

1.0

1.5

1.5

1.0

0.5

0.0

0.5

1.0

1.5

loss

loss

two encoders. Furthermore, although SREA uses a single network training, it utilizes a decoder for the unsupervised reconstruction task of the original time series, which significantly increases training time on the FD-A dataset with longer sequences. running time is higher than Co-teaching. The Scale-teaching paradigm uses multiple encoders for training and has an additional noisy label correction module, which is expected to increase the training time. Nevertheless, the larger the sampling scale (coarse scale) of the training data used by the Scale-teaching paradigm, the lower the training elapsed time of its model. For example, with input scales of [1, 8, 32], the training time of Scale-teaching is lower than that of CULCU and SREA. On the SleepEEG dataset with the largest number of samples, we find that FINE with an encoder has a higher running time because FINE using all training samples to select clean labels is time-consuming when the sample size is large. In contrast, the runtime of Scale-teaching is lower than FINE. Also, when the input scales are set to [1,8,32], the runtime of Scale-teaching is lower than SREA.

It is worth noting that when Scale-teaching is trained using two scales, such as [1,2] or [1,16], its training run time decreases further. From the analysis in Appendix E, it is clear that Scale-teaching using three different scales generally performs better than two scales for classification with noisy labels. In addition, the classification performance of [1,2,4], [1,4,16], and [1,8,32] when Scaleteaching is trained using three different scales has less difference in classification performance on datasets with longer sequences (e.g., FD-A and Sleep-EDF). The above results indicate that the Scale-teaching paradigm has a greater advantage in runtime on time-series datasets with longer sequences.

Robustness analysis. Three time-series supervised classification methods (Boss [15], Rocket [60] and FCN [57]) and the Scale-teaching paradigm are chosen for robustness analysis against time-series noise labels. Boss [15] is a time series classification method based on similarity search, which can effectively mitigate the negative impact of noise (e.g., adding Gaussian noise) in time series values on classification. Rocket [60] uses a large number of randomly initialized convolution kernels to extract time series features, and employs the extracted features to classify time series using a machine learning classifier (e.g., Ridge classifier). FCN [57] is the encoder used by Scale-teaching, which is a time series classification method based on DNNs. As shown in Table 10, the classification performance of the Scale-teaching paradigm using FCN as encoders is better than that of Boss, Rocket and FCN in the presence of noisy labels. It is worth noting that both Boss and Rocket training processes are independent of the optimization of DNNs. However, their classification performance is still reduced due to the influence of noisy labels. In addition, the encoder of the Scale-teaching paradigm can be designed flexibly, such as using ResNet [1], InceptionTime [65] and OS-CNN [20] in the field of time series classification. In other words, using better robustness encoders, the classification performance of Scale-teaching can be further improved with time-series noise labels.

Peer reviews:
comments: SUMMARY:
This paper proposes a deep learning paradigm, namely scale-teaching, to cope with time series noisy labels, with designing a fine-to-coarse cross-scale fusion mechanism for learning discriminative patterns by using time series at different scales to train multiple DNNs simultaneously. Additionally, each network is trained in a cross-teaching manner by using complementary information from different scales to select small-loss samples as clean labels. Meanwhile, this paper introduces a multi-scale embedding graph learning method via label propagation to correct their labels from unselected large-loss samples.  Extensive experiments demonstrate the superior performance of the proposed method.

STRENGTHS:
1. The research problem is interesting and important.
2. The idea is simple and effective.
3. The paper is well written.
4. Extensive experiments with good results.

WEAKNESSES:
1. The meanings of some equations or symbols are not clear, like the operator || in Eq. (2).
2. The problem definition in Label Correction for Eq. (5) is difficult to understand.
3. The proposed method is relatively complex, and its procedure is not clear, including the Algorithm 1.
4. In Figure 2, it is better to introduce the flowchart of the proposed method.

QUESTIONS:
1. In Eq. (2), how the operator || to operate the r_i^k, v_i^k and so on?
2. How to obtain the Eq. (5)?

LIMITATIONS:
The proposed Scale-teaching paradigm only utilizes three scales, and its training time is increasing with the increase of the number of scales.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-29

comments: SUMMARY:
This paper a paradigm to improve the small-loss criterion for time series noisy labels, which mainly addresses the problem that external noises easily distort time series’ discriminative patterns. The motivation is clear, and experiments have shown good results. However, the reviewer is still concerned about some issues.

STRENGTHS:
1. DNNs with different random initializations have high consistency in classification results for clean labeled samples in the early training period; while there is disagreement in the classification of noisy labeled samples, the designation of cross-scale fusion using inputs with different scales is lighter than other methods using different networks like co-teaching which are time-consuming.

WEAKNESSES:
1. Although the proposed method is useful, the network is similar to multi-scale feature learning like FPN and the label propagation part is also not novel.
2. The way of downsampling methods needs more comparison.
3. Momentum updating is also useful for other methods, although the goal of using it in this paper is reasonable, and scale-teaching achieves SOTA even w.o. momentum updating. But it is still an unfair comparison.

QUESTIONS:
1. What is the fundamental difference between the proposed method with FPN and traditional label propagation is graph learning?

LIMITATIONS:
The authors seem to discuss some limitations of the method in terms of computation efficiency. The reviewer suggests more discussion in energy consumption for a more general usage.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a time series classification algorithm that works across multiple scales with a noise-robust loss function. The authors claim improvements due to the multi-scale network architecture and the objective function capable to handle label noise on a variety of datasets in UCR128 and beyond.\n\nSTRENGTHS:\nThe paper is mostly well organized. Statistical tests were performed in all experiments, and the paper has a good mix of summary statistics for large evaluation runs (like in Table 1), and more qualitative results showing properties of the algorithm (e.g. the training dynamics in Figure 5). The authors also perform ablation studies and attempt to test the effect of different components of their algorithm.\n\nWEAKNESSES:\n- presentation can be generally improved. In section 3, it is not fully clear which components are preliminaries and present in existing work, and which are the new algorithmic components of the new approach. According to title and abstract, these should be the multi-scale and label noise corrections parts of the model. Maybe the headings could be further adapted to make this clear (e.g. right now, it is not clear why \"cross-scale fusion\" mostly addresses the multi-scale aspect, while \"multi-scale embedding graph learning\" mostly covers the label noise robustness part).\n\n- The figure caption for Figure 2 could be a bit longer and actually explain the figure content. \n\n**Minor**:\n\n- typo: emebeddings (l. 56)\n- the definition of the considered noise model comes too late and should be directly mentioned in Section 3.\n- Table 1 has a lot of numbers. Given that you ran a statistical test, consider to e.g. gray out the non-signficant results for easier readability.\n- sec 4.5 has a few typos (missing spaces, etc)\n\nQUESTIONS:\n- The work seems large incremental, given that the method (3.4) is composed of techniques used in [44], [40], and label propagation. Could you highlight the main conceptual novelty of your algorithm compared to previous approaches?\n- What is the test performed in Table 1? I would suggest to include this directly into the caption.\n- l. 327 in the conclusion claims \"... can utilize the multi-scale properties of time series to effectively handle noisy labels\". Where is this particular point validated, in your opinion? \n- How are hyperparameters selected? Can this be made more explicit in the main paper?\n- What are additional limitations of the assumed noise model and correction mechanism?\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a \"scale-teaching\" framework aimed at addressing label noise in time series classification tasks. The proposed approach involves multiple encoders to extract multi-scale time series features. These features are then concatenated for the final loss calculation, enabling the selection of clean samples. The identified clean and noisy samples are further processed using a label propagation algorithm, incorporating graph learning techniques to correct the noisy labels. The entire dataset is subsequently trained using cross-entropy as the loss function. Experimental evaluations are conducted on the UCR 128 archive and UEA 30 archive datasets, both containing synthetic label noise.\n\nSTRENGTHS:\n1. While the problem of learning with noisy labels (LNL) has been extensively studied in the domain of image classification, the exploration of label noise in time series classification remains relatively limited in current literature. This work makes significant contributions by conducting extensive experiments to investigate how the LNL experience gained in image classification can be effectively generalized to time series classification.\n\n2. The utilization of multi-scale operations is particularly noteworthy in the context of time series classification with label noise, considering the inherent differences in modality between images and time series data.\n\n3. The paper demonstrates a clear and coherent writing style, ensuring ease of comprehension for readers. Additionally, the release of the accompanying code promotes reproducibility, while the appendix provides comprehensive details on the conducted experiments.\n\nWEAKNESSES:\n1. While I appreciate the contribution made by generalizing learning with noisy labels (LNL) from image classification to time series classification, the level of technical novelty in this work appears somewhat limited. Specifically, the utilization of a multi-scale framework has already been explored in time series forecasting tasks, and the application of a semi-supervised approach to enhance performance is not novel in the context of general LNL tasks [R1].\n\n2. The paper lacks sufficient ablation studies to thoroughly examine the individual components of the proposed method. For instance, in order to demonstrate the efficacy of the multi-scale framework in sample selection, it would be valuable for the authors to conduct controlled experiments using only one scale. While accuracy is reported in Table 3, it is not necessarily the most appropriate criterion for evaluating sample selection. Therefore, I suggest that the authors also report selection quality metrics such as precision, recall, and F1 score. Additionally, it would be worthwhile to explore alternative label correction techniques, such as pseudo-labeling or solely utilizing clean samples for training, to ascertain their competitiveness in comparison to the label propagation method employed in the paper.\n\n3. It would be insightful to evaluate the performance of the \"scale-teaching\" framework on datasets that do not contain label noise. I noticed that such experiments were conducted in [R2], and I encourage the authors to perform similar settings to provide a comprehensive analysis in their work.\n\n\n[R1] DivideMix: Learning with Noisy Labels as Semi-supervised Learning\n\n[R2 ]Estimating the Electrical Power Output of Industrial Devices with End-to-End Time-Series Classification in the Presence of Label Noise\n\nQUESTIONS:\nSee **weaknesses**\n\nLIMITATIONS:\nThis work extensively addresses its limitations, and based on the information provided, it appears that the paper does not pose potential negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-08"}