Scale-teaching: Robust Multi-scale Training for Time Series Classification with Noisy Labels
Authors: Zhen Liu
Peitian Ma
Dongliang Chen
Wenbin Pei
Qianli Ma
Venue: NeurIPS 2023 poster (2023)
Keywords: time series classification
deep neural networks
noisy labels
TL;DR: We propose a deep learning paradigm called Scale-teaching for time-series classification with noisy labels.

Abstract: Deep Neural Networks (DNNs) have been criticized because they easily overfit noisy (incorrect) labels. To improve the robustness of DNNs, existing methods for image data regard samples with small training losses as correctly labeled data (small-loss criterion). Nevertheless, time series' discriminative patterns are easily distorted by external noises (i.e., frequency perturbations) during the recording process. This results in training losses of some time series samples that do not meet the small-loss criterion. Therefore, this paper proposes a deep learning paradigm called Scale-teaching to cope with time series noisy labels. Specifically, we design a fine-to-coarse cross-scale fusion mechanism for learning discriminative patterns by utilizing time series at different scales to train multiple DNNs simultaneously. Meanwhile, each network is trained in a cross-teaching manner by using complementary information from different scales to select small-loss samples as clean labels. For unselected large-loss samples, we introduce multi-scale embedding graph learning via label propagation to correct their labels by using selected clean samples. Experiments on multiple benchmark time series datasets demonstrate the superiority of the proposed Scale-teaching paradigm over state-of-the-art methods in terms of effectiveness and robustness.

Scale-teaching: Robust Multi-scale Training for Time Series Classification with Noisy Labels
Zhen Liu
Peitian Ma
South China University of Technology
South China University of Technology Guangzhou, China
Guangzhou, China
cszhenliu@mail. scut, edu.
ma__cuuerr163.c com
Dongliang Chen
Wenbin Pei
South China University of Technology
Dalian University of Technology Dalian, China
Guangzhou, China
ytucdl@foxmall com
peiwenbinodlut. edu. cn
Qianli Ma"
South China University of Technology Guangzhou, China
gianlima@sen edu. cn
Abstract
Deep Neural Networks (DNNs) have been criticized because they easily overfit noisy (incorrect) labels. To improve the robustness of DNNs, existing methods for image data regard samples with small training losses correctly labeled data (small-loss criterion). Nevertheless, time series' discriminative patterns are easily distorted by external noises (i.c., frequency perturbations) during the recording process. This results in training losses of some time series samples that do not meet the small-loss criterion. Therefore, this paper proposes deep learning paradigm called Scale-teaching to cope with time series noisy labels. Specifically, we design fne-to-coarss cross-scale fusion mechanism for learning discriminative patterns by utilizing time series at different scales to train multiple DNNs simultaneously. Meanwhile, each network trained mentary information from different scales to select small-loss samples clean labels. For unselected large-loss samples, we introduce multi-scale embedding graph learning via label propagation correct their labels by using selected clean samples. Experiments on multiple benchmark time series datasets demonstrate the superiority of the proposed Scale-teaching paradigm over state-of-the-ant methods in terms of effectiveness and robustness.
tross-teaching manner by using comple-
Introduction
Time series classification has recently received much attention in deep learning 2 Essentially, the success of Deep Neural Networks (DNNs) is driven by large amount of well-labeled data. However, human errors 3 and sensor failures 4 produce noisy (incorrect) labels in time series datasets. For example, electrocardiogram diagnosis [5]. physicians with different experiences tend to make inconsistent category judgments. In recent studies 71. DNNs have shown their powerful learning ability, which, however, makes relatively easier to overfit noisy labels and inevitably degenerate
QQanli Ma is the corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Finer
1000
Vilue
Val
Valun
Valun
Value
Jake Valter
20C
430
600
700
40
50
120
160
30
80
80
49
10
26
20
40
50
Valun
100
Value
Coarser
300
Time
Time
Time (c) ElectiiDeerios
Time
(a) PLAID
(b) Lightning?
(d) Modical
Figure 1: Illustration of time series samples from the same category at different time scales. Among all samples in the same category, red indicates the one with the largest variance, and blue indicates few samples with the smallest variance.
the robustness of models. Moreover, time series data has complex temporal dynamics that make challenging to manually correct noisy labels [8].
To cope with noisy labels, existing studies on label-noise learning 9 10 use the memory effect
of DNNs to select samples with small losses for training. DNNs memorize the data with clean
labels first, and then those with noisy labels in classification training (small-loss criterion) [TT]. is worth noting that the small-loss criterion is not affected by the choice of training optimizations and network structures [2]. and is widely utilized for label-noise learning in computer vision [ 14] However, the small loss criterion cannot always be applied to time series because the discriminative For example, in a smart grid, distortions may occur due to sampling frequency perturbations, imprecise sensors, or random differences in energy consumption [I7] Such distortions can make it difficult for DNNs learn the appropriate discriminant patterns of time series, resulting in large training losses for some clean labeled samples. addition, the small-loss criterion only utilizes the data's label information and does not consider the inherent properties of time series features (i.e., multi-scale information).
patterns of time series data are easily distorted by external noises 15| 16
Multi-scale properties are crucial in time series classification tasks. In recent years, multi-scale convolution [16]. dynamic skip connections [8] 19] and adaptive convolution kernel size [20] have been utilized to learn discriminative patterns of time series. Furthermore, according to related studies 22 20 21), the selection of appropriate time scales for time series data can facilitate DNNs to learn class- characteristic patterns. With correct labels, the above studies indicate that the multi- properties of time series data can help DNNs learn appropriate discriminative patterns for mitigating the negative effects of time series recording noises. Nevertheless, it remains an open challenge as to how the multi-scale properties of time series can be used for label-noise learning.
To this end, we propose deep learning paradigm, named Seale-teaching. for time-series classification with noisy labels. In particular, we design a fine-to-coarse cross-scale fusion mechanism for obtaining robust time series embeddings in the presence of noisy labels. We select four time series datasets from the UCR archive (22) to explain our motivation. As shown in Figure L in the single scale case (top row), the red and blue samples from the same category have large differences in certain local regions (the green rectangle in Figure I By downsampling the time series from fine to coarse, some local regions between the red and blue samples did become similar. Meanwhile, existing studies [12 show that multiple DNNs with random initialization have classification divergence for noisy labeled samples, but are consistent for clean labeled samples. The above findings inspire us to utilize multiple DNNs to combine robust embeddings at different scales to deal with noisy labels. Nonetheless, the coarse scale discards many local regions in the fine scale (as in Figure [c)), which may degenerate the classification performance. Hence, we propose the Scale-teachine paradigm, which can better preserve the local discriminative patterns of fine scale while dealing with distortions.
More specifically, the proposed Scale-teaching paradigm performs the cross-scale embedding fusion in the finer-toccoarser direction by utilizing time series at different scales to train multiple DNNs simultaneously. The cross-scale embedding fusion exploits complementary information from different scales to learn discriminative patterns. This enables the learned embeddings to be more robust to distortions and noisy labels. During training, clean labels are selected through cross-teaching on those networks with the learned embeddings. The small-loss samples in training are used as (clean) labeled data, and the unsclected large-loss samples used (noisy) unlabeled data. Moreover,

---

multi-scale embedding graph learning is introduced to establish relationships between labeled and unlabeled samples for noisy label correction. Based on the multi-scale embedding graph, the label propagation theory [24] is employed to correct noisy labels. This drives the model to better fit time series category distribution. The contributions are summarized as follows:
We propose deep learning paradigm, called Scale-teaching, for time-series label-noise learning. In particular, a cross-scale fusion mechanism is designed to help the model select more reliable clean labels by exploiting complementary information from different scales.
We further introduce multi-scale embedding graph learning for noisy label correction using the selected clean labels based on the label propagation theory. Unlike conventional image label-noise learning methods focused on sample loss levels, our approach uses well-learned sclti-scale time series embeddings for noise label correction at sample feature levels.
Extensive experiments on multiple benchmark time series datasets show that the proposed Scale-teaching paradigm achieves a state-of-the-ar classification performance. In addition, multi-scale analyses and ablation studies indicate that the use of multi-scale information can effectively improve the robustness of Scalc-teaching against noisy labels.
2 Related Work
Label-noise Learning. Existing label-noise learning studies focus mainly on image data 10. These studies can be broadly classified into three categories: (1) designing noise-robust objective
functions 25 26 or regularization strategies (27 28]; (2) detecting and correcting noisy labels [3] 29 30) (3) transition -attia-
32] and semi-supervised based [14] 33) methods. In contrast to the methodologies in the first and third categories, approaches categorized under the second category have received considerable attention in recent years 7 34). Methods of the second category can be further divided into sample selection and label correction. The common methods of sample selection are the Co-teaching family [12 13 23 and FINE 35 Label correction [36] 37] attempts to correct noisy labels by either using prediction results of classifiers or pseudo-labeling techniques. Recently, SREA 4 utilizes pseudo-labels generated based on a clustering task to correct time-series noisy labels. Although the above methods can improve the robustness of DNNs, how the multi-scale properties of time series are exploited for label-noise learning has not been explored.
Multi-scale Time Series Modeling, In recent years, multi-scale properties have gradually gained attention in various time series downstream tasks (18] 38), such as time series classification, prediction, and anomaly detection [39]. For example, Cui et al. [16] employ multiple convolutional network channels of different scales to learn temporal patterns that facilitate time series classification. Chen
et al. [19] design time-aware multi-scale RNN model for human action prediction. Wang et al. [40 introduce multi-scale one-class RNN for time series anomaly detection. Also, recent 43 44] indicate that multi-scale properties can effectively improve the performance of long-term time series prediction. Unlike prior work, we utilize multiple DNNs with identical architectures to separately capture discriminative temporal patterns across various scales. This enables us to acquire robust embeddings for handling noisy labels via a cross-scale fusion strategy.
studies [1 42
Label Propagation. Label propagation (LP) is graph-based inductive inference method 24
45
that can propagate pseudo-labels to unlabeled graph nodes using labeled graph nodes. Since I. can utilize the feature information of data to obtain pseudo-labels of unlabeled samples, related works employ LP in few-shot learning [46] and semi-supervised learning [47] 48). Generally speaking, DNNs have the powerful capability for feature extraction, and the learned embeddings tend to be similar within classes and different between classes. Each sample contains feature and label information. Intuitively, the embeddings of samples with noisy labels obtained by DNNs closely align with the true class distribution when the DNNs do not fit noisy labels in the early training stages. Naturally, we create a nearest-neighhor graph based on well-learned multi-scale time series embeddings at the feature level. Subsequently, we employ LP theory to correct the labels of unselected noisy samples using the labels of clean samples chosen by the DNNs. This approach leverages robust multi-scale embeddings to address the issuc of noisy labels.
3

---

Complementary Information
Embedding Fusion
Clean Labcled
Conrected Labelod Samples
Incorred/ Unlabcled
Sumples
Samples
Finer
Encoder
Fine Scale Embeddiings A
Embeddings A
Granh Leaming
Clean Labels
Multii-scle E Embedding Graph Learning
Cross-Scale Fusion
Label Progegatiu
Encoder
Medium Scale
Embeddings (A O B) Graph Leaming
Cleen Labels
Embeddings B
Cross- Scale Pusion
Clean Labels
Encoder
Coarse Scale Embeddings
Embeddings (A O B
Coarser
Graph Leaming
The Scale-teaching paradigm's general architecture comprises two core processes: (i) clean label selection and (ii) noisy label correction. In the clean label selection phase, networks A, B, and C engage in cross-scale fusion, moving from fine to coarse (A-B, B-C)). They employ clean labels acquired through cross-teaching (A-B. B-xC C-+A) to guide their respective classification training. In the noisy label correction phase, pseudo labels derived from multi-scale embeddings graph learning are employed as corrected labels for time series not selected as clean labeled samples.
Figure 2:
Proposed Approach
3.1 Problem Definition
Given noisy labeled time series dataset D 0:)) it contains N time series, where X, RLXT, L denotes the number of variables, and T is the length of variable. E (1, (1...., C} is the observed label X; with probability of being noisy label. Our goal to enable the DNNs trained the noisy labeled training dataset Dirain correctly predict the ground-truth labels of the given time series in the test set. Specifically, the problem to be addressed in this paper consists of two steps. The first is select clean labeled time series from Draann and the second is perform noisy label correction for time series in Derain that have not been selected as clean labels.
3.2 Model Architecture
The overall architecture of Scale-teaching is shown in Figure 2 While this figure illustrates Scale- teaching with input time series three scales, can be extended to models with more scales, exceeding three. We utilize a consistent structural encoder to learn embeddings for each input scale sequence. Each encoder undergoes training at two levels: embedding learning for clean sample selection at the feature level and label correction with the multi-scale embeddings. For embedding learning, we propose a cross-scale fusion (Section 3.3) mechanism from fine to coarse to obtain robust embeddings. This approach enables the selection of more dependable clean labels through the small-loss criterion. Specifically, embeddings (A @ B C C) encompass multi-scale information from fine, medium, and coarse scale sequences derived from the same time series. Regarding noisy label correction, we introduce multi-scall embedding graph learning (Section 3.4) based on label propagation, utilizing the selected clean samples correct the labels of unselected large-loss samples.
3.3 Cross-scale Fusion for Clean Label Selection
After downsampling the original time series at different scales, it eliminates some of the differences in local regions between samples of the same category (as in Figure []. However, the downsampled sequences (i.e., coarse scale) discard many local regions of the original time series. This tends to degrade the model's classification performance if the downsampled sequences are used directly for classification (please refer to Table(2]in in the Experiments section). Meanwhile, existing studies [12 23 on label-noise learning show that DNNs with different random initializations have high consistency in classification results clean labeled samples in the early training period, while there is disagreement in the classification of noisy labeled samples. Based on the above findings, we utilize multiple DNNs (or encoders) with different random initializations to learn embeddings of different downsampled scale sequences separately, and perform cross-scale fusion. On the one hand, exploit complementary

---

information between adjacent scale embeddings to promote learned embeddings to be more robust for classification. On the other hand, we leverage the divergence in the classification of noisy labeled samples by different DNNs to mitigate the negative impact of noisy labels in training. In this way, we can utilize the cross-scale fusion embeddings for classification, thus better using the small loss criterion [1 [9] for clean label selection. Specifically, downsampling is employed to generate different scale sequences from the same time series. Given time series X; {2 22,,..,T7 supposing the downsampling ratio is k. Then, we only keep data points in X; as follows:
x = (mkes] ,J =1,2 K
where & [1,T/2], and larger k indicates that X, is coarser. As shown Figure 2 time series with multiple downsampling intervals (i.e., k= = 1, 2, 4) is treated as the input data for training. To better utilize the small-loss criterion for clean label selection, each time series sample performs cross-scale fusion from fine to coarse (i.e., A- B B-C) in the embedding space, which is mathematically defined as:
x = (r+ 10 | (r. -x (r. "
(2)
where r represents the single-scale embedding acquired by learning X through an encoder. Mean- while, of (or ot denotes the embedding of the time series X (or x;-²) after performing cross-scale fusion. Here, t denotes the interval between adjacent downsampling ratios, and signifies the concatenation of two vectors to form a new vector. Notably, when & = 1, we employ the single- scale for classification training, resulting in o = r. By combining (r - o and (r of=²) for vector concatenation, of can capture more nuanced discriminative information between rf and v k-t than that of simply concatenating r with u.=² The function f(-) represents two-layer nonlinear network mapping function for fusing information of rf and of Additionally, of has the same dimension as rf and serves the input data for the multi-scale embedding graph learning process.
3.4 Multi-scale Embedding Graph Learning for Noisy Label Correction
We now present the multi-scale embedding graph learning module for correcting noisy labels. This module incorporates selected clean labels using label propagation theory. The process consists of two stages: graph construction and noisy label correction.
Graph Construction. It is assumed that the of cross-fusion embeddings obtained from
v. batch of time series is defined as V = {vf, ......, J. where M is the batch size. Intuitively, samples close to each other in the feature space have high probability of belonging to the same class. However, in label-noise learning, o obtained from the current iterative training of the model may have unstable information, resulting in large deviations in the information of the nearest-neighbor samples of uft. To address this issue, the proposed approach performs momentum update [49 on of during training, which is defined as:
Jर aot + (1 a- 1].
where e is the current training epoch and denotes the momentum update parameter.
The multi-scale embeddings nearest-neighhor graph can be created by using Euclidean distance among different o². A common approach is the use of the Gaussian similarity function 45 to obtain the nearest-neighbor graph edge weight, which is defined as:
exp (
We
-
(4)
where d(.) the Euclidean distance function and a is a fixed parameter. W XMMx is a symmetric adjacency matrix, and the element Wij denotes the nearest-neighhor edge weight between the embedding of and oy (note that larger values indicate closer proximity). Then, W is normalized based on the graph laplacians (50) to obtain Q = D-1/2W D-1/2, where D = diag(W11 is a diagonal matrix. Specifically, the K neighbors with the largest values in each row of Q are employed to create the nearest-neighhor graph. It is noteworthy that the embeddings in each mini-batch are utilized to generate the nearest-neighhor graph, thus obtaining Q within short computational time.
5

---

Noisy Label Correction. Specifically, small training loss samples acquired by DNNs in the early training period can be considered as clean samples, while samples with large training losses are considered as noisy ones [12 [4]. The above learning pattern of DNNS has been mathematically
validated [1 (see Appendix A for details). Under this criterion, prior studies [12 13 23 have typically employed samples with small losses after a Cwarm warm-up training as clean labels. Following [29], we extend the small-loss sample selection process to operate within each class, thereby enhancing the overall quality of the chosen clean labels. In our method, samples chosen with clean labels are considered labeled data, whereas unselected samples are treated as unlabeled data.
We utilize clean samples selected from time series at different scales in cross- cros-tee manner (as in Figure 2). This could explore complementary information from different scale fusion embeddings to deal with noisy labels. It is supposed that there is a corresponding one-hot encoding matrix Y E MMC (Y,, E (0,1)) for the cross-fusion embeddings V. If yi is identified as a clean label, we employ YI to set Y; as one-hot encoded label. Otherwise, all the elements in Y; are identified as zero. Through Y. the pseudo-label of each node in the nearest-neighhor graph Q can be obtained in an iterative way based on the label propagation theory. The specific solution formula is defined as:
F FQF; + (1-
(5)
where Fi E RMXC denotes the predicted pseudo-label of the f-th iteration and B E (0, 1) is hyperparameter. Naturally, F, has closed-form solution (24 defined as follows:
F -1-133 (1))
(6)
where F RM is the final pseudo-labels and / denotes the identity matrix. Finally, the corrected label obtained for an unselected large-loss sample X; is defined as:
V/ arg max
However, F is the estimated pseudo-labels, which inevitably contain some incorrect labels. To address this issue, two strategies are used to improve the quality of pseudo-labels in F. For the first strategy, the model continues training Cupdate epochs by using small-loss samples after ewarm epochs warm-up training to improve the robustness of the multi-scale embeddings. Then, the noisy label correction is performed after (Ewarm Eupdate) epoch. For the second strategy, dynamic threshold (e(ee) maxx(a,) 7 is utilized for each class [51] select the pseudo-labels with a high confidence for noisy label correction, where (c) is the number of labeled samples contained in class c in the e-th epoch, and 7 is constant threshold.
8_(c)
Overall Training. Finally, each encoder utilizes the selected clean samples in combination with multi-scale embedding graph learning to perform noisy label correction for unselected large-loss samples. Combining the training data of the selected clean labels and those of corrected labels, the proposed Scale-teaching paradigm utilizes cross-entropy for time-series label-noise learning. Please refer Algorithm I in the Appendix for the specific pseudo-code of Scale-teaching.
4 Experiments
4.1 Experiment Setup
Datasets. We use three time series benchmarks (four individual large datasets 3,552 53 UCR
128 archive [22] and UEA 30 archive (54) for experiments. Among the four individual large
5x3 is datasets, HAR [52 and UniMiB-SHAR 3 are human activity recognition scenarios; FD-A
the mechanical fault diagmosis scenario; Sleep-EDF [52] belongs to the sleep stage classification scenario. The UCR archive [22 contains 128 univariate time series datasets from different real-world scenarios. The UEA archive (54] contains 30 multivariate time series datasets from real-world scenarios. For details on the above datasets, please refer to Appendix B Since all the datasets in three time series benchmarks are correctly labeled, we utilize a label transtormation matrix to add noises to the original correct labels 44) where Tiş is the probability of label being flipped to j. We use three types of noisy labels for evaluations, namely Symmetric (Sym) noise, Asymmetric (Asym) noise, and Instance-depender (Ins) noise. Symmetric (Asymmetric) noise randomly replaces a true label with other labels with an equal (unequal) probability. Instance noise [5 means that the noisy label is -nstance-dependent. Like 44 12 23 we use the test set with correct labels for evaluations.

---

as the encoder of Scale-teaching Table 1: Test classification accuracy results compared with baselines on three time series benchmarks. The best results are bold, and the second best results are underlined. When P-value < 0.05, it indicates
that the performance of Scale-teaching is statistically significant than the baseline.
Dataset
Noise Ratio
Metric
Standard
Mixup
Co-teaching
FINE
SREA
SELC
CULCU
Seale-teaching
Sym 20%
Avg Rank
4.75
4.75
4.50
7.50
6.50
4.50
2.50
1.00
Four individual
Sym 50%
Avg Rank
4.75
4.50
4.75
7.25
5.75
4.50
3.25
large datasets
Asym 40%
Avg Rank
5.00
5.50
3.75
7.50
5.75
4.00
3.25
1,00)
Ins 40%
Avg Rank
4.75
4.25
4.25
7.25
6.00
4.75
3.50
1,00
Avg Rank
4.15
4.33
3.61
7,50
6,16
3.48
3,54
3,02
Sym 20%
P-value
1.90F-04
4.06F-05
1 190E-03
1.49F-34
1.70E-17
3.04F-03
8,57E-03
Sym 50%
Avg Rank
4.31
4.57
4.05
6.43
5.89
3.56
3.86
3.11
UCR 128 archive
P-value
3.15E-05
1.70E-05
4.02E-04
7.48E-19
1.22E-15
1.40E-02
4.93E-03
Avg Rank
4.38
4.80
3,93
6.91
5.91
3.30
3.67
2.95
Asym 40%
P-value
1662E-05
3.53E-07
6.10E-04
1.93E223
9.82E-14
1.89E-02
2.24E-02
Avg Rank
4.05
4.52
4.02
7.04
6.18
3.30
3.77
2.95
Ins 40%
P-value
1.43E:05
1.815:06
2.43E:04
9.818:26
26
2.36F
17
3.276:02
1.54E:02
Avg Rank
5.03
5.20
3.83
6.37
4.77
3.73
4.00
Sym 20%
P-value
6.61E-04
3.33E-04
2.69E-02
2.37E-05
1.14E-02
2.63E-02
3.93E-02
Avg Runk
5.17
5.73
4.23
6,23
3.93
3.83
4.30
2.43
Sym 50%
UEA 30 archive
v-vuue
2.98E 04
7.40E
05
1. 1.59F-02
9.35E-05
1.67E02
1. 1.08E:
02 3.75E:02
Asym 40%
Avg Rank
5.60
4.77
4,40
6,13
4,20
4.00
3.97
2.73
P-value
3881E-03
6.17E-03
1,63E-02
9,33E-05
1.361-00
2.62F-02
3.88E-02
Avg Rank
5.20
4.77
4.33
6.60
4.27
4,20
3.77
2.60
Ins 40%
P-value
6.08E-04
2,92E-03
1220-002
2,55E-05
5,52E-03
1.08E-02
3.47E-02
Baselines. We select seven methods for comparative analyses, namely 1) Standard: direct training of the model using cross-entropy with all noisy labels; 2) Mixup [56]: 3) Co-teaching [12]: 4)
FINE (35): 5) SREA 44: 6) SELC (37;; and 7) CULCU (23. Among them, Standard, Mixup. and
Co-teaching are the benchmark methods for label-noise learning. FINE, SELC, and CULCU are the state-of-ther of-the-art methods that do not need focus on data types, and SREA is the offte-of-thhear method in time series domain. In addition, for fair comparisons, all the baselines and the proposed Scale-teaching paradigm use the same encoder and classifier. We focus on the ability of different label-noise learning paradigms to cope with time series noise labels, rather than the classification performance achieved by using fully correct labels. Hence, considering the trade-off between the running time and classification performance, we choose FCN [57] For more details of baselines, please refer to Appendix
Implementation Details. Based the experience 119 44 in time series modeling, utilize three different sampling intervals 1, 2, 4 as the input muti-scale series data for Scale-teaching. We use Adam the optimizer. The learning rate is set 1e-3, the maximum batch size is set to 256, and the maximum epoch is set to 200. Ewarm is set to 30 and Eupdate is set to 90. a in Eq. 3 set to 0.9, o in Eq. 4 is set to 0.25, ,3 in Eq. is set to 0.99, the largest neighbor K is set to 10, and y is set to 0.99. In addition, following the parameter settings suggested in (23), we linearly decay the learning rate to zero from the 80-th epoch 200-th epoch. For comprehensive understanding of the hyperparameter selection and the implementation of the small-loss criterion applied to tealh-tea please consult Appendix C To reduce random errors, we utilize the mean test classification accuracy of the last five epochs of the model on the test set as experimental results. All the experiments are independently conducted five times with five different seeds, and the average classification accuracy and rank are reported. Finally, we build our model using Py PyTorch 1.10 platform with 2 NVIDIA GeForce RTX 3090 GPUs Our implementation of Scale-teaching is available at https: //gtthub...tt.....m wtcchhu...om//ggee cow/qianl ima--iiiniimmmiim wab//aaaa--eaaahhiaa
4.2 Main Results
We evaluate each time series benchmack using four noise ratios, Sym 20%, Sym 50% Asym 40%,
and Ins 40%. Due to space constraints, we only give the average ranking of all the methods on each benchmark in Table I Please refer to Appendix D for the specific test classification accuracies. Besides, for UCR 128 and UEA 30 archives, we use the Wilcoxon signed rank test (P- (P-value) (58) to analyze the classification performance of baselines. As shown in Table the proposed Scale-teaching paradigm achieves the best Avg Rank in all the cases. It is found that Mixup (56) and FINE [35]
perform worse than the Standard method in most cases. For Mixup, the complex dynamic properties
7

---

Pinescale
Modium scale Srale teaching
Coarsossaale
[Fine Medium) scale
(Fine Medium Coarsel stale
Cross-scale
wib
leaching
Cross
Scale teaching
VS.
VS.
375
196
71
106
147
116
17
3R
26
33
2077
361
Fine - Medium
Medium Coarse
(c)
Figure 3: Venn diagram of the average number of correctly classified samples for the different scale sequences of UCR 128 archive with Sym 20% noisy labels. The numbers in the figure indicate the complements and intersections of classification results different scales.
Table The test classification accuracy (%) results of different scale classifiers on UCR 128 archive. The best results are bold, and the second best results are underlined. When P-value < 0.05, it indicates that the performance of Scale-teaching's coarse scale classifier is significant than other classifiers.
Method
w/o Cross-scale fusion
Scale-teaching
Noise Ratio
Metric
Fine
Medium
Coarse
Fine
Medium
Coarse
Avg Acc
65.13
30.11
28.17
59.67
68.17
68.70
Sym 20%
Avg Rank
2.38
5.09
5.37
3.20
2.17
2.11
P-value
1.89--03
2.85E-37
2.07E-40
1.58E-09
3.74E-02
Avg Acc
49.61
29.01
28.87
47.75
51.93
52.87
Asym 40%
Avg Rank
2.64
4.75
3.01
2.45
2.27
P-value
1.94E-03
6.78E-25
1,59E-27
1.80E-07
2.80E-02
of the original time series are destroyed probably due to the mixture of two different time series mechanisms. FINE uses embeddings of the input data to select clean labels. Although FINE achieves advanced classification performance for image data, difficult be used directly for time series data because its discriminative patterns are easily distorted by external noises. SREA [4] has good performance on UEA 30 archive, while it performs poorly on the other benchmarks. Meanwhile, Co-teaching [12,, SELC (37). and CULCU (23) are more robust against time series noisy labels in different cases, further indicating that the small-loss criterion is also applicable to time scries.
4.3 Multi-scale Analysis
To explain the multi-scale mechanism in the Scale-tcaching paradigm, we add an ablation study based on Scale-teaching (w/o cross-scale fusion). We select the UCR 128 archive analyze the classification results obtained by the fine, medium, and coarse scale classifiers. As shown in Figure 3 the classification results of different scale sequences have evident complementary information. Scale- teaching can effectively use complementary information between cross-scale to obtain more robust embeddings and clean labels. In response to the tendency the coarse scale ignore discriminative patterns in fine scale (please see Table our proposed cross-scale fusion mechanism can effectively improve the classification performance Please refer to Appendix E for the specific classification results of Figure Jnd Table In Appendix we also analyze the order and size the downsampled input scale sequence for Scale-tcaching.
medium and coarse scales while retaining complementarity,
Scale-teaching utilizes multi-scale embeddings to generate the nearest-neighhes graph, and uses clean labels selected for noisy label correction. To explore the distribution of different classes of embeddings, we employ t-SNE [59] for dimensionality reduction visualization. Specifically, we utilize the UniMiB-SHAN
dataset containing Sym 20% noisy labels for visualization. As shown in Figure 4 we find that the embeddings learned by Scale-teaching are more discriminative across classes than the Standard and CULCU methods that use single scale series for training. The above results suggest that Scale-teaching can effectively exploit the complementary information between different scales, prompting the learned embeddings to more discriminative between classes. In addition, choose the FD-A dataset for t-SNE visualization, and please refer to Appendix E

---

8
40
T-NN
40
-60
40
10 (a) Sandord (05.43%)
0
20
-00
20
-60 -10
20
8
TSNE
T-NEE
T:SNE
(b) CULCU (8339%)
Scale traching (91.32%)
Figure 4: t-SNE visualization of the learned embeddings on the UnimiB-SHAL dataset with Sym 20% noisy labels (values in parentheses are the test classification accuracies).
Stenber (Cress Entrrgy)
Scale teaching (Ous)
Stankket (Cruas Entropy)
Scale-teaching (Ours
3.55
Dean
clerr
way
Less volue
3.70
3.50
Loas velue
1.6
2.6
3.85
1.4
2.4
360
3:30
1.2
1.2
3.50
3.23
345
1.0
155
50
150
200
50
250
200
350
200
Epoch
Epoch
Ppoch
Bpoch
(a) NanitwaaivveeeaaBBOOToeee
(b) ECG5000
Figure 5: The change of loss values for clean and noisy time series samples under Aysm 40% noise labels. The solid line and shading indicate the mean and standard deviation loss values of all clean (or noisy) training samples within each epoch.
4.4 Small-loss Analysis
To analyze the application of the small-loss criterion time series data, we visualize the change of loss values for ground-truth clean and noisy time series samples during training. Specifically, Figure shows the change in loss values of the models trained by the Standard method and Scale-teaching two UCR time series datasets. When the model is trained with the Standard method, differences can be found in the loss values of clean and noisy samples in the network early training, especially in Figure 5 (b). The Standard method makes the model gradually fit the noisy samples as the training proceeds, while Scale-teaching improves the ability of the model handle noisy labels. To further prove its efffectiveness, we selected two other UCR datasets for the loss value change analysis, which have the same pattern as Figure 5 Also, report the HAR and UniMiB-SHAR dataset's loss value probability distributions of clean and noisy samples. For more details, please refer to Appendix
4.5 Ablation Study
To verify the robustness of each module in Scale-teaching, the ablation experiments have been conducted in the HAR and UniMiB-SHAR datasets, and the results are shown in Table Specifically, (I) w/o cross-scale fusion: the cross-scale embedding fusion from fine to coarse mechanism is ablated; (2) only single scale: only the original time series is used for training; (3) w/o graph learning: the multi-scale embedding graph learning module for noisy label correction is ablated; (4) w/o moment: the embedding momentum update mechanism (Eq. is ablated; (5) w/o dynamic threshold:
using a dynamic threshold to select high-quality propagation pseudo-labels is ablated.
As shown in 3 the cross-scale fusion strategy (w/o cross- fusion) and the clean labels cross- teaching mechanism (only single scale) can effectively improve the classification performance of Scale-teaching, especially on the UniMiB-SHA. in terms of label correction based on multi-scale embedding graph learning, the results of the corresponding ablation module show that improving the stability of embedding (w/o moment) and
dataset with large number of classes. Meanwhile,

---

Table 3: The test classification accuracy (%) results of ablation study (values in parentheses denote drop accuracy).
HAR
UniMiB SHAR
Method
Sym 50%
Asym 40%
Sym 50%
Asym 40%
Scale- Scale-teaching w/o cross-scale fusion
90.17
89.62
81.31
70.68
88.47 (-1.70)
87.64 (-1.98)
73.32 (-7.99)
61.62 (-9.06)
only single scale
89.01 (-1.06)
88.11 (-1.51)
69.89 (-11.42)
60.32 (-10.36)
w/o graph learning
88.06 (-2.11)
87.65 (-1.97)
79.72 (-1.59)
68.87 (-1.81)
w/o moment w/o dynamic threshold
89.76 (-0.41)
88.76 (-0.86)
80.57 (-0.74)
69.85 (-0.83)
89.12 (-1.05)
88.75 (-0.87)
77.42 (-3.89)
69.53 (-1.15)
Table 4: The test classification accuracy (%) results on four individual large datasets without noisy labels. The best results are bold, and the second best results are underlined.
Dataset
Standard
Mixup
Co-teaching
FINE
SREA
SELC
CULCU
Scale-teaching
HAR 93.29
95.42
93.77
93.13
93.02
93.76
94.75
94.72
UniMiB-SHAR
89.14
84.84
88.24
88.14
65.51
89.28
89.46
93.61
FD-A
99.93
99.96
68.22
90.25
99.82
99.95
99.96
Sleep-EDF
84.93
84.67
85.37
84.62
79.42
84.82
85.54
85.34
selecting high-quality pscudo-labels (w/o dynamic threshold) can effectively improve the performance of label correction based on graph learning.
Furthermore, we select the four individual large datasets without noisy labels for evaluation. As shown in Table Scale-teaching's classification performance still better than most baselines. It's worth mentioning that SREA [4] employs an unsupervised time series reconstruction loss as an auxiliary task, which reduces the model's classification performance without noisy labels. We also provide the corresponding test classification results for Tables Saaleesaandgg [and[Junder the Fl-score metric in Appendix G Additionally, we find the running time of teachin...ing. which is faster than FINE, SREA and CULCU for datasets with larger number of samples longer length of the sequence. We further analyze the classification performance the proposed Scale-teaching paradigm and time series classification methods [15 60) in Appendix G
Conclusions
The input scales of our proposed Scale-teaching paradigm can only select fixed number of scales for training, and the running time will increase as the number of scales increases.
Limitations.
Conclusion. In this paper, we propose a deep learning paradigm for time-series classification with noisy labels called Scale-teaching. Experiments on the three time series benchmarks show that the Scale-teaching paradigm can utilize the multi-scale properties of time series to effectively handle noisy labels. Comprehensive analyses on multi-scale and ablation studies demonstrate the robustness of the Scale-teaching paradigm. In the future. will explore the design of scale-adaptive time-series label-noise learning models.
Acknowledgments
We thank the anonymous reviewers for their helpful feedbacks. We thank Professor Eamonn Keogh and all the people who have contributed the UCR 128 archive, UEA 30 archive, and the four large individual time series classification datasets. The work described in this paper was partially funded by the National Natural Science Foundation China (Grant Nos. 62272173, 61872148. 62206041), the Natural Science Foundation of Guangdong Province (Grant Nos. 2022A15150101 0019A151501011 the Science and Technology Planning Project Guangdong Province (Grant No. 2023A0505050106) the Fundamental Research Funds for the Central Universities under grants D30 The authors would like to thank Siying Zhu, Huawen Feng. Yu Chen, and Junlong Liu from SCUT for their review and helpful suggestions.
10

---

References
[1]
Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre- Alain Muller. Deep learning for time series classification: review. Data mining and knowledge discovery, 33(4):917-963, 2019.
[2] Mingyue Cheng, Qi Liu, Zhiding Liu, Zhi Li, Yucong Luo, and Enhong Chen. Formertime: Hierarchical multi-scale representations for multivariate time series classification. arXiv preprint arXiv:2302.0985 098188 2023.
[3] Gentry Atkinson and Vangelis Metsis. Tsar: a time series assisted relabeling tool for reducing label noise. In The 14th PErvasive Technologies Related to Assistive Environments Conference, pages 203-209, 2021.
[4] Andrea Castellani, Sebastian Schmitt, and Barbara Hammer. Estimating the electrical power output of industrial devices with end-to-end time-series classification in the presence of label noise. In Joint European Conference on Machine Learning and Knowledge Discovery in Dutabases, pages 469-484. Springer, 2021.
[5] Benoît Frénay, Gaël de Lannoy, and Michel Verleysen. Label noise-tolerant hidden markov models for segmentation: application to ecgs. Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 455-470. Springer, 2011.
[6] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107- 115, 2021.
[7] Sheng Liu, Zhihui Zhu, Qing Qu, and Chong You. Robust training under label noise by over-parameterization. arXiv preprint arXiv:2202.14022 2022.
[8] Gentry Atkinson and Vangelis Metsis. A survey of methods for detection and correction of noisy labels in time series data. In IFIP International Conference on Artificial Intelligence Applications and Innovations, pages 479-493. Springer, 2021.
[9] Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International conference on machine learning, pages 2304-2313. PMLR, 2018
[10] Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey, IEEE Transactions on Neural Networks and Learning Systems, 2022.
[11] Xian-Jin Gui, Wei Wang, and Zhang-Hao Tian. Towards understanding deep learning from noisy labels with small-loss criterion. arXiv preprint arXiv:2106.09291 2021.
[12] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems, 31, 2018.
[13] Xingrui Yu, Bo Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? In International Conference on Machine Learning, pages 7164-7173. PMLR, 2019.
[14] Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. arXiv preprint ar arXiv:2002.07339, 2020.
[15] Patrick Schäfer. The boss is concerned with time series classification in the presence of noise. Data Mining and Knowledge Discovery, 10-155055300 2015.
[16] Zhicheng Cui, Wenlin Chen, and Yixin Chen. Multi-scale convolutional neural networks for time series classification. arXiv preprint 1rXiv:1603.0699 2016.
[17] Patrick Schäfer and Ulf Leser. Fast and accurate time series classification with weasel. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pages 637-646, 2017.
11

---

[18] Shiyu Chang, Yang Zhang, Wei Han, Mo Yu, Xiaoxiao Guo, Wei Tan, Xiaodong Cui, Michael Witbrock, Mark A Hasegawa-Johna and Thomas S Huang. Dilated recurrent neural networks. Advances in neural information processing systems, 30, 2017.
[19] Zipeng Chen, Qianli Ma, and Zhenxi Lin. Time-aware multi-scale mns for time series modeling. In IJCAI, pages 2285-2291, 2021.
[20] Wensi Tang, Guodong Long, Lu Liu, Tianyi Zhou, Michael Blumenstein, and Jing Jiang. Omni- scale cnns: simple and effective kernel size configuration for time series classification. In International Conference Learning Representations, 2022.
[21] Víctor Campos, Brendan Jou, Xavier Giró-i Nieto, Jordi Torres, and Shih-Fu Chang. Skip rnn: Learning to skip state updates in recurrent neural networks. arXiv preprint arXiv: 1ee..17000000 06834, 2017.
[22] Hoang Anh Dau, Anthony Bagnall, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, and Eamonn Keogh. The ucr time series archive. IEEE/CAA Journal of Automatica Sinica, 6(6): 193::19931335,, 2019.
[23] Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Jun Yu, Gang Niu, and Masashi Sugiyama. Sample selection with uncertainty of losses for learning with noisy labels. In International Conference on Learning Representations, 2022.
[24] Dengyong Zhou, Olivier Bousquet, Thomas Lal, Jason Weston, and Bernhard Schölkopf. Learning with local and global consistency. Advances in neural information processing systems, 16, 2003.
[25] Yisen Wang. Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey, Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 322-330, 2019.
[26] Xiong Zhou, Xianming Liu, Junjun Jiang. Xin Gao, and Xiangyang Ji. Asymmetric loss functions for learning with noisy labels. In International conference machine learning, pages 12846-12856. PMLR, 2021.
[27] De Cheng, Yixiong Ning, Nannan Wang, Xinbo Gao. Heng Yang. Yuxuan Du, Bo Han, and Tongliang Liu. Class-dependent label-noise learning with cycle-consistency regularization. Advances in Neural Information Processing Systems, 35:1 1141110-11116,, 2022.
[28] Kilian Fatras, Bharath Bhushan Damodaran, Sylvain Lobry. Remi Flamary, Devis Tuia, and Nicolas Courty. Wasserstein adversarial regularization for learning with label noise. IEEE Transactions Pattern Analysis and Machine Intelligence, 2021.
[29] Nazmul Karim, Mamshad Nayeem Rizve, Nazanin Rahnavard, Ajmal Mian, and Mubarak Shah. Unicon: Combating label noise through uniform selection and contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2022.
[30] Deep Patel and PS Sastry. Adaptive sample selection for robust learning under label noise. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2023.
[31] Bo Han, Jiangehao Yao, Gang Niu, Mingyuan Zhou, Ivor Tsang, Ya Zhang, and Masashi Sugiyama. Masking: A new perspective of noisy supervision. Advances in neural information processing systems, 31, 2018.
[32] Zhaowei Zhu, Jialu Wang, and Yang Liu. Beyond images: Label noise transition matrix estimation for tasks with lower-quality features. In International Conference on Machine Learning, pages 27633-27653. PMLR, 2022.
[33] Sheng Liu, Jonathan Weed, Narges Razavian, and Carlos Brrmandez-Grande Early- learning regularization prevents memorization of noisy labels. Advances in neural information processing systems, 13:20331-20343 2020.
12

---

[34] Mingchen Li, Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with early stopping provably robust to label noise for overparameterized neural networks. In International conference on artificial intelligence and statistics, pages 4313-4324. PMLR, 2020.
[35] Taehyeon Kim, Jongwoo Ko, JinHwan Choi, Se- Young Yun, et al. Fine samples for learning with noisy labels. Advances in Neural Information Processing Systems, 34:24137-24149 2021.
[36] Eric Arazo, Diego Ortego, Paul Albert, Noel O' O'Conoor, and Kevin McGuinness. Unsupervised label noise modeling and loss correction. In International conference on machine learning, pages 312-321. PMI.R, 2019.
[37] Yangdi Lu and Wenbo He. Selc: Self-ensemble label correction improves learning with noisy labels. arXiv preprint arXiv:2205.01156 2022.
[38] Lifeng Shen, Zhuocong Li, and James Kwok. Timeseries anomaly detection using temporal hierarchical one-class network. Advances in Neural Information Processing Systems, 33:13016- 13026, 2020.
[39] Qianli Ma, Zhen Liu, Zhenjing Zheng, Ziyang Huang, Siying Zhu, Zhongzhong Yu, and James Kwok. A survey time-series pre-trained models. arXiv preprint arXiv:2305.10711 2023.
[40] Zhiwei Wang, Zhengzhang Chen, Jingchao Ni, Hui Liu, Haifeng Chen, and Jiliang Tang. Multi- scale one-class recurrent neural networks for discrete event sequence anomaly detection. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pages 3726-3736 2021.
[41] Junchen Ye, Zihan Liu, Bowen Du Leilei Sun, Weimiao Li, Yanjie Fu, and Hui Xiong. Learning the evolutionary and multi-scale graph structure for multivariate time series forecasting. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2296-2306. 2022.
[42] Amin Shabani, Amir Abdi, Lili Meng, and Tristan Sylvain. Scaleformer: iterative multi-scale refining transformers for time series forecasting. arXiv preprint arXiv:2206.04036 2022.
[43] Huiqiang Wang, Jian Peng, Feihu Huang, Jince Wang, Junhui Chen, and Yifei Xiao. Micn: Multi-scale local and global context modeling for long-term series forecasting. In The Eleventh International Conference on Learning Representations, 2023.
[44] Tian Zhou, Ziqing Ma. Qingsong Wen, Liang Sun, Tao Yao, Rong Jin, al. Film: Fre- quency improved legendre memory model for long-term time series forecasting. arXiv preprint 2.Xiv:2205.08897, 2022.
[45] Fei Wang and Changshui Zhang. Label propagation through linear neighborhoods. In Proceed- ings of the 23rd international conference on Machine learning, pages 985-999-97 2006.
[46] Yanbin Liu, Juho Lee, Minseop Park, Saehoon Kim, Eunho Yang, Sung Ju Hwang, and Yi Yang. Learning to propagate labels: Transductive propagation network for few-shot learning. arXiv preprint 1805 10002, 2018.
[47] Konstantinos Kamnitsas, Daniel Castro, Loic Le Folgoe. Ian Walker, Ryutaro Tanno, Daniel Rueckert, Ben Glocker, Antonio Criminisi, and Aditya Nori. Semi sumi-supervised learning via compact latent space clustering. In International conference on machine learning, pages 2459-2468. PMLR, 2018
[48] Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. Label propagation for deep semi-supervised learning. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 5070-5079, 2019.
[49] Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning, arXiv preprint arXiv:1610.02242, 2016.
[50] Fan RK Chung. Spectral graph theory, volume 92. American Mathematical Soc., 1997.
13

---

[51] Bowen Zhang, Yidong Wang. Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting cemi-suuer learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems, 34:18408-18419 2021.
[52] Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting. arXiv preprint arXi::216.. 14112, 2021.
[53] Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik. Self-supervised contrastive pre-training for time series via time-frequency consistency. arXiv preprint arXiv:2206.8849, 2022.
[54] Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The uea multivariate time series classification archive, 2018. arXiv preprint
2018.
[55] Jiacheng Cheng, Tongliang Liu, Kotagiri Ramamohanarao, and Dacheng Tao. Learning with bounded instance and label-dependent label noise. In International Conference Machine Learning, pages 1789-1799. PMLR, 2020.
[56] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 1710.0942, 2017.
[57] Zhiguang Wang, Weizhong Yan, and Tim Oates. Time series classification from seratch with deep neural networks: A strong baseline. In 2017 International joint conference on neural networks (IJCNN), pages 1578-1585, IEEE, 2017.
[58] Janez Demšar. Statistical comparisons of classifiers over multiple data sets. The Journal of Machine learning research, 7:1-30 2006.
[59] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9111), 2008
[60] Angus Dempster, François Petitjean, and Geoffrey Webb. Rocket: exceptionally fast and accurate time series classification using random convolutional kernels. Data Mining and Knowledge Discovery, 34(5))1454-1444) 2020.
[61] Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra Perez, and Jorge Luis Reyes Ortiz. public domain dataset for human activity recognition using smartphones. In Proceedings of the 21th international European symposium on artificial neural networks, computational intelligence and machine learning, pages 437-442, 2013.
[62] Daniela Micucci, Marco Mobilio, and Paolo Napoletano. Unimib shar: A dataset for human activity recognition using acceleration data from smartphones. Applied Sciences, 7(10):1101, 2017.
[63] Christian Lessmeier, James Kuria Kimotho, Detmar Zimmer, and Walter Sextro. Condition monitoring of bearing damage in clectromechanical drive systems by using motor current signals electric motors: benchmark data set for data-driven classification. In PHM Society European Conference, volume 3, 2016.
[64] Ralph G Andrzejak, Klaus Lehnertz, Florian Mormann, Christoph Rieke, Peter David, and Christian E Elger. Indications of nonlinear deterministic and unite-dimensional structures in time series of brain electrical activity: Dependence on recording region and brain state. Physical Review E, 64(6):061 1907, 2001.
[65] Hassan Ismail Fawaz, Benjamin I Lucas, Germain Forestier, Charlotte Pelletier, Daniel F Schmidt, Jonathan Weber, Geoffrey Webb, Lhassane Idoumghar, Pierre-Alain Muller, and François Petitjean. Inceptiontime: Finding alexnet for time series classification. Data Mining and Knowledge Discovery, 34(6):1936-1962, 2020.
14

---

Supplementary Material:
Scale-teaching: Robust Multi-scale Training for Time Series Classification with Noisy Labels
A
Small-loss Criterion
DNNs have been widely known to first learn simple and generalized patterns, which is achieved by learning clean data. After that, the networks gradually overfit noisy ones. In other words, when we train model with dataset containing incorrectly labeled samples, we can consider the samples with small training losses as clean ones and use them to update the model. Formally, let f be the target concept which determines the true label of and model g" (a;; O*) minimizing the expected loss, i.c.
θ* arg (a. (< 0).
8
Then, the small-loss criterion can be stated as follows[1]:
Theorem 1. Suppose y is e-close to g", i.e., llg - 90 for two examples (æ1, D) and (222 V). assume f (x1) = y and fr (x2) # y. if satisfies the diagonally-dominan condition Ta > max {maxj+s Tıj. max Tzs}: Vi, and loe (g (1z2), .0).
(Tyg (=a))))) then Ice (g ( <
The work [ provides the proof of this theorem. It shows that during training, the model can select clean samples according the loss values. The reason is that the loss values of clean samples among the samples with the same observed labels are smaller.
worth noting that the theorem is under the assumption of the class-dependent noise type and requires the transition matrix satisfy the miggonally-dominae condition. Additionally, the finite data may also make the conditions of the theorem difficcult to hold because the model g may be far away from g".
B Dataset Information
To evaluate the robustness our proposed Scale-teaching and baselines on the time-series label-noiss learning task, we selected three benchmark time-series datasets for experimental analysis.
B.1 Four individual large datasets
The statistical information of the four individual time series datasets is shown in Table 5 And the specific dataset information is as follows:
Human Activity Recognition (HAR)
The HAR dataset (52 61 is collected from 30 students performing six human actions (i.e., walking,
walking upstairs, downstairs, standing, sitting, and lying down) by wearing sensors.
University of Milano Bicoeca Smartphone-based Human Activity Recognition (UniMiB SHAR)
The UniMiB SHAR dataset 3 62 human activity information collected sampling rate of 50 Hz from volunteers with a smartphone with an accelerometer sensor in the front pocket of their pants. Specifically, each accelerometer entry is labeled by specifying the type of ADL (e.g., walking, sitting. or standing) or the type of fall (c.g., forward, fainting, or backward).
Faulty Detection Condition A (FD-A)
The FD-A dataset 52
is generated by an electromechanical drive system that monitors the condition of rolling bearings and detects their failure. Each rolling bearing can be classified into three categories: undamaged, inner damaged, and externally damaged.
63
Sleep Stage EEG Signal Classification (Sleep-EDF)
The Sleep-EDF dataset (52 64 includes the whole night PSG sleep recordings, which contain five
EEG sleep signal recordings: Wake (W), Non-rapid eye movement (N1, N2, N3), and Rapid Eye Movement (REM).
15

---

Table 5: A summary of four individual large time series datasets used in the experiments.
Dataset
# Train
# Test
Length
# Variables
# Classes
HAR
7352
2947
128
9
6
Sleep-EDF
25612
8910
3000
1
5
FD-A
8184
2728
5120
1
3
UniMiB-SHER SHAR
9416
2354
453
1
17
B.2 UCR 128 Archive
The UCR time series archive 22 contains 128 univariate datasets and is widely used for classification in the time series mining community. Each UCR dataset includes single training and a single test set, and each time series sample has been z-normalized. In addition, we uniformly use the mean- imputation method to preprocess the datasets that contain missing values. For detailed information about UCR datasets, please refer to https: / /www. cs ucr reai//eamonnnttne testttimmmeeeeesee Series_date 2018//
B.3 UEA 30 Archive
The UEA time series archive [54] contains 30 multivariate datasets. mainly derived from Human Activity Recognition, Motion classification, ECG classification. EEG/MEG classification, Audio Spectra Classification, and other realistic scenarios. Each dataset contains partitioned train- ing set and test set. In addition, we use the mean-imputation method deal with datasets with missing values. For detailed information about UEA datasets, please refer to https com com/dataset..pie
7/www. timeseriesclassiiiaaaee
C Baselines
To analyze the performance and effectiveness of Scale-teaching on time-series label-noise learning, we selected seven baselines for comparative analysis. The specific information is as follows.
Standard directly employs all samples in the training set containing noisy labels and performs supervised classification training using cross-entropy loss. Then, the trained model used to make predictions on the test set.
Mixup [6 trains a neural network on convex combinations of pairs of time series samples and their labels (whatever is clean noisy). For the specific open source code, please refer to /tthtps:///iiii thub htt.ahuww.....aoaaooooeoeeoeoeee httm///faaboooreerrrrrramamaaaaa cifar
Co-teaching [2 trains two deep neural networks simultancously, and lets them teach each other given every mini-batch with selected clean labels based small-loss crite- rion. For the specific open source code, please refer to sattps://gii thub. com/ com/bhanML/ Co-teaching
FINE (35] utilizes novel detector for clean label selection. Especially, FINE focus each data point's latent representation dynamics and measures the alignment between the latent distribution and each representation using the eigen decomposition of the data gram matrix. For the specific open source code, please refer
official
SREA employs novel multi-task deep learning approach for time series noisy label correction that jointly trains a classifier and an autoencoder with a shared embedding representation. For the specific open source code, please refer to htttss://gituut com/ Caste144/SREA
SELC 37 utilizes simple and effective method self-ensemble label correction (SELC) to progressively correct noisy labels and refine the model. For the specific open source code, please refer to
Com/MacLLL/SSER
CULCU 23 incorporates the uncertainty of losses by adopting interval estimation instead of point estimation of losses to select clean labels based on Co-teaching, CULCU has two
16

---

versions: CNLCU-S and CNLCU-H, where CNLCU-S uses soft labels for training and CNLCU-H uses hard labels for training. According to the original paper's 23 experimental results, CNLCU-S has a better performance. Hence, we use CNI.CU-S as a baseline. For the specific open source code, please refer to
Finally, based on the source code of the above baselines, we provide the reproduction source code of all baselines, as well as the source code of our proposed Scale- Scale-teaching (refer to Algorithm For the specific open-source code, please refer to our GitHub repository https:: petpp://githun com/ mamllimm-lab/salaee teaching
[].
Our experiment contains 162 datasets. It would be time-consuming to perform hyperparameter selection for each dataset. Therefore, the hyperparameters of Scale-teaching are not carefully tuned for each dataset, and most of the hyperparameters are set based on the default hyperparameters of related works. The learning rate and maximum epoch are set based on the parameters of existing noise-label learning methods, such as FINE and CULCU. a in Eq. a in Eq. 4 and in Eq. 5 are set based on the default hyperparameters of related label propagation works. Gwarm is based on FINE settings. Cupdate: Y and batch size are based on manual empirical settings without specific hyperparameter analysis. The largest neighbor K is set based on human experience, and we had a simple test on several datasets, and found that a larger value of does not improve the classification performance, but instead increases the running time of the model.
For the implementation of small-loss criterion in Scale-teaching, we select small-loss samples within each class from the mini-batch data as clean labeled data. For stduies [12] 13), they use warm-up training to decrease X(e) from 1 1-17. 17. X(e) denotes the selection ratio of small- samples within the mini-batch data without considering the difference of class, and 17 is the ratio of noise labels in the divergence to calculate difference d between the classification result p; of sample X2 and the observation label yi- Following (29). for each class c, we consider the observed label of X as clean label when the d
training set. Based on the above criterion, the current work (29 uses the Lensen-Shanme
of the training sample X; is less than dārg after Emarm warm-up training. davg denotes the average of ds of all the training samples when the epoch e. We observed that using the Jensen-Shannon divergence method [29] and directly employing stduies [12 [3] for clean sample selection within each class have distinct strengths and weaknesses when applied to various time series datasets. In our study, we implemented the strategy of stduies [12 [3] for clean sample selection within each class on four individual large datasets and the UCR 128 archive. Meanwhile, the
divergence method 29 was applied to the UEA 30 archive for clean sample selection within each class.
D Details of Main Results
For the four individual large time series datasets, the specific classification results of our proposed
Scale-teaching paradigm and baselines are shown in Table ō For the UCR 128 archive, the specific classification results for all methods with different noise ratios are shown in Table TT (Sym 20%),, [2 (Sym 50%), [3 (Asym 40%), and [4 (Ins 40%). For the UEA 30 archive, the specific classification results for all methods at different noise ratios are shown in Tables (Sy 20%). (Sym 50%) ). [7 (Asym 40%), and 18 (Ins 40%). For layout and reading convenience, we only give the average classification accuracy for multiple runs of all methods without standard deviation on the UCR 128 archive and UEA 30 archive.
E Details of Multi-scale Results
To analyze the multi-scale mechanism in the Scale-teaching paradigm, we provide the classification performance of classifiers corresponding to fine, medium and coarse scales, as shown in Tables 19
and 2T And the classification results by ablation cross-scale fusion mechanism based on the Scale-
and [2] such as a_t b_f. b bccfcttt t,, and c_____, where a denotes fine classifier, b denotes medium classifee, and e denotes coarse classifier, and t and f represent correct and incorrect classification results, respectively. For example, a_t_b_ indicates the number of samples correctly predicted by the fine classifier and incorrectly predicted by the medium classifier. In addition, we provide t-SNE (59 visualization on the FD-A dataset with Sym 50% noisy labels (as in Figure ( to explore the distribution of different
teaching are shown in Tables 20 and 22 For the abbreviations in Tables 19 20 21
classes of embeddings. Figure 6 shows that the cross-scale fusion mechanism in Scale-teaching for
17

---

Algorithm The proposed Scale-teaching paradigm.
Input: encoders [Wa wB, wc). classifiers [CA, CB, ccl, fine-scale series ZA, medium-scat series
BB. and coarse-scale series #C
Output: [WA, WB, we] and [CA, CB,co]
Note: For clarity, our analysis utilizes three distinct scales for training, but this approach can be extended to incorporate multiple scales.
Step one: Obtain single-scale embeddings r', TB. TC:
A Ax));
rB WB (x
TC wc(())
2: Step two: Obtain ccoss-scale embeddings VAs UB. tci
AA T BrB VA]:
UB Eq.
cc Eq.
rC, UB);
3: Step three: Obtain clean labels y.A. ya. yC for cross-teaching training;
VA <c(C)) via small loss criterion;
YB = CA (A( via small loss criterion;
Uc Cp (UB) via small loss criterion;
4: Step four: Obtain corrected labels yca, yep, yee for classification training;
yсл Eq. 6
U. va) via label propagation;
= Eq. 6 UB. MB via label propagation;
gcc = Eq. 6
vc. (C. via label propagation;
5: Step five: Overall training;
Update encoder WA and classifier сд via cross-entropy loss(UA, VA yca):
Update encoder wyy and classifier cg via cross-entropy loss(UB, & yCB); Update encoder we and classifier co via cross-entropy loss(uc, yC ycc).
Table 6: The detailed test classification accuracy (%) compared with baselines on four individual
large datasets (values in parentheses are standard deviations). The best results are in bold.
Darsset
Noise
Standard
Mixup
Co-teaching
PINE
SREA
SELC
CULCU
Scale-teaching
Sym 20%
92. (0.64)
92.52 (1.05)
92.28 (0.67)
92 (0.55)
(2.43(1.411
92.88 (0.82)
92.66 (0.37)
93.93 (0.66)
Sym 50%
83.99 (2.89)
76.75 (1.88)
89.90
88.42 (3.83)
91.38 (0.59)
90.37 (0.73)
89.91 (2.19)
90.17 (0.67)
Asym
87.67
88.98
(2.39)
Ins 40%
83.56 (2.82)
73.86 (0.39
90.98 (0.96)
90.77 (0.33)
(1.25
91.02 (1.53)
91.15 (1.43)
91.58 (1.47)
Sym 20%
87.07 (0:95)
82.13 08)
80:54
26.63 (3.07)
68.52 (1.48(3655)
81.80(1.77)
(1.69
UniMiB-SHMM
50%
79.37
43.32
Asym
(1.93)
51.16
(1.59)
(1.65)
Ins 40%
55.83 (8. 14)
56.97 (6.48)
54.99 (1. 79)
11.18 (4-75)
51.5
$4.62 (6.63)
$3.90(4.75
71.14 (3.99)
Sym 107%
98.89 (11785)
19, (0106)
19.83 (011R)
78.
(0.92
99767 (0.19))
99.85 (0.18)
99.93 (1.14)
Sym
(17.53)
Asym
96.
(0.50.1..00
97.06
61.08
90.23 (0.02)
(0.58)
98.
(0.36)
Ins 40%
99.36 (0.47)
99.55 10)
99.51 (0.19)
67.81 (12.95)
88.63 (0.02)
99.36 (0:23)
99. (0.22)
99.82 (0.06)
Sym 207%
85.01 (0.00)
84.31 36)
84.8 (0.14)
81.21 (0.25)
12.79'0199
84.33 (0.33)
83.23 23.33.111 (0.14)
(0.35)
Sleep-EDP
25)
Asym
79.62 (2.39)
77.40 (1.92)
82.87
61.77 (2.10)
72.23 (0.89)
82.50 (1 (1.07)
83.05 (0.64)
83.87 (0.38)
Ins 40%
84.35 (0.38)
84.25 (0.31)
84.62 (0) 28)
79.68 (2.55)
71.99 (1.24)
83.78 (0.28)
84. 84.86 (0.23)
85.03 (0.61)
18

---

80
-20
-00
-40
-0
0
20
40
60
-80
-00
20
21
40
(
80
-60
-10
20
0
20
T-NN_ 1
T-NNE
1
T-NEE
1
(x) Sumdard (96.73%)
(b) CULCU 77%)
Scale-traching (99.73%)
Figure 6: t-SNE visualization of the learned embeddings the FD-A dataset with Sym 50% noisy labels (values in parentheses are the test classification accuracies).
0.925
0.81
0.915
0.79
0.78
0.9
0.77
0.885
0.89
0.885
[1223344
8,16
0.75
11.233445)
[1.24,8116]]
0.88
0.74
(a) HAR Dataset
(b) UniMiB UniMiB-SHAR SHAR Dataset
Figure 7: Multi-scale sampling strategies analysis under Sym 50% noisy labels.
time-series label-noise learning can make the embeddings of different classes more discriminative, thus facilitating clean sample selection and noisy label correction.
Impact of downsampling scale sequence list. Scale-teaching can be performed using a variety of different downsampling scales for label-noise learning. Based on the experience of [19] 44]
time series classification and prediction tasks, we utilize the downsampling scales of [1,2,4] for the experimental analyses of Scale-teaching However, for real-world scenarios that actually contain noisy labels, it is generally not possible to perform hyperparametric analyses using llcen-labeled validation To facilitate the analysis, in this paper, we use the classification performance of the test set for multi-scale hyperparameter analyses. However, to avoid test set information leakage, we do not use the hyperparameter analysis result for Scale-teaching in our experiments. We use two multi-scale sampling strategies for analyses, which are (1) [[1,2], [1,2,3], [1,2,3,4], [1,2,3,455]]; (2) ([1,3], [1,2,4], [1,2,4.8], [1,2.4.8,166). From Figure we find that Scale-teaching using four different scales for training has the highest classification accuracy, which indicates that more input scales do not necessarily make the classification performance better. In addition, using three or four scales of sequences can effectively improve the classification performance of Scale-teaching compared with using two different scales.
Impact of input scales of sequences order. Scale-teaching employs finer-to-coarser strategy for cross-scale embedding fusion. Intuitively, when single scale is used for classification, the original single scale (finer) time series better overall because does discard the original sequence information compared to coarser scale time series. Therefore, Scale-teaching is trained using the finer-ta to-coarser cross-scale fusion strategy. To analyze the difference in classification performance between different fusion directions, we subtract the classification accuracy using the finer-to-coarse and coarser-to-fine training approaches, and the specific results are shown Figure g We can find that the classification performance of finer-to-coarser is better overall, which is due to its ability to use single fine-scale sequence with an excellent classification performance from the beginning to gradually promote the classification performance of multiscale fusion embeddings.
19

---

0.025
0.0219
Accuracy Difference
0.02
0.015
0.0095
0.01
0.007
0.005
-0.0084
HAR
Univ SHAR
FD-A
Sleep-EDF
-0.005
-0.01
Dataset
Figure 8: The cross fusion direction of input scale series analysis under Sym 50% noisy labels.
Standard (Cross Entrppy)
Scale-tsaching (Ours)
Standard (Cross Entropy)
Scale-teaching (Ours)
nosy
3.60
deaa nolry
2.5
dean
reay
3.50
Loss value
3.35
Loxs value
23
2
3.50
211
3.30
3.20
3.0
1.9
1.7
60
150
200
50
150
200
200
Epoch
Epoch
Epoch
Epuch
(a) Adiac
(b) CricketZ
Figure 9: The change of loss values for clean and noisy time series samples under Aysm 40% noise labels. The solid line and shading indicate the mean and standard deviation loss values of all clean (or noisy) training samples within each epoch.
F Small-loss Visualization
The small-loss criterion has been extensively validated for clean label selection in label-noise learning for computer vision. To further analyze the application of the small-loss criterion in time series data, we provide the change of loss values of the models trained by the Standard method and Scale-teaching on Adiac and CricketZ. UCR datasets (as in Figure 9 Also, we visualize the probability distributions of the ground-truth clean and noisy (corrupted) sample loss values on the test set with different training strategies. Specifically, Figures 10 and IT show the loss probability distributions of the models trained by different strategies on the HAR dataset and UniMiB-SHAR with Aysm 40% noisy labels. Both red (clean) and blue (corrupted) indicate that some correctly labeled samples still difficult learn (large loss) and some incorreetly labeled samples are also easy to learn (small loss). Compared with the Standard method (Figure IT] (a) and Figure II (a)), Scale-teaching (Figure (b) and Figure (b)) clearly distinguish clean and noisy samples by the loss value distribution, further validating the robustness of the multi-scale embeddings to cope with time-series noisy labels.
Figure T0 and Figure IT contain two peaks, which
G Other Analysis
The test F1-score results ablation study. Following (4,
we select the averaged Fl-score on the test set as new metric for ablation analysis in Section 4.5 Hence, we give the corresponding test classification Fi-score (%) in Tables and
Running time analysis. We select two datasets for running the time-consuming analysis, the FD-A dataset with the largest sequence length and the Sleep-EDF dataset with the largest samples. We performed the running time statistics on the NVIDIA GeForce RTX 3090 GPU using all baselines, and the results are shown in Table 9 the FD-A dataset with the longest sequence length, Co- teaching and CULCU take essentially twice long to run the Stanard method because they use
20

---

clean density
clean density
3.0
corrupted density
corrupted density
2.5
Samp e densits
2.0
0.0
loss
loss
(a) Standard
(b) teaching
Figure 10: The loss value probability distributions visualization on HAR dataset with Asym 40% noisy labels.
ceean density
clean density
corrupted density
2.0
density
sample density
1.5
1.0
-1.
-10
-0.5
0.0
0.5
1.0
1.5
-1.5
-1.0
-0.5
0.0
0.5
1.0
1.5
loss
lass
a Standard
(b) Scale-teaching
Figure 11: The loss value probability distributions visualization on UniMiB-SHAR dataset with Asym 40% noisy labels.
Table 7: The test classification F1-score (%) results of ablation study (values in parentheses denote drop F1-score).
HAR
UniMiB-SHAR
Method
Sym 50%
Asym 40%
Sym 50%
Asym 40%
Scale-teaching
90.05
89.14
77.56
65.89
w/o cross-scale fusion
88.16 (-1.89)
87.05 (-2.09)
68.23 (-9.33)
57.76 (-8.13)
only single scale
87.56 (-2.49)
86.75 (-2.39)
66.87 (-10.69)
54.12 (-11.77)
w/o graph learning
87.79 (-2.26)
87.41 (-1.73)
74.62 (-2.94)
63.15 (-2.74)
w/o moment
89.34 (-0.71)
88.27 (-0.87)
76.67 (-0.89)
64.92 (-0.97)
w/o dynamic threshold
88.93 (-1.12)
88.29 (-0.85)
73.11 (-4.45)
64.76 (-1.17)
Table 8: The test classification F1-score (%) results on four individual large datasets without noisy labels. The best results are bold, and the second best results are underlined.
Dataset
Standard
Mixup
Co-teaching
FINE
SREA
SELC
CULCU
Scale-teaching
HAR
93.27
95.39
93.75
93.19
92.91
93.71
94.72
94.18
UniMiB-SHAn SHAR
86.37
80.17
84.43
84.03
66.54
89.19
86.45
93.62
FD-A
99.93
99.91
99.96
64.05
90.14
99.82
99.95
99.96
Sleep-EDD
81.99
82.11
82.52
83.07
77.67
82.17
83.26
84.76
21

---

Table 9; Training time (hours) analysis using the FD-A and Sleep-EDF datasets with Asym 40% noisy labels.
Scale-teaching
Dataset
Standard
Mixup
Co-teaching
FINE
SREA
SELC
CULCU
[1,2,4]
[1,4,16]
[1,8,2]]
FD-A Sleep-EDF
0.37
0,42
0.79
0.63
0.90
0.42
0.87
1.06
0.86
0.82
0.54
0.83
1.09
2.04
1.64
0.73
1.47
2.02
1.76
1.60
Table 10: Comparison with classification methods without label noise learning strategy. The best test classification accuracy (%) results are bold, and the second best results are underlined.
Dataset
HAR
FD-A
Method
Sym 20%
Sym 50%
Asym 40%
Ins 40%
Sym 20%
Sym 50%
Asym 40%
Ins 40%
Boss
72.34
62.53
56.
53.29
52.34
69.75
64.75
57.95
61.99
62.25
Rocket [0)
95.29
92.93
90.04
82.53
90.43
99.99
99.71
97.01
89.75
97.98
FCN
93,74
92.13
83,99
75.59
83.56
99.56
98.89
96.63
96.
99.36
Scale-teaching
94.72
93,93
90.17
89.62
91.58
99.98
99.93
99.38
99.55
99,82
two encoders. Furthermore, although SREA uses a single network training, it utilizes decoder for the unsupervised reconstruction task of the original time series, which significantly increases training time on the FD-A dataset with longer sequences. running time is higher than Co-teaching. The Scale-teaching paradigm uses multiple encoders for training and has an additional noisy label correction module, which is expected to increase the training time. Nevertheless, the larger the sampling scale (coarse scale) of the training data used by the Scale-tcaching paradigm, the lower the training clapsed time of its model. For example, with input scales of [1. 8, 32], the training time of Scale-teaching is lower than that of CULCU and SREA. On the SleepEEG dataset with the largest number of samples, using all training samples to select clean labels is consuminn when the sample size is large. In contrast, the runtime of Scalenge is lower than FINE. Also, when the input scales set to [1,8,22], the runtime of Scale-teaching is lower than SREA.
find that FINE with encoder has higher running time because FINE
It is worth noting that when Scale-teaching is trained using two scales, such as [1,2] or [1,16], its training run time decreases further. From analysis in Appendix E, it clear that Scale-teaching using three different scales generally performs better than two scales for classification with noisy labels. In addition, the classification performance of [1,2,4], [1,4,16] and [1,8,32] when Scale- teaching is trained using three different scales has less difference in classification performance on datasets with longer sequences (e.g., FD-A and Sleep-EDF). The above results indicate that the Scale-teaching paradigm has greater advantage in runtime semees datasets with longer sequences.
Robustness analysis. Three time-series supervised classification methods (Boss [5], Rocket [60] and FCN (57) and the Scale-teaching paradigm are chosen for robustness analysis against time-series noise labels. Boss [15] is a time series classification method based on similarity search, which can effectively mitigate the negative impact of noise (e.g., adding Gaussian noise) in time series values on classification. Rocket (60) uses large number of randomly initialized convolution kernels to extract time series features, and employs the extracted features to classify time series using machine learning classifier (e.g., Ridge classifier). FCN (57) is the encoder used by Scale-teaching, which is a time series classification method based on DNNs. As shown in Table 10 the classification performance of the Scale-teaching paradigm using FCN as encoders is better than that of Boss, Rocket and FCN in the presence of noisy labels. It is worth noting that both Boss and Rocket training processes independent of the optimization due to the influence of noisy labels. In addition, the encoder of the Scale-teaching paradigm can be designed flexibly, such as using ResNet [1, Inception Time 655 and OS-CNN (00 in the field of time series classification. In other words, using better robustness encoders, the classification performance of Scale-teaching can be further improved with time-series noise labels.
DNNs. However, their classification performance is still reduced
22

---

Table 11: The test classification accuracy (%) on UCR archive with Sym 20% noisy labels.
Allac
13:04
11.51
2.30
16.18
13.96
52.41
4066
43.0
33.37
36.00
10.00
66.43
46.35
56.66
05.90
01:1
03:31
16.30
1120
5100
47-03
66.31
BMB
1805
11:33
17:33
1607
18.07
BuaChacoct
05.00
01.33
2003
15.00
88300
$3.90
95.90
60.33
59.40
09.87
25.01
23.33
07.33
Chanatons
7391
Caccooriaso
02:00
01.3
63.27
27/00
31.00
64.51
56.99
25.00
Chengaten
72055
23.49
71.46
57.04
13.00
14.00
71.35
72.90
56.90
44.5
31.54
13.58
46.74
50.25
Coclez
51.39
37.46
$1.93
26.4
45.15
35.71
6308
11235
61.54
69-73
07.33
70.13
59.34
72.00
99.30
75.31
48.35
42.5
15.23
19.1
66.78
3014
02.00
43.71
50455
32.17
68364
06.15
Sua
Bodgel
BOG20n
7000
15.32
$2.89
59.43
73.20
19.00
7711
53.09
26.34
19:11
91.13
75.44
RoGkkelnnnallgge god
11.7
10.36
40.01
41.41
11
Bistlopaioss
20/55
48.3
26.59
15.04
73.51
00.67
Electe
37.52
2009
25.55
53.20
57.13
Feccali
$1.30
1043
71.58
85.0
60-47
30
TicislERR
77.40
05.3
16.35
34.33
68-1
75.35
MUT
FillWWothh
09.33
13,90
15.00
00.55
52.1
66.76
Farth
25.30
75.6
76.31
60.91
54.55
90.26
95.55
02.33
01.00
76.21
$9.20
03.01
6
Fing
16-07
13.13
21.55
11.66
11.15
46.90
26
51.21
26000
13.00
32.41
16.11
16.9
11:00
5:15
3000
25.34
13
26.55
15.74
75.99
25.04
81.77
76.33
25.57
53.54
51.03
13.01
58.25
06.45
47.47
96.45
06.3
0536
Hes
6362
25.19
07.24
46.71
$1.00
6877
05.02
Peplics
36-11
39.35
3005
19.8
26.2
11.25
36.51
Barning
33.06
18.91
57.58
20122
25.44
25.34
15.35
15.55
15.4
25.3
20.0
72.17
13.35
41.65
05.35
45.55
11m
05.50
19810
7512
75.33
$4.0
4501
$3.45
16.55
25.76
15.71
1331
80:20
8118
25
Lightsng?
50755
35.00
36.5
21:0
3155
29,04
51.00
550
Moc
12.00
66.5
55.35
$3.33
11.07
01.35
$2.35
Medicatzrage
0611
31.49
65.60
Melloaen Proturicn
11.55
56.07
50115
3049
43.7
57.25
40-45
41.09
19.31
56.22
Niallc''s dlan TW
53.12
12.44
$4.48
25.05
55.4
65.00
54.35
12.45
09.53
30.59
35.6
19.47
77.06
mus
75.30
01.5
50.75
79.55
$0.56
85.33
80
36.11
3478
13.50
10.45
31.60
42.02
KLA
Onredil
-4.57
42.0
42.00
4003
41.0
4000
27.0
34.2
27.3
1975
11.15
23.05
27.00
13.56
13.3
11.5
1.85
13.00
3002
11
23.54
3.30
13.28
0555
188.0
11
4534
5217
$5.35
$1.5
05.30
76.5
76.54
25.27
07.55
19.30
79.5
$1.9
50.3
$4.1
54.1
47.35
Scrantippe
6514
01.58
63.6
35.35
42.13
63.56
01.19
47.40
4003
47.00
3233
36.15
23.47
45.00
30/33
57.0
Sane
61.01
10.44
30.91
44.00
75.22
60.0
10c
Skacketiz
61.11
71.75
70.7
5000
45.30
60.0
0001
5000
Ini
75.53
16
76.45
68.75
77-1
5000
Alfil Aeeetlibbwrrreeeee
87.07
05.36
6100
53.6
S4
107
2641
47.00
24
110
Surtebarty
91.37
12.3
$2.90
70.05
1530
02.17
9544
111
Sprands
76.77
53.00
22.44
60.16
112
Sy Creahakchendd
07:01
1430
$6.11
50.59
116
72.25
3443
03.25
43.00
54.4
13.0
12.12
13.45
116
Teal eall
96.90
33.03
$3.55
49.95
$4.3
65.00
$6.11
60.9
1IT
$5.00
$3.07
77.51
70.0
119
24.12
55.55
7225
12.35
12.02
$150
62.12
131
62/11
13
01.41
12.51
42.08
66.30
65
Wala
16.55
$7.09
96.55
13.33
91.11
5007
Wise
09:31
41.1
1009
46.11
128
Wezso
02-42
07.38
06.3
22.05
31.4
6327
62:75
Yog.
23.56
21.33
41.80
11/64
Ang
4003
4130
400%
1001
9116
6671
(8.0
Keek
Fostls
1/KE-44
4.88416
L.KE-43
1.48-34
1.98-11
3J4E-0
8-725-23
23

---

Table 12: The test classification accuracy (%) on UCR archive with Sym 50% noisy labels.
47.34
Alc
12:48
8.50
51.45
2014
22.85
01:32
51.20
13.00
12.17
34.03
19:01
70.14
38.08
13:03
11.54
14
13.40
13.63
62.45
DMB
19.33
BuaChacoct
5000
45.00
01.33
2000
15.00
63300
02.30
40.33
15.33
23.33
Chanatons
0627
66.91
11:55
94.37
Caccooriaso
-243
44.00
34.54
24-11
11.44
45:55
31.00
Chengaten
09.44
11.03
54.34
47.36
93.40
51.76
47.00
835
45.31
3347
29.3
23.85
43.55
Coclez
2019
26.90
51.7
16.70
66.34
6333
47.38
55.55
50.65
63.50
51,32
52.53
31.99
48.33
45.35
48.06
30.55
5015
4071
45.00
31.5
40.37
30.1
50200
53.33
Bodgel
BOG20n
-0.0
43.48
52.33
52.90
41.5
50.20
53.00
91.20
11.0
19:31
51.6
56:43
13:5
11
Bistlopaioss
53.30
55.00
66.38
63.55
11.00
61.33
Electe
32.2
25.57
31.35
25.36
25.20
23.76
12.91
5502
Feccali
56.3
91.9
1943
10:12
16.00
50.0
30
TicislERR
46086
42.33
42.95
11.3
18.6
44.51
45.64
41.57
FillWWothh
28.99
51.38
3033
15.1
18.00
35.41
52.3
26.38
Farth
09.53
46.53
25.05
18.5
30.15
53.99
28.55
54.20
65.28
54,60
45.05
6
Fing
15.99
1730
16.33
15.35
10.55
15.59
1740
13.32
15.46
13.5
10.56
12:55
35.9
13.0
-8.75
24.00
19:97
11.53
01.00
56.0
5009
50.30
-0.54
51.13
06.30
55.44
46.88
43.49
31.54
55.00
NUSE
Hes
46.23
47.20
54.9
41.35
40.14
53.05
Peplics
26-11
29.39
2535
20.99
21.55
19.00
26.35
28
Blrning
99.38
21.88
36.59
55.50
48.45
35.01
5555
2043
19.0
19.99
15.4
15.1
15.24
21.01
18.75
65.3
7526
51.0
5558
15.50
61-01
45.34
47.8
30.45
3005
51.54
48.02
20.00
2015
Lung Apelicccs
57/11
40.3
(0
25
Lightsng?
3041
35.42
1520
32.33
43.51
4507
4155
Moc
65.00
43.35
56,00
33.33
25,00
43
Medicatzaage
42.44
31.49
Melloaen Peoturico
07.37
-5.7
4370
35.79
34.10
31.02
42.0
54.11
4643
Nidllc''s dlan TW
00.79
13356
25.30
$5.64
13.54
56.55
$0.80
57.25
-1.57
21.09
23.55
4612
43.45
50.64
47.55
44.0
43:50
83
3147
1453
3.0
1550
Onredil
36,32
35.00
34.7
35.00
34.00
15.33
ana
26.45
2013
15.35
17.54
20.22
21.72
1975
443
100
100
145
1.55
7.06
11.01
131
2515
07.57
K3.06
6513
67.4
09.6
T3.56
65.66
50.0
7501
09.2
22.34
07.55
15.51
12.99
Refiggratic
43.30
4554
53.33
45.5
43.50
4542
45.77
Scrantippe
4213
4784
43.33
1611
34.55
10.5
45.76
415
42.42
3352
42.90
35.11
57.55
40.00
35.93
30.44
Sane
2013
13
15.01
16.45
47.81
38.4
10c
Skacketiz
51.75
52.18
2034
3000
53.00
32300
25.90
50.9
Ini
0502
1733
63.45
K5.00
01
63.07
47.30
31.41
5133
71.00
41-11
01.34
107
110
Surt
40022
47.55
45.55
40.5
41.43
4346
45455
45.35
111
Sprands
-6.22
5034
43.05
23.32
59.55
45.00
40.85
112
62:25
54.5
92.99
0110
116
51.44
5615
X3.35
63.99
15.00
5450
0032
4175
116
Teal eall
42.54
54.44
47.50
40.90
11.4
51m
30.99
4442
1IT
7628
63.07
76.11
-3.6
4543
36.34
45.6
119
40.5
3372
28.35
12.35
12.52
SAIT
47:41
2331
131
-9.90
5053
6113
12.51
4159
$3.55
29.0
46.1
Wala
3001
45.3
23.45
59.12
12.14
46.00
4632
Wise
16.30
55.80
128
Wezso
05270
36.35
3001
26:25
45.5
6136
0001
5571
Yog.
54.1
90.2
4015
Ang
2507
3154
3001
8678
Keek
1.6
Fostls
3158-4
1.30446
40225-04
T
1,325-15
1446-0
4308--3
24

---

Table 13: The test classification accuracy (%) results on UCR archive with Asym 40% noisy labels.
ACSE:
-1:13
435
3.76
3.31
20.17
25.17
56.00
31.06
33.47
28.13
12:12
18.23
58.33
46.33
Altrasehiiradd
76.80
18:35
18.76
11:00
16.15
76.71
17.03
02.29
Assovikad
12.43
47:00
Reat
7667
BuaChacoct
3000
37.00
$1.03
2003
90.00
6040
30093
75.0
5712
93.47
54.53
2607
14.33
66/91
Chanatons
66.6
51.0
Caccooriaso
56772
45.33
43.23
24.57
11.17
$3.55
45.76
Chengaten
53.1
00.33
55.84
63.00
60.30
90.35
45.90
Chidurt
41.39
35.52
25.05
17.00
54.2
Coclaz
52.45
35.15
30.32
3.0
11:18
10m0
8740
5016
40.33
76-45
31.5
36.54
41.5
SLS
30.5
18.04
63.59
43.00
40.99
6439
25.1
30.22
41.30
52.33
40.37
31.30
54.00
22115
Bodgel
BOG20n
-6.90
4135
63.99
41.00
98.59
46.20
66722
25.20
52.5
67.50
RoGkkeinnnallgge gaal
2013
96.9
8.57
3007
11
Bistlopaioss
57.37
53.5
26.30
56.11
19.56
72.04
Electe
-0.45
23.08
31.04
27.36
25.04
23.30
27.30
Feccali
47.30
06.55
15:0
30
TicislERR
5011
41.38
4644
3375
40.51
54.50
SLA
FillWWothh
06.30
33.49
20.04
15.09
13.14
51.6
05.67
44.5
40.35
72:43
Farth
55.56
62.64
$7.00
05.55
72.30
00.03
60.25
02.11
11.75
01.30
75.4
6
Fing
36.88
3343
43.00
11.54
11.00
34:30
22.43
22.31
2.55
50.56
11:51
15
01.5
36.66
15.5
11%
06.15
50.51
47.25
54.55
51.55
05.23
0130
20.43
26.42
47.47
MLAS
55.1
Hes
-7.39
44.35
24.2
4183
10.20
94.5
5L.
Peplics
30.53
32.45
5005
2085
21.56
1054
26.51
Barning
46.1
2530
06.31
58.0
05.00
03.91
9050
25.35
23.04
23.44
1782
17.19
21m0
25.85
07722
75.1
16.90
10.5
11m
7661
5550
57.45
25.09
71.55
51.34
65.54
63.01
07.91
63/A
25
Lightsng?
34.23
35.55
33.59
23.55
3144
4027
35.32
4154
Moc
61.47
62.47
36.43
40.35
33.33
40.77
50.33
Medicatzrape
24.37
2006
Melloaen Peoturicn
13.00
35.43
36.36
28.57
69.5
4150
51.71
Nidllc'ss alan TW
05.59
38.03
40.0
12.86
51.33
10.11
$1.36
47.9
57.23
55.54
19.14
35.56
54.00
56.76
31.55
50.55
$3.50
03.05
80
2600
15.58
22.00
12.04
Onredil
30.53
36.00
36.3
3000
$5.33
15.33
33.35
2015
21.34
33.02
2121
12.3
15.50
19.70
6.12
19:11
22.14
1271
5.0
196
4.1
1.5%
4.11
19.11
10.00
15.01
2024
4587
16.11
36.55
4533
47.3
65.11
73.99
5171
0004
01.32
50.36
23.59
53.33
7102
Refiggeatii
56.2
38.99
42.1
55.39
44.5
14.75
41.5
Scrantippe
-9.74
41.6
42.1
2407
16.44
47
10.57
5-50
4003
34.34
40.29
20.55
23.54
(3.01
30.39
Sane
45.88
41.03
3842
15.04
5650
10m
$2.1
10c
Skacketiz
53.22
30.50
3059
3000
$3.56
23.75
31.00
2000
Ini
50.34
67.06
1745
60.4
(0.30
69.84
41-11
15.13
09-55
107
63.9
62.61
110
Surt
71412
3433
7455
41.41
0435
TAMO
05.06
111
Sprands
55.45
45.00
52.31
$6.22
5010
30.50
45.3
112
Sy Sreahakchendd
65.41
16.7
78495
7515
3235
116
S615
13.00
04.59
26.31
06.99
5462
05.72
TK
116
Teal eall
72.36
43.4
21.56
11.6
13.06
71.90
9611
1IT
65.84
7648
03.56
40.50
41.41
$1.34
119
30.55
32.06
41.31
12.45
12:18
5314
42.51
5240
131
-2.35
49
47.58
12.99
31.7
58776
47453
Wala
7005
34.00
$3.99
1043
91.33
16.10
MUT
Wise
51-11
$2.99
128
Wezso
32.31
13.00
33.31
2319
4131
16.6
2034
5806
Yog.
35.17
35.17
33.55
41.66
53.3
9635
51.90
Ang
-200
0711
0117
914
Keek
4.1
Fostls
1428-45
3.31-40
6100-04
1308-23
1886-0
1248-22
25

---

Table 14: The test classification accuracy (%) results on UCR archive with Ins 40% noisy labels.
61:40
Alc
1.29
4.5
19.33
27:16
56.90
55.13
52:31
24.82
10.00
16.00
32.75
45.78
51445
15:15
11:47
15:13
11:01
33.91
12.01
19676
Assovikad
15.11
62.21
BuaChacoct
6600
70.00
$3.00
2003
50.00
73.00
07/01
49.93
41.0
36.07
26.33
45,0
Chanatons
6236
02.13
8856
Caccooriaso
3011
4142
41.25
16.30
31.12
58.00
4505
4633
Chengaten
41.0
55.16
51.3
47.25
$1.00
54.00
56.35
56.
48.34
36.55
31.36
15.38
47.66
Coclaz
41.77
13.39
63.35
11.00
05.03
31.50
13.13
45.77
38.12
23
06.07
51.60
33.33
14.22
59.57
6812
16.75
50.22
36.3
33.4
40.37
52.17
16.23
52.01
5635
Bodgel
BoG20n
01.43
53.25
01.43
52.00
04.22
63.40
67
53.66
1099
11.0
RoGkkelnnnallgge god
07.9
15.53
18.75
11
Bistlourios
0013
18.30
71.44
76.80
7453
66.99
73.91
Electe
25.15
11.35
32.34
24.86
25.12
23.76
52.41
Feccali
56.99
17.49
55.41
50.5
30
TicislERR
-0.11
42.43
45.35
14.55
58.26
46.11
FillWWothh
50.9
33.6
10.23
13.54
11.40
55.00
6622
71.7
80.39
Farth
61.02
17.03
(6.5
71.31
34.3
2009
52.15
11
01.05
6
Fing
73.38
1941
24.65
26.51
11:08
11.55
19:34
15.00
12.12
25.17
13.51
13466
53.00
11.2
1801
56.75
50
13.35
16.30
11.00
15.15
52.16
59.01
57.55
51.55
-1.62
60.57
57.31
53.1
0611
00.99
40.37
19.0
71.22
90.66
Hes
59.11
17.45
56.33
48.57
49.3
5930
56.57
5130
Peplics
20.30
28.08
26.9
19.11
23.56
18.33
26.92
Barning
21.9
57.04
63.54
46.41
$4.59
13.36
48.35
1067
15.55
19.49
16:00
14.1
18.0
2016
07:34
11.53
5665
13.9
14.00
05.00
3630
07.35
55.55
01.09
4001
69.89
65.23
96.33
Lung Apelicccs
25
Lightsng?
26115
13.02
33.55
10.15
25.00
15.40
36.77
M012
Moc
46.00
45.33
43.5
53.35
33.35
45.07
Medicatzrage
A231
Melloaen Peoturico
09.00
10.50
13.00
7855
46.52
35.5
$1.9
27.35
41.09
4530
46.71
15
Nidllc'ss alan TW
43.71
29.64
40.3
21.30
14.0
15.00
2500
50.80
11.06
4641
21.01
21.38
63.00
63.20
51.5
75.32
70.8
00.23
72.13
13.00
76.33
83
31.1
500
11.33
310
11.51
13.05
4180
Onredil
-4.3
41.0
4000
41.0
18.0
30.00
1512
21.35
1631
19.39
11:21
7001
1577
22.79
305
11:48
7446
158
4.57
11.05
12a0
6.15
5.5
1355
67:45
55.00
63.31
56.55
55.00
65.55
71.40
63.41
38.50
45055
35.37
65.31
23.59
$1.00
Refiggeatii
4506
4933
$3.39
44.3
45:00
45.00
48.7
Scrantippe
52.44
$3.54
4609
33.35
36.75
41.41
41.00
4611
5029
35.33
39.45
51.57
13.15
19.20
35.11
41.66
Sane
-0.55
26.87
35.40
16.30
47.31
36.0
10c
Skacketizz
3054
45.11
43.5
3000
30.0
45.00
20.55
56.0
Ini
66/15
43.00
75.6
13.00
11.1
71.01
51.5
05.55
35177
$2.84
16.99
14.56
110
Surtebart
66.11
34.0
75.24
25:71
52.5
18.7
0143
56.0
111
Sprands
43.54
42.48
41.54
41.31
42.2
45.0
39.35
112
7625
1697
18007
6609
116
00.35
72.45
71.54
43.99
73.5
11.38
06.15
0035
116
Teal eall
01.35
57.53
50.99
49.95
63.65
59.74
55.03
51.1
1IT
65.7
87.37
-007
17.55
4234
50.0
119
32-41
2133
3034
12.42
12.18
65146
45.11
5434
131
1665
45.3
41.33
12.24
12.00
15:1
45.71
6643
Wala
$1.0
35.23
90.33
38.77
91.11
918
Wise
5161
128
Wezso
01.35
03.5
02.55
21:27
45.33
6466
38.1
2001
Yog.
36.7
35.13
55,00
Ang
18
9111
Keek
Fostls
1.28-45
1111/4
2411-04
W818-26
2356-17
321640
1.548-22
26

---

Table 15: The detailed classification accuracy (%) noisy labels.
results on UEA 30 archive with Sym 20%
1D
Dataset
Standard
MixUp
Co-teaching
FINE
SREA
SELC
CULCU
Scale-teaching
ArticularyWoniie condiecnnntttoonn
89.33
81.04
81.77
41.07
90.13
87.87
84.33
92.73
AtriaFiibrilation
26.67
25.67
26.67
33.33
26.67
26.67
22.
28
BasicMotioos
97.10
97
99.25
91.00
97.50
98. 10.00
95.50
Characte/Tnjectrriss
98.16
92.59
7.45
96.54
98.93
98.93
Cricket
94.72
94.17
91.53
41.11
97.50
96.67
93.89
97.28
Deeee
48.40
52.48
48.60
24.40
46.80
50.00
49.60
45.60
7
FiigeWWomss
53.56
62.88
34.66
61.80
64.58
64.10
61.25
Epilepsy
84.96
80.28
93.33
82,46
88.70
86.38
93,33
87.59
9
ConanotConnnnttattinn
24.15
25
26.43
24.71
25.11
24.87
24.39
27.42
ERing
73.11
73.70
70.04
16.67
64.59
72.59
61.70
72.77
11
FaceDDttection
52.90
51.93
$2.36
$1.96
50.93
$2.30
52.69
$3.09
12
FingerMovementts
50.80
50.44
52.76
$2.20
50.20
52,80
52.30
54.68
HundMovvemantiinccion
28.92
32.22
35.14
24.05
19:46
28.11
29.16
35.35
14
Handwriting
35.71
28.55
30.71
25.31
16.63
28. 19.19
34.12
41.35
15
Heartbeat
52.00
$2.93
$7.46
72,10
62.75
50.17
66.39
51.76
16
InscetWingbeat
62.31
$3.90
64.74
63.92
51.56
63
64.75
63.85
Japanese Vowels
18
Libras
74,60
70.07
77.00
10.78
73.22
76.78
70.83
79.67
20
MotorImagery
51.00
50.88
$1.60
50.80
55.92
$2.80
50.10
50.84
NATOPS
22
PenDigits
96.04
97.58
97.33
98.30
97.90
98.27
98. 16.16
98.12
PEMS-SF
24
PhonerneSpectra
21.89
23.46
22.25
18.00
6.23
22.82
23.74
26.50
RacketSports
26
SellRegulatioSCP1
76.81
77.49
78.91
$6.86
49.83
78.43
77.55
81.28
SellRegulationSCPP
28
SpokonArabbiDigiis
96.17
96.21
98.83
27.79
98.52
98.67
98.71
99.18
StandWalkJunn
30
UWavGeetureLtbrryy
76.90
76.97
74.61
12.50
74.29
77.35
66.97
78.12
Avg
64.55
Avg Rank
5.03
5,20
3.83
6.37
4,77
3,73
4,00
2.73
P-value
Table 16: The detailed classification accuracy (%) results on UEA 30 archive with Sym 50% noisy labels.
ID
Dataset
Standard
MixUp
Co-teaching
FINE
SREA
SELC
CULCU
Scale-teaching
66.64
55.57
52.12
7.53
67.33
$7.40
49.63
68.68
AtrialFibrilltion
28,00
26.67
29.33
33.33
29.33
30.67
32.00
36.00
BasicMotions
56.00
$8.20
70.75
$4.50
81.00
59.00
71.25
60.20
Character Trajectrraees
90.62
87.56
67.75
6.73
95.28
93.82
97.00 97.03
97.23
Crickel
67.44
60.00
68.89
8.33
84.28
66.67
87.27
DueeeeckGeesse
32.00
36,64
29.60
26.00
34.00
33.20
33.20
33.84
EigenWWorns
49.62
45.37
55.65
32.98
38.78
56.79
56.09
49.28
Epilepsy
57.39
57.25
64.16
36.38
62.62
6U.00
63.55
64.36
EthanolConcentratio
25.20
25.32
26.05
27.56
25.00
24.58
26.27
ERing
44.30
44.39
40.63
16.67
39.35
44.52
38.54
45.10
FaceDetecttin
49,30
48.94
49.11
48.58
50.72
12
FingerMovementss
50.40
50.20
$1.80
$1.40
50.60
50.80
$1.30
$4.40
HandMovemmanDireccio
25.41
26.
23.51
26.57
24.35
27.08
14
Handwriting
19.54
19.53
19.99
13.41
7.53
19.18
18.28
21,06
Heartheat
55.32
53.52
52.20
55,40
$3.27
62
16
InngheWinghent
49.30
32.34
58.02
$2.25
31 07
53.97
58.13
$2.01
Japanese
18
Libras
47.09
43.98
$1.39
6.67
44.78
50.
40.06
49.78
20
MotorImagery
50.84
51.48
50. 10
$1.40
$2.00
50.60
49.70
49,80
NATOPS
22
PenDigits
93.38
83.29
95.92
92.85
92.74
96.80
96.53
93.89
PEMS-SF
24
PhonemneSpectra
19.08
19.11
19.94
11.48
3.92
19.69
19.23
20.09
25
RacketSports
$2.80
56.58
26
SedfRegulaiioSCPP1
47.41
42.68
48.86
48.60
49.97
48.33
$7.93
$8.08
SedfRogulationSCPP
47.61
50.00
28
SpokenArnbieDighee
85.64
69.95
96.55
96.16
99.23
95.66
97.59
97.69
StandWalkJung
37.33
30
UWaveGesmmeeLibaan
50.41
48.60
45.45
12.50
53.91
49.00
37.94
$2.61
Avg Ace
50.81
$1.75
Avg Rank
5.17
5.73
4.23
6.23
3,93
3.83
4,30
2.43
P-value
2.98E-04
7-40E-05
1.59E-02
9.35E-05
1.67E-022
1.08E-02
3.75E-02
27

---

Table 17: The detailed test classification accuracy (%) results on UEA 30 archive with Asym 40% noisy labels.
ID
Dutaset
Standard
MixUp
Co-teaching
FINE
SREA
SFLC
CULCU
Scale-teaching
Articulary BeeeiiaayWWrrReeeee WordRecognitio
66.33
62.40
55.87
17.27
69.40
63.67
53.73
70.44
AtrialFibrillation
3
BasicMotions
66.00
62.30
67.25
49.50
69.00
64,00
61.75
65. 15.10
Character Trajectories
61.35
Cricket
72.22
71.56
70.97
50.00
73.44
72.78
68.61
80.56
DuceDuckGeesee
44.60
7
EigenWorms
41.75
34.75
51.34
37,86
43.56
41.68
50.38
42.47
Epilepsy
62.32
63.01
63,48
47.25
61.01
61.45
64.71
58.70
9
EthanolConocatration
23.04
23.85
23.61
25.02
24.78
24.33
25.57 25.57
27.70
ERing
60.30
60.37
42.74
45.56
43.96
61.74
11
FaceDetection
49.88
50.64
51.12
$1,06
50.32
$1.U7
50.12
$1.61
FingerMovements
47.76
49.92
48.50
48.20
50.
$0.96
13
HundMoovemennDirectioo
28.97
31.24
30.41
38
31.46
29.73
29.03
14
Hondwriting
21.61
23.92
24.74
10.67
22.99
26.98
Heartheat
55.22
$7.46
55.61
72.20
61.52
55.51
$6.68
16
Winghhal
43.40
38.07
45.34
46.32
48.78
47.81
50.34
$1.87
17
JapaneseNowel
61.62
58.63
62.46
36.81
65.97
64.27
73.76
70.02
18
Libras
57.47
57.00
53.39
8.67
54.33
59.44
45.72
63.22
1.SST
42.11
42.77
41.70
43.67
32.79
43.67
43.74
29, 10
20
Motorlmagery
48.80
50.24
51.32
53.00
52,60
49.60
49.70
$3.20
21
NATOPS
57.00
$5.29
$8.65
16.67
64.89
55,89
63.22
65.13
22
PenDigits
78.76
67.36
92.78
84,05
91.07
89.18
92.23
93.57
23
PEMS-SF
50.20
$1.38
42.60
14.45
50.87
50.76
47.86
51.45
24
PhonemeSpectra
17.71
19.05
18.70
14,41
5.11
18.02
18.65
19,52
25
RacketSports
57.50
59.13
$4.30
27.50
55.26
58.16
56.32
$4.21
26
SolfRegulationSCII
63.47
66.21
64.94
60.96
49.83
66.42
68.10
66.30
SelfRegulationSC22
51.20
52.11
28
SpolenArabieDigis
64.19
60.64
79.13
72.11
99.04
79.42
88.
93.85
Stund Jump
30
UWaveGestureLiima
53.56
53.36
55.38
12.50
57.81
53.69
45.53
53.84
Avg Ace
50.16
49.76
51.08
38.06
51.47
52.17
51.75
$4.60
Avg Rank
5.60
4,77
4,40
6,13
4,20
4,00
3,97
2.73
P-value
1.63E-02
1.36B-02
2.62E-02
3.88E-02
Table 18: The detailed test classification accuracy (%) labels.
results on UEA 30 archive with Ins 40% noisy
ID
Dataset
Standard
MixUp
Co-teaching
FINE
SREA
SELC
CULCU
Scale-teaching
Articulary WordRnoognition
67.27
60.93
9,20
61.73
57.10
75.40
2
AtrialFibrillarin
28.00
29.33
30.00
33.33
32.00
28.00
34.67
32.00
BasicMotions
77.00
81.75
39.00
80.90
77.50
74.75
CharaaterTrajectorrs
82.52
69.46
66.33
5.22
85.38
81.92
83 48
87.47
Cricket
80.28
78.28
79.31
26.11
92.50
81.11
76.81
92.78
DeekDeckGeese
38,80
41.60
36.40
20.00
39.20
39.60
42.00
37.20
EigenWorns
8
Epilepsy
65.80
64.55
72.03
67. 10
73.04
71.74
78. 64
79.36
EthanalConcentration
10
ERing
$3.69
$2.48
48.48
36.79
49.82
56.44
44.59
$4.15
FaceDetection
12
FingerMovementss
$1.40
51.60
$1.10
$1.20
49.60
$1.40
48.60
49.60
HandMovemennDinection
26.49
28.92
14
Handwriting
21.35
22.54
22.92
17.93
6.17
21.51
23.19
23.01
Heartbeat
56.20
$9.02
16
InseceWingbeut
47.94
39.18
$7.07
$5.93
36.95
$7.12
59.40
$8.32
JapaneseNowar
73.35
78.97
81.54
18
Libras
46.04
51.56
47.00
9.33
48.67
49, 44
41.11
50.1 18
LSST
46.43
20
MocorImagery
49.20
$1.00
49.90
$0.00
$1.00
$2.00
$1.90
47.80
21
NATOPS
57.78
56.80
58.78
26.67
67.24
$7.67
58.34
$9.98
22
PenDigits
81.99
70.68
93.43
91.81
92.59
91. 18
93.29
96.69
23
PEMS-SF
42.89
43.86
35.14
16.42
47.86
43.82
41.49
44.35
24
PhonemeSpeate pectra
16.50
15.82
17.37
8.40
3:22
17.15
17.48
17.42
25
RacketSporrt
59.08
59.55
64.68
27.50
60.11
58.55
58.62
$9.89
26
SelfRegulationSCP1
60.63
60.38
$6.02
48.12
49.90
65,80
66.00
$4.54
27
SelfRegnlationSCPP
51.69
50.87
50.63
$1.11
50.00
$1.89
50.22
$2,11
28
Spoken
74.85
67.24
89.70
83.37
98.76
86.68
87.87
97.53
29
StundWalkJuup
40.00
42.67
39.33
32.00
42.67
38.67
42.00
40.00
30
UWaveGesturLLibraa
67.19
67.90
60.56
12.50
67.74
67.69
$5.69
69.96
Avg Acc.
52.27
51.10
53.76
36.014
53.38
54.51
34.61
56.90
Avg Rank
5.20
4.77
4.33
6.60
4.27
4.20
3.77
2.60
P-value
6.08F:04
2.92F-03
1.20F-02
2.55F-05
5.52F-.33
1.88--02
3.47F-02
28

---

Table 19: Multi-scale analysis of Scale-teaching using UCR 128 archive with Sym 20% noisy labels.
c..43
Adlic
0.631
0.850
135341
12
175
221
0.480
0,352h
0.3905
135
152
22
11
955
140
73
367
0.863
06609
0.581
190
100
31
30
-30
161
55
756
0330
07553
(33831
110
0.760
0.880
10
03800
15500
(1530)
10
18
Car
04403
05633
035733
20
Chanaoo
0.304
(3299
03391
Chiorian
2559
1965
15
0.597
0.380
0.330
141
136
555
34
100
457
145
93
Competers
01684
0.750
0.399
168
178
12
13723
118877
133
CREEY
0.4542
15
15
318
146
29
Ohckeet
0.4344
0.9955
153
11
13
340
118
03305
03364
388
355
208
0.7943
07706
03313
210
26
01800
116394
27
03542
0.330
20
15
16
51
22
15
47
Dedggil
ECG200
0.333
0.260
03731
76
0
15
03411
13
4021
163
0.001
500
ROGMeetieee eegleeeeeeetiiggaa Higral
0.206
011.9
33
1.6
76
30
11
Sc
0746
0.388
100
3.
3
67
0.233
0.5628
033712
19
207
259
52
0.603
01723
0.353
236
1000
40
11
1966
335
10
SM
02544
1101
111
63
1584
ST8
54
1003
41
Fas
04114
0.014
0.211
117
36
68
136225
10
1388
03134
65
15
505
da
0.5022
(55316
018477
250
2305
12
2564
72
258
2124
46
Pongi
2061
03300
03531
31
10
36
13
0.423
03111
0.3315
01661
03115
22055
0.875
01377
136
176
13
18
116
0.3127
0.3321
199
152
Gasheasttaa
2.504
03071
015321
334
56
Hans
03005
D4236
0.3311
4
43
25
38
Hapki
0.2551
(1300)
0.333)
10
109
72
Horlig
DA716
15
0.302
015331
78
61
0.1585
(C291
0.232)
20
34
125
113
44
43
60
Thain
10000
1/80
1.091
300
3/M
3/9
135
0.2811
03802
0.3935
913
300
66
Largli dysttrane
43317
11805
118677
118
66
Lishnine"
0.352
03397
35
20
19
Mizi
0.333
62100
01133
31
71
311
115
(1851
05518
2166
75
0.0154
0.5360
0.491
26
10
3.
71
16
34
60
75
Mall liffe a
0.515
0.251
0.591
15
17
Mise
Misuittupusiina Trais
0.437
0.360
033345
116
112
1666
134
0.331
03591
102
200
202
100
575
1
675
1980
101
0.084
6.7710
01001
1669
140
217
1371
15.4
18
Olockl
0.400
0.000
018867
17
03334
0.336
115
33
Pioncne
02346
11284
013333
114
176
100
90
425
119
141
87
3.061
01961
10
0.3123
0.031
01881
90
1000
1/000
105
60
Presie www.laeetberieeeegeonno
23537
0.2004
165
161
17
16.
03354
07361
03331
165
161
33506
01529
03355
15
196
121
177
Seaser Typs
0.0355
0.585
035344
41
24
157
17
11
204
25
47
190
0.4522
(1903)
0.547
189
16
259
97
37
171
300
Sorvet
25
113
03300
01533
30
302
Shapdatiin
035867
01215
0.530
12
153
25
10
134
22
157
304
(1800
23
263
740
21
34
0.5087
305
01533
30
115
307
Songeaiteen sichecte Boad.
(1975
14
34
307
seawheezy
03235
00381
(1534)
336
344
3
300
111
Sontob
0.350
0.006
(1539)
930
577
112
(0967
119531
299
30
033566
114
Teckagmanatiett
03882
D.5215
033155
106
3
107
100
115
03613
01956
0.8999
142
11
760
24
706
01536
3387
334
LND
03561
0/875
15
56
35
113
035491
11200
013115
112
100
1838
85
201
2671
1008
136
1814
131
(3811
06031
01344
313
16
111
17A
3144
408
360
1833
123
Wates
0.822)
(1934
0.997
151
3346
134
Wine
(.99%)
0150
125
Watto
0.823
07714
033552
10
26
12
125
Yoga
0.839
0620
0.548)
183
1934
135
156
1812
Ang Vall
13566
0/617
29

---

Table 20: Multi-scale analysis of w/o cross-scale fusion based on Scale-tcaching using UCR 128 archive with Sym 20% noisy labels.
Inc
S.UJJ
E.L.J k.Ex. IUJU
SLU
CLU
AESII
0.5500
E00000
1325931
Adse
0.1014
00021
0.007
206
0.583
DU863
0.2944
363
13
25
38
30
0
5
121
Alklewkace creecci
0.335
04114
15
34
35
19
45
54
0.210
Raod
03360
133391
Soakett
03500
0.300
0.399)
10
10
CTI
03888
0.311
01151
533
768
76
Chauason
03114
07791
03338
790
N6
Chiviiat coctcation
035671
0.760
0333
700
H4R
16
Coffee
10000
0.861
0.8623
13
0
13
15
13
0,492
0,1723
01599)
125
35
04785
(1856
(11149
M63
21
Cass
0.000
012255
10215
256
1832
1757
748
11
183
1153
166
0.7655
0.122
0.891
31
13
18
50
46
Denaltha ateeeeeeee
[.977
148
137
21
0.765
13389)
26
Decpolooppter
0.343
0.112
0.3391
19
15
16
30
75
wwwkrreerwwweer Wieben!
00044
0.30
03504
W
56
16
36
30
0.559
02252
0.501
240
1256
1354
63
4383
08155
0.909
35
0.353
D0061
(19911
100
15
33
16
13
15
34
35
Elcendikokk koz
0.567
0.593)
5458
73
157
5183
530
013415
121
116
118
125
223
05155
36
0.444
0:391
13
5
11
40
FillWiads
0.114
09068
0.093
142
18
178
06264
(1464)
624
432
57
0.715
03532)
235
356
40
100
30
PreoecSeal Train
03336
0,420
(1530)
1167
606
164
1008
126
1363
750
02531
02286
0.5963
32
Gornen abuubo
(1251
015055
GosscMaaiaat
30
03253
0.3195
114
20
13
32
33
C
Gealhore
1000
09551
0.801
143
60
74
76
14
0.355
0.4377
131
Gedhwer Corheerbtanbmmelorr
0.290
135
19)
03333
196
105
150
165
57
HanDheliee
0.3741
0/605
0.635
106
230
11
306
Haring
03154
014562
26
60
Hon cllentte
0.525
04200
0.4302
50
60
03311
06926
01001
19
118
140
0
0
307
66
0.3157
01125
0.0943
539
146
240
173
14
70
368
116
516
0.333
01388
125
116
67
Lighning:
0.0085
0.310
30
9
15
02500
01500
01111
631
130
704
47
54
0.130
Medicallragee
0.655
0.536
305
26
101
00079
1939
76
03193
04655
033364
51
182
40
130
23
28
143
Misdtt Traae
0.5803
000144
0.3341
525
3956
316
77
Atteehhupustial Treas
0.336
03753
01134
3381
26
599
7M
Neatrasie Twerst
02333
02005
412
36
38
12
0
480
13
51
0.5825
01196
0.122
0
202
0.430
PLAID
0.314
61711
41
S4.
0.7883
012066
031331
190
29
407
160
26
0.3100
0.000
01300
3
0.1130
00085
0.5012
00230
01200
11
PowerCCaae
03397
03367
0.5301
67
54
0
10
70
B3
0.315
0.2113
196
46
W7
01061
03335
136
035266
100
Rock
3.3600
01100
0.300
13
0.3007
0.977
0.390
167
13
200
72
283
138
302
035111
02116
01330
30
St SkaleCei Winemnd
0.4800
10860
0.390
21
0
19
Stapddidien
303
Shapeokd
03500
0/26
(1200)
325
15
0
0
210
Send
353
0.500
(3333
0.3931
16
27
96
55
305
Bucel
047755
01131
56
305
0.540
0,250
03311
3750
21
2253
2305
4344
0
45
3124
4409
113
Saadiil ofr
5005
07660
531
14
24
516
11
0.5548
0.134)
114
153
SynthetcCient
01967
113
0.7737
0.710
109
108
110
103
115
Fraco
15500
0/800
03300
57
30
70
15
157
03634
D1221
03111
3382
1015
234
2800
118
03881
0L303
03351
2385
135
Unovereennt Bosun
0.160
03305
194
360
12
144
348
122
0.559
01126
0.122
1855
640
404
2312
300
(U801
113
5183
134
Wine
03000
(15301
125
0.323
02223
0.150)
135
120
79
22
42
138
59
139
03555
0.571
0.514
40
44
15
123
Yoga
0.853
0.387
403
1254
173177
119
30

---

Table 21: Multi-scale analysis of Scale-teaching using UCR 128 archive with Asym 50% noisy labels.
Adlic
0.014
0.185
132517
36
12
95
03334
018633
0,4883
-1
56
129
15
19
9m
09
11
220
03134
0.664
0.1824
70
94
130
11
4
307
133
26
211
0.500
0.300
0.380
13111
10
03000
0.200
033501
13
13
Car
04002
04633
01493
Chanaoo
0.3811
01861
231
Chiorian
1S5
1767
15
03312
03352
03229
114
352
28
21
435
110
131
335
Conpelers
03133
0,877
0.489)
121
119
18
CREEY
03531
0.331
033822
116
29
Ohckeet
0.114
03384
0.552
10
138
71
17
7390
2.555
0.500
0355)
178
17M
179
0.043
(1942
0.364
105
130
01681
0/601
27
03823
033455
100
34
59
Dedggil
2CG200
0.280
06260
0.599)
60
59
0.075
03000
3590
0551
76
940
PocHeetioeettig Higral
0.200
17
136
115
Sc
0.347
0755
0.388
104
105
0
I0L
0.340
0.902
36
106
183
57
0.4453
0.5511
0.4314
106
35
42
731
M
SM
0.387
241
134
68
77
908
211
94
715
41
Fas
0.1371
0.447
19
59
60
18
13
1131
1134
0.561
0.591
(15351
15
da
0.145
0.354
21
2101
31
2115
69
46
2000
46
Pongi
0.125
012911
10
36
30
14
45
13
0.755
033895
76
01:85
0.011
0 0.890
10
74
13
69
0.507
0-801
03377
25
136
140
0.344
0.016
011555
250
56
Hans
03345
033443
23
5
24
38
Hapki
0.222
0.325
0.342
04
60
Horleg
0.475
03755
107
10
61
0.1835
01280
0.599
92
30
42
16
112
54
25
60
Thain
0.5600
0.56
03534
14
136
43
D
50
5
56
105
0.311
01985
0.8995
599
18
STY
66
Lagglr Aystiai
015471
0.500
306
36
66
Lishtine"
04447
D4110
04434
0
30
25
Mizi
0.390
DA167
03533
35
71
Modcalinngo
03967
415
159
03666
05541
163
75
0.1133
01623
0.790
24
23
7
23
75
Mal liffee Tu
0.5615
047771
0.110
72
10
65
Mise
Misuittupusiina Traiss
0.351
033611
(1459)
125
857
332
649
2.480
0.000
01499)
0
577
ST
1067
101
0086
(15517
115
30
198
662
60
Olookl
0.400
04333
04491
12
03137
01154
33
Pioncne
02003
01072
013963
91
150
35
319
16
16
277
87
111088
1344
18
17
15
18
13
0.377
25
28
0.033
90
0.063
0.557
60
Precie www.keeetterreeeeeeggro
01285
03171
166
147
5
18
16.
03563
06637
(1583)
129
0.000
013457
136
196
15
17
135
177
Seaser Type
03115
01583
03153
17
133
6
122
11
23
11
190
Song Sa
0.313
04120
0.499
15
14
170
46
26
140
300
0.770
18
107
St aasotannottiaaaa
03300
(35301
302
Stapdadiin
03500
0,200
0.330
0
40
50
304
01011
0/966
(1081)
316
756
15
17
3/4
110
305
07511
10
36
307
Songaaitt tichecte Boad
11001
0.7766
111
434
177
307
seawheezy
03412
(12052
15
15
232
25
222
111
Sontob
0.315
04223
0453)
377
137
314
112
03233
023773
230
34
04667
114
07882
01785
03354
D00
115
03533
019200
01511
31
175
30
26
02767
1067
169
100
2058
334
LNDD
03331
015312
113
UWaeeli wew.eiooaaaealaaaageeeoooo All
034883
(1300
0338
148
236
108
159
1229
144
218
1548
131
0.881
313
316
1815
60
100
1628
330
272
1854
123
Wates
0.821
0.053
0.588
2009
72
5499
134
Wine
0.980
11533
125
Watto
03323
0.667
0.3936
36
35
125
Yoga
0.317
0.300
0.190)
1533
1330
Ang Vall
0.475
0058
31

---

Table 22: Multi-scale analysis of w/o cross-scale fusion based on Scale-tcaching using UCR 128 archive with Asym 40% noisy labels.
Inc
S,UJ
E.L. k.Ex. S.U
AESPI
0.0001
EJ0000
123391
Adse
08865
(0086
0.0003
0
8
0.220
0.1364
0,2383
337
78
67
83
15
12
31
28k
6
Alklesksck crweeci
0.381
03822
0,522
20
35
0.333
Raod
03331
033867
Scaketty
04400
0.300
0.3301
10
CTI
3.504
0.311
0331
305
38
Chouason
03335
03375
03335
20
120
Chaviia coctcation
0.386
01181
1996
1000
764
1977
16
Coffee
03355
0.961
0.863
13
13
0.792
03754
0.3335
0.338
01314)
21
Cass
0883
D0000
01234
7553
1TM
870
1040
ST6
385
8364
03865
0.524
65
20
54
Denalhha aneteaattteenneeee
0.57
0.633
033535
26
Decpolooppter
0.100
0,007
0.193
15
9
10
12
75
wwwrrreerwwweerd Wieben!
03734
0.308
03504
25
16
13
30
0.385
01195
0.591
5115
4300
0.909
761
703
35
03833
012911
54
35
25
7
25
D0
35
Elcendiookk keze
0.0003
011131
45TS
407
4333
301
03350
03334
125
03508
244
36
04435
00360
03335
41
18
6
27
14
40
FillWiads
0.2133
01123
0.699
53
30
20
100
03807
0.766
6.6
133
AT
50
03447
03534)
51
400
33
163
350
65
PreoecSeal Train
02504
0.768
(1530)
143
172
1354
775
160
1333
101
0.1199
00015
0.5963
01785
017467
GosscMaaiaat
50
04434
11180
0.3435
53
C
Gealhore
0.971
0.111
20
17
17
75
04542
0.4377
Gedhwer CorheetMtanmmmeeoro
190
04500
03333
165
150
165
57
HanDheliies
03400
0/566
06815
164
117
136
230
113
100
130
Haring
6.665
014522
26
60
Hon cllentte
03353
0.214
0.4302
54
48
60
00000
0.7716
0.533
111
138
42
118
99
160
66
0.1622
00742
0.183
283
177
223
14
139
219
M22
thay Pheceht bhroand
516
0.8055
0.265
0333
67
Lighning:
0.8353
0.5410
0.310
15
24
0
35
13
24
01311
0200
376
30
16
501
Medicallragee
0.596
03371
106
314
100
01000
1773
195
15
14
76
0.030
0
033344
17
184
105
34
15T
Misddt Traas
0.390
D1851
0.124
1725
312
3527
Atteehbupustial Troas
04444
0.115
012511
975
7M
Neatradiet Twerst
0.1100
(15422
182
42
83
0
D0
13
51
0.001
(11994
0.089
36
18
148
0.4301
PLAID
02134
0.165
46
S4.
03642
04142
038111
145
110
408
0
336
11
146
307
0.133
04060
13
0
2
0.8544
0.5092
00009
01217
11
PowerCCane
03335
01300
033355
61
10
50
45
0.340)
04638
199
195
0.333
011951
132
Rock
0.800
0.300
22
0.200
03777
130
123
200
78
215
160
302
03301
01141
112341
100
72
351
St SkaleCei Winemnd
033300
01:00
0
18
Stapddidiin
303
Shapeokd
03311
134
12
15
0
144
31
353
08133
03333
0.333
25
21
Buoll
367
0.7241
06711
01231
S85
514
305
0.831
02250
03372
5707
90
2225
2355
4334
0
24
4130
113
Saatiil off
07077
01301
114
47
13
14
SEE
16
011648
163
153
SynthetcCient
113
0.205
0170
105
115
Traco
03300
01100
03300
13
157
03127
001500
033%%
384
1090
328
3224
118
0330
0.303
(33131
251
135
Unovereennt Bosus
0332
01136
01115
184
430
464
430
3
186
1984
36
122
0.313
(1159
0.081
196
402
125
1813
200
250
5387
134
Win
0.500
0.500
(1530
125
0.342
01396
0,095
85
46
30
148
339
0.334
0.511
01514
14
11
44
11
123
Yoga
03333
357
1050
(730)
116
32


Reviews:
[{"review_id": "Wlhfqx8wPq", "reviewer": "Reviewer_Rktg", "summary": "This paper presents a \"scale-teaching\" framework aimed at addressing label noise in time series classification tasks. The proposed approach involves multiple encoders to extract multi-scale time series features. These features are then concatenated for the final loss calculation, enabling the selection of clean samples. The identified clean and noisy samples are further processed using a label propagation algorithm, incorporating graph learning techniques to correct the noisy labels. The entire dataset is subsequently trained using cross-entropy as the loss function. Experimental evaluations are conducted on the UCR 128 archive and UEA 30 archive datasets, both containing synthetic label noise.", "questions": "See **weaknesses**", "limitations": "This work extensively addresses its limitations, and based on the information provided, it appears that the paper does not pose potential negative societal impacts.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. While the problem of learning with noisy labels (LNL) has been extensively studied in the domain of image classification, the exploration of label noise in time series classification remains relatively limited in current literature. This work makes significant contributions by conducting extensive experiments to investigate how the LNL experience gained in image classification can be effectively generalized to time series classification.\n\n2. The utilization of multi-scale operations is particularly noteworthy in the context of time series classification with label noise, considering the inherent differences in modality between images and time series data.\n\n3. The paper demonstrates a clear and coherent writing style, ensuring ease of comprehension for readers. Additionally, the release of the accompanying code promotes reproducibility, while the appendix provides comprehensive details on the conducted experiments.\n", "weaknesses": "1. While I appreciate the contribution made by generalizing learning with noisy labels (LNL) from image classification to time series classification, the level of technical novelty in this work appears somewhat limited. Specifically, the utilization of a multi-scale framework has already been explored in time series forecasting tasks, and the application of a semi-supervised approach to enhance performance is not novel in the context of general LNL tasks [R1].\n\n2. The paper lacks sufficient ablation studies to thoroughly examine the individual components of the proposed method. For instance, in order to demonstrate the efficacy of the multi-scale framework in sample selection, it would be valuable for the authors to conduct controlled experiments using only one scale. While accuracy is reported in Table 3, it is not necessarily the most appropriate criterion for evaluating sample selection. Therefore, I suggest that the authors also report selection quality metrics such as precision, recall, and F1 score. Additionally, it would be worthwhile to explore alternative label correction techniques, such as pseudo-labeling or solely utilizing clean samples for training, to ascertain their competitiveness in comparison to the label propagation method employed in the paper.\n\n3. It would be insightful to evaluate the performance of the \"scale-teaching\" framework on datasets that do not contain label noise. I noticed that such experiments were conducted in [R2], and I encourage the authors to perform similar settings to provide a comprehensive analysis in their work.\n\n\n[R1] DivideMix: Learning with Noisy Labels as Semi-supervised Learning\n\n[R2 ]Estimating the Electrical Power Output of Industrial Devices with End-to-End Time-Series Classification in the Presence of Label Noise", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1**:  The utilization of a multi-scale framework has already been explored in time series forecasting tasks, and the application of a semi-supervised approach to enhance performance is not novel in the context of general LNL tasks [R1].\n\n**A1**: Thanks for your comment. \nIn Section 2, we have discussed existing work related to multi-scale time series modeling. In particular, the multi-scale properties of time series have been widely used for various time series downstream tasks, such as time series prediction,  classification, and anomaly detection tasks.\nThe above studies show that exploiting the multi-scale properties of time series can improve the performance of downstream tasks. Nevertheless, the above studies do not consider how to utilize the multi-scale properties of time series for label-noise learning. Unlike the above work, we employ multiple DNNs with the same structure to learn discriminative patterns of the time series at different scales separately, and design a cross-scale fusion strategy to obtain robust embeddings for handling noisy labels.\n\nDivideMix is a classical work for image label-noise learning that combines MixMatch semi-supervised learning method for noisy label classification. In contrast, our proposed Scale-teaching paradigm utilizes learned multi-scale embeddings to cope with time series noisy labels.\nIn other words, Scale-teaching focuses on utilizing robust feature representations for label-noise learning rather than a semi-supervised learning paradigm for label-noise learning.\nIn particular, Scale-teaching utilizes its learned multi-scale embeddings to select more reliable clean labels. Further, Scale-teaching uses the selected clean labels in conjunction with multi-scale graph learning to achieve noisy label correction. Hence, the focus of Scale-teaching exploits the multi-scale nature of time series for label-noise learning, and does not rely exclusively on the semi-supervised learning paradigm for noisy label classification.\n\n[R1] DivideMix: Learning with Noisy Labels as Semi-supervised Learning\n\n\n**W2**: (i) It would be valuable for the authors to conduct controlled experiments using only one scale. (ii) I suggest that the authors also report selection quality metrics such as precision, recall, and F1 score. (iii) Additionally, it would be worthwhile to explore alternative label correction techniques, such as pseudo-labeling or solely utilizing clean samples for training.\n\n**A2**: Thanks for your comment. \n\n(i) From Table 3 in Section 4.5, we have conducted controlled experiments using only one scale. For details, please refer to the *only single scale* method in Table 3.\n\n(ii) That's a good suggestion. Following [R2], we select averaged F1-score on the test set as a new metric for ablation analysis. Like Table 3 in Section 4.5, we give the corresponding test classification F1-score (%) as follows:\n\n|Dataset | HAR |  | UniMiB-SHAR |  |\n|:---:|:---:|:---:|:---:|:---:|\n| Method  | Sym 50% | Asym 40% | Sym 50% | Asym 40% |\n| Scale-teaching | **90.05** | **89.14** | **77.56** | **65.89** |\n| w/o cross-scale fusion | 88.16 (-1.89) | 87.05 (-2.09) | 68.23 (-9.33) | 57.76 (-8.13) |\n| only single scale | 87.56 (-2.49) | 86.75 (-2.39) | 66.87 (-10.69) | 54.12 (-11.77) |\n| w/o graph learning | 87.79 (-2.26) | 87.41 (-1.73) | 74.62 (-2.94) | 63.15 (-2.74) |\n| w/o moment | 89.34 (-0.71) | 88.27 (-0.87) | 76.67 (-0.89) | 64.92 (-0.97) |\n| w/o dynamic threshold | 88.93 (-1.12) | 88.29 (-0.85) | 73.11 (-4.45) | 64.76 (-1.17) |\n\n(iii) In Section 3.4, we claim that $\\mathcal{F}$ in Eq. (7) is the estimated pseudo-labels which inevitably contain some incorrect labels. To address the above issue, we utilize a dynamic threshold pseudo-labeling strategy to promote label correction performance. \nFurther, we add an ablation study method (w/o dynamic threshold) in Table 3. The result shows that the dynamic threshold pseudo-labelling strategy can improve classification performance.\nFor solely utilizing clean samples for training, please refer to the ablation study method named *w/o graph learning* in Table 3.\n\n[R2] Estimating the Electrical Power Output of Industrial Devices with End-to-End Time-Series Classification in the Presence of Label Noise\n\n**W3**: It would be insightful to evaluate the performance of the \"scale-teaching\" framework on datasets that do not contain label noise.\n\n**A3**: Thanks for your comment. \nWe select the four individual large datasets without noisy labels for evaluation. The detailed test classification accuracy (Acc, %) and F1-score (F1, %) are shown as follows:\n\n|  | Standard  |  | Mixup  |  | Co-teaching  |  | FINE |  |  SREA  |  | SELC  |  | CULCU  |  | Scale-teaching |  |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| Dataset | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 |\n| HAR | 93.29 | 93.27 | **95.42** | **95.39** | 93.77 | 93.75 | 93.13 | 93.19 | 93.02 | 92.91 | 93.76 | 93.71 | 94.75 | 94.72 | 94.72 | 94.18 |\n| UniMiB-SHAR | 89.14 | 86.37 | _84.84_ | _80.17_ | 88.24 | 84.43 | 88.14 | 84.03 | _65.51_ | _66.54_ | 89.28 | 89.19 | 89.46 | 86.45 | **93.61** | **93.62** |\n| FD-A | 99.93 | 99.93 | 99.91 | 99.91 | **99.96** | **99.96** | _68.22_ | _64.05_ | _90.25_ | _90.14_ | 99.82 | 99.82 | 99.95 | 99.95 | **99.96** | **99.96** |\n| Sleep-EDF | 84.93 | 81.99 | 84.67 | 82.11 | 85.37 | 82.52 | 84.62 | 83.07 | _79.42_ | _77.67_ | 84.82 | 82.17 | **85.54** | 83.26 | 85.34 | **84.76** |\n\nAs shown in the above table, Scale-teaching's classification performance is still better than most baselines. In addition, SREA employs an unsupervised time series reconstruction loss as an auxiliary task, which reduces the model's classification performance without noisy labels.\n\n"}, {"review_id": "RkVD8Sl695", "reviewer": "Reviewer_QaFC", "summary": "The paper proposes a time series classification algorithm that works across multiple scales with a noise-robust loss function. The authors claim improvements due to the multi-scale network architecture and the objective function capable to handle label noise on a variety of datasets in UCR128 and beyond.", "questions": "- The work seems large incremental, given that the method (3.4) is composed of techniques used in [44], [40], and label propagation. Could you highlight the main conceptual novelty of your algorithm compared to previous approaches?\n- What is the test performed in Table 1? I would suggest to include this directly into the caption.\n- l. 327 in the conclusion claims \"... can utilize the multi-scale properties of time series to effectively handle noisy labels\". Where is this particular point validated, in your opinion? \n- How are hyperparameters selected? Can this be made more explicit in the main paper?\n- What are additional limitations of the assumed noise model and correction mechanism?", "limitations": "n/a", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is mostly well organized. Statistical tests were performed in all experiments, and the paper has a good mix of summary statistics for large evaluation runs (like in Table 1), and more qualitative results showing properties of the algorithm (e.g. the training dynamics in Figure 5). The authors also perform ablation studies and attempt to test the effect of different components of their algorithm.", "weaknesses": "- presentation can be generally improved. In section 3, it is not fully clear which components are preliminaries and present in existing work, and which are the new algorithmic components of the new approach. According to title and abstract, these should be the multi-scale and label noise corrections parts of the model. Maybe the headings could be further adapted to make this clear (e.g. right now, it is not clear why \"cross-scale fusion\" mostly addresses the multi-scale aspect, while \"multi-scale embedding graph learning\" mostly covers the label noise robustness part).\n\n- The figure caption for Figure 2 could be a bit longer and actually explain the figure content. \n\n**Minor**:\n\n- typo: emebeddings (l. 56)\n- the definition of the considered noise model comes too late and should be directly mentioned in Section 3.\n- Table 1 has a lot of numbers. Given that you ran a statistical test, consider to e.g. gray out the non-signficant results for easier readability.\n- sec 4.5 has a few typos (missing spaces, etc)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**:  The work seems large incremental, given that the method (3.4) is composed of techniques used in [40], [44], and label propagation. Could you highlight the main conceptual novelty of your algorithm compared to previous approaches?\n\n**A1**: Thanks for your question. \nLiterature [40] uses original sample features for graph construction, thus for semi-supervised learning. Literature [44] designs an unsupervised reconstruction loss for semi-supervised learning using a momentum update strategy.\nIn contrast to [40], we construct the graph using learned multi-scale embeddings. Unlike [44], we perform graph construction using momentum-updated multi-scale embeddings to exploit robust multi-scale embeddings for noisy label correction.\nCompared to existing noisy label learning methods, our conceptual innovation is to exploit the complementary information between sequences of different scales and design a cross-scale fusion mechanism for learning more robust multi-scale embeddings. Furthermore, we use multi-scale embeddings to introduce graph learning based on label propagation for noisy label correction.\n\n[40] Label propagation through linear neighborhoods. ICML, 2006.\n\n[44] Temporal ensembling for semi-supervised learning. arXiv, 2016.\n\n**Q2**: What is the test performed in Table 1? I would suggest to include this directly into the caption.\n\n**A2**: Thanks for your question and suggestion. \nThe results in Table 1 show the test classification accuracy statistics of different baselines on four individual large datasets, the UCR 128 archive and the UEA 30 archive.\nThe Avg. Rank indicates the average ranking of the corresponding baseline's test classification accuracy among all baselines for each benchmark. In the next version, we will update the description in the caption of Table 1.\n\n**Q3**: l. 327 in the conclusion claims \"... can utilize the multi-scale properties of time series to effectively handle noisy labels\". Where is this particular point validated, in your opinion?\n\n **A3**: Thanks for your question. \nIn Section 4.3, we analyze the cross-scale fusion mechanism for Scale-teaching. Figure 3 in Section 4.3 illustrates that the classification results of different scale sequences exhibit evident complementary information. Specifically, Figure 3 (c) demonstrates that Scale-teaching (larger shaded circle area) has significantly more samples with correct category predictions for the classifiers corresponding to the coarse and fine scales on the UCR 128 archive than w/o cross-scale fusion (smaller shaded circle area). Additionally, Scale-teaching on the coarse $\\rightarrow$ fine scale also shows a significantly larger cross-circle area (525) compared to w/o cross-scale fusion (207). These results indicate that Scale-teaching can improve the model's classification performance by effectively utilizing the complementary information between different scales in the presence of noisy labels.\n\nFurthermore, the t-SNE visualization in Figure 4 reveals that the multi-scale embeddings learned by Scale-teaching are more discriminative across categories. Hence, we claim that Scale-teaching can effectively leverage the multi-scale properties of time series to handle noisy labels.\n\n\n**Q4**: How are hyperparameters selected? Can this be made more explicit in the main paper?\n\n**A4**: Thanks for your question.\nOur experiment contains 162 datasets. It would be time-consuming to perform hyperparameter selection for each dataset. Therefore, the hyperparameters of Scale-teaching are not carefully tuned for each dataset, and most of the hyperparameters are set based on the default hyperparameters of related works.\nThe learning rate and epoch are set based on the parameters of existing noise-label learning methods, such as FINE and CULCU.\n$\\alpha$ in Eq. 3, $\\sigma$ in Eq. 4 and $\\beta$ in Eq. 5 are set based on the default hyperparameters of related label propagation works. \n$e_{warm}$ is based on FINE settings. $e_{update}$, $\\gamma$ and batch size are based on manual empirical settings without specific hyperparameter analysis.\nThe largest neighbor $K$ is set based on human experience, and we had a simple test on several datasets, and found that a larger value of $K$ does not improve the classification performance, but instead increases the running time of the model. In our next release, we will explain in detail the basis for choosing each hyperparameter.\n\n**Q5**: What are additional limitations of the assumed noise model and correction mechanism?\n\n**A5**: Thanks for your question. \nIt may be difficult for our proposed method to achieve good classification performance with noise ratios greater than 50%. In particular, the literature [R3] suggests that the small-loss criterion selects clean labeled samples that still contain many noisy labels when the noise ratio is greater than 50%. Therefore, existing work on label-noise learning based on the small-loss criterion (e.g., Co-teaching, CULCU) has experimentally set label noise ratios of less than 50%.\n\n[R3] Towards understanding deep learning from noisy labels with small-loss criterion. IJCAI, 2021.\n\n**W1**:  presentation can be generally improved.\n\n**A6**: Thanks for your comment.\nWe think your suggestions are excellent. In our next release, we will update the title and organization of Section 3 based on your suggestions.\n\n**W2**:  The figure caption for Figure 2 could be a bit longer and actually explain the figure content.\n\n**A7**: Thanks for your comment. We will update the caption of Figure 2 in the next version.\n\n**W3**:  Minor: typo: emebeddings (l. 56), etc.\n\n**A8**: Thanks for your comment. We will fix the above issues in the next version."}, {"review_id": "r5384p0yd5", "reviewer": "Reviewer_L5T6", "summary": "This paper a paradigm to improve the small-loss criterion for time series noisy labels, which mainly addresses the problem that external noises easily distort time series’ discriminative patterns. The motivation is clear, and experiments have shown good results. However, the reviewer is still concerned about some issues.", "questions": "1. What is the fundamental difference between the proposed method with FPN and traditional label propagation is graph learning?", "limitations": "The authors seem to discuss some limitations of the method in terms of computation efficiency. The reviewer suggests more discussion in energy consumption for a more general usage.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. DNNs with different random initializations have high consistency in classification results for clean labeled samples in the early training period; while there is disagreement in the classification of noisy labeled samples, the designation of cross-scale fusion using inputs with different scales is lighter than other methods using different networks like co-teaching which are time-consuming.", "weaknesses": "1. Although the proposed method is useful, the network is similar to multi-scale feature learning like FPN and the label propagation part is also not novel.\n2. The way of downsampling methods needs more comparison.\n3. Momentum updating is also useful for other methods, although the goal of using it in this paper is reasonable, and scale-teaching achieves SOTA even w.o. momentum updating. But it is still an unfair comparison. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1 & Q1**:  The network is similar to multi-scale feature learning like FPN and the label propagation part is also not novel. Also, what is the fundamental difference between the proposed method with FPN and traditional label propagation in graph learning?\n\n**A1**:  Thanks for your comment.  Based on the keyword *FPN*, we searched for the article [R1], which is expected to be the FPN. First, we summarize the differences between Scale-teaching and FPN as follows:\n```\n(1) FPN and Scale-teaching solve different problems. FPN is used for image object detection, while Scale-teaching is used for time series classification with noisy labels. In addition, time series data is a collection of data points sorted in chronological order, which is different from image data. In general, it is hard to use the image model directly for time series.\n(2) FPN takes a single-scale image as input and uses a top-down pyramid network to learn the multi-scale embeddings of the image. Scale-teaching takes different scale sequences of the time series as input and uses the cross-scale fusion mechanism to learn the multi-scale embeddings of the time series.\n(3) For different scale embeddings, FPN uses shared classifiers/regressors for object detection. In contrast, Scale-teaching employs multiple classifiers with different initializations for classification training, thus exploiting the divergence information of the DNNs on the noisy labels to obtain more robust multi-scale embeddings.\n(4) FPN uses one randomly initialized DNN to learn multi-scale embeddings of an image. Scale-teaching uses several different randomly initialized DNNs to learn multi-scale embeddings of a time series, and uses selected clean labels for cross-teaching classification training.\n```\n\nFor the differences between Scale-teaching and Traditional Label Propagation (TLP) in graph learning, we summarize as follows:\n```\n(1) TLP is generally used for semi-supervised learning. Scale-teaching solves the time-series noise label learning problem with label propagation.\n(2) TLP generally uses single-scale data for graph construction. Scale-teaching utilizes learned multi-scale embeddings for graph learning.\n(3) TLP requires explicitly correctly labeled data for modeling. Scale-teaching utilizes selected small-loss data as correctly labeled data for modeling.\n```\n\n[R1] Feature Pyramid Networks for Object Detection. CVPR, 2017.\n\n\n**W2**: The way of downsampling methods needs more comparison.\n\n**A2**: Thanks for your comment. In our supplementary material, we have provided the comparison analysis of downsampling methods in Section E. \n\nFirstly, we analyze the impact of the downsampling scale sequence list. From Figure 2 in Section E, we find that Scale-teaching has the highest classification accuracy using four different scales for training, indicating that more input scales do not necessarily improve the classification performance. In addition, using three or four scales of sequences can effectively improve the classification performance of Scale-teaching compared with using two different scales.\n\nThen, we analyze the impact of input scales of sequence order. As shown in Figure 3 in Section E, We can find that the classification performance of finer-to-coarser is better overall, which is due to its ability to use a single fine-scale sequence with an excellent classification performance from the beginning to promote the classification performance of multi-scale fusion embeddings gradually.\n\n\n**W3**: Momentum updating is also useful for other methods.\n\n**A3**: Thanks for your comment. \nExisting noise-label learning methods rarely consider utilizing robust feature representations to cope with noisy labels.\nSpecifically, existing noise-label learning methods mainly design robust learning paradigms at the loss level, while Scale-teaching focuses on handling noise labels with robust multi-scale feature representations.\nTherefore, Scale-teaching designs the cross-scale mechanism and combines it with momentum updating to learn robust multi-scale feature representations.\n\n"}, {"review_id": "RxmBQmi1Vy", "reviewer": "Reviewer_oEKq", "summary": "This paper proposes a deep learning paradigm, namely scale-teaching, to cope with time series noisy labels, with designing a fine-to-coarse cross-scale fusion mechanism for learning discriminative patterns by using time series at different scales to train multiple DNNs simultaneously. Additionally, each network is trained in a cross-teaching manner by using complementary information from different scales to select small-loss samples as clean labels. Meanwhile, this paper introduces a multi-scale embedding graph learning method via label propagation to correct their labels from unselected large-loss samples.  Extensive experiments demonstrate the superior performance of the proposed method.", "questions": "1. In Eq. (2), how the operator || to operate the r_i^k, v_i^k and so on?\n2. How to obtain the Eq. (5)?", "limitations": "The proposed Scale-teaching paradigm only utilizes three scales, and its training time is increasing with the increase of the number of scales.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The research problem is interesting and important.\n2. The idea is simple and effective.\n3. The paper is well written.\n4. Extensive experiments with good results.\n", "weaknesses": "1. The meanings of some equations or symbols are not clear, like the operator || in Eq. (2).\n2. The problem definition in Label Correction for Eq. (5) is difficult to understand.\n3. The proposed method is relatively complex, and its procedure is not clear, including the Algorithm 1.\n4. In Figure 2, it is better to introduce the flowchart of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1 & Q1**:  The meanings of the operator || in Eq. (2), and how the operator || to operate the r_i^k, v_i^k.\n\n**A1**: Thanks for your comment. The operator || in Equation (2) means to concatenate two vectors into a new vector. For example, vector **a** = [0,1], vector **b** = [2,3], then **a** || **b** = [0,1,2,3].\nr_i^k represents the single-scale k embedding, and v_i^k is the multi-scale ([1, 2, …, k], where k is downsampling scale) fusion embedding via Eq.(2). Particularly if k = 1, we use the single-scale for classification training and let v_i_k = r_i_k. If k > 1, v_i_k = f(r_i_k || v_i_k-t||(r_i_k - v_i_k-t)||(r_i_k * v_i_k-t)), where v_i_k-t denotes the multi-scale ([1,2, …, k-t], where k-t is downsampling scale) fusion embedding. And the source code for the implementation of Eq (2) is given below:\n\n```\n f = nn.Sequential(\n            nn.Linear(128 * 4, 256),\n            nn.BatchNorm1d(256),\n            nn.ReLU(inplace=True),\n            nn.Linear(256, 128),\n        )  ### the function of f(.)\n\nfea_concat = torch.cat((embed_x, scale_x, (embed_x - scale_x), (embed_x * scale_x)), dim=1) ### the function of operator ||. where embed_x is r_i^k, scale is v_i_k-t.\n\nv_i_k = f(fea_concat)\n```\n\n\n**W2 & Q2**:  The problem definition in Label Correction for Eq. (5) and how to obtain the Eq. (5)?\n\n**A2**: Thanks for your comment. In Eq.(5), **Y** contains labeled and unlabeled samples, Y_ij=1 if x_i is labeled as y_i =j, otherwise Y_ij = 0. Label propagation theory utilizes the diffusion process [R1] to get the pseudo-labels of unlabeled samples via labeled samples and nearest-neighbor **Q**. \nHence, we defined the sequence {F_t} to denote the diffusion process and described it as Eq.(5).\nWhen *t = 0*, *F_0* is initialized as **Y**. Further, when timestamp *t* in Eq. (5) is large, through the label information of **Y** and the graph edge weight of nearest-neighbor **Q**, the predicted pseudo-label *F_t* quality is better. Conveniently, we can know the sequence {F_t} has a closed-form solution [R1], which can be defined as Eq. (6).\n\n[R1] Learning with local and global consistency. NIPS, 2003.\n\n\n**W3 & W4**: The proposed method is relatively complex, and it is better to introduce the flowchart of the proposed method in Figure 2.\n\n**A3**: Thanks for your comment. The proposed Scale-teaching paradigm includes two processes: (i) clean label selection; (i) multi-scale graph embedding learning. \nFor clean label selection,  \nnetworks A, B, and C use clean labels provided by cross-teaching (A$\\rightarrow$B, B$\\rightarrow$C, C$\\rightarrow$A) to guide each other's classification training.\nFor multi-scale graph embedding learning, Scale-teaching first performs cross-scale fusion from fine to coarse (A$\\rightarrow$B, B$\\rightarrow$C).\nThen, we utilize pseudo labels obtained via multi-scale embeddings graph learning as corrected labels for those time series not selected as clean labelled data.\nIn addition, we give a pseudo-code for the Scale-teaching paradigm in Figure 2:\n\n```\nStep 1: Obtain single-scale embeddings r_A, r_B, r_C\nr_A = encoder_1(x_A)\nr_B = encoder_2(x_B)\nr_c = encoder_3(x_C)\nStep 2: Obtain cross-scale embeddings v_A, v_B, v_C\nv_A = r_A\nv_B = Eq.5(r_B, v_A)\nv_C = Eq.5(r_C, v_B)\nStep 3: Obtain clean labels y_A, y_B, y_C for cross-teaching training \ny_A = classifier_3(v_C)  ## Using small loss criterion\ny_B = classifier_1(v_A)  ## Using small loss criterion\ny_C = classifier_2(v_B)  ## Using small loss criterion\nStep 4: Obtain corrected labels yc_A, yc_B, yc_C for classification training\nyc_A = Eq.6(v_A, y_A)\nyc_B = Eq.6(v_B, y_B)\nyc_C = Eq.6(v_C, y_C)\nStep 5: Overall training\nUpdate encoder_1 and classifier_1 via cross-entropy loss(v_A, yc_A)\nUpdate encoder_2 and classifier_2 via cross-entropy loss(v_B, yc_B)\nUpdate encoder_3 and classifier_3 via cross-entropy loss(v_C, yc_C)\n```"}]

Author rebuttal:

Meta-review: This is a borderline paper. The core idea is to learn robust multi-scale representations, select small loss examples as confident examples, and build graphs to correct labels of large-loss examples, handling the issue of noisy labels in time series.

Most reviewers acknowledge the empirical performance of this method. However, reviewers have concerns about its technical contribution. 

AC has carefully checked the content of this paper, all comments from reviewers, and the authors' responses. After consideration, AC recommends accepting this paper. The reasons are as follows:
- The paper skillfully melds diverse techniques like co-training, multi-scale feature learning, graph learning. This combination, driven by a clear rationale and motivation, surpasses the performance of  prevalent baselines. Harmoniously integrating these methods, especially when each brings its own complexity, is not easy.
- It targets a practical and significant problem: reducing the negative impact of time series label errors.  Targeting this issue holds  implications for real-world applications, ensuring more reliable data interpretations and outcomes.

Seen through this lens, the method proposed by this paper represents a novel combination of well-known techniques and contributes a solution to a important problem. With the proposed post-acceptance modifications (clarifying definitions, distinguishing from existing methods, and additional experiments), the paper can be further enhanced, making it a valuable contribution to the conference. 

For the camera-ready version, the authors should integrate the reviewers' feedback and implement the suggested changes. Additionally, for the readers' convenience, it would be beneficial if the related work section paints a broader overview of methods for learning with noisy labels, e.g., also covering semi-supervised-based methods and transition-matrix-based methods.
Decision: Accept (poster)