SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Datasets. We use three time series benchmarks (four individual large datasets [3, 52, 53], UCR 128 archive [22], and UEA 30 archive [54]) for experiments. Among the four individual large datasets, HAR [52] and UniMiB-SHAR [3] are human activity recognition scenarios; FD-A [53] is the mechanical fault diagnosis scenario; Sleep-EDF [52] belongs to the sleep stage classification scenario. The UCR archive [22] contains 128 univariate time series datasets from different real-world scenarios. The UEA archive [54] contains 30 multivariate time series datasets from real-world scenarios. For details on the above datasets, please refer to Appendix B. Since all the datasets in three time series benchmarks are correctly labeled, we utilize a label transformation matrix  $T$  to add noises to the original correct labels [4], where  $T_{ij}$  is the probability of label  $i$  being flipped to  $j$ . We use three types of noisy labels for evaluations, namely Symmetric (Sym) noise, Asymmetric (Asym) noise, and Instance-dependent (Ins) noise. Symmetric (Asymmetric) noise randomly replaces a true label with other labels with an equal (unequal) probability. Instance noise [55] means that the noisy label is instance-dependent. Like [4, 12, 23], we use the test set with correct labels for evaluations.
Table 1: Test classification accuracy results compared with baselines on three time series benchmarks. The best results are bold, and the second best results are underlined. When P-value  $< 0.05$ , it indicates that the performance of Scale-teaching is statistically significant than the baseline.
Dataset	Noise Ratio	Metric	Standard	Mixup	Co-teaching	FINE	SREA	SELC	CULCU	Scale-teaching
Four individual large datasets	Sym 20%	Avg Rank	4.75	4.75	4.50	7.50	6.50	4.50	2.50	1.00
Sym 50%	Avg Rank	4.75	4.50	4.75	7.25	5.75	4.50	3.25	1.25
Asym 40%	Avg Rank	5.00	5.50	3.75	7.50	5.75	4.00	3.25	1.00
Ins 40%	Avg Rank	4.75	4.25	4.25	7.25	6.00	4.75	3.50	1.00
UCR 128 archive	Sym 20%	Avg Rank	4.15	4.33	3.61	7.50	6.16	3.48	3.54	3.02
P-value	1.90E-04	4.06E-05	1.90E-03	1.49E-34	1.70E-17	3.04E-03	8.57E-03	-
Sym 50%	Avg Rank	4.31	4.57	4.05	6.43	5.89	3.56	3.86	3.11
P-value	3.15E-05	1.70E-05	4.02E-04	7.48E-19	1.22E-15	1.40E-02	4.93E-03	-
Asym 40%	Avg Rank	4.38	4.80	3.93	6.91	5.91	3.30	3.67	2.95
P-value	1.62E-05	3.53E-07	6.10E-04	1.93E-23	9.82E-14	1.89E-02	2.24E-02	-
Ins 40%	Avg Rank	4.05	4.52	4.02	7.04	6.18	3.30	3.77	2.95
P-value	1.43E-05	1.81E-06	2.43E-04	9.81E-26	2.36E-17	3.27E-02	1.54E-02	-
UEA 30 archive	Sym 20%	Avg Rank	5.03	5.20	3.83	6.37	4.77	3.73	4.00	2.73
P-value	6.61E-04	3.33E-04	2.69E-02	2.37E-05	1.14E-02	2.63E-02	3.93E-02	-
Sym 50%	Avg Rank	5.17	5.73	4.23	6.23	3.93	3.83	4.30	2.43
P-value	2.98E-04	7.40E-05	1.59E-02	9.35E-05	1.67E-02	1.08E-02	3.75E-02	-
Asym 40%	Avg Rank	5.60	4.77	4.40	6.13	4.20	4.00	3.97	2.73
P-value	3.81E-03	6.17E-03	1.63E-02	9.33E-05	1.36E-02	2.62E-02	3.88E-02	-
Ins 40%	Avg Rank	5.20	4.77	4.33	6.60	4.27	4.20	3.77	2.60
P-value	6.08E-04	2.92E-03	1.20E-02	2.55E-05	5.52E-03	1.08E-02	3.47E-02	-
Baselines. We select seven methods for comparative analyses, namely 1) Standard: direct training of the model using cross-entropy with all noisy labels; 2) Mixup [56]; 3) Co-teaching [12]; 4) FINE [35]; 5) SREA [4]; 6) SELC [37]; and 7) CULCU [23]. Among them, Standard, Mixup, and Co-teaching are the benchmark methods for label-noise learning. FINE, SELC, and CULCU are the state-of-the-art methods that do not need to focus on data types, and SREA is the state-of-the-art method in time series domain. In addition, for fair comparisons, all the baselines and the proposed Scale-teaching paradigm use the same encoder and classifier. We focus on the ability of different label-noise learning paradigms to cope with time series noise labels, rather than the classification performance achieved by using fully correct labels. Hence, considering the trade-off between the running time and classification performance, we choose FCN [57] as the encoder of Scale-teaching. For more details of baselines, please refer to Appendix C.
Implementation Details. Based on the experience [19, 44] in time series modeling, we utilize three different sampling intervals 1, 2, 4 as the input multi-scale series data for Scale-teaching. We use Adam as the optimizer. The learning rate is set to 1e-3, the maximum batch size is set to 256, and the maximum epoch is set to 200.  $e_{warm}$  is set to 30 and  $e_{update}$  is set to 90.  $\alpha$  in Eq. 3 is set to 0.9,  $\sigma$  in Eq. 4 is set to 0.25,  $\beta$  in Eq. 5 is set to 0.99, the largest neighbor  $K$  is set to 10, and  $\gamma$  is set to 0.99. In addition, following the parameter settings suggested in [23], we linearly decay the learning rate to zero from the 80-th epoch to 200-th epoch. For a comprehensive understanding of the hyperparameter selection and the implementation of the small-loss criterion applied to Scale-teaching, please consult Appendix C. To reduce random errors, we utilize the mean test classification accuracy of the last five epochs of the model on the test set as experimental results. All the experiments are independently conducted five times with five different seeds, and the average classification accuracy and rank are reported. Finally, we build our model using PyTorch 1.10 platform with 2 NVIDIA GeForce RTX 3090 GPUs. Our implementation of Scale-teaching is available at https://github.com/qianlima-lab/Scale-teaching.
We evaluate each time series benchmark using four noise ratios, Sym  $20\%$ , Sym  $50\%$ , Asym  $40\%$ , and Ins  $40\%$ . Due to space constraints, we only give the average ranking of all the methods on each benchmark in Table 1. Please refer to Appendix D for the specific test classification accuracies. Besides, for UCR 128 and UEA 30 archives, we use the Wilcoxon signed rank test (P-value) [58] to analyze the classification performance of baselines. As shown in Table 1, the proposed Scale-teaching paradigm achieves the best Avg Rank in all the cases. It is found that Mixup [56] and FINE [35] perform worse than the Standard method in most cases. For Mixup, the complex dynamic properties
Figure 3: Venn diagram of the average number of correctly classified samples for the different scale sequences of UCR 128 archive with Sym  $20\%$  noisy labels. The numbers in the figure indicate the complements and intersections of classification results at different scales.
Table 2: The test classification accuracy (%) results of different scale classifiers on UCR 128 archive. The best results are **bold**, and the second best results are **underlined**. When P-value < 0.05, it indicates that the performance of Scale-teaching's coarse scale classifier is significant than other classifiers.
Method	w/o Cross-scale fusion	Scale-teaching
Noise Ratio	Metric	Fine	Medium	Coarse	Fine	Medium	Coarse
Sym 20%	Avg Acc	65.13	30.11	28.17	59.67	68.17	68.70
Avg Rank	2.38	5.09	5.37	3.20	2.17	2.11
P-value	1.89E-03	2.85E-37	2.07E-40	1.58E-09	3.74E-02	-
Asym 40%	Avg Acc	49.61	29.01	28.87	47.75	51.93	52.87
Avg Rank	2.64	4.78	4.75	3.01	2.45	2.27
P-value	1.94E-03	6.78E-25	1.59E-27	1.80E-07	2.80E-02	-
of the original time series are destroyed probably due to the mixture of two different time series mechanisms. FINE uses embeddings of the input data to select clean labels. Although FINE achieves advanced classification performance for image data, it is difficult to be used directly for time series data because its discriminative patterns are easily distorted by external noises. SREA [4] has a good performance on the UEA 30 archive, while it performs poorly on the other benchmarks. Meanwhile, Co-teaching [12], SELC [37], and CULCU [23] are more robust against time series noisy labels in different cases, further indicating that the small-loss criterion is also applicable to time series.
4.3 Multi-scale Analysis
To explain the multi-scale mechanism in the Scale-teaching paradigm, we add an ablation study based on Scale-teaching (w/o cross-scale fusion). We select the UCR 128 archive to analyze the classification results obtained by the fine, medium, and coarse scale classifiers. As shown in Figure 3, the classification results of different scale sequences have evident complementary information. Scale-teaching can effectively use complementary information between cross-scale to obtain more robust embeddings and clean labels. In response to the tendency of the coarse scale to ignore discriminative patterns in fine scale (please see Table 2), our proposed cross-scale fusion mechanism can effectively improve the classification performance of medium and coarse scales while retaining complementarity. Please refer to Appendix E for the specific classification results of Figure 3 and Table 2. In Appendix E, we also analyze the order and size of the downsampled input scale sequence for Scale-teaching.
Scale-teaching utilizes multi-scale embeddings to generate the nearest-neighbor graph, and uses clean labels selected for noisy label correction. To explore the distribution of different classes of embeddings, we employ t-SNE [59] for dimensionality reduction visualization. Specifically, we utilize the UniMiB-SHAR dataset containing Sym  $20\%$  noisy labels for visualization. As shown in Figure 4, we find that the embeddings learned by Scale-teaching are more discriminative across classes than the Standard and CULCU methods that use a single scale series for training. The above results suggest that Scale-teaching can effectively exploit the complementary information between different scales, prompting the learned embeddings to be more discriminative between classes. In addition, we choose the FD-A dataset for t-SNE visualization, and please refer to Appendix E.
(a) Standard  $(86.43\%)$
(b) CULCU  $(83.39\%)$
(c) Scale-teaching  $(91.32\%)$
Figure 4: t-SNE visualization of the learned embeddings on the UnimiB-SHAR dataset with Sym  $20\%$  noisy labels (values in parentheses are the test classification accuracies).
(a) NonInvasiveFetalECGThorax1
Figure 5: The change of loss values for clean and noisy time series samples under Aysm  $40\%$  noise labels. The solid line and shading indicate the mean and standard deviation loss values of all clean (or noisy) training samples within each epoch.
(b) ECG5000
4.4 Small-loss Analysis
To analyze the application of the small-loss criterion to time series data, we visualize the change of loss values for ground-truth clean and noisy time series samples during training. Specifically, Figure 5 shows the change in loss values of the models trained by the Standard method and Scale-teaching on two UCR time series datasets. When the model is trained with the Standard method, differences can be found in the loss values of clean and noisy samples in the network early training, especially in Figure 5 (b). The Standard method makes the model gradually fit the noisy samples as the training proceeds, while Scale-teaching improves the ability of the model to handle noisy labels. To further prove its effectiveness, we selected two other UCR datasets for the loss value change analysis, which have the same pattern as Figure 5. Also, we report the HAR and UniMiB-SHAR dataset's loss value probability distributions of clean and noisy samples. For more details, please refer to Appendix F.
To verify the robustness of each module in Scale-teaching, the ablation experiments have been conducted in the HAR and UniMiB-SHAR datasets, and the results are shown in Table 3. Specifically, (1) w/o cross-scale fusion: the cross-scale embedding fusion from fine to coarse mechanism is ablated; (2) only single scale: only the original time series is used for training; (3) w/o graph learning: the multi-scale embedding graph learning module for noisy label correction is ablated; (4) w/o moment: the embedding momentum update mechanism (Eq. 3) is ablated; (5) w/o dynamic threshold: using a dynamic threshold to select high-quality propagation pseudo-labels is ablated.
As shown in Table 3, the cross-scale fusion strategy (w/o cross-scale fusion) and the clean labels cross-teaching mechanism (only single scale) can effectively improve the classification performance of Scale-teaching, especially on the UniMiB-SHAR dataset with a large number of classes. Meanwhile, in terms of label correction based on multi-scale embedding graph learning, the results of the corresponding ablation module show that improving the stability of embedding (w/o moment) and
Table 3: The test classification accuracy (%) results of ablation study (values in parentheses denote drop accuracy).
Method	HAR	UniMiB-SHAR
Sym 50%	Asym 40%	Sym 50%	Asym 40%
Scale-teaching	90.17	89.62	81.31	70.68
w/o cross-scale fusion	88.47 (-1.70)	87.64 (-1.98)	73.32 (-7.99)	61.62 (-9.06)
only single scale	89.01 (-1.06)	88.11 (-1.51)	69.89 (-11.42)	60.32 (-10.36)
w/o graph learning	88.06 (-2.11)	87.65 (-1.97)	79.72 (-1.59)	68.87 (-1.81)
w/o moment	89.76 (-0.41)	88.76 (-0.86)	80.57 (-0.74)	69.85 (-0.83)
w/o dynamic threshold	89.12 (-1.05)	88.75 (-0.87)	77.42 (-3.89)	69.53 (-1.15)
Table 4: The test classification accuracy (%) results on four individual large datasets without noisy labels. The best results are bold, and the second best results are underlined.
Dataset	Standard	Mixup	Co-teaching	FINE	SREA	SELC	CULCU	Scale-teaching
HAR	93.29	95.42	93.77	93.13	93.02	93.76	94.75	94.72
UniMiB-SHAR	89.14	84.84	88.24	88.14	65.51	89.28	89.46	93.61
FD-A	99.93	99.91	99.96	68.22	90.25	99.82	99.95	99.96
Sleep-EDF	84.93	84.67	85.37	84.62	79.42	84.82	85.54	85.34
selecting high-quality pseudo-labels (w/o dynamic threshold) can effectively improve the performance of label correction based on graph learning.
Furthermore, we select the four individual large datasets without noisy labels for evaluation. As shown in Table 4, Scale-teaching's classification performance is still better than most baselines. It's worth mentioning that SREA [4] employs an unsupervised time series reconstruction loss as an auxiliary task, which reduces the model's classification performance without noisy labels. We also provide the corresponding test classification results for Tables 3 and 4 under the F1-score metric in Appendix G. Additionally, we find the running time of Scale-teaching, which is faster than FINE, SREA and CULCU for datasets with a larger number of samples or longer length of the sequence. We further analyze the classification performance of the proposed Scale-teaching paradigm and time series classification methods [15, 60] in Appendix G.
Limitations. The input scales of our proposed Scale-teaching paradigm can only select a fixed number of scales for training, and the running time will increase as the number of scales increases.
Conclusion. In this paper, we propose a deep learning paradigm for time-series classification with noisy labels called Scale-teaching. Experiments on the three time series benchmarks show that the Scale-teaching paradigm can utilize the multi-scale properties of time series to effectively handle noisy labels. Comprehensive analyses on multi-scale and ablation studies demonstrate the robustness of the Scale-teaching paradigm. In the future, we will explore the design of scale-adaptive time-series label-noise learning models.
We thank the anonymous reviewers for their helpful feedbacks. We thank Professor Eamonn Keogh and all the people who have contributed to the UCR 128 archive, UEA 30 archive, and the four large individual time series classification datasets. The work described in this paper was partially funded by the National Natural Science Foundation of China (Grant Nos. 62272173, 61872148, 62206041), the Natural Science Foundation of Guangdong Province (Grant Nos. 2022A1515010179, 2019A1515010768), the Science and Technology Planning Project of Guangdong Province (Grant No. 2023A0505050106), the Fundamental Research Funds for the Central Universities under grants DUT22RC(3)015. The authors would like to thank Siying Zhu, Huawen Feng, Yu Chen, and Junlong Liu from SCUT for their review and helpful suggestions.
Supplementary Material: Scale-teaching: Robust Multi-scale Training for Time Series Classification with Noisy Labels
A Small-loss Criterion
DNNs have been widely known to first learn simple and generalized patterns, which is achieved by learning clean data. After that, the networks gradually overfit noisy ones. In other words, when we train a model with a dataset containing incorrectly labeled samples, we can consider the samples with small training losses as clean ones and use them to update the model. Formally, let  $f^{*}$  be the target concept which determines the true label of  $x$  and model  $g^{*} = g(\boldsymbol{x};\Theta^{*})$  minimizing the expected loss, i.e.,
$$
\Theta^ {*} = \underset {\Theta} {\arg \min } \mathbb {E} _ {(\boldsymbol {x}, \tilde {\boldsymbol {y}})} \left[ \ell_ {C E} (g (\boldsymbol {x}; \Theta), \tilde {\boldsymbol {y}}) \right]. \tag {8}
$$
Then, the small-loss criterion can be stated as follows[1]:
Theorem 1. Suppose  $g$  is  $\epsilon$ -close to  $g^*$ , i.e.,  $\| g - g^* \|_{\infty} = \epsilon$ , for two examples  $(\pmb{x}_1, \tilde{y})$  and  $(\pmb{x}_2, \tilde{y})$ , assume  $f^*(\pmb{x}_1) = \tilde{y}$  and  $f^*(\pmb{x}_2) \neq \tilde{y}$ , if  $T$  satisfies the diagonally-dominant condition  $T_{ii} > \max \{ \max_{j \neq i} T_{ij}, \max_{j \neq i} T_{ji} \}$ ,  $\forall i$ , and  $\epsilon < \frac{1}{2} \cdot (T_{\tilde{y}\tilde{y}} - T_{f^*(\pmb{x}_2)\tilde{y}})$ , then  $\ell_{CE} \left( g(\pmb{x}_1), \tilde{y} \right) < \ell_{CE} \left( g(\pmb{x}_2), \tilde{y} \right)$ .
The work [11] provides the proof of this theorem. It shows that during training, the model can select clean samples according to the loss values. The reason is that the loss values of clean samples among the samples with the same observed labels are smaller. It is worth noting that the theorem is under the assumption of the class-dependent noise type and requires the transition matrix to satisfy the diagonally-dominant condition. Additionally, the finite data may also make the conditions of the theorem difficult to hold because the model  $g$  may be far away from  $g^{*}$ .
To evaluate the robustness of our proposed Scale-teaching and baselines on the time-series label-noise learning task, we selected three benchmark time-series datasets for experimental analysis.
B.1 Four individual large datasets
The statistical information of the four individual time series datasets is shown in Table 5. And the specific dataset information is as follows:
Human Activity Recognition (HAR)
The HAR dataset [52, 61] is collected from 30 students performing six human actions (i.e., walking, walking upstairs, downstairs, standing, sitting, and lying down) by wearing sensors.
University of Milano Bicocca Smartphone-based Human Activity Recognition (UniMiB SHAR)
The UniMiB SHAR dataset [3, 62] is human activity information collected at a sampling rate of 50 Hz from volunteers with a smartphone with an accelerometer sensor in the front pocket of their pants. Specifically, each accelerometer entry is labeled by specifying the type of ADL (e.g., walking, sitting, or standing) or the type of fall (e.g., forward, fainting, or backward).
Faulty Detection Condition A (FD-A)
The FD-A dataset [52, 63] is generated by an electromechanical drive system that monitors the condition of rolling bearings and detects their failure. Each rolling bearing can be classified into three categories: undamaged, inner damaged, and externally damaged.
Sleep Stage EEG Signal Classification (Sleep-EDF)
The Sleep-EDF dataset [52, 64] includes the whole night PSG sleep recordings, which contain five EEG sleep signal recordings: Wake (W), Non-rapid eye movement (N1, N2, N3), and Rapid Eye Movement (REM).
Table 5: A summary of four individual large time series datasets used in the experiments.
Dataset	# Train	# Test	Length	# Variables	# Classes
HAR	7352	2947	128	9	6
Sleep-EDF	25612	8910	3000	1	5
FD-A	8184	2728	5120	1	3
UniMiB-SHAR	9416	2354	453	1	17
B.2 UCR 128 Archive
The UCR time series archive [22] contains 128 univariate datasets and is widely used for classification in the time series mining community. Each UCR dataset includes a single training set and a single test set, and each time series sample has been z-normalized. In addition, we uniformly use the mean-imputation method to preprocess the datasets that contain missing values. For detailed information about UCR datasets, please refer to https://www.cs.ucr.edu/~eamonn/time_series_data_2018/.
B.3 UEA 30 Archive
The UEA time series archive [54] contains 30 multivariate datasets, mainly derived from Human Activity Recognition, Motion classification, ECG classification, EEG/MEG classification, Audio Spectra Classification, and other realistic scenarios. Each dataset contains a partitioned training set and a test set. In addition, we use the mean-imputation method to deal with datasets with missing values. For detailed information about UEA datasets, please refer to https://www.timeseriesclassification.com/dataset.php.
To analyze the performance and effectiveness of Scale-teaching on time-series label-noise learning, we selected seven baselines for comparative analysis. The specific information is as follows.
- Standard directly employs all samples in the training set containing noisy labels and performs supervised classification training using cross-entropy loss. Then, the trained model is used to make predictions on the test set.
- Mixup [56] trains a neural network on convex combinations of pairs of time series samples and their labels (whatever is clean or noisy). For the specific open source code, please refer to https://github.com/facebookresearch/mixup-cifar10.
- Co-teaching [12] trains two deep neural networks simultaneously, and lets them teach each other given every mini-batch with selected clean labels based on a small-loss criterion. For the specific open source code, please refer to https://github.com/bhanML/Co-teaching.
- FINE [35] utilizes a novel detector for clean label selection. Especially, FINE focus on each data point's latent representation dynamics and measures the alignment between the latent distribution and each representation using the eigen decomposition of the data gram matrix. For the specific open source code, please refer to https://github.com/Kthyeon/FINE_ official.
- SREA [4] employs a novel multi-task deep learning approach for time series noisy label correction that jointly trains a classifier and an autoencoder with a shared embedding representation. For the specific open source code, please refer to https://github.com/Caste144/SREA.
- SELC [37] utilizes a simple and effective method self-ensemble label correction (SELC) to progressively correct noisy labels and refine the model. For the specific open source code, please refer to https://github.com/MacLLL/SELC.
- CULCU [23] incorporates the uncertainty of losses by adopting interval estimation instead of point estimation of losses to select clean labels based on Co-teaching. CULCU has two
versions: CNLCU-S and CNLCU-H, where CNLCU-S uses soft labels for training and CNLCU-H uses hard labels for training. According to the original paper's [23] experimental results, CNLCU-S has a better performance. Hence, we use CNLCU-S as a baseline. For the specific open source code, please refer to https://github.com/xiaoboxia/CNLCU.
Finally, based on the source code of the above baselines, we provide the reproduction source code of all baselines, as well as the source code of our proposed Scale-teaching (refer to Algorithm 1). For the specific open-source code, please refer to our GitHub repository https://github.com/qianlima-lab/Scale-teaching.