SlowGuess/NeurIPS_2023_OCR huggingface dataset:
cszhenliu@mail.scut.edu.cn
Peitian Ma
ma_scuter@163.com
Dongliang Chen
ytucdl@foxmail.com
Wenbin Pei
peiwenbin@dlut.edu.cn
Qianli Ma*
qianlima@scut.edu.cn
Deep Neural Networks (DNNs) have been criticized because they easily overfit noisy (incorrect) labels. To improve the robustness of DNNs, existing methods for image data regard samples with small training losses as correctly labeled data (small-loss criterion). Nevertheless, time series' discriminative patterns are easily distorted by external noises (i.e., frequency perturbations) during the recording process. This results in training losses of some time series samples that do not meet the small-loss criterion. Therefore, this paper proposes a deep learning paradigm called Scale-teaching to cope with time series noisy labels. Specifically, we design a fine-to-coarse cross-scale fusion mechanism for learning discriminative patterns by utilizing time series at different scales to train multiple DNNs simultaneously. Meanwhile, each network is trained in a cross-teaching manner by using complementary information from different scales to select small-loss samples as clean labels. For unselected large-loss samples, we introduce multi-scale embedding graph learning via label propagation to correct their labels by using selected clean samples. Experiments on multiple benchmark time series datasets demonstrate the superiority of the proposed Scale-teaching paradigm over state-of-the-art methods in terms of effectiveness and robustness.
Time series classification has recently received much attention in deep learning [1, 2]. Essentially, the success of Deep Neural Networks (DNNs) is driven by a large amount of well-labeled data. However, human errors [3] and sensor failures [4] produce noisy (incorrect) labels in time series datasets. For example, in electrocardiogram diagnosis [5], physicians with different experiences tend to make inconsistent category judgments. In recent studies [6, 7], DNNs have shown their powerful learning ability, which, however, makes it relatively easier to overfit noisy labels and inevitably degenerate
*Qianli Ma is the corresponding author.
(a) PLAID
(b) Lightning7
Figure 1: Illustration of time series samples from the same category at different time scales. Among all samples in the same category, red indicates the one with the largest variance, and blue indicates a few samples with the smallest variance.
(c) ElectricDevices
(d) MedicalImages
the robustness of models. Moreover, time series data has complex temporal dynamics that make it challenging to manually correct noisy labels [8].
To cope with noisy labels, existing studies on label-noise learning [9, 10] use the memory effect of DNNs to select samples with small losses for training. DNNs memorize the data with clean labels first, and then those with noisy labels in classification training (small-loss criterion) [11]. It is worth noting that the small-loss criterion is not affected by the choice of training optimizations and network structures [12], and is widely utilized for label-noise learning in computer vision [13, 14]. However, the small loss criterion cannot always be applied to time series because the discriminative patterns of time series data are easily distorted by external noises [15, 16]. For example, in a smart grid, distortions may occur due to sampling frequency perturbations, imprecise sensors, or random differences in energy consumption [17]. Such distortions can make it difficult for DNNs to learn the appropriate discriminant patterns of time series, resulting in large training losses for some clean labeled samples. In addition, the small-loss criterion only utilizes the data's label information and does not consider the inherent properties of time series features (i.e., multi-scale information).
Multi-scale properties are crucial in time series classification tasks. In recent years, multi-scale convolution [16], dynamic skip connections [18, 19] and adaptive convolution kernel size [20] have been utilized to learn discriminative patterns of time series. Furthermore, according to related studies [2, 20, 21], the selection of appropriate time scales for time series data can facilitate DNNs to learn class-characteristic patterns. With correct labels, the above studies indicate that the multi-scale properties of time series data can help DNNs learn appropriate discriminative patterns for mitigating the negative effects of time series recording noises. Nevertheless, it remains an open challenge as to how the multi-scale properties of time series can be used for label-noise learning.
To this end, we propose a deep learning paradigm, named Scale-teaching, for time-series classification with noisy labels. In particular, we design a fine-to-coarse cross-scale fusion mechanism for obtaining robust time series embeddings in the presence of noisy labels. We select four time series datasets from the UCR archive [22] to explain our motivation. As shown in Figure 1, in the single scale case (top row), the red and blue samples from the same category have large differences in certain local regions (the green rectangle in Figure 1). By downsampling the time series from fine to coarse, some local regions between the red and blue samples did become similar. Meanwhile, existing studies [12, 23] show that multiple DNNs with random initialization have classification divergence for noisy labeled samples, but are consistent for clean labeled samples. The above findings inspire us to utilize multiple DNNs to combine robust embeddings at different scales to deal with noisy labels. Nonetheless, the coarse scale discards many local regions in the fine scale (as in Figure 1 (c)), which may degenerate the classification performance. Hence, we propose the Scale-teaching paradigm, which can better preserve the local discriminative patterns of fine scale while dealing with distortions.
More specifically, the proposed Scale-teaching paradigm performs the cross-scale embedding fusion in the finer-to-coarser direction by utilizing time series at different scales to train multiple DNNs simultaneously. The cross-scale embedding fusion exploits complementary information from different scales to learn discriminative patterns. This enables the learned embeddings to be more robust to distortions and noisy labels. During training, clean labels are selected through cross-teaching on those networks with the learned embeddings. The small-loss samples in training are used as (clean) labeled data, and the unselected large-loss samples are used as (noisy) unlabeled data. Moreover,
multi-scale embedding graph learning is introduced to establish relationships between labeled and unlabeled samples for noisy label correction. Based on the multi-scale embedding graph, the label propagation theory [24] is employed to correct noisy labels. This drives the model to better fit time series category distribution. The contributions are summarized as follows:
- We propose a deep learning paradigm, called Scale-teaching, for time-series label-noise learning. In particular, a cross-scale fusion mechanism is designed to help the model select more reliable clean labels by exploiting complementary information from different scales.
- We further introduce multi-scale embedding graph learning for noisy label correction using the selected clean labels based on the label propagation theory. Unlike conventional image label-noise learning methods focused on sample loss levels, our approach uses well-learned multi-scale time series embeddings for noise label correction at sample feature levels.
- Extensive experiments on multiple benchmark time series datasets show that the proposed Scale-teaching paradigm achieves a state-of-the-art classification performance. In addition, multi-scale analyses and ablation studies indicate that the use of multi-scale information can effectively improve the robustness of Scale-teaching against noisy labels.
Label-noise Learning. Existing label-noise learning studies focus mainly on image data [10]. These studies can be broadly classified into three categories: (1) designing noise-robust objective functions [25, 26] or regularization strategies [27, 28]; (2) detecting and correcting noisy labels [13, 29, 30]; (3) transition-matrix-based [31, 32] and semi-supervised-based [14, 33] methods. In contrast to the methodologies in the first and third categories, approaches categorized under the second category have received considerable attention in recent years [7, 34]. Methods of the second category can be further divided into sample selection and label correction. The common methods of sample selection are the Co-teaching family [12, 13, 23] and FINE [35]. Label correction [36, 37] attempts to correct noisy labels by either using prediction results of classifiers or pseudo-labeling techniques. Recently, SREA [4] utilizes pseudo-labels generated based on a clustering task to correct time-series noisy labels. Although the above methods can improve the robustness of DNNs, how the multi-scale properties of time series are exploited for label-noise learning has not been explored.
Multi-scale Time Series Modeling. In recent years, multi-scale properties have gradually gained attention in various time series downstream tasks [18, 38], such as time series classification, prediction, and anomaly detection [39]. For example, Cui et al. [16] employ multiple convolutional network channels of different scales to learn temporal patterns that facilitate time series classification. Chen et al. [19] design a time-aware multi-scale RNN model for human action prediction. Wang et al. [40] introduce a multi-scale one-class RNN for time series anomaly detection. Also, recent studies [41, 42, 43, 44] indicate that multi-scale properties can effectively improve the performance of long-term time series prediction. Unlike prior work, we utilize multiple DNNs with identical architectures to separately capture discriminative temporal patterns across various scales. This enables us to acquire robust embeddings for handling noisy labels via a cross-scale fusion strategy.
Label Propagation. Label propagation (LP) is a graph-based inductive inference method [24, 45] that can propagate pseudo-labels to unlabeled graph nodes using labeled graph nodes. Since LP can utilize the feature information of data to obtain pseudo-labels of unlabeled samples, related works employ LP in few-shot learning [46] and semi-supervised learning [47, 48]. Generally speaking, DNNs have the powerful capability for feature extraction, and the learned embeddings tend to be similar within classes and different between classes. Each sample contains feature and label information. Intuitively, the embeddings of samples with noisy labels obtained by DNNs closely align with the true class distribution when the DNNs do not fit noisy labels in the early training stages. Naturally, we create a nearest-neighbor graph based on well-learned multi-scale time series embeddings at the feature level. Subsequently, we employ LP theory to correct the labels of unselected noisy samples using the labels of clean samples chosen by the DNNs. This approach leverages robust multi-scale embeddings to address the issue of noisy labels.
Figure 2: The Scale-teaching paradigm's general architecture comprises two core processes: (i) clean label selection and (ii) noisy label correction. In the clean label selection phase, networks A, B, and C engage in cross-scale fusion, moving from fine to coarse  $(\mathrm{A}\rightarrow \mathrm{B},\mathrm{B}\rightarrow \mathrm{C})$ . They employ clean labels acquired through cross-teaching  $(\mathrm{A}\rightarrow \mathrm{B},\mathrm{B}\rightarrow \mathrm{C},\mathrm{C}\rightarrow \mathrm{A})$  to guide their respective classification training. In the noisy label correction phase, pseudo labels derived from multi-scale embeddings graph learning are employed as corrected labels for time series not selected as clean labeled samples.
Given a noisy labeled time series dataset  $\mathcal{D} = \{(\mathcal{X}_i, \widehat{y}_i)\}_{i=1}^N$ , it contains  $N$  time series, where  $\mathcal{X}_i \in R^{L \times T}$ ,  $L$  denotes the number of variables, and  $T$  is the length of variable.  $\widehat{y}_i \in \{1, \dots, C\}$  is the observed label of  $\mathcal{X}_i$  with  $\eta$  probability of being a noisy label. Our goal is to enable the DNNs trained on the noisy labeled training dataset  $\mathcal{D}_{train}$  to correctly predict the ground-truth labels of the given time series in the test set. Specifically, the problem to be addressed in this paper consists of two steps. The first is to select clean labeled time series from  $\mathcal{D}_{train}$ , and the second is to perform noisy label correction for time series in  $\mathcal{D}_{train}$  that have not been selected as clean labels.
The overall architecture of Scale-teaching is shown in Figure 2. While this figure illustrates Scale-teaching with input time series at three scales, it can be extended to models with more scales, exceeding three. We utilize a consistent structural encoder to learn embeddings for each input scale sequence. Each encoder undergoes training at two levels: embedding learning for clean sample selection at the feature level and label correction with the multi-scale embeddings. For embedding learning, we propose a cross-scale fusion (Section 3.3) mechanism from fine to coarse to obtain robust embeddings. This approach enables the selection of more dependable clean labels through the small-loss criterion. Specifically, embeddings  $(\mathrm{A} \odot \mathrm{B} \odot \mathrm{C})$  encompass multi-scale information from fine, medium, and coarse scale sequences derived from the same time series. Regarding noisy label correction, we introduce multi-scale embedding graph learning (Section 3.4) based on label propagation, utilizing the selected clean samples to correct the labels of unselected large-loss samples.
3.3 Cross-scale Fusion for Clean Label Selection
After downsampling the original time series at different scales, it eliminates some of the differences in local regions between samples of the same category (as in Figure 1). However, the downsampled sequences (i.e., coarse scale) discard many local regions of the original time series. This tends to degrade the model's classification performance if the downsampled sequences are used directly for classification (please refer to Table 2 in the Experiments section). Meanwhile, existing studies [12, 23] on label-noise learning show that DNNs with different random initializations have high consistency in classification results for clean labeled samples in the early training period, while there is disagreement in the classification of noisy labeled samples. Based on the above findings, we utilize multiple DNNs (or encoders) with different random initializations to learn embeddings of different downsampled scale sequences separately, and perform cross-scale fusion. On the one hand, we exploit complementary
information between adjacent scale embeddings to promote learned embeddings to be more robust for classification. On the other hand, we leverage the divergence in the classification of noisy labeled samples by different DNNs to mitigate the negative impact of noisy labels in training. In this way, we can utilize the cross-scale fusion embeddings for classification, thus better using the small loss criterion [11, 29] for clean label selection. Specifically, downsampling is employed to generate different scale sequences from the same time series. Given a time series  $\mathcal{X}_i = \{x_1, x_2, \ldots, x_T\}$ , supposing the downsampling ratio is  $k$ . Then, we only keep data points in  $\mathcal{X}_i$  as follows:
$$
\mathcal {X} _ {i} ^ {k} = \left\{x _ {k * j} \right\}, j = 1, 2, \dots , \frac {T}{k}, \tag {1}
$$
where  $k \in [1, T/2]$ , and a larger  $k$  indicates that  $\mathcal{X}_i^k$  is coarser. As shown in Figure 2, time series with multiple downsampling intervals (i.e.,  $k = 1, 2, 4$ ) is treated as the input data for training. To better utilize the small-loss criterion for clean label selection, each time series sample performs cross-scale fusion from fine to coarse (i.e.,  $\mathrm{A} \to \mathrm{B}$ ,  $\mathrm{B} \to \mathrm{C}$ ) in the embedding space, which is mathematically defined as:
$$
v _ {i} ^ {k} = f \left(r _ {i} ^ {k} \| v _ {i} ^ {k - t} \| \left(r _ {i} ^ {k} - v _ {i} ^ {k - t}\right) \| \left(r _ {i} ^ {k} \cdot v _ {i} ^ {k - t}\right)\right), \tag {2}
$$
where  $r_i^k$  represents the single-scale embedding acquired by learning  $\mathcal{X}_i^k$  through an encoder. Meanwhile,  $v_i^k$  (or  $v_i^{k - t}$ ) denotes the embedding of the time series  $X_{i}^{k}$  (or  $X_{i}^{k - t}$ ) after performing cross-scale fusion. Here,  $t$  denotes the interval between adjacent downsampling ratios, and  $\| \cdot \|$  signifies the concatenation of two vectors to form a new vector. Notably, when  $k = 1$ , we employ the single-scale for classification training, resulting in  $v_{i}^{k} = r_{i}^{k}$ . By combining  $(r_i^k -v_i^{k - t})$  and  $(r_i^k\cdot v_i^{k - t})$  for vector concatenation,  $v_{i}^{k}$  can capture more nuanced discriminative information between  $r_i^k$  and  $v_{i}^{k - t}$  than that of simply concatenating  $r_i^k$  with  $v_{i}^{k - t}$ . The function  $f(\cdot)$  represents a two-layer nonlinear network mapping function for fusing information of  $r_i^k$  and  $v_{i}^{k - t}$ . Additionally,  $v_{i}^{k}$  has the same dimension as  $r_i^k$  and serves as the input data for the multi-scale embedding graph learning process.
3.4 Multi-scale Embedding Graph Learning for Noisy Label Correction
We now present the multi-scale embedding graph learning module for correcting noisy labels. This module incorporates selected clean labels using label propagation theory. The process consists of two stages: graph construction and noisy label correction.
Graph Construction. It is assumed that the set of cross-fusion embeddings obtained from a batch of time series is defined as  $V = \{v_1^k, v_2^k, \ldots, v_M^k\}$ , where  $M$  is the batch size. Intuitively, samples close to each other in the feature space have a high probability of belonging to the same class. However, in label-noise learning,  $v_i^k$  obtained from the current iterative training of the model may have unstable information, resulting in large deviations in the information of the nearest-neighbor samples of  $v_i^k$ . To address this issue, the proposed approach performs a momentum update [49] on  $v_i^k$  during training, which is defined as:
$$
\bar {v} _ {i} ^ {k} [ e ] = \alpha v _ {i} ^ {k} [ e ] + (1 - \alpha) \bar {v} _ {i} ^ {k} [ e - 1 ], \tag {3}
$$
where  $e$  is the current training epoch and  $\alpha$  denotes the momentum update parameter.
The multi-scale embeddings nearest-neighbor graph can be created by using Euclidean distance among different  $\overline{v}_i^k$ . A common approach is the use of the Gaussian similarity function [45] to obtain the nearest-neighbor graph edge weight, which is defined as:
$$
W _ {i j} = \exp \left(- \frac {1}{2} d \left(\frac {\bar {v} _ {i} ^ {k}}{\sigma}, \frac {\bar {v} _ {j} ^ {k}}{\sigma}\right)\right), \tag {4}
$$
where  $d(\cdot)$  is the Euclidean distance function and  $\sigma$  is a fixed parameter.  $W \in R^{M \times M}$  is a symmetric adjacency matrix, and the element  $W_{ij}$  denotes the nearest-neighbor edge weight between the embedding  $v_i^k$  and  $v_j^k$  (note that larger values indicate closer proximity). Then,  $W$  is normalized based on the graph Laplacians [50] to obtain  $Q = D^{-1/2}WD^{-1/2}$ , where  $D = \text{diag}(W1_n)$  is a diagonal matrix. Specifically, the  $K$  neighbors with the largest values in each row of  $Q$  are employed to create the nearest-neighbor graph. It is noteworthy that the embeddings in each mini-batch are utilized to generate the nearest-neighbor graph, thus obtaining  $Q$  within short computational time.
Noisy Label Correction. Specifically, small training loss samples acquired by DNNs in the early training period can be considered as clean samples, while samples with large training losses are considered as noisy ones [12, 14]. The above learning pattern of DNNs has been mathematically validated [11] (see Appendix A for details). Under this criterion, prior studies [12, 13, 23] have typically employed samples with small losses after a  $e_{warm}$  warm-up training as clean labels. Following [29], we extend the small-loss sample selection process to operate within each class, thereby enhancing the overall quality of the chosen clean labels. In our method, samples chosen with clean labels are considered labeled data, whereas unselected samples are treated as unlabeled data.
We utilize clean samples selected from time series at different scales in a cross-teaching manner (as in Figure 2). This could explore complementary information from different scale fusion embeddings to deal with noisy labels. It is supposed that there is a corresponding one-hot encoding matrix  $Y \in R^{M \times C}$  ( $Y_{ij} \in \{0,1\}$ ) for the cross-fusion embeddings  $V$ . If  $y_i$  is identified as a clean label, we employ  $y_i$  to set  $Y_i$  as a one-hot encoded label. Otherwise, all the elements in  $Y_i$  are identified as zero. Through  $Y$ , the pseudo-label of each node in the nearest-neighbor graph  $Q$  can be obtained in an iterative way based on the label propagation theory. The specific solution formula is defined as:
$$
F _ {t + 1} = \beta Q F _ {t} + (1 - \beta) Y, \tag {5}
$$
where  $F_{t} \in R^{M \times C}$  denotes the predicted pseudo-label of the  $t$ -th iteration and  $\beta \in (0,1)$  is a hyperparameter. Naturally,  $F_{t}$  has a closed-form solution [24] defined as follows:
$$
\mathcal {F} = (I - \beta Q) ^ {- 1} Y, \tag {6}
$$
where  $\mathcal{F} \in R^{M \times C}$  is the final pseudo-labels and  $I$  denotes the identity matrix. Finally, the corrected label obtained for an unselected large-loss sample  $X_{i}$  is defined as:
$$
y _ {i} = \arg \max  _ {c} \mathcal {F} _ {i} ^ {c}, \tag {7}
$$
However,  $\mathcal{F}$  is the estimated pseudo-labels, which inevitably contain some incorrect labels. To address this issue, two strategies are used to improve the quality of pseudo-labels in  $\mathcal{F}$ . For the first strategy, the model continues training  $e_{update}$  epochs by using small-loss samples after  $e_{warm}$  epochs warm-up training to improve the robustness of the multi-scale embeddings. Then, the noisy label correction is performed after  $(e_{warm} + e_{update})$  epoch. For the second strategy, a dynamic threshold  $\varphi_e(c) = \frac{\delta_e(c)}{\max(\delta_e)}\gamma$  is utilized for each class [51] to select the pseudo-labels with a high confidence for noisy label correction, where  $\delta_e(c)$  is the number of labeled samples contained in class  $c$  in the  $e$ -th epoch, and  $\gamma$  is a constant threshold.
Overall Training. Finally, each encoder utilizes the selected clean samples in combination with multi-scale embedding graph learning to perform noisy label correction for unselected large-loss samples. Combining the training data of the selected clean labels and those of corrected labels, the proposed Scale-teaching paradigm utilizes cross-entropy for time-series label-noise learning. Please refer to Algorithm 1 in the Appendix for the specific pseudo-code of Scale-teaching.