# Cras: Curriculum Regularization and Adaptive Semi-Supervised Learning with Noisy Labels
## 2 Related Work
### Semi-Supervised Learning with Noisy Labels  
SSL techniques are widely adopted in the field of LNL. ELR is a method that adopts a technique known as temporal ensemble [47] in SSL. ELR exploits the early-learning phenomenon [24], whereby networks predict true classes during the early stages of learning, even when trained on datasets that contain noisy labels, to set target probabilities. Here, we consider a classification problem in which \(C\) is the number of classes. Given \(N\) samples, \(\{\mathbf{x}_{i},\mathbf{y}_{i}\}_{i=1}^{N}\), \(x_{i}\in\mathbb{R}^{d}\) represents the \(i\)th input sample (with dimension \(d\)), and \(\mathbf{y}_{i}\in\{0,1\}^{C}\) represents the one-hot label that corresponds to \(x_{i}\). In this case, by passing the output of DNN for input \(x_{i}\) through a softmax layer and obtaining the network's predicted probability \(\mathbf{p}_{i}\), the loss function commonly used for training classification networks is expressed as follows. \[\mathcal{L}_{CE}=\frac{1}{N}\sum_{i=1}^{N}H(\mathbf{y}_{i},\mathbf{p}_{i}), \tag{4}\]  
where \(H(\cdot)\) denotes cross-entropy. When the network is trained on a dataset that contains noisy labels, (4) inverts the gradient because element \(y_{i,c}\) of the label for the true class \(c\) iszero. As a result, the network remembers noisy labels, leading to a poor generalization performance. ELR takes advantage of the early-learning phenomenon and adds a regularization term that maximizes the inner product of the network output and the target, under the assumption that the target remains unaffected by overfitting to noisy labels. \[\mathcal{L}_{ELR}=\mathcal{L}_{CE}+\frac{\lambda}{N}\sum_{i=1}^{N}\log\left(1- \langle\mathbf{p}_{i},\mathbf{t}_{i}\rangle\right), \tag{5}\]  
where \(\mathbf{t}_{i}\) denotes the target probability set for \(\mathbf{x}_{i}\). The gradient on the regularization term of the ELR cancels out the gradient of the cross-entropy on the noisy sample. This mitigates its effect and implicitly suppresses overfitting to the noisy labels. The number of samples in the temporal ensemble determines the target probability. The following equation sets the target probability, using the moving average as the temporal ensemble method. \[\mathbf{t}_{i}^{(k)}=\beta\mathbf{t}_{i}^{(k-1)}+(1-\beta)\mathbf{p}_{i}^{(k)}, \tag{6}\]  
where \(0\leq\beta\leq 1\) denotes the momentum. Combining the following two methods yields more refined targets than ELR alone. One approach is to use two different networks and estimate the target for one network from the output of the other network, as seen in the Co-teaching and related methods [22; 23; 25]. Another approach estimates the target using the average weights of the network used in SSL, to reduce confirmation bias [48]. The technique known as ELR+ incorporates weight averaging, employs two networks, and uses the MixUp method. The most promising approach in LNL that has been proposed in recent years is the combination of sample selection with SSL. In this approach, the training of the network is regularized by the SSL strategy, which discards only the labels of the sample sets that are likely to have noisy labels and are ignored by sample selection, and uses them as unlabeled data. [PERSON] et al. [49] and [PERSON] et al. [50] demonstrated the effectiveness of the SSL strategy in LNL scenarios. Nevertheless, these methods struggle to perform well under conditions with high noise rates (i.e., on data that have a large proportion of noisy labels). The most successful approach that combines sample selection with SSL is DivideMix, which first performs a warm-up using all training data with noisy labels, as shown in (4). In contrast with the approach of [PERSON] et al. [40], a GMM is then fitted to the loss distribution. The training data are used as a set of labeled samples with a high probability of being clean (having a clean label for each sample), \(\mathcal{X}=\{x_{b}:b\in(1,...,B)\}\), and a set of unlabeled samples with a low probability of being clean, \(\mathcal{U}=\{u_{b}:b\in(1,...,B)\}\).