# DYMatch: Semi-Supervised Learning with Dynamic Pseudo Labeling and Feature Consistency
## 2 Related Works  
### Consistency Regularization and Pseudo-Labeling Methods in SSL  
The SSL task addressed in this paper can be defined as a classification task with \(C\) categories. We define \(\mathcal{X}=\{(X_{b},p_{b}):b\in(1,\cdots,B)\}\) as the batches of labeled training samples, where \(\mathbf{X}_{b}\) and \(p_{b}\) represent the \(b\)-th batch of labeled samples and the labels corresponding to these samples, respectively. \(B\) represents the number of these batches, and \(b\) denotes the index of an unlabeled sample in a batch. In addition, let \(\mathcal{U}=\{U_{b}:b\in(1,\cdots,B)\}\) be the batches of unlabeled data. We use \(N_{L}\) and \(N_{U}\) to denote the number of labeled and unlabeled samples in each batch of training data, typically with \(N_{U}\)= \(\sigma N_{L}\). \(P_{model}(y|z_{i};\theta)\) is denoted as the prediction score by the model \(p\) for input \(z_{i}\) with parameters \(\theta\), where \(z_{i}\) represents the \(i\)-th sample in a training batch. In SSL, the motivation of the consistency regularization often follows the smoothness assumption or the low-density assumption [6], which means that the model should generate similar predictions for a specific unlabeled sample when subjected to various perturbations. Therefore, the consistency regularization method can be viewed to find a smooth manifold on which the whole dataset lies by utilizing the unlabeled data [24]. A common strategy for introducing perturbations is employing data augmentation methods [25; 26], which apply image transformations while preserving the semantic information of the data. Specifically, consistency regularization demands that an unlabeled sample \(u_{i}\) should exhibit similarity to its corresponding augmented sample, i.e., \(Augment(u_{i})\). Therefore, most SSL models employ the consistency regularization loss term in Equation (1) to a batch of unlabeled data:  
\[\frac{1}{N_{U}}\sum_{i=1}^{N_{U}}||P_{model}(y|Augment_{1}(u_{i});\theta)-P_{ model}(y|Augment_{2}(u_{i});\theta)||_{2}^{2} \tag{1}\]  
where \(Augment_{1}(\cdot)\) and \(Augment_{2}(\cdot)\) represent two different data augmentation operations, \(u_{i}\) denotes the \(i\)-th unlabeled sample in a batch. Mean Teacher approach [27] contains a teacher model and a student model. The student model closely resembles the regular mode of \(\Pi\)-Model [13], and the teacher model has the same network structure as the student model but employs an exponentially moving average of the student's weights for parameter update. In Mean Teacher, the predictions of the student model and the teacher model are used to calculate the consistency regularization loss. Specifically, \(Augment_{2}(\cdot)\) in Equation (1) is defined as the predictions of the model updated with the Exponential Moving Average (EMA) method. This operation yields a relatively stable feature representation, significantly improving the classification performance. Dual Student [28] is an enhanced teacher-student model that overcomes the effect of the tightly coupled teacher-student models by introducing another student model to replace the teacher model. These two student models have different initial states and are trained with distinct optimization strategies during the training process. Furthermore, Dual Student also introduces two innovative concepts, that is, stable sample and stabilization constraint, to prevent performance bottlenecks associated with a coupled EMA teacher-student model. Virtual Adversarial Training (VAT) [29] proposed the concept of adversarial training based on consistency regularization, which aims to generate adversarial transformations of unlabeled data to perturb the predictions of the model. Essentially, it is a training strategy where \(Augment_{2}(\cdot)\) in Equation (1) represents an adversarial transformation. Consistency regularization method is performed between the predictions of the original unlabeled data and the perturbed one.