# DYMatch: Semi-Supervised Learning with Dynamic Pseudo Labeling and Feature Consistency
## 3 DYMatch
### Dynamic Pseudo-Label Estimation Method Based on Gaussian Mixture Model
#### 3.2.1 Dynamic Local Confidence Threshold Estimation  
Dynamic local confidence threshold estimation aims to estimate class-specific local thresholds to account for the inter-class diversity. Furthermore, the dynamic local confidence threshold is steadily increased during training to ensure the discarding of unlabeled data with incorrect prediction. Therefore, the Gaussian Mixture model is employed to distinguish the positive from negative samples in model's prediction scores for specific categories. Here, positive samples correspond to unlabeled data that the model predicts correctly, while negative samples correspond to unlabeled data with incorrect prediction. The model's prediction scores used to distinguish positive and negative samples can be regarded as the local thresholds for the corresponding categories. Specifically, we assume that the prediction scores \(s^{c}\) for category \(c\) are sampled from a Gaussian mixture distribution \(P(s^{c})\) with two distributions, positive and negative. Local thresholds are dynamically generated by fitting a Gaussian Mixture Model to the prediction scores. \[P(s^{c})=w_{n}^{c}N\Big{(}s^{c}|\mu_{n}^{c},(\sigma_{n}^{c})^{2}\Big{)}+w_{p}^ {c}N\Big{(}s^{c}|\mu_{p}^{c},(\sigma_{p}^{c})^{2}\Big{)} \tag{19}\]  
where \(N\big{(}\mu,\sigma^{2}\big{)}\) denotes a Gaussian distribution. The parameters \(w_{n}^{c}\), \(\mu_{n}^{c}\), \((\sigma_{n}^{c})^{2}\) and \(w_{p}^{c}\), \(\mu_{p}^{c}\), \((\sigma_{p}^{c})^{2}\) represent the weight, mean and variance of the negative and positive sample distributions, respectively. Then, the EM algorithm is employed to infer the posterior probability \(P(pos|s^{c},\mu_{p}^{c},(\sigma_{p}^{c})^{2})\). This posterior probability can be directly used to generate the pseudo label, and the dynamic local threshold \(\widetilde{\eta}^{l}(c)\) corresponding to category \(c\) can be defined as:  
\[\widetilde{\eta}^{l}(c)=\underset{s^{c}}{\text{argmax}}P(pos|s^{c},\mu_{p^{c}} ^{c},(\sigma_{p}^{c})^{2}) \tag{20}\]  
By using the EMA method, we obtain the final dynamic local threshold, which reflects the learning status of the model for a specific category:  
\[\eta_{t}^{l}(c)=\left\{ \begin{aligned} 1/C&,& \text{if }t=0\\ \lambda\widetilde{\eta}_{t-1}^{l}(c)+(1-\lambda)\widetilde{\eta}_{t}^{l}(c),&\text{otherwise}\end{aligned} \right. \tag{21}\]  
where \(\lambda\in(0,1)\) denotes the momentum decay in EMA, \(t\) represents the \(t\)-th iteration during the training process. Meanwhile, \(\widetilde{\eta}_{t}^{l}(c)=DLTE\big{(}DA(q_{i}(c)_{s})\big{)}\), where \(DLTE(\cdot)\) represents the dynamic local threshold estimation method based on GMM, and \(q_{i}(c)_{s}\) denote the prediction scores of unlabeled data with class \(c\). \(DA(\cdot)\) represents the distribution alignment strategy from ReMixMatch, which balance the distributions of prediction scores. The local thresholds are initialized to \(1/C\), where \(C\) represents the number of categories. Finally, \(\eta_{t}^{l}=\left[\eta_{t}^{l}(1),\eta_{t}^{l}(2),\cdots,\eta_{t}^{l}(c) \right]\) contains confidence thresholds for all categories. In the implementation of the dynamic local threshold estimation method, we maintain a queue \(m(c)\in\mathbb{R}^{Q}\) of prediction scores with dimension \(Q\) for each category. The queues of all categories form a prediction memory bank \(M=\{m(1),m(2),\cdots,m(c)\},M\in\mathbb{R}^{C\times Q}\), where \(C\) represents the number of categories. Specifically, the queue \(m(c)\) stores the prediction scores \(s^{c}\) of the unlabeled data predicted by the model as category \(c\), and employs these prediction scores to fit a Gaussian mixture model. Subsequently, the dynamic threshold estimation method can dynamically adjust the threshold \(\eta_{t}^{l}(c)\) corresponding to category \(c\) based on the fitted Gaussian mixture model. As a result, \(\eta^{l}(c)\) can align with the model's classification ability at different training stages. Moreover, the EM algorithm has a negligible effect on training time and does not impose an additional burden to the training.