Non-Stationary Predictions May Be More Informative: Exploring Pseudo-Labels with a Two-Phase Pattern of Training Dynamics
Authors: Hongbin Pei
Jingxin Hai
Yu Li
Huiqi Deng
Denghao Ma
Jie Ma
Pinghui Wang
Jing Tao
Xiaohong Guan
Venue: ICML 2025 poster (2025)
Keywords: Pseudo-Labels
Two-phase pattern
Semi-supervised learning
Self-training
Primary area: deep_learning->selfsupervised_learning
TL;DR: In contrast to existing pseudo-labeling methods, this paper explores a new type of pseudo-labels --- predicted labels that do not exhibit high confidence scores and training stationary.

Abstract: Pseudo-labeling is a widely used strategy in semi-supervised learning. Existing methods typically select predicted labels with high confidence scores and high training stationarity, as pseudo-labels to augment training sets. In contrast, this paper explores the pseudo-labeling potential of predicted labels that **do not** exhibit these characteristics. We discover a new type of predicted labels suitable for pseudo-labeling, termed *two-phase labels*, which exhibit a two-phase pattern during training: *they are initially predicted as one category in early training stages and switch to another category in subsequent epochs.* Case studies show the two-phase labels are informative for decision boundaries. To effectively identify the two-phase labels, we design a 2-*phasic* metric that mathematically characterizes their spatial and temporal patterns. Furthermore, we propose a loss function tailored for two-phase pseudo-labeling learning, allowing models not only to learn correct correlations but also to eliminate false ones. Extensive experiments on eight datasets show that **our proposed 2-*phasic* metric acts as a powerful booster** for existing pseudo-labeling methods by additionally incorporating the two-phase labels, achieving an average classification accuracy gain of 1.73% on image datasets and 1.92% on graph datasets.

Non-Stationary Predictions May Be More Informative: Exploring Pseudo-Labels with a Two-Phase Pattern of Training Dynamics
Hongbin Pei!
Jingxin Hai
Yu
Huiqi Deng' Jing Tao! Xiaohong Guan
Denghao M 2 Jie Ma
Pinghui Wang'
Abstract
partial label learning (Tian et al., 2024), to address the prac- tical challenge of labeled data scareity. This strategy uses predicted labels of unlabeled samples as pscudo-labels to augment training set, facilitating model training via reduc- ing model uncertainty (also called epistemic uncertainty) (Hüllermeier & Waegeman, 2021) and promoting a robust decision boundary in low-density regions (Chapelle & Zien, 2005). Pseudo-labeling has achieved success in various tasks, e.g. computer vision (Rizve et al., 2021), text mining (Yang et al., 2023), and graph learning (Sun et al., 2020).
Pseudo-labeling is a widely used strategy in semi- supervised learning. Existing methods typically select predicted labels with high confidence scores and high training stationarity, pseudo-labels to augment training sets. In contrast, this paper ex- plores the pseudo-labeling potential of predicted labels that do not exhibit these characteristics. We discover a new type of predicted labels suitable for pseudo-labeling, termed two-phase labels, which exhibit two-phase pattern during training: they are initially predicted one category in early training stages and switch to another category in subsequent epochs. Case studies show the two- phase labels are informative for decision bound- aries. To effectively identify the two-phase labels, we design 2-phasic metric that mathematically characterizes their spatial and temporal patterns. Furthermore, propose a loss function tailored for two-phase gpeudo-labeling learning, allowing models not only to learn correct correlations but also to eliminate false ones. Extensive experi- ments on eight datasets show that our proposed 2-phasic metric acts as a powerful booster for existing pseudo-labeling methods by additionally incorporating the two-phase labels, achieving an average classification accuracy gain of 1.73% on image datasets and 1.92% on graph datasets.
High-quality pseudo-labels are essential to this strategy, yet they are challenging to obtain. The confidence score the most commonly used and intuitive metric for selecting pseudo-labele however, it suffers from poor calibration is- sue, especially on out-of-distribution data (Guo et al., 2017; Kage et al., 2024). Recently, researchers discovered training dynamics-i.. the trajectory of model predictions during training- contains rich information about prediction uncer- tainty (Swayamdipta al., 2020; Jia al., 2023). They show that if model makes consistent predictions for an unlabeled sample throughout training, that predicted label is highly likely to be correct. Consequently, training station- arity metric is proposed to identify these predicted labels to act as pseudo-labele (Song et al., 2019; Zhou et al., 2020; Chen et al., 2021; 1; Pleiss et al., 2020; Pei et al., 2024b).
In this paper, in contrast to existing works, we explore pseudo-labeling potential of new type of predicted labels that do not exhibit high confidence score and high training stationary, as illustrated by type II labels in Fig. I(A). These predicted labels are rarely exploited for pseudo-labeling, as they are associated with atypical patterns and tend to have a high risk of misclassification. Incorrect pseudo-labels can significantly degrade model performance due to introducing noise and misleading patterns (Wang al., 2023). How- ever, we find that despite their low correctness, correctly predicted labels of this type could offer greater information gain² for labelig,.,,lng, than the commonly used type
1. Introduction
Pseudo-labeling (Leee et al., 2013) is widely employed strat- egy in many semi-supervised learning methods, e.g., self- training (Amini et al., 2025), co-training (Blum & Mitchell, 1998), consistency regularization (Sohn et al., 2020), and
'MOE KLINNS Lab, Xi'an Jiaotong University, China: Beejing Information Science and Technology University, China. Correspondence to: Jing Tao xjtao@mail.xxil wwwe@aaaaiileeeeeeeee mww...jt...duu.n.. Pinghui Wang <phwang@ @ mail mail.xjtu.cdu.cc>>
'In Fig. 1(A), we use an inverse indicator, non-stationary, which can be easily calculated, as detailed in the Appendix A. "We quantify the information gain by measuring the gradient change from pseudo labels, detailed in Appendix E.
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Non-Stationary Predictions May Be More Informative
A. Spatial distribution
Type II Label: relatively high confidencc and low training
B. Comparison
Correctness Information gain
Correctly predicted label
stationary; Rarely explored to date
Incorrectly predicted labe)
0.4
Non-stationary in training
0.5
0.3
Type
Type II
Two-phase (Ours)
C. Overlap of pseudo-labels
0.2
CIFARI00 dataset
Cora dataset
0.1
Type Label: high confidence and training stationary; Well exploited in existing pseudo-laheling works
6106
18987
3109
248
641
199
0.0
0.2
0.4 Confidence score
0.6
0.8
1.0
Confidence
Overlap
2-phasic (Ours)
Figure 1. (A) Distribution of predicted labels for CIFAR 100 dataset, hased on pre-trained Vision Transformers (ViT) classifier. The z-axis represents confidence score, the y-axis denotes non- stationary during training. Colors indicate correctuess of predicted labels. Type I labels have high confidence and high training stationarity, corresponding to samples with typical patterns; these labels have been well exploited existing pseudo-labeling works. Type II labels exhibit relatively high confidence and low training stationarity, corresponding to samples with complicated patterns; these labels are rarely explored for pseudo-labeling. (B) Type I labels show high correctness, while Type labels provide high information gain. Our proposed two-phase labels combine both high correctness and relatively high information gain. (C) Overlap of effective pscudo-label, (75% accuracy) selected by confidence and our 2-phasic metrie. The pseudo-labels only identified by 2-phasic metric (blue) indicate that many two-phase labels do not have high confidence scores.
I labels, as shown in Fig. (B). This finding raises a new research problem: How can we leverage the significant information gain from rype II labels for pseuli-labeling while minimizing the risk of misclussification?
We validate the proposed 2-phusic metric-based pseudo- labeling method on eight benchmark datasets, including four image datasets and four graph datasets. Experimental results show: (1) Our proposed 2-phusic metric can widely enhance existing pseudo-labeling methods
booster by additionally incorporating the two-phase labels, achieving average classification accuracy gain of 1.73% on image data and 1.92% on graph data. (2) Tow-phase labels are often overlooked by commonly used confidence metrics, and they exhibit high quality in terms of information gain and correctness. We analyze the proposed loss function by an ablation study and examine method limitations. The experiment code is released in our Github repository?.
Toward exploring this problem, this paper uncovers subset within type II labels, termed nwo-phase labels. The two- phase labels refer to predicted labels whose training dynam- ies exhibit two-phase pattern: they are initially predicted as one category in early training stage (the first phase), and switch to another category in subsequent epochs (the second phase). as shown in Fig. 2(A). They show strong potential for labelo,a exhibiting high information gain and relatively high correctness, as shown in FB). Notably, two-phase labels cannot be easily identified by confidence scores, evidenced by non-overlaps in Fig. ICC).
In summary, our contributions in this paper are three-fold:
We uncover two-phase labels, which have strong po- tential for plement to pseudo-labels provided by existing meth- ods. We analyze the rationale of two-phase labels for pscudo-labeling from different perspectives.
We propose the 2-phasic metric to effectively identify the two-phase labels by mathematically characterizing their spatial and temporal patterns. We further analyze the ratio- nale behind using two-phase labels for pseudo labelinn,, (1) Two-phase labels provide valuable information about deci- sion boundaries, two-phase samples are positioned much closer to the boundaries than samples with high confidence; (2) They can enable models to not only learn correct correla- tions, but also eliminate false correlations. To fully release the two potentials, we design a specialized pseudo-labeling loss function tailored for two-phase labels.
and can be valuable com-
We propose the 2-phasic metric to effectively and ef- ficiently identify two-phase labels by mathematically characterizing their spatial and temporal patterns. Fur- thermore, we propose loss function specifically de- signed for two-phase pseudo-labeling learning.
'RRL: http:///gttaaaoomm/////c/cannane wwo.passiefforpeedoo-lliiigg
2

---

Non-Stationary Predictions May Be More Informative
We conduct extensive experiments to validate the 2- phasic metric-based pseudo-labeling method on eight image and graph datasets. Experimental results show that the proposed 2-phasic metric acts as powerful booster for existing pseudo-labeling methods.
2020). A popular method for measuring the uncertainty is Monte Carlo dropout (Gal & Ghahramani, 2016; Rizve et al., 2021), which measures the variance of predicted la- bels during multiple dropout operations in testing, Another important uncertainty metric used for pseudo-lahel selec- tion is training stationarity (also called time-consistency), which is based on the training dynamics (Song et al., 2019; Zhou et al., 2020; Chen et al., 2021; Pleiss al., 2020; Pei et al., 2024b). The metric can be viewed self-ensemble method, where models at different epochs act as ensemble members (Liu et al., 2022b). It relies on the observation that if a model consistently predicts the same label for an unlabeled sample throughout training, that label is likely to be correct. Unlike existing works, this work is, to the best of our knowledge, the first to explore predicted labels with non-stationary training dynamics for pseudo-labeling,
2. Related Works
2.1. Pseudo-labeling Approach
The pseudo-labeling approach dominant in semi- supervised learning, and it involves class of methods that assign predicted labels unlabeled samples that are then acted labeled samples for training (Kage et al., 2024).
In self-training, pseudo-labels are generated by a model trained on labeled data and subsequently used to further train the same model (Lee et al., 2013; McClosky et al., 2006). This process can be conducted iteratively to achieve curriculum learning (Cascante- Bonilla al., 2021; Xie et al., 2020). In co-training, pseudo-labels are generated by the teacher model and then used to augment the student model's training set, allowing for complementary learning (Wang & Zhou, 2013), knowledge distilling (Hinton et al., 2015), and iteratively optimizing pseudo-label assignment policy through the teacher model (Blum& & Mitchell, 1998; Pham et al., 2021). Pseudo-labeling is also employed in con- sistency regularization, where the objective is to enforce consistency among the pseudo-labels (soft or hard) of per- turbed gennratee by various data augmentation methods-and and the original sample (Berthelot et al., 2020: Sohn et al., 2020; Hu et al., 2021). In partial label learn- ing, pseudo-labels are leveraged to disambiguate among candidate labels, thereby identifying the true label of each sample (Jin& & Ghahramani, 2002: Tian et al., 2024). This paper uncovers& new type of pseudo-labels, the two-phase labels, which are under-explored in existing research and thus may broadly benefit the aforementioned methods.
2.3. Training Dynamics-based Analysis
Training dynamics has been applied in various analysis tasks, as it contains rich information about both models and data. In addition to pseudo-labeling. training dynamics is used to identify samples that are frequently forgotten to address catastrophic forgetting issues (Toneva et al., 2019; Pan et al., 2020). It is employed in active learning to find ambiguous samples for expert querying, helping to refine the training set and support oott---diittibution generaliza- tion (Kye et al., 2023; Wang et al., 2022a). It is also used to measure learning difficulty in curriculum learning (Bal- dock et al., 2021; Zhang et al., 2025), identify important samples to reduce the training set (Paul et al., 2021), and detect mislabeled samples for data cleaning (Jia et al., 2023; Swayamdipta et al., 2020). Notably, in computer vision, many studies utilize training dynamics to enhance the em- bedding space; such approaches are commonly referred to as memory bank methods (Liu et al., 2022c).
3. Problem Definition
2.2. Metrics for Pseudo-label Selection
Let D2 = ((x(0) ( be a labeled dataset with N samples, where the vector (1)) represents input features of sample i. Each sample belongs to one of C' categories and is labeled with the one-hot vector y E {0, 1) Let Du = {<<( )= ples, which does not include sample labels. The pseudo- labeling approach trains a parameterized model fo on both the labeled samples in Dr. and pseudo-labeled samples in Dp (x)(), ) where g(() denotes pseudo- label. These pseudo-labeis are generated from the unlabeled dataset Dy. An essential issue of this approach is the selec- tion of pseudo-labels, which aims to identify unlabeled sam- ples whose predicted labels can be confidently assigned as pseudo-labels, thereby enhancing the training of the model fo. These predicted labels may be given by the model fo from a previous iteration in self- trainin or by a teacher
Pseudo-label selection aims to identify which samples should be assigned pseudo-labels. It is essential to pseudo- labeling methods. In literature, the most commonly used metric for pseudo-label selection is the confidence score de- rived from softmax distribution (Sun et al., 2020; Cascante- Bonilla et al., 2021; He et al., 2023). However, this metric suffers from poor calibration, which means high confidence scores are often assigned to incorrectly predicted labels, leading to incorrect pseudo-labels (Guo et al., 2017). More- over, it has been argued that confidence scores should not be trusted for fit-o--ddittribtionn data (Gal& & Ghahramani, 2016). Confidence scores are also susceptible to manipula- tion by adversarial examples (Nguyen et al., 2015).
No be an unlabeled dataset with No sam-
Prediction uncertainty is also a critical metric for select- ing pseudo-labels (Gawlikowski et al., 2023; Zhao et al.,
3

---

Non-Stationary Predictions May Be More Informative
Training dynamics (prediction trajetory)
Spatial chara- cteristic
C. Temporal chara- cteristic
4.1.1. SPATIAL MEASURE
We summarize the spatial characteristics of training dynam- ics by averaging multiple predictions made by the model throughout model training, as follows:
Calegory
Node 876
62
p(0) 1/ EIeT
Here, the set denotes a memory bank, in which each el- ement E T specifies a training epoch used for averaging, and T is expected to cover the training process unbias- edly. o (4) softmax(z, ( represents the softmax distri- bution given by the model at epoch i, with z
Category
Node 121
6.0
(() being the
Category
C-dimensional logit vector of sample i.
2
42-
We find that, as two-phase samples (i.e., samples with two- phase labels) undergo a change in predicted labels during training, their distribution p(1) exhibits a clear bimodal pattern, with probabilities of two categories significantly higher than those of others, as shown in Fig. This spatial pattern effectively distinguishes two-phase samples from many other types. For example, the distribution p(() of node 121 is unimodal, which is preferred by training stationarity- methods (Pei et al., 2024b), while the distribution of node 1016 resembles a uniform distribution.
Node-1016
Calegory
Node 2359
6.-
022
Epoch
Category
Cutegory
Figure 2. (A) Training dynamics with two-phase pattern (Node 876), stationarity (Node 121), and frequent oscillations (Nodes 1016 and 2359), generated by a Graph Convolutional Networks (GCN) Cora dataset. (B) and (C) summarize spatial and tempo- ral characteristics of corresponding training dynamics, respectively
We specifically design spatial measure to characterize and quantify the bimodal pattern of p(o), as
(8) Pspotial HLMM - CLMO p!()) log p!()
model in co-training. The pseudo-label selection problem can be modeled as designing metric that evaluates the suitability of predicted labels to act as pseudo-labels.
Here, we propose a new entropy measure, termed Leave- Maximum Out entropy (LMO entropy for short), denoted by HIMO [P()] to measure bimodal distributions. Unlike traditional entropy, the uniqueness of LMO entropy lies in that it excludes the category with the highest probability from the entropy calculation. Specifically, the set CLMO includes every category c, except the one with the highest probability in the distribution p(), The smaller the spatial (i) measure 1 Espatial the clearer the bimodal pattern.
4. Method
In this section, we first design the 2-phasic metric to effec- tively and efficiently identify two-phase labels, then ana- lyze the rationale behind using two-phase labels for pseudo- labeling, and finally propose a pseudo-labeling loss that can fully extract the information within two-phase labels.
As empirically validated in Appendix B, the proposed LMO entropy effectively captures bimodal patterns. Its rationale is as follows: by removing the category with the highest probability, a bimodal distribution transforms into sharply unimodal distribution, resulting in high entropy.
4.1. 2-phasic Metric
The key to designing a metric to identify two-phase labels lies in mathematically characterizing the two-phase pattern: the label is initially predicted as one class by the model in the first training phase and then shifts to different class in the second phase, as shown in Fig. 2A. Our proposed 2- phasic metric characterizes this pattern through both spatial and temporal measures, defined as follows:
4.1.2. TEMPORAL MEASURE
We further design the temporal measure to complementally capture the two-phase pattern. We summarize temporal char- acteristics of training dynamics by accumulating changes of successive predictions during training,
2 phasic (t) == (1 " (premporat)""
Q EV.'ET lo (1)
(i())) where F spactial and fl Htemporal represent the measures of spa- tial and temporal pattern, respectively, of the training dynam- ics for sample i. The parameters 7/, and 7/2 are exponential weights. The product operator implies that both the spatial and temporal patterns must be satisfied simultancously.
where t' and 1" are successive epochs in set T, with being the immediate predecessor of t". The C-dimensional vector Q captures the magnitude of prediction changes throughout training, reflecting the evolution of predictions.
4

---

Non-Stationary Predictions May Be More Informative
We observe two distinguishing temporal patterns in two- phase samples. Observation 1: The magnitude of changes in prediction probabilities is relatively small and occurs within the two bimodal categories of p(() rather than other categories. Observation 2: The changes in prediction prob- abilities are directional, shifting from the predicted category in the first phase to the one in the second phase.
Input images
Attribution maps
Dog
Tiger
Dog
Modelling Observation 1. We first define two deltas,
4 1 ( + 2
A (4)
1 - oz Q
C
Bed Ground truth
Chair Predicted labels in 1st phase
Bed Predicted labels in 2nd phase
where A captures the average change of the two bimodal categories C1 and C22 which correspond to the predicted label in the first phase and the second phase, respectively. A -be represents the average change of other categories. We further apply a sigmoid function to the deltas to model tolerance to the changes in Observation 1, as follows
Figure 3. Atributtion maps of two-phase samples in CIFARIOU (Left) Input images. (Middle) and (Right) attribution maps based on ViT model in the first and second phases, respectively.
Pattern Learning Perspective. The two-phase pattern in training dynamics implies a significant shift in the patterns learned by the model, transitioning from patterns associated with category e₁ (the predicted label in the first phase) to those corresponding to category 22 (the predicted label in the second phase). Existing studies have shown that models ini- tially learn simple and typical patterns, gradually progress to learning more complex patterns (Arpit et al., 2017: Siddiqui et al., 2023). Given that patterns learned in later training stages more stable and reliable (Liang et al., 2020), we conclude that two-phase samples contain two types of pat- terns: simple and non- con-class-exclend
bi 1 1 -mn) 1 e (4 -E mn)
Here, Cbi and C-di are thresholds that control the sensitivity of the sigmoid functions. We set én >> 6-bi >> 0 to allow for greater tolerance to changes within the two bimodal categories compared to the other categories. The measure o can be used to filter out samples whose predicted labels frequently switch, such as node 2359 in Fig.2, even if they exhibit a bimodal distribution p(i of spatial pattern.
Modelling Observation 2. To measure the directional change of prediction probabilities, we first calculate the difference in probabilities from the bimodal category 02 C1,
patterns associated with category Cı; complex and class-exclusive palterns as- sociated with category C2.
10) (02) o (c1).
To visualize these shifts of learned patterns, we present two representative images whose training dynamics exhibit clear two-phase pattern, as shown in Fig.3. We use Grad- CAM (Selvaraju et al., 2017), widely used attribution interpretability technique, to highlight the key regions in two-phase samples that the model relies on for classification. Details of the attribution technique are provided in Appendix C. In the first phase, the model primarily focuses on simple and non-class-xxclnsive patterns. For example, in the "Bed" image, the model initially emphasizes the headboard, which visually resembles the backrest of chair, resulting in a misclassification as "Chair". In the second phase, however, the model shifts its attention to more complex, detailed, and class-exclusive patterns, such as the footboard in the "Bed" image and the nose and ears in the "Dog" image. This shift enables the model to finally make correct classifications.
Notably, the difference Jt epoch t. We then measure the directional change by
(8) is signed and defined at each
(8) - 9T ( - Imin and Imin min(g] 101 (TT),
where denotes the latest epoch in set T. A large ! indicates directional shift in prediction probabilities from C1 c2, which can filter out samples whose predicted labels remain consistently ambiguous between e₁ and C2 through- out training. These samples cannot be detected by either the measure bs or the spatial measure.
Finally, we design the temporal measure of the two-phase pattern as a combination of measures ( and (
Ptemporal == (0 y (1/O@))².
where parameters 1 and 222 are exponential weights.
This shift suggests two-phase labels are well-suited for pseudo-labeling in the following two ways: (i) Two-phase labels can help eliminate f'alse correlations between labels and simple, non-exclusive patterns, i.e., the Ci category and the patterns learned in the first phase, thereby reducing the risk of misclassification. (ii) Two-phase labels can provide
4.2. Rationale Analysis to Two-phase Labels
From two distinct perspectives, we analyze why our pro- posed two-phase labels are well-suited for pseudo-labeling.
5

---

Non-Stationary Predictions May Be More Informative
A. CIFARI00 dataset
cross-entropy loss for a two-phase label:
((7 (ggg))) - (v!2 log(u + (1 o( log(1 1!4 )),
where 7(") denotes two-phase pseudo-label, specifically with the bimodal categories ye I and y = 0, and j() = fo (=(()) represents the prediction probability given by the model. In this loss function, the first 02 term is designed to learn correct correlations between complex, class-exclusive patterns and category c22 and the second ej term enforces the climatation to false correlations between simple, non- exclusive patterns and category c1. The two terms are further analyzed in the following ablation study, A pseudo-labeling algorithm using 2-phasic metric is given in Appendix D.
Confidence
2-phasic
Data sample
B. Cora dataset
5. Experiment
We empirically validate our proposed 2-phasic metric and pseudo- algorithm on both image and graph datasets, by answering four key questions. Q1 (Booster Test): Can the 2-phasic metric enhance existing pseudo-labeling meth- ods as a booster by additionally incorporating two-phase labels? Q2 (Complementary Analysis): Are two-phase labels high-quality pseudo-labels that are missed by existing -seudo-labeling metrics? Q3 (A blation Study): How do the two terms in our specialized loss function contribute to performance? Q4 (Parameter Sensitivity): How do the parameters in the 2-phasic metric impact performance?
Figure 4. Two-phase samples are closer to decision boundaries in 2D representation space. Blue triangles and red stars denote sam- ples with the highest confidence and 2-phasic metric, respectively. additional information to strengthen correct correlations between labels and complex, elasu-xxe patterns, i.e., the 0g category and the patterns learned in the second phase. To fully release the two potentials of two-phase labels, we design specialized pseudo-labeling loss function tailored for two-phase labels, as presented in the next subsection.
Decision Boundary Perspective. To observe how two- phase samples are positioned in representation space, we conduct a visualization study. We extract representations from the last layer of a ViT model for images in CIFAR 100 dataset and from the last layer of a GCN (Kipf & Welling, 2017) for nodes in the Cora dataset. These high-dimensional representations are projected into 2D space using I-SNE (Van der Maaten & Hinton, 2008) for visualization.
Baselines. We evaluate our proposed method based five state-of-the-art baselines, including both confidence-based and uncertainty- pseudo-labeling methods. On graph datasets, we use four baselines: Confidence score, AUM (Sosea & Caragea, 2022), MoDis (Pei et al., 2024b), and DR-GST (Liu et al., 2022a); on image datasets, we use four baselines: Confidence score, MoDis, UPS (Rizve et al., 2021), and Softmatch (Chen et al., 2023).
As shown in Fig.4, samples with high confidence scores (blue triangles) are distant from decision boundaries, es- pecially on Cora dataset, indicating they are safe choices as lseudo-labeling but may not contribute much additional information to model. In contrast, two-phase samples se- lected by 2-phasic metric (red stars) are positioned much closer to the decision boundaries, suggesting they can offer valuable information about boundaries. Existing studies have shown that samples near decision boundaries generally have a greater influence on model performance than those positioned farther away (Cortes, 1995; Wei et al., 2021).
Datasets. Experiments are conducted on eight bench- mark datasets, including four image datasets, CIFAR-100 (Krizhevsky, 2012), AT (Helber et al., 2019), Semi- Aves (Su & Maji, 2021), and STL-10 (Coates et al., 201 1), and four graph datasets, Cora (McCallum et al., 2000), Cite- seer (Sen et al., 2008), Pubmed (Namata et al., 2012), and AmazonComputers (McAuley et al., 2015). Details of base- lines, datasets, and metrics are provided in Appendix E.
5.1. Experiment 1: Booster Test
Experimental protocol. To evaluate the pseudo-labeling potential of two-phase labels, we test whether the 2-phasic metric can enhance baselines as a booster by addi- tioanally incorporating two-phase labels. We design a controlled experiment with a baseline trial and a 2-phasic trial following a three-stage protocol, as illustrated in Fig.5. In stage 1, base model established by either fine-tuning pre-trained model or training a model on labeled data. In
4.3. Two-phase Pseudo-labeling Loss
As analyzed above, two-phase labels can both enhance cor- rect correlations and eliminate false correlations. To fully leverage this valuable information to train model, we design a specialized gpeudo-Labeling loss function. For a general multi-label classification task, we present modified binary
6

---

Non-Stationary Predictions May Be More Informative
Stage 1
Stage
Stage 3
Table I. Classification accuracy of pseudo- learning (%). L/C denotes the number of labeled nodes per category; # Labels denotes the total number of labeled images used.
Pscudo-Inbeling
(Our method)
Taining the model our two-plase pecudo-fabels
Graph data
Cora
Citeseer
Buse model
Pseudo-Jabeling neaseline method)
L/C
labeled data finetuning
Taining model
Taining model pseudo-labe laudll given basel
Comparing
Confidence
+2-phase
70.41
66.21
71.38
74.07
77.17
73.73
64.63
60.91
68.09
65.23
69.59
67.16
Pseudo-labellin
mhddc))
(Baselime method)
AUM
63.50
69.86
75.88
61.10
68.27
67.87
"Taining the model on more pseudo labels given by the baseline
+2-phuse
66.95
71.84
77.71
64.99
69.72
69.64
MoDis
68.61
71.85
76.32
65.74
69.17
71.88
Training epoch.
+2-phase
70.10
72.83
78.25
68.63
71.92
72.01
DR-GST
71.01
77.07
81.12
61.05
69.62
73.55
Epoch 0
Epoch
Epoch
Epoch
+2-phase
73.88
78.53
81.56
84
71.02
74.50
Figure 5. An illustration of the protocol of experiment 1.
Graph data
PubMed
Amazon
L/C
3
stage 2, the model is trained using pseudo-labels generated by a baseline. In stage 3, for the baseline trial, the baseline continues to generate pscudo-labels for model training; for the 2-phasic trial, the model is trained on two-phase labels given by our 2-phasic metric. Finally, we compare the per- formance of models trained in the two trials to evaluate whether the 2-phasic metric improves the baseline.
Confidence
64.02
69.94
72.78
73.55
75.28
80.75
+2-phase
65.99
72.32
74.01
75.80
77.00
82.49
AUM
65.19
70.22
72.14
73.65
75.28
80.84
+2-phase
66.38
72.24
73.22
75.49
77.02
81.77
MoDis
65.11
71.56
74.14
75.31
76.14
81.34
+2-phase
65.69
72.66
74.43
76.64
78.23
82.04
DR-GS- DR-GST
68.88
72.82
77.20
79.76
80.00
81.21
+2-phase
73.06
74,56
77.76
80.04
81.30
82.45
We employ the widely used GCN as the base model for graph datasets, which initially trained using only 3, 5, and 10 labeled nodes per category (L/C), following the existing setup (Liu al., 2022a) to create challenging scenarios with minimal label information. We employ a pre-trained VIT model for image datasets and fine-tune it using few labeled images. The total number of the labeled images (# Labels) is set according to (Li et al., 2023; Wang et al., 2022b), as shown in Table 1. Additionally, we introduce a more challenging scenario with only 100 labeled images on CIFAR-100. More details can be found in Appendix E.
Image data
CIFARI00
Semi-Aves
# Labels
100
200
400
3959
Confidence
53.70
66.82
75.37
50.88
+2-phase
55.40
67.24
77.33
52.03
UPS
52.62
66.84
76.02
50.25
+2-phase
54.40
68.43
77.65
52.40
Modis
54.06
66.99
75.24
50.92
+2-phase
58.40
68.28
76.78
51.08
SoftMatch
63.54
75.68
81.24
51.88
+2-phase
65.31
77.11
82.45
54.09
Image data
Euro-SAT
STL-10
Labels
20
40
40
100
Confidence
75.88
85.59
75.63
87.77
Table 1 summarizes the classification accuracies after stage 3 of the experimental procedure eight image and graph datasets. We repeat the experiments 10 times and report average results on the graphs to minimize the impact of randomness. We observe that incorporating two-phase la- bels through our 2-phasic metric generally improves the performance of these baselines. Specifically,
+2-phase
77.31
88.55
76.73
89.29
UPS
75.93
86.55
76.99
88.30
+2-phase
76.29
87.90
78.45
88.50
Modis
76.09
85.24
74.42
87.01
+2-phase
76.76
88.74
75.31
89.96
SoftMatch
91.30
92.48
85.26
89.06
+2-phase
95.20
95.65
87.65
90.19
The 2-phase trials consistently outperform the base- line trials across baselines, datasets, and base models. The average accuracy gain reaches 1.73% for image datasets and 1.92% for graph datasets, indicating addi- tional information is provided by the two-phase labels in stage 3, compared to the baselines.
5.2. Experiment 2: Complementary Analysis
We analyze the complementarity of two-phase labels with pseudo-labela labels generated by existing methods, i.e., whether two-phase labels are high-quality pseudo-labels that are cur- rently overlooked. this end, we compare pseudo-labels generated by 2-phasic metric and confidence score in terms of correctness, information gain, and their overlap. The correctness refers to the proportion of correct pseudo-labels among all generated heeudo-labels. We employ the infor- mation gain measure proposed in (Pei et al., 2024b), which quantifies the magnitude of changes in model gradients after adding pseudo-labels to training. We adopt Intersection over Union (IoU), also known Jaccard similarity, measure
The improvements from 2-phasic become signifi- cant when the number of labeled samples is lower. This suggests that two-phase labels play more important role in scenarios with extremely limited labels.
The SoftMatch achieves strong performance because it well combines both pseudo-labeling module and unsupervised contrastive learning module.
7

---

Non-Stationary Predictions May Be More Informative
Cora (L/C-3)
CIFARIO0 (# Label=400)
Table 2. Complementary analysis of two-phase labels. We selected the top-k pseudo-labels with the highest confidence and 2-phasic metrics to compare their accuracy, information gain, and overlap.
70.5
77.5
Accuracy
Eb C-b
Graph data L/C
Cora
Citeseer
10
3
10
68.5
75.5
Correctness of pscudo-lahels (%)
2
2
2
2
2
2
2
2
Confidence
89.55
91.75
96.40
84.15
82.85
86.10
70.5
77.5
2-phasic
90.50
91.00
96.95
84,65
83.15
87.00
Aecurade Mo
Information gain of pseudo-labels
7/1
Confidence
2.81
3.60
1.78
2.18
2.48
2.76
PP
2-phasic
3.78
4.88
2.67
3.10
3.03
3.66
PP2
67.0
75.5
Overlap of the two pscudo-label sets
0.01
0.2
0.4
0.6
0.8
0.01
0.2
0.4
0.6
0.8
I
LoU
0.61
0.48
0.60
0.60
0.68
0.64
Image data
CIFARI00
Euro-SAT
Figure 6. Parameter sensitivity analysis. The g-axis shows the classification accuracy on Cora (Left) and CIFARI00 (Right): the 2-axis denotes the values of parameters in 2-phasic metric.
# Labels
100
200
400
20
40
Correctness of pseudo-lahed (%)
Confidence
56.09
69.10
83.47
77.29
88.60
2-phaxic
61.83
75.69
86.91
80.05
91.31
Information gain of pseudo-lahels
Ablation study results are summarized in Table 3, which shows that: (1) Both the e1 and cy terms contribute to the overall performance, with the 02 term being the dominant factor; (2) The improvement by the ei term is more signif- icant when the number of labeled samples is lower. This indicates that, analyzed earlier, eliminating false correla- tions between simple, non-exclusive patterns and labels is especially important when labeled data is scarce.
Confidence
252
344
344
356
528
2-phasic
614
554
453
426
551
Overlap of the two pseudo-lahe sets
LoU
0.16
0.17
0.21
0.41
0.42
Table Classification accuracy of pscudo-labeling learning (%) in ablation study. "ei & e2 terms" denote the complete loss function: "O2 term" refers the one with only c2 term.
Dataset
Cora
CIFARION
L/C| # Labels
3
10
100
200
400
5.4. Experiment 4: Parameter Sensitivity
Confidence
66.21
71.38
73.73
53.70
66.82
75.37
+ term
68.96
72.91
77.06
54,33
67.07
76.31
We analyze the sensitivity of the parameters in the proposed 2- 2-phasic metric following the protocol of experiment 1. The analysis involves three weight parameters (1/, (P1. and y22) and two bias parameters (Eg and e-6).. Notably, the weight 1/2 is dependent on y and 422 in practice and therefore excluded from our analysis. We adopt Cora ((//==3) and CIFAR 100 (# Labels 400) experimental scenarios and use the confidence score the baseline. This is controlled experiment where we adjust only the target parameter and observe the resulting changes in model performance, while keeping all other hyperparameters as fixed values.
+ terms
70.16
74.07
77.17
55.40
67.24
76.52
6) improvement
1,20
1.16
0.11
1.07
0.17
0.21
the overlap of pseudo-labels generated by different methods. We use the model at the last epoch of stage 2 to generate pscudo-labels that were not previously used, and then we select the top-k pseudo-labels with the highest value of the corresponding metric for analysis. The number & is set to 200 for graph datasets and 10,000 for image datasets.
The comparison results are summarized in Table 2, which shows that: (1) The two metrics are at the same level in terms of pscudo-label correctness; (2) Two-phase labels generated by the proposed 2-phasic metric are more informative for the model at the last epoch of stage 2: (3) The overlap of the two pseudo-lahel sets is not high, indicating that two-phase labels are often overlooked by the confidence score.
As shown in Fig.6, all parameters in 2-phasic metric influ- ence model performance, indicating that each component of 2-phasic metric is effective and essential. In the figure, none of the curves oscillate up and down, indicating that the parameters can be linearly searched to quickly obtain optimized values. The optimal value of bais es precedes that of E-b. which aligns with our modeling for Observation 1, as discussed in subsection 4.1.2.
5.3. Experiment Ablation Study
Here, we evaluate the contribution of the two terms in the specialized loss function introduced in Section 4.3. Follow- ing the protocol of experiment 1, we modified the training process in stage 3 by using the loss function with only the C2 term and with both the ej and e2 terms, respectively. We adopt Cora and CIFAR 100 as two representative datasets and use the confidence score as the baseline.
6. Discussion
Calculating the proposed 2-phasic metric introduces addi- tional space complexity due to the use of a memory bank to store training dynamics. The additional space complexity is (NN[T)), where N denotes the number of unlabeled sam- ples and [T] is the number of epochs sampled to capture
8

---

Non-Stationary Predictions May Be More Informative
training dynamics. Our experiments show that small [T] is sufficient effectively identify two-phase samples. For instance, in the experiment of booster test, [T] is set to 50, meaning the additional space complexity is ((N)
References
Akiba, T., Sano, S., Yanase, T., Ohta, T., and Koyama, M. Optuna: A next-generation hyperparameter optimization framework. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2019.
As shown in Fig.6, the optimal values of the parameters vary between the two datasets. Manually tuning these parameters can be tineconnuming in practice. In future work, we plan to automatically learn the parameters by adopting the strategy proposed in (Kendall et al., 2018). 8). Future work also includes applying the proposed method to cutting-edge models such as MTGCN (Pei et al., 2024a) on graph data, and extending it to broader learning scenarios, such as active learning and contrastive learning.
Amini, M.-R., Feotanov, V., Pauletto, L., Hadjadj, L.. De- vijver, É., and Maximov, Y. Self-training: A survey, Neurocomputing, 616: 128904, 2025.
Arpit, D., Jastrzebski, S., Ballas, N., Krueger, D., Bengio. E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al. A closer look at memorization in deep networks. In International Conference on Machine Learning, pp. 233-242, 2017.
7. Conclusion
Baldock, R., Maennel, H.,, and Neyshabur, B. Deep learn- ing through the lens of example difficulty. In Advances in Neural Information Processing Systems, pp. 10876- 10889, 2021.
This work uncovers two-phase labels, new type of pseudo- labels, which are highly complementary to existing pseudo- labeling methods. These labels are high-quality pscudo- labels that are often overlooked by current methods. Be- hind this discovery lies an important novel problem - How can significant information from predicted labels with non-stationary training dynamics be leveraged for pseudo- labeling? The problem may inspire further research in this underexplored area. We designed the 2-phasic metric to identify two-phase labels and proposed a specialized loss function in which the two-phase labels help models both learn correct correlations and eliminate false correlations. Extensive experiments demonstrated that incorporating two- phase labels significantly enhances existing gseudo-labeling methods due to their high information gain and correctness.
Berthelot, D., Carlini, N., Cubuk, E. D., Kurakin, A., Sohn, K. Zhang, H., and Raffel, C. Remixmatch: Semi- supervised learning with distribution matching and aug- mentation anchoring. In International Conference on Learning Representations, 2020.
Blum, A. and Mitchell, T. Combining labeled and unlabeled data with co-training. In Eleventh Annual Conference on Computational Learning Theory, pp. 92-100, 1998.
Cascante-Bonilla, P., Tan, F., Qi, Y., and Ordonez, V. Cur- riculum labeling: Revisiting pseudo-labeling for semi- supervised learning. In AAAI Conference on Artificial Intelligence, pp. 6912-6920, 2021.
Acknowledgements
Chapelle, O. and Zien, A. Semi-supervised classification by low density separation. In International Workshop on Artificial Intelligence and Statistics, pp. 57-64, 2005.
The authors would like to thank all the anonymous reviewers and chairs for their constructive comments. This work was supported by the National Natural Science Foundation of China under grant 62202369, 62372362, and 62306229.
Chen, C., Dong, S., Tian, Y., Cao, K., Liu, L.., and Guo, Y. Temporal self-ensembling teacher for semi-supervised object detection. IEEE Transactions on Multimedia, pp. 3679-3692, 2021.
Impact Statement
This paper may have broad impact on sseudo-label-based semi-supervised learning methods, as it expands the scope of pseudo-labels to include not only high-confidence, vrining-stationary predictions but also those exhibiting non-stationary training dynamics. This expansion offers valuable additional information for training, which is particularly significant given the widespread use of pseudo- labeling methods across various algorithms and applications. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Chen, H., Tao, R., Fan, Y., Wang, Y., Wang. J., Schiele, B., Xie, X., Raj, B., and Savvides, M. Softmatch: Addressing the quantity-qualle tradeoff in sumi-supervised learning. In International Conference on Learning Representations, 2023.
Coates, A., Ng. A., and Lee, H. An analysis of single-layer networks in unsupervised feature learning. In Interna- tional Conference on Artificial Intelligence and Statistics, pp. 215-223, 2011.
Cortes, C. Support-vector networks. Machine Learning. 1995.
9

---

Non-Stationary Predictions May Be More Informative
Gal, Y. and Ghahramani, Z. Dropout bayesian approxi- mation: Representing model uncertainty in deep learning- In International Conference Machine Learning, pp. 1050,1059, 2016.
Kipf, T. N. and Welling, M. Semi-supervised classifica- tion with graph convolutional networks. In International Conference on Learning Representations, 2017.
Krizhevsky, A. Learning multiple layers of features from tiny images. University of Toronto, 2012.
Gawlikowski, J., Tassi, C. Ali, M., Lee, J., Humt, M., Feng, J., Kruspe, A., Triebel, R., Jung, P., Roscher, R., et al. A survey of uncertainty in deep neural networks. Artificial Intelligence Review, pp. 1-77, 2023.
Kye, S. M., Choi, K., Byun, H., and Chang, B. Tidal: Learn- ing training dynamics for active learning. In JEEE/CVF International Conference on Computer Vision, pp. 22335- 22345, 2023.
Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q.Q. On calibration of modern neural networks. In International Conference Machine Learning, pp. 1321-1330, 2017.
Lee, D.-H. et al. Pscudo-label: The simple and efficient semi-supervised learning method for deep neural net- works. In ICML 2013 Workshop on Challenges Repre- sentation Learning, pp. 896, 2013.
He, H., Aminian, G., Bu, Y., Rodrigues, M., and Tan, V. Y. How does pseudo-labeling affect the generalization error of the semi-supervised gibbs algorithm? In International Conference Artificial Intelligence and Statistics, pp. 1494-8520, 2023.
Li, M., Wu, R., Liu, H., Yu, J.. Yang, X., Han, B., and semi-supervised learning with instance- dependent thresholds. In International Conference on Neural Information Processing Systems, 2023.
Liu, T. Instant:
Helber, P., Bischke, B., Dengel, A., and Borth, D. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Se- lected Topics in Applied Earth Observations and Remote Sensing, 12(7):211-2226, 2019.
Liang. R., Li. T., Li, Wang. J.. and Zhang, Q. Knowledge consistency between neural networks and beyond. In International Conference on Learning Representations, 2020.
Hinton, G. E., Vinyals, O., and Dean, J. Distilling the knowledge in neural network. ArXiv, abs/1503.02531 2015.
Liu, H., Hu, B., Wang, X., Shi, C., Zhang, Z., and Zhou, J. Confidence may cheat: Self-training on graph neural net- works under distribution shift. In ACM Web Conference, pp. 1248 1258, 2022a.
Hu, Z., Yang, Z., Hu, X., and Nevatia, R. Simple: Similar pseudo label exploitation for semi-supervised classifica- tion. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15099-15108, 2021.
Liu, J., Qi Z., Wang, B., Tian, Y., and Shi, Y. Self-Ilp: Self-supervised learning from label proportions with self- ensemble. Pattern Recognition, 129:108767, 2022b.
Hüllermeier, and Waegeman, W. Aleatoric and epistemic uncertainty in machine learning: An introduction to con- cepts and methods. Machine Learning, 1 110()):577-506 2021.
Liu, J., Sun, Y., Zhu, F., Pei, H., Yang. Y., and Li, W. Learning memory-augmente unidirectional metrics for cross-modality person re-identification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 19366-19375 2022c.
Jia, Q., Li, X., Yu, I.., Bian, J., Zhao, P., Li, S., Xiong, H., and Dou, D. Learning from training dynamics: Identify- ing mislabeled data beyond manually designed features. In AAAI Conference on Artificial Intelligence, pp. 8041- 8049, 2023.
McAuley, J., Targett, C., Shi, Q. and van den Hengel, A. Image-based recommendations on styles and substitutes. In International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 43-52, 2015.
Jin, R. and Ghahramani, Z. Learning with multiple labels. In Advances in Neural Information Processing Systems, 2002.
McCallum, A. K., Nigam, K., Rennie, J., and Seymore, K. Automating the construction of internet portals with ma- chine learning. Information Retrieval, 3: 1:127-163, 2000.
Kage, P., Rothenberger, C., Andreadis, P., and Diochnos, for computer vision.
McClosky, D.,, Charniak, E., and Johnson, M. Effective self- training for parsing, In Human Language Technology Conference of the NAACL, pp. 152- 2006.
D. review of ArXiv preprint arXiv:2408.0722 07221, 2024.
Kendall, A., Gal, Y., and Cipolla, R. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In IEEE conference on computer vision and pattern recognition, pp. 7482-7491, 2018.
Namata, G. M., London, B., Getoor, L., and Huang, B. Query-driven active surveying for collective classification. In Workshop Mining and Learning with Graphs, 2012.
10

---

Non-Stationary Predictions May Be More Informative
Nguyen, A., Yosinski, J., and Clune, Deep neural net- works are easily fooled: High confidence predictions for unrecognizable images. In IEEE Conference Com- puter Vision and Pattern Recognition, pp. 427-436, 2015.
Sohn, K., Berthelot, D., Carlini, N., Zhang, Z.,, Zhang, H., Raffel, C. A., Cubuk, E. D., Kurakin, A., and Li, C.-L. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In Advances in Neural Infor-
mation Processing Systems, pp. 596-608, 2020.
Pan, P., Swaroop. S., Immer, A., Eschenhagen, R., Turner, R., and Khan, M. E. E. Continual deep learning by func- tional regularisation of memorable past. In Advances in Neural Information Processing Systems, pp. 4453 4464, 2020.
Song, H., Kim, M., and Lee, J.-G. SELFIR: Refurbishing unclean samples for robust deep learning, In International Conference on Machine Learning, pp. 5907-5915, 2019.
Sosea, T. and Caragea, C. Leveraging training dynamics and self-training for text classification.. In Findings of the Association for Computational Linguistics, pp. 4750- 4762, 2022.
Paul, M., Ganguli, S., and Dziugaite, G. K. Deep learning data diet: Finding important examples early in training. In Advances in Neural Information Processing Systems, pp. 20596-20607, 2021.
Su, J.-C. and Maji. S. The semi-supervised inaturalist-aves challenge at fgvc7 workshop. ArXiv, abs/2103.06933 2021.
Pei, H., Li, Y., Deng, H., Hai, J., Wang, P., Ma, J., Tao, J., Xiong. Y., and Guan, X. Multi-track message pass- ing: Tackling oversmoothing and oversquashing in graph learning via preventing heterophily mixing. In Forty-first International Conference Machine Learning, 2024a.
Sun, K., Lin, Z., and Zhu, Z. Multi-stage self-supervised learning for graph convolutional networks on graphs with few labeled nodes. In AAAI Conference on Artificial Intelligence, pp. 5892-58928888 2020.
Pei, H., Xiong, Y., Wang, P., Tao, J., Liu, J., Deng, H., Ma, J., and Guan, X. Memory disagreement: A pseudo-labeling measure from training dynamics for semi-supervised graph learning. In ACM Web Conference, pp. 434 445, 2024b.
Swayamdipta, S., Schwartz, R., Lourie, N., Wang, Y., Ha- jishirzi. H., Smith, N. A., and Choi, Y. Dataset cartog- raphy: Mapping and diagnosing datasets with training dynamics. In Conference on Empirical Methods in Natu- ral Language Processing, pp. 9275-9293, 2020.
Pham, H., Dai, Z.,, Xie, Q., and Le, Q. Meta pseudo labels. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11557-11568, 2021.
Tian, S., Wei, H., Wang, Y., and Feng. L. Crosel: Cross selection of confident pseudo labels for partial-label learn- ing. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19479-19488, 2024.
Pleiss, G., Zhang, T., Elenberg, E., and Weinberger, K. Q. Identifying mislabeled data using the area under the mar- gin ranking. In Advances in Neural Information Process- ing Systems, pp. 17044-17056, 2020.
Toneva, M., Sordoni, A., des Combes, R. T., Trischler, A., Bengio, Y., and Gordon, G. J. An empirical study of example forgetting during deep neural network learning. In International Conference Learning Representations, 2019.
Rizve, M. N., Duarte, K., Rawat, Y. S., and Shah, M. In de- fense of lseudo-labeling: An uncertainty-awant pseudo- label selection framework for semi-supervised learning. In International Conference on Learning Representations, 2021.
Van der Maaten, and Hinton, G. Visualizing data using t-sne. Journal of Machine Learning Research, 9(11): 2579-2605, 2008
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. Grad-cam: Visual explanations from deep networks via gradient-based localization. In IEEE International Conference on Computer Vision, pp.
Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liō, P., and Bengio, Y. Graph attention networks. In International Conference on Learning Representations, 2018.
618 00-6226,22 2017.
Sen, P., Namata, G., Bilgic, M., Getoor, L., Gallagher, B., and Eliassi-Rad T. Collective classification in network data. Al Magazine, 29(3):93-106, 2008
Wang, B., Li, J., Liu, Y., Cheng, J., Rong, Y., Wang, W., and Tsung, Deep insights into noisy pseudo labeling on graph data. In Conference on Neural Information Processing Systems, 2023.
Siddiqui, S. A., Rajkumar, N., Maharaj, T., Krueger, D., and Hooker, S. Metadata archaeology: Unearthing data subsets by leveraging training dynamics. In The Eleventh International Conference on Learning Representations, 2023.

---

Non-Stationary Predictions May Be More Informative
Wang, H., Huang, W. Wu, Z., Tong, H., Margenot, A. J., and He, J. Deep active learning by leveraging training dynamics. In Advances in Neural Information Processing Systems, pp. 25171-251811 2022a.
Xie, Q.. Luong, M.-T., Hovy, E.,, and Le, Q. V. Self-training with noisy student improves imagenet classification. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10687-10698, 2020.
Wang, W. and Zhou, Z.-H. Co-training with insufficient views. In Asian Conference on Machine Learning, pp. 467-482, 2013.
Yang. W. Zhang, R., Chen, J., Wang, L.,, and Kim, J. Prototype-guided pseudo labeling for semi-supervised text classification. In Annual Meeting of the Association for Computational Linguistics, pp. 16369- 16382, 2023.
Wang. Y., Peng. J., and Zhang, Z. Uncertainty pseudo label refinery for domain adaptive semantic segmentation. In IEEE/CVF International Conference on Computer Vi- sion, pp. 9092-9101, 2021.
Zhang, S., Zhang, H., Lou, S., Wang, Z., Zeng, Z., Wang, Y., and Luo, M. Ptcl: Pseudo-label temporal curriculum learning for label-limited dynamic graph. arXiv preprint arXiv:2504 17641, 2025.
Wang, Y., Chen, H., Fan, Y., Sun, W., Tao, R., Hou, W., Wang, R., Yang, I.., Zhou, Z., Guo, L.., Qi, H., Wu, 7 Li, Y., Nakamura, S., Ye, W. Savvides, M., Raj, Shinozaki, T., Schiele, B., Wang, J., Xie, X., and Zhang, Y. USB: A unified semi-supervised learning benchmark for classification. In Advances in Neural Information Processing Systems, 2022b.
Zhao, X., Chen, F., Hu, S., and Cho, J.-H. Uncertainty aware semi-supervised learning on graph data. In Advances in Neural Information Processing Systems, pp. 12827- 12836, 2020.
Zhou, T., Wang, S., and Bilmes, J. Time-consistent self- supervision for semi-supervised learning. In International Conference on Machine Learning, pp. 11523-11532 2020.
Wei, C., Shen, K., Chen, Y., and Ma, T. Theoretical analysis of self-training with deep networks on unlabeled data. In International Conference Learning Representations, 2021.
12

---

Non-Stationary Predictions May Be More Informative
Appendix
200
Bimodal distribution
Unimodal distribution
This appendix organized into following six sections, pre- sented in the order as they are referenced in the main paper. Notably, the Code to replicate the experimental results is available in our GitHub repository.
Number of distributions
Random distribution
100
threshold
In Appendix A, we present details for plotting the Fig.1(A) in the main paper.
In Appendix B, we validate the effectiveness of our proposed LnO-entropy to detect bimodal distribution. In Appendix C, we provide details of the attribution method used in Section 4.2 in the main paper.
0.0
1.0
2.0
LMO-entropy Value
Figure 7. A histogram of LMO entropy values of the generated dis- tribution samples, with distribution types distinguished by colors. The horizontal axis represents the LMO-entropy values, and the vertical axis indicates the number of distribution samples.
In Appendix D, we propose a pseudo-labeling algo- rithm based on the proposed 2-phasic metric.
In Appendix E. we provide details of the experiments in the main paper, including experimental protocols, baselines, datasets, and evaluation metrics.
short), a key component for calculating the spatial measure in 2-phasic metrie. Specifically, we test the ability of LMO entropy distinguish bimodal distributions (corresponding to two-phase labels) from other types of distribution, such as unimodal and random distribution.
In Appendix F, we present supplementary experiments to further analyze the proposed 2-phasic metric.
A. Details for Plotting the Figure 1(A)
We provide details on visualizing the distribution of pre- dicted labels, shown in Fig. in the main paper. Specifically, we first fine-tune a pre-trained Visual Trans- former (ViT) model (Wang et al., 2022b) on the CIFARI00 dataset using 400 labeled samples, achieving an accuracy of 71.30%, while recording the training dynamics of the VIT.
To this end, we synthesize three types of distributions: ran- dom, unimodal, and bimodal distributions. Each of the distribution is discrete and with ten categories. We gener- ale unimodal distributions using a normal distribution with random mean and random variance between 1 and 2. We generate bimodal distributions by applying the softmax function to the combination of two normal distributions, each with random means and random variances between 3 and 5. We generate random distributions by randomly per- turbing uniform distributions. For each distribution type, we generate 1,000 distribution samples, and then calculate and analyze the LMO entropy of the the distribution samples.
In the figure, red points represent labels that are incorrectly predicted in the final model output, while blue points indi- cate correctly predicted labels. Using the recorded training dynamics, we calculate two properties of the predicted la- bels for each unlabeled sample: the confidence score and non-stationarity, which corresponds to the x-axis and y-axis in the figure, respectively. Here, the confidence score is defined as the average predicted probability of the predicted label's category across epochs by
Fig.7 presents the histogram of LMO entropy values for the generated distribution samples. The results clearly show the I.O entropy values of bimodal distributions are sig- nificantly lower than those of other types of distributions. This indicates that a simple threshold of LMO entropy can effectively distinguish bimodal distributions (corresponding to two-phase labels) from other types of distribution.
1 T Σ Po.
Ur (1)
where specifies a epoch, O, denotes the model parameters at epoch f, and y/² denotes the predicted label for the model after training. The non-stationary is defined as the standard deviation of the prediction probability of the predicted la- bels's ' category throughout training,
C. Details of Attribution Method
In Section 4.2 in the main paper, we employed Grad-CAM (Selaaraju et al., 2017) to plot the attribution map Fig.3, i.e., to highlight the key regions in two-phase samples that the first-phase and second-phase models rely on for classifica- tion. Grad-CAM is widely used attribution technique for explaining the attention mechanisms of DNNs.
(
V
-i=
Poo
x
m())
2
We use the non-stationary as an inverse indicator of the training stationary, which is easy to calculate.
B. Validation of LMO Entropy
Grad-CAM begins by computing the gradients of the clas- sification confidence score for a specific category w.r.f. all feature activations of each channel in the final convolutional
We design this experiment to validate the effectiveness of the proposed Leave- Maxieum- Out entropy (LMO entropy for
13

---

Non-Stationary Predictions May Be More Informative
layer. By applying a global average pooling operation on these gradients, it derives the importance weights of each channel (each feature map) for the given category. These weights for different channels are then used to perform a weighted combination of feature map activations, followed by applying a ReLU activation function, to generate the Grad-CAM attribution map. The resulting attribution map highlights the regions of the image that contribute most significantly to the prediction of the target category.
Algorithm 1 2-phasic based Pseudo-label Selection
Input: Labeled samples Dr, unlabeled samples Du. and initial model fo
Output: Pseudo-label samples Dr Parameter: Threshold epoch indexes T I: Initialize model parameters 6, and Dp + Ø
2: for training epoch t = to max epoch do
3: Update using the gradient calculated on Dr.
4: if epoch t E T then
5: for each sample x(1) in Dy do
In experiments, we adopt ViT model as the base model, which is trained on the CIFARI00 dataset with 400 labeled samples provided. We record the models trained after 250 steps (achieving an accuracy of 66.02%) as the first-phase model, and models trained after 6250 steps (achieving an accuracy of 74.55%) as the second-phase model. We extract two-phase image samples along with their corresponding predicted categories. Then, the Grad-CAM method is em- ployed to visualize the critical image regions for classifica- tion of the first-phase and second-phase, respectively.
Compute af via fo and record 7: end for
8: end if
9: end for
10: for each sample (0 in Du do
11: Calculate its 2-phasic metric by using Eq. (1) in the
main paper
12:
Record its bimodal categories bel() = [(11,2)
13: if 2-phasiic < < then
14: ÿ = fo(()))); Dp t DDU bo
15: end if
D. 2-phasic based Pseudo-labeling Algorithm
16: end for
The procedure of the entire pseudo-labeling algorithm is as follows: (1) Base model training: train a base model fo on the labeled dataset Dr, and record the training dynam- ies of the model. (2) Two-phase label generation: Select two-phase samples using the recorded training dynamics and the proposed 2-phasic metric. Predict the two-phase labels based on the trained base model. (3) bbeudo-labeling learning: The two-phase labels used to augment the labeled dataset for further training of the model, utilizing the proposed two-phase pseudo-labeling loss (described in Section 4.3 of the main paper). In the following, we first present the 2-phasic based pseudo-label selection algorithm, detailed in Algorithm 1. We then incorporate this selection algorithm into a pseudo-labeling learning framework, as described in Algorithm 2.
17: Return Dp
Algorithm 2 2-phasic based Pseudo-labeling Learning
Input: Labeled samples Dr, unlabeled samples Du, and initial model fe Output: Predicted labels D = {=x (()))= E Dut
1: D" < Ø
2: for each stage k- = 1 to K do
3: Dr = Algorithm 1(Dr,Du-Jo
4: Dr + Dr Dp: D + D* U UD
5: Update Du according to DL
end for
Train model fo on augmented Dr
8: for each sample (0 E Dy do
9: T(D) ()); D + D U
In Algorithm 1, the base model fo is trained on the labeled data Dr.. Throughout the training, we record the softmax distributions of of predictions at specific epochs (or batch ID in batch training) in the memory bank, T (Line 4-8). Subsequently, the recorded training dynamics are used to calculate 2-phasic metric for every unlabeled sample using Eq. (1) from the main paper (Line 10-11). Finally, unlabeled samples with a 2-phasic value below the threshold T are selected as pseudo-labeles (Line 13-15). The output pseudo-labels and the corresponding bimodal categories Dp then used augment the limited labeled data.
10: end for
11: Return D
datasets and K > 1 for experiments on image datasets. In each stage, pseudo-labels are generated by Algorithm 1 by leveraging currently available labels in Dr (Line 2-3). Then, the labeled data D,. is augmented by these pseudo-labels, and the augmented labeled data serves as the foundation for the next stage of pseudo-labels generation (Line 4-5), After iterating through the K stages, the final model is trained using the ultimately augmented label set D,, (Line 7). The final model is then used to predict labels for all unlabeled samples (Line 8- 10). Notably, we add an equal number of pseudo-labels for each category in every stage so as to avoid the issue of label imbalance.
In Algorithm 2, we present 2-phasic based pseudo-labeling learning method by incorporating Algorithm 1 into multi- stage pseudo-labeling learning framework. This algorithm comprises K stages, with K = 1 for experiments on graph
14

---

Non-Stationary Predictions May Be More Informative
E. Details of Experiments
E.3. The Calculation of Information Gain
E.1. Details of Baselines
In Table 2 of the main paper, we employ information gain to evaluate the quality of pseudo-labels. Specifically, measure the information gain from pscudo-labels by sum- marizing the positive contribution of correct pseudo-labek, and the adversarial impact of incorrect pseudo-labels,
We choose the confidence score and MoDis as baselines in experiments on both graph and image datasets.
Confidence score(Lee et al., 2013): The likelihood or probability assigned the predicted category, which is widely used metric for selecting pseudo-labels.
p(
where p(t) and p أ( denote the positive contribution and the adversarial impact, respectively. Taking positive contri- bution p(+)) as an example, the information gain from the inclusion of correct pseudo-labels is calculated based on model perturbations. Specifically, we quantify the gradient changes induced by the inclusion of correct pseudo-labels
Memory Disagreement (MoDis) (Pei et al., 2024b): By leveraging training dynamics to quantify the predic- tion uncertainty, which favors unlabeled samples with higher prediction consistency during training.
In experiments on image datasets, we additionally choose the following two basclines:
(Drs x++) = - :.)):))) -IVE (o))
Uncertainty-aware Pseudo-label Selection (UPS) (Wang et al., 2021): UPS utilizes prediction uncer- tainty to reduce noise from poorly calibrated models, there mitigating overconfidence in pseudo-labeling.
where Ve(.) denotes the model's gradient, D2 represents the labeled samples, and D is the set of correct pseudo- labels. Here ||||| signifies the Frobenius norm. To calculate adversarial impact pس-), we replace the set Dft) with D/-;, which denotes the set of incorrect pseudo-labels.
SoftMatch (Chen et al., 2023): SoftMatch proposes a uniform alignment approach that weights samples based their confidence levels, thereby enhancing the utilization of weak learners.
In Fig. we compare the information gain of Type and Type 2. The information gain caculated based on sam- ples within each type and divided by the mean information gain of all samples. In the complementary analysis in 5.2, we compute the average information gain per sample by dividing the information gain of the pseudo-label set by the number of samples in the set.
In experiments graph datasets, additionally choose the two following two baselines:
Area Under the Margin (AUM) (Sosea & Caragea, 2022): AUM captures the divergence between the logit of annotated labels and predicted labels in training.
Distribution Recovered Graph Self- Self-Training (DR-GST) (Liu et al., 2022a): DR-GST introduces distribu- tional correction mechanism into confidence based self-training on graphs.
E.4. Details of Booster Test
The booster experiment (Section 5.1 in the main paper) is structured into three stages, as illustrated in Fig.5 in the main paper. We conducted the experiment on both graph and image datasets. For image classification, we selected CI- FAR 100, EuroSAT, 100 and Semi-Aves datasets, with detailed descriptions in Appendix E. We utilized the pre- trained ViT model provided in the (Wang et al., 2022b) as the backbone. In the first phase, we directly loaded the pre-trained checkpoint of the ViT model. During the second phase, in accordance with the settings from the USB, we set the batch size to 8 and employed a baseline algorithm to select pseudo-labels for training while recording the train- ing dynamics within batches. A total of 4 epochs were conducted in the stage 2. In stage 3, the experiment was divided into two trials. We use pseudo-labels selected by the 2-phasic metric and the baseline to train the model, respec- tively, and report the final experimental results of both trials. In the bascline trial, the model continued to use the bascline algorithm to select pseudo-labela for model training. In the two- two-phase trial, we leveraged the training dynamics ob- tained from stage 2 to select two-phase pseudo-labels using 2-phasic according to Algorithm 1.
E.2. Details of Datasets
Our experiments are conducted on eight benchmark datasets, including four graph datasets, specifically Cora (McCal- lum et al., 2000), Citeseer (Sen et al., 2008), PubMed (Na- mata et al., 2012), and AmazonComputer (McAuley et al., 2015), and four image datasets, specifically CIFAR100 (Krizhevsky, 2012), EuroSAT (Helber et al., 2019), STL. 10 (Coates et al., 201 1), and Semi-Aves(Su & Maji, 2021). Statistics of image and graph datasets are summarized in Table 4 and Tables 5, respectively.
In experiments on graph datasets, we follow the setup in (Liu et al., 2022a; Pei et al., 2024b), use only 3, 5, and 10 labeled nodes per class (L/C), which presents challenging scenarios with minimal label information. In experiments on image datasets, we use different configurations, as shown in Table 5, according to existing works (Wang et al., 2022b).
In experiments, we employ Optuna (Akiba et al., 2019) to searche parameters in 2-phasic metric, with the parameter search ranges detailed in the Table 6.
15

---

Non-Stationary Predictions May Be More Informative
Table Statisties of graph datasets.
Graph dataset
# Nodes
Edges
Categories
Features
# Labels used training pre category
Cora
2,708
5,429
1,433
375710 10
Citeseer
3,327
4,732
6
3,703
3 5 10
PubMed
19,717
44,338
3
500
3/
AmazanComputer
13,752
245,778
767
3/5/
Table 5. Statistics of image datasets.
Image dataset
# Images in training set
# Images in test set
# Categories
Labels used for training pre category
CIFAR-IOU
50,000
10,000
100
1/2/4
STL-10
5,000
8,000
10
4/10 10
EuroSat
16,200
5,400
10
2 2/4
Semi-Aves
3,959
4,000
200
15
Table Parameter search setup. We use Optuna (Akiba et al., 2019) to search for parameters within the following ranges.
Table 8. The node classification accuracy of pseudo-labeling algo- rithms graphs (%) with the backbone GAT. L/C represents the number labeled nodes per category.
Parameters
Range for graphs
Range images
(0, 0.1)
(0, 0.1)
Graph Data
Cora
Citeseer
6-66
(0.1, 2)
(0.1, 2)
L/C
5
10
5
10
7/1
(0,
(0, 1)
Confidence
67.04
75.12
75.37
61.16
65.33
67.86
1/2
depends on yoi and y02
+2- +2-phase
70.72
77.00
78.53
62.56
67.48
68.63
(P1
(0, I)
(0, 1)
AUM
68.99
73.14
76.00
59.38
63.45
66.83
(0, 1)
(0, 1)
+2-phase
69.70
77.21
79.44
59.70
66.60
68.65
5c-3 for Amazon
(Ie-4, le-3)
MoDis
72.67
74.21
75.13
63.09
66.98
67.27
(Ic-5, 1e-3) for others
+2-phase
74.42
76.06
77.22
64.10
68.37
68.66
Mean of 2-phasic all the unlabeled
(0.8, 1.8)
DR-GST
73.58
79.42
82.35
52.47
68.13
72.91
+2-phase
74.28
80.13
83.01
56.86
70.84
73.39
Graph Data
Pubmed
Amazon
Table Classification accuracy of pseudo-labeling learning (%).
L/C
3
5
10
3
5
10
Image Data
CIFAR100
Confidence
63.06
64.46
68.
75.66
77.09
80.81
Labels
200
400
+2-phase
63.43
70.08
72.16
75.75
78.63
81.56
Confidence
66.84
75.38
AUM
60.48
65,27
68.01
74.52
77.34
80.29
+2-phase
68.22
77.98
+2-phase
63.92
69.55
72.16
75.20
78.19
80.73
MoDis
65.37
69.22
71.80
74.70
77.18
81.43
According to the experimental results in (Wang al., 2022b), the baseline models are convergent around the tenth epoch. To speed up our comparison experiments, we did not train for 200 epochs as the seeting in (Wang et al., 2022b); Instead, we run epochs in stage 3, during which models prove the conver-
+2-phase
66.00
69.67
73.03
77.06
78.20
82.38
DR-GST
66.51
73.93
76.48
78.96
80.56
82,66
+2-phase
67.89
74.26
76.84
81.54
81.78
83.02
ViT trained on the CIFAR100 dataset, Experiments are conducted with 100, 200, and 400 pseudo-labels.
both trials successfully converged. gence, we train model for 200 epochs using one baseline. The results are reported in Table where the perfoermance very closed to the results in Table I in the main paper.
F. Supplementary Experiments
F.1. Additional Booster Test
E.5. Details of Ablation Study
We also use the widely used graph learning model, Graph Attention Networks (GAT) (Veličković et al., 2018) as an additional base model for booster test. All experimental configurations are the same as the experiment booster test, described in the main paper and Appendix E.4.
We conduct ablation study evaluate the contribution of each component in the loss functions in Section 4.3 of the main paper, ie., Ci and C2 terms. The loss is defined as:
C
- log 1
As shown in Table 8, the two-phase labels generated by the 2-phasic metric consistently improve classification ac- curacy across four graph datasets and four baselines, with average enhancement of 1.76%. These results further demonstrate that adding two-phase labels boosts the perfor- mance of existing aseudo-labeling methods, regardless of the base model used, highlighting the broad applicability of our proposed 2-phasic metric.
where ( denotes the two-phase pseudo-Label, (0) fo(x(()) is the model's prediction probability. The loss with both and 02 terms are the same as the loss functions in the main paper. For graph data, we select the GCN model trained on the Cora dataset, as the base model. Our experi- ments are performed on three partitions of Cora, specifically, L/C 3, 5, and 10. For image data, the base model
16

---

Non-Stationary Predictions May Be More Informative
Table Classification accuracy of pseudo-labeling learning (%) in component ablation study. "Temporal only" denotes that only Ptempooaal is used the metric and "Spatial only" denotes that only Hapatial is used. "2-phasic" represents the complete metric.
Dataset
Cora
L/C
3
10
Temporal only
69.76
73.40
75.99
Spatial only
68.24
72.70
73.87
2-phasic
70.16
74.07
77.17
F.2. Ablation Study on 2-phasic Metric
To further analyze the contributions of the spatial measure and the temporal measure in the 2-phasic metric, we con- ducted an ablation study. Specifically, we use GCN the base model on Cora dataset with L/C=3, 5, and 10. In the booster test experiments, we employ confidence pseudo-labeling metric in stage 2, and use either (temporal or Mapatial alone as pseudo-labeling metric in stage 3. The experimental results are summarized in Table 9.
The results demonstrate that both temporal and spatial com- ponents contribute positively the overall performance, with the temporal features having more significant impact.
17


Reviews:
[{"review_id": "RwZJ5AlTrF", "reviewer": "Reviewer_xfQV", "summary": "This paper introduces a novel type of pseudo-labels that hold significant potential for enhancing pseudo-labeling strategies and complementing existing methods. The authors further propose a metric to efficiently identify these two-phase labels. Extensive experiments on eight datasets demonstrate that the 2-phasic metric significantly boosts the performance of existing pseudo-labeling methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your detailed and constructive comments! We have carefully studied them and revised the paper accordingly. \n\n**Q1**: *The paper identifies limitations, including parameter sensitivity and computational overhead. Could you discuss any ongoing or planned future work to address these issues?*\n\nResponse: \nThank you for raising these important points. Below is our response to these limitations.\n\n1. Hyperparameter determination\n\nAs shown in Figure 6, although the model’s performance exhibits fluctuations with certain hyperparameter variations, the overall performance remains relatively stable. This indicates that some hyperparameters have low sensitivity and that there may be correlations among them. Therefore, in future work, we plan to reduce the number of hyperparameters by employing hyperparameter fusion strategies, aiming to simplify the deployment of the 2-phasic metric while retaining its performance advantages.\n\n2. Computational Overhead\n\nCalculating the 2-phasic metric requires recording training dynamics but does not introduce significant computational overhead. Specifically, it introduces an additional space complexity of O(N|T|) and time complexity of O(NC|T|), where N is the number of unlabeled samples, C is the class count, and |T| is the number of recorded training dynamics. \nEmpirical validation (e.g., C=100 and |T|=50 for CIFAR-100) shows that this complexity is not too high in real-world datasets.\n\nMoreover, the complexity limitations can be effectively mitigated through two strategies:\n\n1. **Parallelization**: The recording of training dynamics and the computation of the 2-phasic metric are decoupled from the backpropagation process of the neural network. This allows for parallel execution, effectively hiding the additional computational latency within the training loop.\n\n2. **Efficient Sampling for Memory Bank**: To further optimize memory usage, we propose an adaptive sampling strategy for memory bank construction. Rather than storing the training dynamics at all epochs, we selectively retain a representative subset by an adaptive sampling strategy based on the recorded training dynamics.\n\n**Q2**: *Have you ever considered evaluating 2-phasic metric across various types of neural networks, such as RNNs and Transformers?*\n\nResponse: The principle of the 2-phasic metric is to capture the transition of neural network learning patterns from simple to complex. The universal law of neural networks learning from easy to hard has been well-established [1][2], endowing the 2-phasic metric with broad applicability across various neural network architectures. In our experiments, we used GCN and ViT as backbones for node classification and image classification tasks, respectively. The ViT represents a Transformer-based architecture. In the appendix, we additionally validated the effectiveness of the 2-phasic metric using GAT as the backbone for node classification. Moving forward, we plan to apply the 2-phasic metric to diverse data types and additional backbone architectures to further verify its generalizability.\n\n[1] Arpit, D., Jastrzebski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al. A closer look at memorization in deep networks. In International Conference on Machine Learning, pp. 233–242, 2017. \n\n[2] Siddiqui, S. A., Rajkumar, N., Maharaj, T., Krueger, D., and Hooker, S. Metadata archaeology: Unearthing data subsets by leveraging training dynamics. In The Eleventh International Conference on Learning Representations,2023.\n\n**Other Comments Or Suggestions**: \n\n1. Ensure figure references are accurate and well-positioned.\n\n2. Enhance captions and explanations for figures and tables to better guide readers\n\nThank you for your valuable suggestions. In response, we have made the following revisions:\n- We carefully reviewed all figure and table references, including the main paper and the Appendix.\n- We examined the captions of all figures and tables and revised some of them, such as the caption of Table 2.", "claims_and_evidence": "NA", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are appropriate for the problem of enhancing pseudo-labeling in semi-supervised learning. The 2-phasic metric and loss function effectively capture the unique characteristics of two-phase labels. The criteria used in the paper are standard and relevant, ensuring applicable results.", "theoretical_claims": "The paper supports its claims through experimental results and qualitative reasoning, rather than formal proofs. It demonstrates that incorporating two-phase labels improves model performance across multiple datasets and explains their effectiveness by their proximity to decision boundaries and ability to capture complex patterns.", "experimental_designs_or_analyses": "The experimental designs and analyses in the paper are sound and valid. It compares the 2-phasic metric against various baselines across diverse datasets, demonstrating notable performance gains. It also includes ablation studies to validate the contributions of different components. Overall, the experiments effectively substantiate the proposed method’s efficacy.", "relation_to_broader_scientific_literature": "The paper’s core contributions are deeply connected to the broader fields of semi-supervised learning and training dynamics.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths\n1. The paper introduces novel two-phase labels and a tailored metric and loss function to enhance semi-supervised learning.\n2. Extensive experiments show significant performance gains, especially with limited labeled data.\n3. The approach is practical, easily integrated into existing frameworks and effective in scenarios with limited labeled data.\n\nWeaknesses\n1. Requires careful parameter adjustment, which can be time-consuming.\n2. Generalizability to other domains (e.g., text, time-series) is untested.", "other_comments_or_suggestions": "1. Add equation numbers for all equations to enhance clarity and referencing.\n2. Ensure figure references are accurate and well-positioned.\n3. Enhance captions and explanations for figures and tables to better guide readers.", "questions_for_authors": "1. The paper identifies limitations, including parameter sensitivity and computational overhead. Could you discuss any ongoing or planned future work to address these issues?\n2. Have you ever considered evaluating two-phasic metrics across various types of neural networks, such as RNNs and Transformers?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hla0wEmiTT", "reviewer": "Reviewer_4ygb", "summary": "This paper discovers a new type of predicted labels suitable for pseudo-labeling, termed two-phase labels, which exhibit a two-phase pattern during training and are informative for decision boundaries.  This finding is different from existing methods which typically select predicted labels with high confidence scores and high training stationarity as pseudo-labels to augment training sets, and thus offers new insights for pseudo-labeling. Besides, this paper proposes a 2-phasic metric to mine the two-phase labels, and a loss function tailored for two-phase pseudo-labeling learning, allowing models not only to learn correct correlations but also to eliminate false ones.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thorough and constructive feedback! Below, we present our responses to each of your concerns and questions.\n\n**Weaknesses**: *The method introduces hyperparameters that must be carefully selected to ensure the effectiveness of the proposed approach.*\n\n**Response:** \nWe agree with you that the sensitivity of hyperparameter in 2-phasic metric is crucial for its practical applicability. As shown in Figure 6, although the model’s performance exhibits fluctuations with certain hyperparameter variations, the overall performance remains relatively stable. This indicates that some hyperparameters have low sensitivity and that there may be correlations among them. Therefore, in future work, we plan to reduce the number of hyperparameters by employing hyperparameter fusion strategies, aiming to simplify the deployment of the 2-phasic metric while retaining its performance advantages.\n\n**Other Comments Or Suggestions**: *A more detailed discussion on the theoretical guarantees of the proposed metric would enhance its reliability for practical use.*\n\n**Response:**\nWe appreciate your insightful feedback. Due to limited time during the rebuttal, we propose a preliminary theoretical idea as a potential direction for addressing this concern.\n\nOur idea is to use the local elasticity hypothesis to prove that adding 2-phase labels can promote feature separability. \nLocal elasticity is introduced: when the model is updated at sample $x$ via SGD, the predicted change  $|f(x', w^+) - f(x', w)|$ of another feature vector $x'$ is positively correlated with the similarity between $x$ and  $x'$ [1]. \n\nReference [2] utilizes the local elasticity assumption to derive conditions for feature separability. \nThis study describes the temporal evolution of features for two classes of samples, denoted as $X_{i}^1(t)$ and $X_{j}^2(t)$.\nFurthermore, Theorem 2.1 in [2] indicates that:\n> Given the feature vectors $X_{i}^1​(t)$,$X_{j}^2​(t)$ for  $i, j \\in [n]$, as $t \\to \\infty$ and large $n$,  \n> 1. if $\\alpha>\\beta$, they are asymptotically separable with probability tending to one,  \n> 2. if $\\alpha\\leq\\beta$, they are asymptotically separable with probability tending to zero.\n\nAs illustrated in Figure 4, our 2-phasic samples tend to lie closer to the decision boundary.\nIntuitively, incorporating these samples may help reduce the inter-class effect $\\beta$, thereby promoting stricter feature separability.\nIn the future, we plan to study the local elasticity about 2-phase samples, estimate the intra-class effect $\\alpha$ and inter-class effect $\\beta$ before and after adding 2-phase samples. This analysis will allow us to verify our hypothesis and provide a theoretical foundation for the effectiveness of our method.\n\n[1] The local elasticity of neural networks. ICLR, 2020.\n\n[2] Imitating deep learning dynamics via locally elastic stochastic differential equations. NeurIPS, 2021.\n\n\n**Q1**: *In pseudo-labeling tasks, what metrics have been proposed by existing works to identify effective samples?*\n\n**Response:** \nPseudo-label selection metrics mainly fall into two categories: confidence-based and uncertainty-based. Confidence-based methods typically use the maximum softmax score as the selection criterion [1]. Enhancements include FlexMatch [3], which applies dynamic class-wise thresholds for more flexible filtering, and SoftMatch [4], which weights samples by confidence to balance pseudo-label quality and quantity.\n\nUncertainty-based metrics prioritize samples with low prediction uncertainty. A representative approach is Monte Carlo dropout [5], which estimates uncertainty via the variance of predictions under multiple dropout runs. Another method leverages training stationarity (or time consistency), selecting labels with stable predictions over time [7]. Additionally, recent work proposes learning a metric via a neural network that models the relationship between label embeddings and feature embeddings to assess pseudo-label quality [8].\n\n[1] Curriculum labeling: Revisiting pseudo-labeling for semi-supervised learning. AAAI, 2021.\n\n[2] Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. NeurIPS, 2021.\n\n[3] Softmatch: Addressing the quantity-quality tradeoff in semi-supervised learning. ICLR, 2022.\n\n[4] Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. ICLR, 2016.\n\n[5] Temporal self-ensembling teacher for semi-supervised object detection. IEEE Transactions on Multimedia, 2021.\n\n[6] Semireward: A general reward model for semi-supervised learning. ICLR, 2024.\n\n**Q2**: *How do you construct an effective memory bank to support the calculation of the proposed metric?*\n\n**Response:**   \nIn our experiments, we adopted a uniform sampling strategy to construct the memory bank. Specifically, model predictions were sampled and stored at fixed training steps. With only 50 recorded training dynamics, we achieved satisfactory accuracy in experiments on image datasets.", "claims_and_evidence": "The authors claim that they discover a new type of predicted labels suitable for pseudo-labeling. First, they analyze the rationale of two-phase labels for pseudo-labeling from different perspectives. Second, extensive experiments were conducted on eight benchmark datasets, including image and graph datasets, and the results show that the use of two-stage labeling can significantly improve the performance of existing pseudo-labeling methods. Specifically, the average classification accuracy of image datasets and graph datasets increased by 1.73% and 1.92%.", "methods_and_evaluation_criteria": "The proposed methods make sense for this task. Experiments are performed on CIFAR100, EuroSAT, STL-10,Semi-Aves, Cora, Citeseer, PubMed, AmazonComputers.  The evaluation citeria contains classification accuracy, correctness,  Information gain, overlap of the two pseudo-labels, which are commonly used benchmarks.", "theoretical_claims": "The authors provide a theoretical rationale for the two-phase labels, and I find no apparent issues with their analysis.", "experimental_designs_or_analyses": "Experiments are performed on the dataset of CIFAR100, EuroSAT, STL-10, Semi-Aves, Cora, Citeseer, PubMed, and AmazonComputers, and clearly verify the effectiveness of the proposed methods.", "relation_to_broader_scientific_literature": "The newly identified type of predicted labels, characterized by a two-phase pattern during training, is not only well-suited for pseudo-labeling and highly informative for decision boundaries, but also provides novel insights for other related learning tasks.", "essential_references_not_discussed": "Key related works are all discussed.", "other_strengths_and_weaknesses": "Strengths: This paper is a well-written and interesting paper. It introduces novel insights into two-phase labels for pseudo-labeling, proposes a two-phase metric to mine these labels, and presents a loss function specifically designed for two-phase pseudo-labeling learning. These new insights have the potential to make a significant impact on the community.\n\nWeaknesses: The method introduces hyperparameters that must be carefully selected to ensure the effectiveness of the proposed approach.", "other_comments_or_suggestions": "A more detailed discussion on the theoretical guarantees of the proposed metric would enhance its reliability for practical use.", "questions_for_authors": "1) In pseudo-labeling tasks, what metrics have been proposed by existing works to identify effective samples?\n2) How do you construct an effective memory bank to support the calculation of the proposed metric?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "miQY0Lqs7A", "reviewer": "Reviewer_JQGf", "summary": "The paper proposes a novel and interesting 2-phasic metric for two-phase pseudo-label learning. The 2-phasic metric characterizes the two-phase pattern through both spatial and temporal measures. Extensive experimental results show the effectiveness of the proposed 2-phasic metric, especially when the number of labeled samples are very small.\n## update after rebuttal\nThanks for the rebuttal!  Most of my concerns have been addressed. I keep my score unchanged.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your constructive comments! We have carefully studied them and revised the paper accordingly.\n\n**Weakness 1**: *More theoretical analysis behind the observations should be presented, which could make the paper more convincing.*\n\n**Response:**\nWe appreciate your insightful feedback. Due to limited time during the rebuttal, we would like to propose a preliminary theoretical idea as a potential direction for addressing this concern.\n\nOur idea is to use the local elasticity hypothesis to prove that adding 2-phase labels can promote feature separability. \nLocal elasticity is the core concept introduced in [1], and it is defined as follows: when the model is updated at sample $x$ via SGD, the predicted change  $|f(x', w^+) - f(x', w)|$ of another feature vector $x'$ is positively correlated with the similarity between $x$ and  $x'$. \nSpecifically, local elasticity describes a phenomenon that if $x$ and $x'$ belong to the same class (similar), the predicted change is significant; if they belong to different classes (dissimilar), the change is small. \n\nReference [2] utilizes the local elasticity assumption to derive conditions for feature separability. \nThis study describes the temporal evolution of features for two classes of samples, denoted as $X_{i}^1(t)$ and $X_{j}^2(t)$, in a binary classification context.  Furthermore, Theorem 2.1 in [2] indicates that:\n> Given the feature vectors $X_{i}^1​(t)$ and $X_{j}^2​(t)$ for  $i, j \\in [n]$, as $t \\to \\infty$ and large $n$,  \n> 1. if $\\alpha>\\beta$, they are asymptotically separable with probability tending to one,  \n> 2. if $\\alpha\\leq\\beta$, they are asymptotically separable with probability tending to zero.\n\nOur two-phasic metric is designed to identify samples that exhibit a two-stage characteristic in training dynamics. As illustrated in Figure 4, the 2-phasic samples tend to lie closer to the decision boundary.\nIntuitively, incorporating these samples may help reduce the inter-class effect $\\beta$, thereby promoting stricter feature separability.\nIn the future, we plan to study the local elasticity about 2-phase samples, estimate the intra-class effect $\\alpha$ and inter-class effect $\\beta$ before and after adding 2-phase samples. This analysis will allow us to verify our hypothesis and provide a theoretical foundation for the effectiveness of our method.\n\n[1] He H, Su W J. The local elasticity of neural networks. In International Conference on Learning Representations, 2020.\n\n[2] Zhang J, Wang H, Su W. Imitating deep learning dynamics via locally elastic stochastic differential equations. Advances in Neural Information Processing Systems, 2021.\n\n**Weakness 2**: *The additional overhead during the computation of spatial and temporal measures makes the proposed method less efficient, especially when the size of dataset is large. The authors could analyze the computational efficiency of the method from an experimental view.*\n\n**Response:**\nThank you for pointing out this concern. We have conducted an empirical analysis of the computational efficiency of our method. Specifically, on the Cora dataset using GCN as the backbone, the measured time costs are as follows:\n- Training dynamics recording time: 0.11625 seconds\n- Two-phasic metric computation time: 0.00253 seconds\n- Total training time: 2.24 seconds\n\nThese results indicate that the combined overhead of recording training dynamics and computing the two-phasic metric accounts for approximately 5% of the total training time. Therefore, we consider this additional computational cost to be acceptable in practice.\n\n**Weakness 3**: *The authors could perform ablation study regarding the design 2-phasic metric and analyze effectiveness of the spatial measure and the temporal measur.*\n\n**Response:**\nThank you for the valuable suggestion. In response, we conducted an ablation study using GCN as the backbone on the Cora dataset to evaluate the contributions of the temporal and spatial characteristics in the 2-phasic metric. The experimental results are summarized in Table A. The results demonstrate that both the temporal and spatial components contribute positively to the overall performance, with the temporal features having a more significant impact. We will include the results and corresponding discussion in the final revised version of the paper.\n\n**Table A: Ablation study on the temporal and spatial characteristics of the two-phasic metric**\n| L/C           | 3     | 5     | 10    |\n| --- | ---- | ---- | ---- |\n| Temporal only | 69.76 | 73.36 | 75.99 |\n| Spatial only  | 68.24 | 72.70 | 73.87 |\n| 2-phasic metric       | 70.27 | 73.40 | 76.24 |\n\n**Other Comments Or Suggestions**: *In the equation of Line 241, there is a typo in the average change of other categories.*\n\n**Response:** \nThank you for pointing out the error. We have corrected $\\neg \\Delta_{b i}^{(i)}$ to $\\Delta_{\\neg b i}^{(i)}$ in the formula on Line 241.", "claims_and_evidence": "The proposed 2-phasic metric is novel for pseudo-label learning.\n\nThe motivation should be further explained in detail. The authors claim that the two-sample samples are important and have more information gain. But the author does not explain why the two-samples are important. \n\nThe authors give many observations to demonstrate the importance of the two-samples and the training dynamics that used to design spatial and temporal measures. I am curious that the observations are general for all kinds of datasets. Moreover, more theoretical analysis behind the observations should be presented, which could make the paper more solid.", "methods_and_evaluation_criteria": "The authors conduct extensive experiments to validate the effectiveness of the propose method. The experimental design is comprehensive.", "theoretical_claims": "The theoretical claims are reasonable. However, more theoretical analysis behind the proposed observations in this paper should be presented.", "experimental_designs_or_analyses": "The authors conduct comprehensive and solid experiments from multiple perspectives, including booster test, complementary analysis, ablation study, and hyperparameter analysis. The experimental results reveal the effectiveness of the proposed method and each component.", "relation_to_broader_scientific_literature": "The paper addresses the pseudo-label learning problem from a new perspective. In contrast to existing works that focus on samples exhibiting high confidence score and high training stationary, this work exploits the potential the two-phase samples. Moreover, the paper designs a 2-phasic metric to identify the two-phase samples with both spatial and temporal measures.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n\n1. The paper is very well written and easy to understand.\n\n2. The proposed 2-phasic metric is novel.\n\n3. The experiment design is comprehensive and experimental results show the effectiveness of the proposed method.\n\n\nWeaknesses:\n\n1. More theoretical analysis behind the observations should be presented, which could make the paper more convincing. \n\n2. The additional overhead during the computation of spatial and temporal measures makes the proposed method less efficient, especially when the size of dataset is large. The authors could analyze the computational efficiency of the method from an experimental view.\n\n3. The authors could perform ablation study regarding the design 2-phasic metric and analyze effectiveness of the spatial measure and the temporal measur", "other_comments_or_suggestions": "N/A", "questions_for_authors": "In the equation of Line 241, there is a typo in the the average change of other categories.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ws8Zso1Z7w", "reviewer": "Reviewer_on6i", "summary": "This paper investigates the potential of a novel type of pseudo-labels—two-phase labels—in semi-supervised learning. Unlike conventional methods that rely on high-confidence and stable predictions, two-phase labels exhibit relatively low correctness and demonstrate a unique two-phase training dynamic: they initially predict one category in the early training stages and switch to another in later epochs. The authors demonstrate that these labels are highly informative for decision boundaries. To effectively identify such labels, they propose a 2-phasic metric for their quantitative characterization. Furthermore, a loss function is designed to enable models to learn correct correlations while simultaneously eliminating false ones. Extensive experiments are conducted to validate the effectiveness of incorporating two-phase labels.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thorough and constructive feedback! Below, we present our responses to each of your concerns and questions.\n\n**Weakness 1**: *In the Booster test experiments, the authors adopted a three-stage protocol. This design is not commonly seen in general pseudo-labeling approaches. The authors should provide a detailed analysis of this hyper-parameter setting to validate its robustness.*\n\n**Response:**\nThe three-stage training protocol was designed to validate whether the proposed two-phase labels can effectively enhance existing pseudo-labeling methods. It is worth emphasizing that the primary contribution of this paper lies in the identification of the two-phase labels, which serve as a complementary enhancement to baseline pseudo-labels.\nDuring Stage 2, we use the baseline pseudo-labeling method while recording training dynamics. These recorded dynamics were subsequently utilized to calculate the 2-phasic metric used to identify pseudo labels in Stage 3. \n\nWe determine the length of Stage 2 based on two robust principles:\n1. In graph datasets, the transition from Stage 2 to Stage 3 is determined by model convergence criteria. Specifically, Stage 2 concludes when additional training iterations using the baseline pseudo-labeling method no longer yield performance improvements, at which point Stage 3 is initiated.\n2. In image datasets, a fixed schedule is adopted, where Stage 3 begins after recording four epochs of training dynamics in Stage 2, followed by six training epochs in Stage 3.\n\n**Weakness 2**: *As shown in Figure 4, on the Cora dataset, two-phase labels are significantly closer to the decision boundaries compared to high-confidence labels. However, this phenomenon is not as evident in the CIFAR-100 image dataset. The authors should further analyze and clarify the reasons behind this discrepancy.*\n\n**Response:** \nThank you for highlighting this critical observation. We find that this phenomenon is primarily attributed to the use of pre-trained models in image datasets. Specifically, pre-trained models learn diverse feature representations from large-scale pre-training datasets, which may cause samples from the same class to be scattered across multiple clusters in the latent space. \nThis dispersion effect reduces the observable distinction between high-confidence pseudo-labels and the decision boundaries.\n\nTo validate this point, we conducted an additional controlled experiment using a ResNet-18 model trained on CIFAR-100, replacing the original pre-trained Vision Transformer (ViT) backbone. This modification eliminates the representational biases introduced by pre-training. Our empirical results demonstrate that, under this setting, high-confidence pseudo-labels exhibit clearer separation from the decision boundaries. We will include these findings and their analysis in the camera-ready version.\n\n\n**Other Comments Or Suggestions**: *The SoftMatch method achieves a balance between the quality and quantity of pseudo-labeled samples by weighting them according to their confidence, leading to significant performance improvements in pseudo-labeling tasks. Could this idea also be applied to the selection of two-phase labels to further enhance their effectiveness?*\n\n**Response:**\nThank you for this insightful suggestion. The SoftMatch method effectively balances pseudo-label quality and quantity by weighting samples based on their confidence scores. Inspired by this motivation, we adopt a similar strategy in our approach.\n\nSpecifically, we first calculate a normalized 2-phasic metric to the [0,1] range by $\\phi=Normalize(-1*2-phasic)$.\nWe then calculated the mean $\\hat{\\mu}$ and variance $\\hat{\\sigma}$ of $\\phi$ and assigned loss weights $\\lambda$ to samples using the formula in SoftMatch\n\n$$\\\\begin{equation} \\lambda(\\mathbf{p})= \\\\begin{cases} \\phi(\\mathbf{p})* \\exp\\left(-\\frac{(\\phi(\\mathbf{p}) - \\hat{\\mu})^2}{2\\hat{\\sigma}^2}\\right), & \\text{if } \\phi(\\mathbf{p}) < \\hat{\\mu},  \\\\\\ \\phi(\\mathbf{p}), & \\text{otherwise}. \\\\end{cases} \\\\end{equation}$$\n\nAs shown in Table A, experimental results show that integrating the SoftMatch weight into our 2-phasic metric led to a slight decrease in classification accuracy. We are conducting further analysis to investigate the underlying causes of this phenomenon.\n\n**Table A: Applying SoftMatch-inspired weighting to our 2-phasic metric**\n\n| L/C                | 3     | 5     | 10    |\n| --- | ----- | ----- | ----- |\n| Confidence         | 66.21 | 71.38 | 73.73 |\n| +2-phasic           | 70.41 | 74.07 | 77.17 |\n| +2-phasic & Softmatch | 67.69 | 73.78 | 76.80 |", "claims_and_evidence": "The claims made in the submission are indeed supported by clear and convincing evidence. First, the authors analyze the rationale of why two-phase labels are effective from both perspectives of pattern learning and decision boundaries. Furthermore, extensive experiments demonstrate that two-phase labels serve as an effective booster for existing pseudo-labeling methods. These experiments cover a wide range of datasets and various state-of-the-art (SOTA) baseline pseudo-labeling methods. Besides, the authors demonstrate that two-phase labels are high-quality pseudo-labels, which are often overlooked by existing pseudo-labeling methods.", "methods_and_evaluation_criteria": "Methods and evaluation criteria are well-suited for the problem and application at hand, including eight image and graph datasets (e.g., Cora and CIFAR-100), the baseline pseudo-labeling methods for comparison (e.g., Confidence and SOTA SoftMatch), and commonly used evaluation metrics (such as pseudo-labeling accuracy and IOU).", "theoretical_claims": "This paper does not propose explicit theoretical claims. The in-depth analysis of the rationale provides valuable insights, helping readers better understand why two-phase labels are well-suited for pseudo-labeling tasks.", "experimental_designs_or_analyses": "The experimental setup and analyses appear to be well-structured and appropriate for assessing the claims made, such as booster test (Table 1), complementary analysis (Table 2),  ablation Study (Table 3), and parameter Sensitivity (Figure 6).", "relation_to_broader_scientific_literature": "This paper focuses on the research problem of pseudo-label selection in semi-supervised learning. Unlike previous works that primarily emphasize the role of high-confidence, high training stationary labels (Type I labels in Figure 1) in pseudo-labeling, this study explores the potential of a novel type of predicted labels (i.e., Type II labels in Figure 1), which exhibit relatively high accuracy but low training stationary. This exploration provides new and interesting insights. \nFurthermore, this paper further identifies a subset within Type II labels, termed two-phase labels, which not only provide significant information gain but also mitigate the risk of misclassification.", "essential_references_not_discussed": "The key contribution is a novel pseudo-label selection method. The literature discussed by the authors is comprehensive and highly relevant to the topic.", "other_strengths_and_weaknesses": "Strengths: \n1.This paper is well-written and highly engaging.\n2.The paper conducts an insightful exploration, investigating the potential of a type of predicted labels in pseudo-labeling—specifically, labels that exhibit relatively high accuracy but low training stability, which have often been overlooked in previous works. This offers valuable inspiration for future research.\n3.This paper uncovers two-phase labels. Extensive experiments demonstrate that incorporating two-phase labels can significantly improve the accuracy of existing pseudo-labeling algorithms (1.73% on image datasets and 1.92% on graph datasets). This indicates that two-phase labels can serve as a valuable complement to pseudo-labels provided by existing methods.\n4.The authors analyze the rationale behind the effectiveness of two-phase labels from both perspectives of pattern learning and decision boundaries. \n\nWeakness: \n1.In the Booster test experiments, the authors adopted a three-stage protocol: in the early stage, pseudo-labels were generated using the baseline method, while in the later stage, pseudo-labels were generated using both the baseline method and the proposed method for comparison. This design is not commonly seen in general pseudo-labeling approaches. However, the authors did not clearly specify how the training epochs were divided into early and later stages, nor did they analyze the impact of this division on the experimental results. Therefore, the authors should provide a detailed analysis of this hyper-parameter setting to validate its robustness.\n2.As shown in Figure 4, on the Cora dataset, two-phase labels are significantly closer to the decision boundaries compared to high-confidence labels. However, this phenomenon is not as evident on the CIFAR-100 image dataset. The authors should further analyze and clarify the reasons behind this discrepancy, such as whether factors like data distribution, model complexity,  play a critical role.", "other_comments_or_suggestions": "The SoftMatch method achieves a balance between the quality and quantity of pseudo-labeled samples by weighting them according to their confidence, leading to significant performance improvements in pseudo-labeling tasks. Could this idea also be applied to the selection of two-phase labels to further enhance their effectiveness?", "questions_for_authors": "Please see Other Strengths And Weaknesses, Other Comments Or Suggestions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces a novel approach to pseudo-labeling in semi-supervised learning by identifying and leveraging "two-phase labels"—predicted labels that exhibit a distinct two-phase pattern during training: initially predicted as one category in early epochs and later switching to another. The authors argue that these labels, despite their low confidence and non-stationary nature, are highly informative for decision boundaries. To operationalize this insight, they propose a 2-phasic metric to quantify the two-phase pattern and a tailored loss function that learns correct correlations while mitigating false ones. Experiments across eight datasets (including image and graph benchmarks demonstrate that incorporating two-phase labels boosts the performance of existing pseudo-labeling methods.

The paper received 4 positive reviews, with reviewers acknowledging its ​​novelty​​, ​​theoretical grounding​​, and ​​empirical effectiveness​​. Thus, I recommend ​​acceptance​​.
Decision: Accept (poster)