ICML 2025

Non-Stationary Predictions May Be More Informative:
Exploring Pseudo-Labels with a Two-Phase Pattern of Training Dynamics
Hongbin Pei 1 Jingxin Hai 1 Yu Li 1 Huiqi Deng 1
Denghao Ma 2 Jie Ma 1 Pinghui Wang 1 Jing Tao 1 Xiaohong Guan 1
Abstract
Pseudo-labeling is a widely used strategy in semi-
supervised learning. Existing methods typically
select predicted labels with high confidence scores
and high training stationarity, as pseudo-labels to
augment training sets. In contrast, this paper ex-
plores the pseudo-labeling potential of predicted
labels that do not exhibit these characteristics. We
discover a new type of predicted labels suitable for
pseudo-labeling, termed two-phase labels, which
exhibit a two-phase pattern during training: they
are initially predicted as one category in early
training stages and switch to another category in
subsequent epochs. Case studies show the two-
phase labels are informative for decision bound-
aries. To effectively identify the two-phase labels,
we design a 2-phasic metric that mathematically
characterizes their spatial and temporal patterns.
Furthermore, we propose a loss function tailored
for two-phase pseudo-labeling learning, allowing
models not only to learn correct correlations but
also to eliminate false ones. Extensive experi-
ments on eight datasets show that our proposed
2-phasic metric acts as a powerful booster for
existing pseudo-labeling methods by additionally
incorporating the two-phase labels, achieving an
average classification accuracy gain of 1.73% on
image datasets and 1.92% on graph datasets.
1. Introduction
Pseudo-labeling (Lee et al., 2013) is a widely employed strat-
egy in many semi-supervised learning methods, e.g., self-
training (Amini et al., 2025), co-training (Blum & Mitchell,
1998), consistency regularization (Sohn et al., 2020), and
1MOE KLINNS Lab, Xi’an Jiaotong University, China;
2Beijing Information Science and Technology University, China.
Correspondence to: Jing Tao <jtao@mail.xjtu.edu.cn...