Learning Dynamics of Logits Debiasing for Long-Tailed Semi-Supervised Learning
Authors: Yue Cheng
Jiajun Zhang
Xiaohui Gao
Weiwei Xing
Zhanxing Zhu
Venue: ICLR 2026 Poster (2026)
Keywords: learning dynamics; semi-supervised learning; long-tailed; logits debiasing
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Long-tailed distributions are prevalent in real-world semi-supervised learning (SSL), where pseudo-labels tend to favor majority classes, leading to degraded generalization. Although numerous long-tailed SSL (LTSSL) methods have been proposed, the underlying mechanisms of class bias remain underexplored. In this work, we investigate LTSSL through the lens of learning dynamics and introduce the notion of baseline images to characterize accumulated bias during training. We provide a step-wise decomposition showing that baseline predictions are determined solely by shallow bias terms, making them reliable indicators of class priors. Building on this insight, we propose a novel framework, DyTrim, which leverages baseline images to guide data pruning. Specifically, we perform class-aware pruning on labeled data to balance class distribution and label-agnostic soft pruning with confidence filtering on unlabeled data to mitigate error accumulation. Theoretically, we show that our method implicitly realizes risk reweighting, effectively suppressing class bias. Extensive experiments on public benchmarks show that DyTrim consistently enhances the performance of existing LTSSL methods by improving representation quality and prediction accuracy.

Published as a conference paper at ICLR 2026
LEARNING DYNAMICS OF LOGITS DEBIASING FOR LONG-TAILE SEMI-SUPERVISER LEARNING
Yue Chengl.2* Jiajun Zhang
Xiaohui Gao
Weiwei Xing
Zhanxing Zhu
Beijing Jiaotong University 2 AAntrroup
Northwweterrn Polytechnical University Univeesstyy of Southampton {yuecheng, junzhang] wwxingebjtu.
edu. en gaitxh@ foxmai1..om.ooooo
www.ijttuuuuuuuuooooooo edu. en
z. zhu ac.
ABSTRACT
Long-tailed distributions are prevalent in real-world semi-supervised learning (SSL), where pseudo-labels tend to favor majority classes, leading to degraded generalization. While many long-tailed semi- superviee learning (LTSSL) meth- ods have been proposed, the mechanisms by which they implicitly debias logits remain poorly understood. In this work, we revisit LTSSL through the lens of learning dynamics and provide a theoretical characterization of logits debiasing. Specifically, we derive a step-wist decomposition of the logits updates, showing that predictions are dominated by class-imbalance bias that reliably reflects la- bel priors. To expose this effect, we use the logits of a task-irrelevant baseline image an indicator of accumulated bias and prove that they converge to the class prior. This provides a unified view where LTSSL remedies such as logit adjustment, reweighting, and resampling correspond to reshaping gradient dy- namics. Based on this insight, we propose DyTrim, a principle-based dynamic pruning framework that reallocates gradient budget through class-aware pruning on labeled data and confidence- vide theoretical guarantees that DyTrim reduces class bias and improves gen- eralization. Extensive experiments on standard LTSSL benchmarks show con- sistent gains across architectures and methods. Code available at: https: / //jia jun 425. github. io/ io/DyTrim
soft pruning on unlabeled data. We pro-
INTRODUCTION
Semi-supervised learning (SSL), exemplified by FixMatch (Sohn et al., 2020) and ReMix- Match (Berthelot et al., 2019), has been proven to demonstrate significant generalization advantages supervised learning, particularly in deep neural networks (Li et al., 2025). However, many isting SSL variants, e.g. Flexhatch (Zhang et al., 9021). FreeMatch (Wang et al., 20236) implicitly assume that both labeled and unlabeled data drawn from balanced class distribution, ie., class imbalance. In practice, real-world datasets commonly exhibit a long-tailed label distribution, lead- ing biased pseudo- toward majority classes. This discrepancy poses significant challenges to the effectiveness of SSL algorithms on real-world datasets.
Recent studies on long-tailes semi-supervised learning (LTSSL) have emerged mitigate the bias introduced by class imbalance in both labeled and unlabeled data. These methods range from distri- bution aligmment (Wei al., 2021; Kim al., 2020), data rebalancing (Fan al., 2022; Lee al., 2021), ), logit adjustment variants (Wei & Gan, 2023; Zhou et al., 2024), to foundation model-based methods (e.g., LADaS; Zheng et al., 2025). In particular, the approach employs a baseline image introduced by Lee & Kim, 2024 simple yet effective tool for quantifying classifier bias, which has garnered significant attention the community (Xing et al., 2025; Yi et al., 2025). Despite these advancements, the underlying mechanisms of how class bias emerges and why existing ap- proaches can mitigate it remain largely unexplored and poorly understood. That also prevents us from exploring principle-based method to improve performance.
"Equal contribution

---

Published as a conference paper at ICLR 2026
In this paper, we analyze the underlying mechanisms of class debiasing through the lens of learning dynamics in long-tailed semi-superrised learning (LTSSL), investigating how inputs, the classifier, and pseudo-labels interact and recursively shape one another during training. Specifically, rive a stepwise decomposition of logit updates in SSI., showing that class imbalance dominates the predictions and prevents the model from leveraging inter-sample similarity, thereby impairing generalization. We further point out that in the learning dynamics of LTSSL, the logits of the base- line image serve as an indicator of the accumulated influence of the network's bias. Building on this framework, we offer unified view of existing debiasing methods, including logit adjustment (LA) (Menon et al., 2021), reweighting (Wang et al., 2017), and resampling (JAPKOWICZ, 2000), which can all be understood through the lens of learning dynamics.
de-
As side product of this analysis, we propose a pruning- based dehiasing framework for long-tailed remedies, named DyTrim. For labeled data, we compute class-wise pruning ratios to rebalance samples. For unlabeled data, we apply lahel-agnostic criterion that prunes low-confidence, incon- sistent samples. Beyond empirical improvements, we provide theoretical guarantees demonstrating how our method alleviates class bias and improves generalization. Extensive experiments demon- strate that method consistently improves LTSSL performance across standard benchmarks and various backbone architectures.
2
PRELIMINARIES
Notions. We consider a labeled dataset X = {(x", ) with N samples and an unlabeled dataset U [ur 1
M with M samples, where x" E R is the n-th labeled sample with label
y" E [C] (1,...CC,. and un E IRd the m-th unlabeled sample. Let N₂ and M, denote the number of labeled and unlabeled samples in class G, such that Ee-1 No = N and > Me- = M. If classes sorted by size, have N1 2 N₂ Z
Ne. and define the imbalance ratios as n = Ni/N. 2 1 and % = max (MM.)/uun //min(M.) (.)) 2 1, respectively. We denote the classifier by fo F + I..... C with parameters and its logits by go(x) E E. where fo (z) arg max. go (r))) and (.), denotes the e-th component. For each iteration of training, we sample minibatches M.X' {(ag. v{²) b E (1,..., B)} C X and MU = {muf") :b E (,....,BB) C l from the training set, where B denotes the minibatch size and denotes the relative size of MM to MX. For brevity, when clear from context we drop the superscript on ug (xg")) and simply write up (2z).
Base SSL algorithms. We use FixMatch (Sohn et al., 2020) as the base SSL algorithm, follow- ing other LTSSL studies. Specifically, FixMatch first predicts the class probability of a weakly augmented unlabeled data point au,) as qb To (y(y))()))) and then generates hard pseudo-label t = arg ma.c. where To(y)-) Softmax(go (go(-)). For consistency regularization, FixMatch uses hard pseudo-label q. only when maxe( 2 т, where denotes a predefined confidence threshold, to improve the quality of the pseudo-labels used for training. We express the training losses of FixMatch C as:
[( 2b. q, 0) (c(xs))) + Leon (A(us), 9o. 0),
where T3 (ug) denotes the b-th labeled (unlabeled) samples in a minibatch MX (MU). A(u)) denotes the strongly augmented of ug. The losses and other SSL algorithms, i.e. FlexMatch (Zhang et al., 2021) 1) and FreeMatch (Wang et al., 2023b), are detailed in Appendix B. to B.3.
Learning dynamics and its per-step decomposition. Inspired by Ren & Sutherland (2025), study how a single gradient update changes the model's confidence on an observation X.. With To æ) denoting the predicted class probability distribution, the learning dynamics become,
AO g 8¹ = - V (ffo v6); A AlogT' (ylxx) e log T (ylær)) -log Ta. (y/æz). (2)
where the update of 0 during step t t + is given by one gradient update the sample pair (Ir, y6) with learning rate 77. L is the loss function, we use the cross-entropy loss H in our setting.
Proposition 1 (Per-step decomposition of learning dynamics; Ren & Sutherland 2025). Let Softmax(z) ()) with z = go (x)) Then the one-step learming dynamics decompose as
A log ( x.)) --:TT'(x..)'"" (xos 2)) + (
where T (xx) Vo Tor (xx) I 1x (xo) only depends the model's current pre- dicted probability, KC ('( vb) (Vo=(x.)) (Voz(1))) is the empirical neural tangent kernel
2

---

Published as a conference paper at ICLR 2026
(eNTK, Jacot al. 2018) of the model, the product of the model's gradients with respect 2g and Tb. (' yo) = Vz yo) is the loss gradient. llp denotes the spectral norm, which bounds the second-order remainder term.
1.0
1T(xp 4)
Before
xt (Ag 4)
Before Alter
x (1gg
4)
Before After
4)
Before After
Aflar
Balanced
Balanced
Imbalanced
Imbalanced
0.4
9.6
X1. and
M1g
(b) Lear Ng. and
M1g
Loarn X6
Mg
Ng with 44
and Up
with qq
with'qa
with 44
Figure 1: Accumulated influence in the MNIST experiment using labeled sample 2 = 0 and an unlabeled sample u6 for training, with To for testing. (a) and (b) shows results from the Balanced experiment (MNIST), (c) and (d) from the Imbalanced experiment (MNIST-LT), (a) and (c) show the influence with accurate pseudo-labels, (b) and (d) with inaccurate pseudo-labels. In (a) and (b), the cumulative influence of pseudo-label authenticity is evident, with the false pseudo-label affecting predictions for similar samples (e.g., probability of 9, 7 and 4). In (c) and (d), the class imbalance masks the influence of false pseudo-label authenticity due to class bias
This decomposition characterizes how each update at (za. y/b) influences predictions at zo. forming the basis for our SSL analysis under class imbalance.
LEARNING DYNAMICS OF LONG-TAILED SEMI-SUPERVISED DEBIASING
3.1 LEARNING DYNAMICS OF SEMI-SUPERVISED LEARNING
In this section, we characterize the learning dynamics of the semi-supervised version gradient descent (GD) for the FixMatch algorithm Eq. (1),
A 2 g = 17 (VLsup(fo(a(z)) yb) + (ccom(ffl((u)))ff((((t))
(4)
Af(x.) - fac11 for (Io).
where T, denotes the observation data point, the update of 0 during step - 6 1 is given by one gradient update on the labeled sample pair (Zy. yo) and unlabeled sample (ux) with learning rate 7.. Previous work (Ren & Sutherland, 2025) showed how single gradient update influences model predictions in supervised learning. We now examine whether such characterization extends to the semi-supervised setting. Since FixMatch (Sohn et al., 2020) update naturally consists of supervised part Crap and a consistency part Loo.. the gradient update can be decomposed accordingly. For an unlabeled sample uy with target o arg maxe ah... where q ne learning dynamics of semi- semi-supervised learning become
(uu)). The per-step
A log n' x'(y/zz)) x.)) A A log log.t/ t.sup (y Ioo (T) + log t.coo o
(y Io u,)
(5)
where Amt f.sup denotes the influence caused by Va and ATT Am denotes the influence caused by us- respectively. Inspired by Definition 1, we now state the decomposition of the per-step influence in semi-supervised learning below:
Proposition 2. For an labeled (unlabeled) sample xs (ux) with target yo (g.). The one-step learning dynamics of SSL decompose as
A log m (.sup (2) xoiws) = '1 (zo) (xo, o(x)))(( (o ((.))) (F
(6)
log Th n..con(y) zo: ub) n/T (xo) (I A(u)) A(1)) gb) + O (
where KC' (C (xx)) and A(ub)))((u))) are eNTK evaluations of the logit network 2(.) =
go(+), with different inputs. (alp)a) V VaLsup (a(xa), yb and Poon (%6, A(us))
V VzLcon (ÿu, Alub))bo, respectively.
3

---

Published as a conference paper at ICLR 2026
As shown in Proposition 2, each update of 9 in FixMatch decomposes into supervised part driven by (Io, 3b) and a consistency part driven by (up: Q). While this decomposition captures the per- (y 2,), in practice training consists of many such steps, and the accumulated
step influence effect is governed by the iterative interaction between labeled and unlabeled updates. The detailed technical proofs are deferred to Appendix C.I.
Accumulated influence and a demonstration on MNIST. To demonstrate this, we train a WRN- 28-2 on MNIST and visualize the accumulated influence in Figure 1. In Figure I(a), when db is correct, the consistency term reinforces the supervised signal, gradually pulling the prediction of æ toward the correct class, i.e. Q6,4, and 96,9.. consistent with the constructive dynamics implied by Eq. (6). contrast, when d incorrect (Figure 1(b)), the consistency update exerts the opposite ef- fect, i.e., 96,4y, 7 and 26.9.. systematically reducing the correct probability of Tg- This illustrates how pseudo-label errors, even if small at each step, can accumulate across iterations into negative loop. The Figure I(c) and (d) show that under class imbalance, such accumulated influence can drive the classifier to consistently predict the majority class (here 96.0 qb,4), regardless of the true label. This confirms the implication of our dynamics analysis: in SSL, the imbalance influence of labeled data is passed to the pseudo-labels through the classifier, so imbalance bias can be amplified rather than averaged out, leading to catastrophic bias.
3.2 LEARNING DYNAMICS ANALYSIS OF ACCUMULATER BIAS UNDER CLASS IMBALANCE
The aforemennioned phenomenon, together with the learning dynamics of the semi-supervised framework, illustrates how class imbalance accumulates into systematic bias. While per-update dynamics capture the influence of individual samples on predictions, they fall short of reflecting the global effect of imbalance. This motivates the search for an indicator that bridges class-imbalance bias with the underlying learning dynamics. Replacing the inputs x, with a task irrelevant baseline image I, we can regard the Eq. (6) such an attributing indicator (Sundararajan al., 2017). To justify this choice, we analyze its theoretical properties in both linear and deep settings, and then incorporate it into the per-step influence decomposition.
Baseline image and its invariance property. with no bias the first layer and a bias vector b E RC in the output layer h(x) = (2 (x),, where (z) = a (W,z) and (2) W2x + b. This setting allows us to isolate and examine the predicted class probability (I) of a baseline image. For baseline image I E R. we have
For simplicity, we first consider a two-layer MLP
h(T) = (2 +b.
In modern neural networks, the explicit bias term b is often absorbed into the normalization layer, egg., BatchNorm, LayerNorm, with other layers typically set without bias. Without loss of gener- ality, take BatchNorm as example for analysis. Since the BatchNorm transformation can be equivalently viewed an affine linear layer with learnable parameters, we may replace (22) with Bat caNoro)))()) layer, i.e.,
M(Z) BatchNorm (Z) = (Z) - 1 (Z)
W2+h.
(8)
(ar((h(( +c
where is a small positive constant that ensures numerical stability. The baseline image is typically a solid color image, which inherently lacks task-related patterns, see Appendix D.1 for more dis- cussions. This representation shows that, for baseline images, the dependence of h(I) on the input is effectively controlled only through the affine parameters (W2, b) of the normalization layer. We now state the main results regarding the prediction To (I) for such baseline images:
Proposition 3 (Invariance of baseline image under affine normalization). Let = k. la be solid color image, where & E {0, 10.1..... }55} and 1, E Id is an all-one vector. Suppose the output of the first hidden transformation is normalized by a normalization layer (e.g., BatchNorm, InstanceNorm, or GroupNorm) with affine parameters (Wz, b). Then the logits hII) are independent of k and reduce to
I(T))= b, T (Z) -Softmax())
One can immediately notice that no (I) in Eq. (9) does not contain any term related to the pixel value l of I. This observation implies that the representation No (Z) of baseline image is entirely
4

---

Published as a conference paper at ICLR 2026
determined by the BatchNorm bias term b, and is invariant to the actual pixel value k. The detailed technical proofs are deferred to Appendix C.2.
Building upon this invariance, now establish direct connection between the bascline image and the underlying class distribution. Specifically, for the classifier formulation in Eq. (8) and Eq. (9), show that the logits of the baseline image encode the class-imbalanc ratio present in the train-
ing data, thus providing a direct bridge between Tt. (Z) and the class prior induced by the long-tailed distribution in training. We empirically validate this connection on CIFARIO-LT by analyzing the distribution of baseline logits: as shown in Figure 2, the baseline logits closely align with the em- pirical class prior. When we remove the bias term in our ablation model, this alignment vanishes, indicating that the baseline logits lose their responsiveness to the class prior.
0.40
0.40
0.40
Laboled catasct
Unlabelec dataset
Full dataset Black image image
Class Probabilitis
0.35
Black image
image
Class Probabilities
0.35
Blck image
hite image
Class Probabilities
0.35
0.30
wia bias
0.30
wit blas
0.30
wil bias
0.25
Last layer blas
0.25
Last layer bias
0.25
0.20
0.20
0.20
0.15
0.15
0.15
0.10
0.10
0.10
0.05
0.05
0.05
0.00
0.00
0.00
0
2
3
4
8
9
Class
Class
Class
(a) Labeled dataset
(b) Unlabeled dataset
(c) Full dataset
Figure 2: Class distributions and measured biaseddegree under m = 100 and YYu
100. The bar
plots show the class distributions for (a) labeled, (b) unlabeled, and (c) full datasets.
Theorem (Bias as the conditional distribution prior). Assume the model h(x) as characterized in Eq. (8) is trained using cross-entropy loss:
L [a [- yd log Softmax(M(=)]]
(10)
At a population risk minimizer (W2, ba) we have
x),
(1 Softmax(b") -Pyl
AO A0 ( -).
(11)
Var
For the baseline image I in Proposition 3, the baseline prediction thus coincides with the conditional class distribution the normalized-zero, feature state, capturing the class prior induced by the long- tailed training distribution. See the detailed to Appendix C.3.
Thus, (T) serves as natural proxy for the accumulated bias of the model, bridging the class imbalance in the training set to the learning dynamics of the classifier.
Per-step influence decomposition of the baseline image. Let (vl-)) denote the estimate of the underlying class prior. Then we can track the change in the model's confidence by observing log no (y( Then the learning dynamics on the baseline image become,
Alog r' (v) log Tg++ (y|Z) log To- (ylZ).
(12)
Proposition 4. Let π = Softmax(z) and z = go(x). The one-step dynamics on the baseline image decompose as
log (y I: = - T7T (I) (I, (Z. (00
(13)
where (I) V2 lo (I) 1n (I), K (I (Vo( (Voz(z)) )r is the eNTK of the logit network z, can be (=x)) or A(ub). y can be ye and au,)). See Appendix C.4 for more details.
Compared with Proposition 2. the main difference is that the (I) and IC' x) term. Since the baseline image I lies far from the data manifold, the coupling kernel K. KC'(Ix is typically small.
5

---

Published as a conference paper at ICLR 2026
first epoch
first 10 epoch
full epoch
bias term 8 =
0.3
0.6
0.4
6.2
Class C9
505 400
Betch
100
Cuns
Epoch
Class
508
420 Epach
2017
150
10
503
Spach
1p
Figure 3: The change of logits's 's probability distribution No (T for the baseline image CIFAR-10 10- LT. The left three panels depict the dynamics of reference logits under FixMatch: at epoch 1, epochs 1-10, and full epochs. The rightmost panel illustrates the dynamics after removing all bias terms.
Thus, the learning dynamics in Eq. (13) mainly governed by the output-sensitivity term ('(Z) and the gradient signal G", with the latter providing both the energy and direction for the model's s adaptation. Under this formulation, the baseline image I serves as an indicator that isolates the model's 's global bias state. Tracking (I) over training therefore provides a direct and interpretable measurement of how class-level bias accumulates during semi-supervised learning. Therefore, as the number of labeled and unlabeled samples from the majority class increases, the output of π (Z) will be progressively squeezed into a biased long-tailed distribution. Even with G guiding the adaptation direction, this process can still be steered by the biased state encoded in (I), further amplifying the long-tailes taig-t shift, as illustrated in Figure 3.
4 DYNAMICS ANALYSIS OF LOGITS DEBIASING SEMI-SUPERVISED LEARNING
Analyzing the dynamics of logits debiasing methods long-tailed semi-supervised learning is chal- lenging because different algorithms such as Logits Adjustment, Reweighting, and Resampling em- ploy distinct formulations. In this section, we propose a unified framework based on the per-step influence decomposition (Proposition 4). This framework enables to analyze how these methods modify the update gradient flow, thereby influencing the model's s bias evolution during training. We also introduce a pruning- based method, Dy Trim, as byproduct of our analysis. It can be integrated in plug-and and-play manner with other logits debiasing methods.
4.1 PER-STEP DECOMPOSITION OF LOGITS ADJUSTMENT
The typical logits debias method used during long-tai tail semi- supervised learning logits adjustment (LA) (Menon et al., 2021), which introduces class-dependent shift in the logits, expressed as:
())))z) Soft))
7 ge (z) XΦ.
(14)
where x >> controls the adjustment strength and & RCC is estimates of the class priors. Thanks to the Σ implemented in CDMAD (Lee & Kim, 2024), the resulting logits adjustment almost identical such simple subtraction, ie.,, Ā(x) = go (x) log w, where (I Thus, the change of the model's prediction on the baseline image I can be represented as,
log ( I: 22) -7 T (z, + ((
(15)
represents the influence of the adjusted logits. Compared with Proposition 4, the main difference is that the gradient term has been modified by class prior म. which allows us to answer how does learning with debiasing affect the gradients for unlabeled samples? When adjusting the model's logits by class prior, the gradient flow will ensure that the model compensates for the class imbalance during training. See more discussions in Appendix C.S. CIFAR 10-LT demonstrate the effectiveness of this debiasing,
where (Z,(Z, y) (a(s)) (A(u)))
We also conducted experiments as illustrated in Figure 3, which illustrates that the bias measured in the baseline image after applying LA to the CDMAD method is alleviated.

---

Published as a conference paper at ICLR 2026
4.2 PER-STEP DECOMPOSITION OF REWEIGHTING
Reweighting is another prevalent debiasing technique in long-tail semi-supervised learning (Lai al., 2022), which introduces class-dependent weights in the loss function, expressed as:
C Suup Σ ((,_Can(a((t))::
Coom Σ Loon (A(u)) 90. T;,T; T;0);
(16)
where wh, (we) is the weight of the k-th class in labeled (unlabeled) samples. For simplicity, assume the class weight distributions are consistent between labeled and unlabeled data, i.e. w and W follow the same proportional relationship and remain fixed during training. Under this reweighting scheme, the gradient signals for both supervised and consistency terms are scaled by their respective class weights. Hence, we can decompose the learning dynamics for reweighting similarly to Eq. (15),
A log m " (y I;x) -1T (I, w " (I w²) (11 (V0(())))))
(17)
where (I, = w Kr and (a y; (x, by scaling both the similarity kernel and the gradient term with the class weight wº. Intuitively, this modulates the strength of interaction between samples and the magnitude of their gradients a class-dependent manner: samples from classes with larger w exert stronger influence on the update of 0, while those from classes with smaller wf contribute less. When wf is designed as a function of class frequency (e.g., inverse frequency). this mechanism increases the effective contribution of under-represented classes and attenuates that of head classes. See more discussions in Appendix C.5.
(*, Thus, reweighting acts
4.3 DYTRIM: A BASELINE IMAGE GUIDED DATA PRUNING FRAMEWORK FOR LTSSL
Under the per-step influence framework of Proposition 4, logits adjustment and reweighting reshape the gradient flow by modifying the update direction or magnitude, while resampling acts directly on the data distribution by changing the frequency with which different classes enter training. Yet all these methods leave the sample set itself intact at each step and ignore the heterogeneous per-step utility of individual samples, allowing redundant head-class examples to continue dominating the learning dynamies. This motivates debiasing at the data-sclection level, where dynamically trolling which samples participate in each update provides a more direct mechanism for mitigating accumulated bias in LTSSI., as illustrated in Figure 4.
Per-step decomposition of dynamic pruning. Differs from logits adjustment, reweighting, or re- sampling, dynamic pruning directly alters the set of samples that participate in each gradient update, instead of modifying the loss or sampling distribution. We define step-dependent scoring func- tions (-) for labeled samples X' and Hi (-) for unlabeled samples U, which dynamically quantify sample utility at training step t. For the dynamic pruning process, samples are discarded by the hee-dependent pruning probabilities Pl and P.V.
(x; ()) = 1 (x), H(); and ("; (u), H.");
(18)
where H and H. are adaptive thresholds, I(.,)) is the indicator function. Under this dynamic pruning mechanism, the one-step decomposition of dynamic pruning decomposes
A logg R 4prume (y I; = -T (") O (11 (V02(()))))
(a.v) Ga (2, Px (x (a,v)
(19)
where
(x H)) (u; Hi!
I E X, x Eu
P((x) -
(20)
This decomposition shows that dynamic pruning reshapes the update dynamics by gating sam- ple participation through Pf and P effectively zeroing out the kernel -gaddint interactions ic' (C'(I=))'' (x. y) of low-utility samples. In contrast logits adjustment and reweighting, which only alter gradient signals, or resampling, which changes the sampling measure, pruning directly removes redundant head-class examples and underlearned unlabeled ones from the optimization

---

Published as a conference paper at ICLR 2026
path, thereby reallocating the model's effective update budget toward samples that meaningfully influence bias correction. Although the kernel (I, x) itself remains unchanged, its operational contribution becomes Eg--p ( (I,,), selectively amplifying informative interactions while suppressing those that drive long-tailed drift. This sample-level intervention yields a more direct and fine-grained control of the learning dynamics than existing debiasing strategies.
Building on this perspective, we now instantiate how dynamic pruning is implemented in practice. We introduce DyTrim, a baseline-guich dynamic pruning framework designed to accommodate the distributional mismatch that real-world LTSSL typically exhibits between labeled and unlabeled data. Since such mismatch renders single participation rule inadequate, Dy DyTrim employs two complementary pruning mechanisms, one tailored to the long-tailed labeled set and the other to the noisy and imbalance-une unknoww unlabeled set. See more details about Appendix C.6.
Dynamic pruning for labeled data. Since the labeled data follow a taig-tailed class distribution, we design class-aware pruning policy Pf guided by my (I). Critically, the classifier's pseudo- labels are primarily influenced by the labeled samples, which introduce bias toward majority classes. Since Proposition 3 shows that the baseline image has invariance to solid-color intensity, from first principles, we leverage the logits from a black image I to calibrate pruning probabilities. Given the labeled dataset X' in the f-th epoch, a class- pruning probability is assigned to each sample based on its score, which is formulated as:
P1(08) - {
Hz H Ha(a)) & H Hr.
(21)
where H! H'rr,, denotes the x Nr smallest scoring values of the class cadd r. = (I)e is the class- aware pruning probability. The labeled scoring function (xā²) is defined using the supervised loss fosup (2i , y(' to quantify sample utility. See more details about Appendix E.
Dynamic pruning for unlabeled data. While the distribution of the label of the unlabeled data and its imbalance ratio "/u unknown. To address the uncertainty and bias of pseudo-labels, design a label-insensitive soft pruning policy PH inspired by (Qin et al., 2024), which introduces randomness and gradient scaling into the pruning process. Specifically, for an unlabeled dataset li at the f-th epoch, a pruning probability is assigned to each sample based on its score, which is formulated as:
Hit and p" (ug² 2 T 2 Hi (ug") < <,,
Pt (u((
(22)
0
where Hi is the adaptive threshold and is randomized pruning rate, is the confidence threshold r and p" p"(n"") = max(sof fox(ssottmxxppp) t. (a((g) ())) denote the debiased pseudo-label confidence. See more details about Appendix E.2.
5 EXPERIMENT
In this section, conducted comprehensive experiments to verify the effectiveness of the pro- posed Drim on CIFARIO-LT, CIFARIO0-LT (Cui et al., 2019), STLI0-LT (Kim et al., 2020), and ImageNet- 127 (Deng al., 2009; Huh et al., 2016) datasets. Due to limited space, defer the detailed experimental settings and additional experiments to the Appendix G.
5.1 RESULTS CIFARIO/100- STLIO-LT AND IMAGENET-LT
Under the consistent condition where Yv is known and matched to n/ the results in Table I show that CISSL algorithms consistently outperform their vanilla SSL counterparts by mitigating class imbal- ance while effectively exploiting unlabeled data. Among them, the proposed DyTrim achieves the best performance across all imbalance ratios. Compared with the state-of-the-a CDMAD, Dy DyTrim improves bACC by 1.2% and GM by 1.4% on average, without incurring additional computational overhead. Furthermore, when integrated into FlexMatch and FreeMatch, DyTrim yields substantial improvements, boosting bACC/GM by 2-3% on average. Table evaluates the methods on CIFAR- 100-LT, which involves more classes and a stronger imbalance. The results demonstrate that Dy consistently outperforms all competing approaches under this more challenging setting.
8

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of bACC/GM CIFAR-IO-LT under different imbalance ratio 7 = 7 Tu
where Yau is assumed to be known. *** indicates our own implementation.
Debiasing Strategy
50
7
7
150
Algorithm
bACC
bACC
GM
bACC
GM
Vanilla
60.05
61.1 #0.09
58.8 +0.11
+0.1+
55.6
44.0++0.88
Re-sampling
40.48
60.6 10.67
55.8 10.47
45.1 1:030
40.05
18,2
LDAM-DRW
68.9 +0.07
67.0 +0.08
62.8 +:0.7
58.9 +0.60
+0.20
50.4
cRT
67.8 1:0.3
66.3 10.15
63.2 1:0.45
59.9 ::0.40
59.3 10.10
54.6.10.72
FixMatch
10.33
77.8 10.36
71.5 10.72
66.8
60.4
59.9 1:0.43
DARP+cRT
85.8 40.43
85.6 10.56
10.26
110111
79.6
78.9 10.35
CReST+LA
80.6
81.9 #0.45
80.70
10.99
72.94
1.74
ABC
85.6
85.2 +0.29
61.1-1.14
...3
1..3
75.6
CoSSL
86.8
86.6
$3.2.0..4
82.7:10.00
10.55
79.6:10.57
SAW+LA
86.2
83,9
80.7 +0.15
77.5+
73.7
71.2
FixMatch
Adsh
82.9 83.4
76.5 +0.35
74.8++034
70.5
68.8
DebiasPL
85.6
85.2 10.23
80.6 10.50
79.9
76.6:10.12
75.8 10.71
UDAL
80,5
80,4 20.26
80.6
10.93
75.8 +0.71
1,2AC
86.6+0031
86.7 +0.30
82.1 +0.57
81,5
77.53
75.8 +0.71
CDMAD
87.3 10.12
87.0 10.15
83.6 10.46
83.1 40.57
80.8 10.86
1.00
DyTrim
88.0 +0.31
87.8 0.32
84.8 +0.48
84.4
82.0 +0009
81.3
FlexMatch*
10.72
10.88
67.7 10.73
63.6
62.6
56.1
FlexMatch
CDMAD*
40.82
73.0 11112
68.4:10.46
66.8 1.0.3
67.0
63.2
DyTrim
+0.42
76,2 +0.44
70.7 +0.49
67.8 +0.70
68.6 +0222
66.3 +0.07
FreeMatch*
71.9
69.4
65.7 #0.18
60.9
62.5 #0.12
$0.53
FreeMatch
CDMAD*
13.63 10.64
69.9 10.65
68.2 10.74
66.2 10.27
63.2
DyTrim
76.9 10.45
75.9 0.52
72.3 10.12
71.4:0..5
69.4 10.35
67.5 10.63
As shown in Table 3, DyTrim con- sistently outperforms prior techniques such CDMAD the large-scale ImageNet-L.T benchmark (Liu et al., 2019), demonstrating its complemen- tary benefits rather than merely over- lapping with existing rebalancing ap- proaches. See more details about large-resolution (224 224) in Ap- pendix H.4. Under the inconsistent condition where 7u was unknown and mismatched to "r. the results in Ta- ble show that DyTrim remains the most effective method overall. When the labeled and unlabeled data distribu- tions deviate, DyTrim consistently out- performs CDMÁD on both CIFAR- 10- LT and STL-10-LT. See more details in Appendix H.2 for the dynamics of base- line image logits during training.
Table Comparison of bACC on CIFAR-100-LL7 un- der different imbalance ratio, where Ya is assumed to be known. "*" indicates our own implementation.
Base SSL Alogrithm
Debiasing Strutegy
20
T 100
FixMatch
49.6 =0.78
42.1
+0.33
37.6++0.8
DARP
50.8
43.1
DARPEcRT
51.4
40.54
40.4
CReST
51.8
44.9
40.1 10.65
CR
52.9-0.07
47.3
42.7
FixMatch
ABC
53.3
46.7
41.2+40.6
CoSSL
53,9=0778
40.6
43.0++061
UDAL
54.1
48.0
43.7 40.1
CPE
52.4
45,6
CDMAD
54.3 =0.4
48.8 +0.75
44.1 #0.29
DyTrim
55.5
50.8 10.80
44.8 10.27
FlexMatch
36.5 10.51
29.6 10..5
25.8 10.79
Fexxaatc
CDMAD*
31,9
DyTrim
40.9 =0.09
33.5 33021
29.8 #:067
FreeMatch
35.9
31.3 10065
24.5 100.6
FreeMatch
CDMAD*
36.9 =0.6
32.8 +0.93
28.0
DyTrim
3-.40
5.2 RESULTS ON VIT BACKBONES
In addition, Table 5 highlights the performance of various al- gorithms under both consistent and inconsistent imbalance settings with ViT backbones. On CIFAR-IO-LT, DyTrim yields the best results, improving bACC 0.6% over CDMAD and nearly 4% over FixMatch when " = Yu = 100. Under the inconsistent condition. DyTrim maintains clear margin, surpassing CDMAD almost 2%. On 1LFAR-100-LT1 although the absolute accuracies are lower due the increased diffi-
Table 3: Comparison of bACC on ImageNet-LT.
Algorithm #
ImageNet-LT
20.0
w/CDMAD*
35.4
w/+DyTrim
37.2

---

Published as a conference paper at ICLR 2026
Table 4: Comparison of bACC/GM on CIFAR-IO-LT and STL-I0-LT under different imbalance
ratio " * Yea where Th is assumed to be unknown. "*" indicates our own implementation.
Base SSL Algorithm
Debiasing Strategy
CIFAR-IO-LT (2)
100, Tv
Unknown)
101- (% Unknnwn)
50
To
150
10
71
20
bACC
GM
bACC
GM
hACC
GM
bACC
GM
FixMatch
73.9:0225
70.5
62.611111
72.9 10.09
69.6
63.4 =0.21
52.6=0.09
DARP
77.3 10.17
75.5 20.21
72.9
69.5 10.18
77.8 1:0.3
76.5
69.9 1.77
65.4 #3.07
DARP+LA
82.3
81.5
78.9
77.7 10.06
78.6
20.4
71.9 20.1
6..7
DARP+oRT
82.7 10.21
20.25
80.7 1.0.4
80.2 10.61
79.3 100.3
78.7
74.1 =0.61
13.1
ABC
82.7 10.61
82.0
77.2 7:.4
79.1 10.46
78.1
73.8
72.1 =0.15
FixMatch
SAW
79.8 10.25
79.1 10.32
74.5 1:0.97
72.5 1:33
78.3 1:0.55
77.0 1:019
71.9 =0.81
69.0 =0.81
SAW+LA
79.1
79,4
73.9 =0.91
SAW+cRT
81.3
81.3 40.32
77.6 10.40
77.1 10.41
78.9 =:0.2
77.8 10.14
72.3 =00.6
69.5 +0.83
CPE
82.1
79.0 +0.05
78.7
77.0 -0.73
CDMAD
85.7 1.036
85.3 10.38
82.3 1:0.23
81.8 2:0.9
79.9 1:0.3
78.9 1:038
75.2 =0.40
73.5 #0.31
DyTrim
1.0.64
1:0.7
76.7
FlexMatch
67.7 10.67
62.8 20:6
63.0
56.3 11.70
62.1 1:029
60.8 :0.3
56,9 =0,90
51.4
HexMatch
CDMAD*
69.2
67.0 10.11
67.0
63.4 10991
65.5
1.002
62.4 1.00
60,5 20999
DyTrim
+0.39
70.7 +0.45
70.3+1.001
67.4 +0.21
68.0 -0.94
66.4 00.8
63.9 =0.16
61.7 =0.28
FrecMatch*
69.39
11.45
63.5 #0.76
56.7 20.77
63.9 00777
62.0 00:90
59.0 11:43
57.6=0.67
FreeMatch
CDMAD*
71.0+098
69.0++1.05
67.1 +0.96
64.3 +0.99
66.1 +-0.32
63.8 +097
61.5 -0.47
59.5-0033
DyTrim
72.3
+1.1
69.9 10.15
67.4 4:037
68.0 00664
66.5 11220
64.6 -0777
62.7 -116
culty, DyTrim still matches or slightly improves upon CDMAD, while consistently outperforming FixMatch. Additional experimental results are provided in Appendix H.
Table 5: Comparison of bACC/GM on CIFAR-IO-LT and CIFAR- 100-LT with Tiny ViT under dif-
ferent imbalance ratio, where is assumed to be known. "*" indicates our own implementation.
CIFAR-10-LT (2)
100)
CIFAR- 100-- (1) 100)
Base SSL Debiasing Strategy
Algorithm
100
No 150
7u
100
bACC
GM
bACC
GM
bACC
GM
#ixMatch
45.5
30.0
45.3 10.12
28.9 10.96
10.13
5,7 10.33
FixMatch
CDMAD
48.7 +0.49
40.5 +0.26
45.4
39.9 +0.10
24.0
9.0 +0.77
DyTrim
10.47
10.36
47.3 10.12
20.77
24.1 10.22
8.9 10.15
5.3 SCALABILITY EVALUATION DYTRIM
DyTrim exhibited robust extensibility universal plug-in component, consistently boosting per- formance across diverse SSL frameworks (CDMAD/CCL), datasets (CIFAR/STLIOO-LT) and im- balance ratios (7 1 150). shown in Table 6. Notably, it achieved up to +1.4% (CDMAD on CIFAR I0-LT) and +2.7% (STLIO-LT, 2=200 gains without architecture-specfficc tuning, validating its versatility in semi-supervised long-tailed scenarios.
Table 6: Comparison of bACC with two SOTA algorithms with and without DyTrim on CIFAR-10,
CIFAR-100, and S0T.- and 1 indicate improvements degradations over the baseline.
FixMatch+
FixMatch+
Dataset
Imbalance ratio
CDMAD
CDMAD+Dy Trim
Gain
CCL
CCL+DyTrri
Gain
Tt % 100
83.6:00.6
84.8 10.48
11.2
+0.35
86.7 10.39
-0.5
CIFARIO-LT
"I = % = 150
80.8 10.86
82.0 +0009
11.2
84.0
:0.21
84.0 =0.26
-0.0
100, 76
87.5 40.46
88.9 10.88
11.4
93.9 10.12
94.1 10.17
-0.2
CIFARIGO-LT
"I = 7/- = 20
54.3 :.0.44
55.5 +0.53
11.2
57.5 10.16
58.1 +:0.49
-0.6
STI.0-LTT
71 10
79.9
80.7 10.64
+1.2
84.8 20.15
85.1 10.33
-0.3
"1 = 20
75.2 4:0.4
77.9 +1.04
12.7
83.1 10.18
83.3 -0.40
-0.2
6 CONCLUSION
In this work, we provide theoretical characterization of class bias in LTSSL through an in-depth analysis of the learning dynamics. We derive step-wise decomposition of logit updates, demon- strating how class imbalance dominates predictions and how debiasing methods, such as logit adjust- ment, reweighting, and resampling. Our theoretical insights bridge the gap between existing meth- ods and their effect on gradient dynamics, highlighting the critical role of sample-level interventions. Based on this foundation, we introduce Dy Trim, dynamic pruning framework that mitigates class imbalance by reallocating gradient budgets. Empirical results across multiple benchmarks and SSL methods demonstrate that Dy DyTrim consistently improves performance.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
This work was supported by the Beijing Natural Science Foundation under Grant (No.L231005), and by the National Key Research and Development Program of China under Grant NN.22024FB3331200 Yue Cheng would like to thank Bochen Lyu, Qianying Tang, and Xiang Wei for the useful discussion.
REFERENCES
Jean-Miche Attendu and Jean- Phiipppl Corbeil. Nlu on data diets: Dynamic data subset selection for nlp classification tasks. arXiv preprint arXiv:2306. 03208, 2023. 15
David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmenta- tion anchoring. arXiv preprint arXiv: 1et.e9991.0977... 2019.
Léon Bottou. Stochastic gradient descent tricks. In Neural networks: tricks of the trade: second edition, pp. 421-436. Springer, 2012. 23
Jiarui Cai, Yizhou Wang, and Jenq-Neng Hwang. Ace: Ally complementary experts for solving loigetailed recogmition in one-shot. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 112-121, 2021. 15
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with abbel-istributtonn-aaar margin loss. Advances neural information processing systems, 32, 2019. 24
Dingshuu Chen, Zhixun Li, Yuyan Ni, Guibin Zhang, Ding Wang. Qiang Liu, Shu Wu, Jeffrey Xu Yu, and Liang Wang. Beyond efficiency: Molecular data pruning for enhanced generalization. In The Tiihth- Annual Conference on Neural Information Processing Systems, 2024. 15
Hao Chen, Ran Tao, Yue Fan, Yidong Wang, Jindong Wang. Bernt Schiele, Xing Xie, Bhiksha Raj, and Marios Savvides. Softmatch: Addressing the quantity-quality tradeoff in semi-supervised learning. In The Eleventh International Conference Learning Representations, 2023. 15
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9268-9277, 2019. 8
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.
Ieee, 2009. 8
Guodong Du, Jia Zhang, Ning Zhang, Hanrui Wu, Peiliang Wu, and Shaozi Li. Semi-supervised imbalanced multi-label classification with label propagation. Pattern Recognition, 150:110358, 2024. 15
Yue Fan, Dengxin Dai, Anna Kukleva, and Bernt Schiele. Cossl: Co-learning of representation and classifier for imbalanced semi-super learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 14574-14584 June 2022. 1, 24
Qianhan Feng, Lujing Xie, Shijie Fang, and Tong Lin. Bacon: Boosting imbalanced semi- remi-supervised learning via balanced feature-level contrastive learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 11970-11978 2024. 15
Zhe Guo and Yu-Feng Li. Class- iass-imbalanced semi-supervised learning with adaptive threshold- ing. In International conference on machine learning, pp. 8082-8094. PMLR, 2022. 15, 24
Xiaoyu Guo, Xiang Wei, Shunli Zhang, Wei Lu, and Weiwei Xing. Derp: Class-aware feature diffusion constraint and reliable pseudo-labeling for imbalanced semi-supervised learning. JEEE Transactions Multimedia,
2024. 24

---

Published as a conference paper at ICLR 2026
Yangyang Guo and Mohan Kankanhalli. Scan: Bootstrapping contrastive pre-training for data effi- ciency. arXiv preprint arXiv:2411.09126. 2024. 15
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016. 23
Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Learning deep representation for imbalanced classification. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5375-5384, 2016. 24
Minyoung Huh, Pulkit Agrawal, and Alexei A Efros. What makes imagenet good for transfer learn- ing? arXiv preprint orXiv:1608.086/1 2016. 8
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and gen- eralization in neural networks. Advances in neural information processing systems, 31, 2018.
JAPKOWICZ.
The class imbalance problem: Significance and strategies. Proc. 2000 Interna- tional Conference on Artificial Intelligence. volume 1, pp. 111-117, 2000. 2, 24
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis
Kalantidis. Decoupling representation and classifier for long-tailed recognition. In International
Conference on Learning Representations, 2020. 24
Kenji Kawaguchi and Haihao Lu. Ordered sgd: A new stochastic optimization framework for em- pirical risk minimization. In International Conference on Artificial Intelligence and Statistics, pp. 669-679. PMLR, 2020. 15
Krishnateja Killamsetty, Sivasubramanian Durga, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer. Grad-match: Gradient matching based data subset selection for efficient deep model training. In International Conference on Machine Learning, pp. 5464-5474. PMLR, 2021. 15
Jachyung Kim, Younghum Hur, Sejun Park, Funho Yang, Sung Ju Hwang, and Jinwoo Shin. Dis- tribution aligning refinery of pscudo-label for imbalanced semi-supervised learning. Advances in neural information processing systems, 33:14567-14579, 2020. 1, 8.8,
Miroslav Kubat. Addressing the curse of imbalanced training sets: one-sided selection. Proceed- ings of the 14th international conference machine learning, pp. 179-186. Morgan Kaufmann, 1997. 24
Zhengfeng Lai, Chao Wang, Henrry Gunawan, Sen-Ching S Cheung, and Chen-Nec Chuah. Smoothed adaptive weighting for imbalanced semi-supervised learning: Improve reliability against unknown distribution data. In Proceedings of the 39th International Conference Ma- chine Learning, volume 162, pp.
2022. 7, 24
Justin Lazarow, Kihyuk Sohn, Cu Lee, Chun-Liang Li, Zizhao Zhang, and Tomas Plister. Unifying distribution alignment as a loss for imbalanced semi-supervised learning. In 2023 IEEE/CVF Winter Conference Applications of Computer Vision (WACV), pp. 5633-5642. IEEE Computer Society, 2023. 24
Doyup Lee, Sungwoong Kim, Ildoo Kim, Yeongjae Cheon, Minsu Cho, and Wook-Shin Han. Con- trastive regularization for semi-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 3911-3920, June 2022. 15
Hyuck Lee and Heeyoung Kim. Cdmad: Class
debiasing for class- imbalanced semi-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23891-23900, 2024. 1, 6, 20, 24
Hyuck Lee, Seungjae Shin, and Heeyoung Kim. Abe: Auxiliary balanced classifier for class-
imbalanced semi-supervised learning. In Advances in Neural Information Processing Systems, volume 34, pp. 7082-7094, 2021. 15, 24
12

---

Published as a conference paper at ICLR 2026
Jingyang Li, Jiachun Pan, Vincent Y. F. Tan, Kim chuan Toh, and Pan Zhou. Towards understand- ing why fixmatch generalizes better than supervised learning. In The Thirteenth International Conference on Learning Representations, 2025.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large- scale long-tailed recognition in an open world. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 2537-2546, 2019. 25
Chengcheng Ma, Ismail Elezi, Jiankang Deng, Weiming Dong, and Changsheng Xu. Three heads are better than one: Complementary experts for long-tailed semi-supervised learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 14229 14237, 2024. 15
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Laig learning via logit adjustment. In Proceedings of the International Conference on Learning Representations, 2021. 2, 6,
Youngtaek Oh, Dong-Jin Kim, and In So Kweon. Daso: Distribution-aware semantics-oriented pseudo-label for imbalanced semi-supervised learning. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 9786-9796, 2022. 24
Ziheng Qin, Kai Wang, Zangwei Zheng, Jianyang Gu, Xiangyu Peng. Daquan Zhou, Lei Shang, Baigui Sun, Xuansong Xie, Yang You, et al. Infobatch: Lossless training speed up by unbiased dynamic data pruning. In The Twelfth International Conference on Learning Representations 2024. 8, 15, 26
Ravi S Raju, Kyle Daruwalla, and Mikko Lipasti. Accelerating deep learning with dynamic data pruning, 2021. 15
Yi Ren and Danica J. Sutherland. Learning dynamics of LLM finetuning. 2025. URL https: / //openeeview../e net htt./formm/// d=t.NNOo2P 2, 19
Yi Ren, Shangmin Guo, and Danica Sutherland. Better supervisory signals by observing learning
paths. In International Conference on Learning Representations, 2022. 19
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza-
tion. arXiv preprint arXiv:1911.0877 2019. 15
Tom Schaul, John Quan, Toannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.005:
2015. 15
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying superviie learning with consistency and confidence. arXiv preprint urXiv:2001.07633 2020. 1, 2, 3, 15, 24
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pp. 3319- 3328, 06-11 Aug 2017. 4
Truong Thao Nguyen, Balazs Gerofi, Edgar Josafat Martinez-Noricga, François Trahay, and Mo- hamed Wahib. Kakurenbo: adaptively hiding samples in deep neural network training. Advances in Neural Information Processing Systems, 36:37900-37999 2023. 15
Renzhen Wang, Xixi Jia, Quanziang Wang, Yichen Wu, and Deyu Meng. Imbalanced semi- supervised learning with bias adaptive classifier. In The Eleventh International Conference on Learning Representations, 2023a. 24
Xudong Wang, Zhirong Wu, Long Lian, and Stella X Yu. Debiased learning from naturally imbal- anced pscudo-labels. In 2022 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14627-14637. IEEE, 2022. 15,
Yidong Wang, Hao Chen. Qiang Heng, Wenxin Hou, Yue Fan, Zhen Wu, Jindong Wang, Marios Sav- vides. Takahiro Shinozaki, Bhiksha Raj. Bernt Schiele. and Xing Xie. Freematch: Self-adaptive thresholding for semi-supervised learning. arXiv preprint arXiv:2205.07246 2023b. 1,2, 15, 16
13

---

Published as a conference paper at ICLR 2026
Yu- Xuoni Wang, Deva Ramanan, and Martial Hebert. Learning to model the tail. In Advances in Neural Information Processing Systems, volume 30, 2017.
Chen Wei, Kihyuk Sohn, Clayton Mellina, Alan Yuille, and Fan Yang. Crest: A class- rebalancing self-training framework for imbalanced semi-supervised learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10857-10866, 2021.
1,
Tong Wei and Kai Gan. Towards realistic long- tailed semi-supervised learning: Consistency is all you need. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3469-3478, 2023. 1, 15,
Kan Wu, Jinnian Zhang, Houwen Peng. Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Tinyvit: Fast pretraining distillation for small vision transformers. In European conference computer vision, pp. 68-85. Springer, 2022. 23
Liuyu Xiang, Guiguang Ding, and Jungong Han. Learning from multiple experts: Self-paced knowl- edge distillation for long-tailed classification. 16th European Conference on Computer Vision, ECCV 2020, pp. 247-263. Springer Nature, 2020. 24
Weiwei Xing, Yue Cheng, Hongzhu Yi, Xiaohui Gao, Xiang Wei, Xiaoyu Guo, Yumin Zhang, and Xinyu Pang. Lege: Learning from consistency gradient conflicting for glass-imbalanced semi-supervised debiasing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 21697-21 2025. 1, 15, 20, 24
Hongzhu Yi, Yue Cheng, Weiwei Xing, and Xiang Wei. Abm: Adaptive bias mitigation for class- imbalanced semi-supervised learning. Neurocomputing, 658:131617, 2025.
Zhuoran Yu, Yin Li, and Yong Jae Lee. Inpl: Pseudo-labeling the inliers first for imbalanced semi-
supervised learning. arXiv preprint arXiv:2303.07269, 2023. 15
Sergey Zagoruyko and Nikos Komodakis.
Wide residual networks.
arXiv preprint
arXiv:160S.07146. 2016. 23
Bowen Zhang, Yidong Wang, Wenxin Hou, HAO WU, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo la- beling. In Advances in Neural Information Processing Systems, volume 34, pp. 18408-18419, 2021. 1, 2, 15, 16
Guibin Zhang, Haonan Dong, Yuchen Zhang, Zhixun Li, Dingshuo Chen, Kai Wang, Tianlong Chen, Yuxuan Liang, Dawei Cheng, and Kun Wang. Gder: Safeguarding efficiency, balancing, and robustness via prototypical graph pruning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 15
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 2017. 32
Na Zheng, Xuemeng Song, Xue Dong, Aashish Nikhil Ghosh, Liqiang Nie, and Roger Zimmer- mann. Language-assistte debiasing and smoothing for foundation model-based semi-supervised learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recogni- tion (CVPR), pp. 15708-25711 June 2025.
Zi-Hao Zhou, Siyuan Fang, Zi-Jing Zhou, Tong Wei, Yuanyu Wan, and Min-Ling Zhang. Continu- contrastive learning for long-tailed semi-supervised recognition. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 1, 15
14

---

Published as a conference paper at ICLR 2026
APPENDIX
A RELATED WORK
A. MORE ABOUT MECHANISMS OF LONG-TAILED DEBIASING
This paper considers learning dynamics to study the debiasing mechanisms of SSL algorithms. We briefly introduce differences between the settings considered here and those in previous works. For debiasing on long-tailed learning, Menon al. (2021) considered a unified framework for debiasing from the perspective of logits adjustment, which requires statistical label frequency. CCL (Zhou al., 2024) considered debiasing from an anfornation-theoretice lens. LCGC (Xing et al., 2025) used gradient flow to analyze the debiasing process. However, these methods only elucidate the model's 's behavior from an ad hoc perspective. We aim to develop more comprehensive framework that enables principle-based lens of the bias generation mechanisms inherent in long-tailed semi- supervised learning.
A.2 MORE ABOUT SEMI-SUPERVISED LEARNING
Modern SSL methods typically integrate diverse strategies for exploiting unlabeled data, such as entropy minimization (Zhou et al., 2024), consistency regularization (Sohn et al., 2020), and con- trastive learning (Zhou et al., 2024; Lee et al., 2022). Among them, most SSL approaches rely on selecting reliable pseudo-labels during training. FixMatch (Sohn et al., 2020) adopts a fixed confidence threshold of 0.95, whereas FlexMatch (Zhang et al., 2021) adapts thresholds per class based on learning difficulty and training progress. FreeMatch (Wang al., 2023b) integrates global and local adjustments with a class-fairness regularizer to promote prediction diversity, while Soft- Match (Chen et al., 2023) employs soft thresholding scheme that reweights samples to balance quantity and quality. In contrast, our method bypasses threshold tuning altogether and directly en- forces class-balanced pseudo-labeling through dynamic pruning.
A.3 MORE ABOUT LONG-TAILED SEMI-SUPERVISER DEBIASING
Existing debiasing methods for LTSSL dominantly rely consistent distribution assumptions (Guo & Li, 2022: Lee et al., 2021) and logit adjustment strategies (Wei & Gan, 2023). Notable ap- proaches include CReST (Wei et al., 2021), which focuses on minority classes through selective self-training, and CoSSL (Cai et al., 2021), which balances representations using tail-class feature augmentation. Recent advances, like BaCon (Feng et al., 2024), utilize contrastive learning for bal- anced features, while SMCLP (Du et al., 2024) exploits collaborative label-instance correlations, and CPE (Ma et al., 2024) employs multiple expert classifiers. Innovative methods such InPL (Yu et al., 2023) and DebiasMatch (Wang et al., 2022) move beyond traditional pseudo-labeling. InPL uses energy scores to detect reliable inliers, whereas DebiasMatch applies adaptive debiasing with marginal loss to reduce long-tailed pseudo-label bias. Despite these advances, LTSSL techniques often demand intricate mechanisms or additional modules (Lee al., 2021), posing challenges in minimizing bias while maintaining simplicity.
A.4 MORE ABOUT DYNAMIC DATASET PRUNING
To reduce training cost on datasets, dynamic dataset pruning methods (Chen et al., 2024; Killamsetty al., 2021: Sagawa al., 2019; Schaul al., 2015; Zhang et al., 2024) aim to reduce the number of training iterations while maintaining performance. Existing methods employ a variety of crite- ria to guide pruning, among which loss-based (Attendu & Corbeil, 2023; Kawaguchi Lu, 2020; Thao Nguyen et al., 2023) method is the most popular. UCB (Raju et al., 2021) applies the cross- entropy loss with exponential moving average (EMA) smoothing to mitigate noise. Infobatch (Qin et al., 2024) randomly prunes low-loss samples and amplifies the gradients of retained ones to pre- serve the expected gradient. SCAN (Guo & Kankanhalli, 2024) categorizes samples as redundant or ill-matched based their loss and gradually increases the pruming ratio using cosine annealing. While thsese methods effectively accelerate training and can yield nearly unbiased results, none have explored their potential to mitigate class imbalance in SSL by pruning.
15

---

Published as a conference paper at ICLR 2026
B MORE BASE SSL ALGORITHMS
MORE ABOUT TRAINING LOSSES OF FIXMATCH
Training losses of FixMatch a minibatch for the labeled set MA and minibatch for the unla- beled set MU can be expressed as follows:
suup(ars 0)
Σ
H H(Ta(vl(((s))
(23)
B
ZEEME
with
Coon (ub. T:
pB Σ 1 (max(()) ((Po)))) (b).
(24)
where ( denote the concatenations Qb- Loup denotes the supervised loss for weakly augmented la- beled data points ug. Leon denotes the consistency regularization loss with the confidence threshold
B.2 MORE ABOUT FLEXMATCH
To overcome the limitation of FixMatch using fixed threshold across all classes, Flex- Match (Zhang et al., 2021) introduces the Curriculum Pseudo Labeling (CPL) strategy. The key idea to dynamically adjust the confidence threshold according to the learning status of each class. Specifically, FlexMatch first predicts the class probability for weakly augmented unlabeled sam- ple 4A as q = ( (us)), and then estimates the learning effect of each class e by (c),), L.e. the number of samples predicted class that exceed the fixed threshold r. After normalization, ratio coefficient (c) obtained, which defines the class-adaptive threshold:
(c Be(c) T.
(25)
In this way, hard-to-lcarn classes receive lower threshold to include more samples in training, while easy-to-learn classes gradually increase their thresholds ensure pseudo-label quality. The unsupervised loss is defined as:
aB
1
Lcon (ub. G. Tr:
Σ b=1
(max(g)) > (arg max(())) H(/, To (un)))
(26)
B.B
where G = arg max. Gb,e denotes the hard pseudo-label, and A(.) is the strong augmentation func- tion. The overall training objective is
Li Loup + XLcon.
(27)
where À is weighting hyperparameter.
B.3 MORE ABOUT FREEMATCH
Unlike FixMatch and FlexMatch, which rely on fixed or indirectly adjusted thresholds, FreeMatch (Wang et al., 2023b) proposes Self-Adaptive Thresholding (SAT) that dynamically deter- mines thresholds based the model's prediction confidence. Specifically, FreeMatch first estimates global threshold using an exponential moving average (EMA) of model confidence:
pB
Tt pT: + p)
max(gh),
(28)
uB
and further refines it with class-specific local statistics Pe(c):
(c(c)
Te(c) max,, P.(c')
(29)
At the early stage of training, thresholds are low to encourage more unlabeled data utilization and faster convergence. As the model becomes more confident, thresholds increase to suppress incorrect pseudo-labels and reduce confirmation bias. The unsupervised loss iteration is thus:
aB
Lcon (4b. g. 0): = BBB
Σ (max(qs) > Te(arg max(qu))) H(q. no (y)A(()))
(30)
16

---

Published as a conference paper at ICLR 2026
In addition, FreeMatch introduces Self-Adaptive Fairness (SAF) regularization L,, which dynami- cally calibrates the prediction distribution to encourage diverse predictions and prevent class collapse during early training. Concretely, let h e RC denotes the normalized class histogram of model pre- dictions at iteration i, and let h* E TRC denotes the target distribution (e.g., a uniform distribution). The SAF regularization is defined as
Cj Dee (he
(31)
where (-1l-) is the Kullback-Leible divergence. The final training objective is:
L Ssup + WnCcon + w Ls
(32)
where wy and wy are weighting hyperparameters.
C PROOF FOR SECTION 3 AND SECTION 4.
C.I PROOF OF PROPOSITION 2
Proposition 1. For an labeled (unlabeled) sample T6 (us) with target yo (4) = arg max, Dhch where ofx = TT= (y))()))). The one-step learning dynamics of SSL decompose as
A log ( (yl Noi
(xo)) (xo, (zx) (at))))))))) w)) (f
(6)
A log Th mo coon (v zoi ub) (x))
(xo)) (x
%6) O Or
where (I,, axxs)) and 1C' ((ux).)((u)))
eNTK evaluations of the logis network z(.)
go(-), with different inputs. Gaup (a 2b) = VaLsup(a(T)) lx and Geou A(us)) VaLcon ((n,) respectively.
Proof. We aim to derive the one-step learning dynamics of SSL for both supervised and contrastive terms. Suppose that want observe the model's prediction "observing example" zig Starting from Eq. (5), we first approximate log logg*+1 (y(zz) using first Taylor expansion (with slight abuse of notation, we write n² for m³)::
log t () = llo'o'
< V (ylo'())zz),, 0++1 - 0r > +0(110 - 1011)3)-
Then, assuming the model updates its parameters using SGD calculated by a "labeled updating example" (It, 36) and an "unlabeled updating example" (A(u)), ()).
Thus, for for supervised learning dynamics, we have, we have
A log o ttIssp (y Toi wb) log π *+1,,u (y) Toi rb) - log " Zoi xz)
V log T' T' xo)| (0 8t) + 0( 1²1)2)
Assuming this step is driven solely by supervised loss, we plug in the definition of SGD and repeat- edly use the chain rule:
(- (a]xo Vo log vh(y x. (0: (0r+1 00) = Vo =o))) o.
= (Vg Voz' (-
V, log (x.)) V02' (,
V2 log n (2x) Voz (Voz'(a VV (zx)) (.)
= (x (( (a())),))))
Similarly, for consistency learning dynamics, the only difference is that the update sample is changed from a(xs) to A(us), and the loss is changed from Lrup to Lem(A(uu (A(us), (.). Note that q arg max, th is treated as a constant in this small step (stop-grad), so the gradient can still be directly caleulated w.r.t. Thus,
gc+1 0¹ V Loon (A(us),88)
17

---

Published as a conference paper at ICLR 2026
Parallel to the above derivation, obtain
A log =0 - (y) Toi ub)
-n/ (2)) Voz (ro) (Voz' (Voz'(A(uu)) foom (Ø6: A(m))((()))))
(((ooA((us))
Soon (A(u
+0 (22)))o((((()))))))
C.2 PROOF PROPOSITION
Proposition (Invariance of baseline image under affine normalization) Let I k la be baseline image, where k E (0.1..... 255} and 1a E R. is an all-one vector. Suppose the output of the first hidden transformation normalized by a normalization layer (e.g.. BatchNorm, LayerNorm, InstanceNorm, or GroupNorm) with affine parameters (W2, b). Then the logits h(I) are independent of le and reduce to
h(I)= b.
To (I) Softmax(b).
(9)
Proof. Consider a neural network with two layers: the first layer is linear transformation, and the second layer is normalization layer followed by affine transformation. For an input I R assume the model has the following structure:
(() ET (I)
(Z) -
- oW(II I(I) =
W + b,
(arlhh()
Let the baseline image I k 1a, where 1a is a vector of ones, and & is a scalar. Our goal is to show that the output hII for the baseline image is independent of k and depends only on the bias term For the baseline image = h la, the output of this neural network is:
( o (k 1-)) = ok W11,) = alk w).
where Wil RM which is constant vector. We see that the output of the first layer depends on and the constant vector u, and it passed through the activation function a. Now, consider the effect of the BatchNorm layer. For the baseline image I la, since (1)) (Z) ak w) is constant vector, the mean (I)()) and variance Var(h (I)] are constants that depend only on w From first principles, we can set & = 0
Note that if the input I is random Gaussian noise or a batch mean, the situation would be different.
Gaussian Noise. Let In ~ N(0,00) E Rd random Gaussian noise vector. After normaliza-
tion:
-E M(I)) - (ZN) W.+ (Z)) +E
Since the input pixel values are random, the mean and variance of the first-layer output depend on the noise distribution characteristics. These statisties fluctuate with the randomness of the input, in contrast to the baseline image, where the normalized output is solely determined by the bias term b.
DB Is E Rdd be the batch mean vector. After normalization, the Batch Mean. Let I, affine transformation:
(II,)) = (L)I)) - (I))
(Var W. bb
Unlike Gaussian noise images, the mean input of data within batch does not contain complete randomness; the and variance relatively stable but still do not solely depend the b.
18

---

Published as a conference paper at ICLR 2026
C.3 PROOF OF THEOREM 1
Theorem 1. (Bias as the conditional distribution prior) Assume the model h(z) as characterized in Eq. (8) is trained using cross-entropy loss:
L [aa [ v" log Softmax(M(=))]
(10)
At a population risk minimizer (W2 b²) we have
p' (I) so Softmax(b") P(y) A - (a)
i' (x) x))
0). (11)
For the baseline image I in Proposition the baseline prediction thus coincides with the conditional class distribution the normalized-zent feature state, capturing the class prior induced by the long-
tailed training distribution.
Proof. Consider the two-layer network fe(x) (0 (x)] (arl Y + 3, where (x) = Wix.
The cross-entropy loss is given by:
L = ((v) [-y] log Softmax Softmax((xx)]
Minimizing the population risk results in (x) For the baseline image I, we analyze the model's output:
(e(ttaax(h(x))) = P(y x).
6r (I) = Softmax(b").
Since - (2) for baseline image with no input signal, the model's output is deter-
(Z)) mined solely by b*.
Thus, we have:
P - (I) (Z) 0)
softmax(b").
[arlhh( +e
Finally, we conclude that the baseline prediction corresponds to the conditional class distribution at the normalized-zene feature state, capturing the class prior induced by the long-tailed distribution.
C.4 PROOF OF PROPOSITION 4
Proposition 3. Let π = Softmax(z) and = = go(z). The one-step dynamics decompose as
log (y = =
+ On
(13)
where (I) log (I) = 1- (Z)
(V V, (Vo2()))))))))) is the empirical neural langent kernel of the logit network z, and (3 v.L(x,)) 1at.
Proof. Inspired by the analysis of the learning dynamic of (Ren al., 2022; Ren & Sutherland, 2025). In this work, we want to observe the classifier's prediction on the baseline image I. Starting from Eq (12), we first approximate log //+1 (y Σ) using first-order Talyor expansion, with slightly abused symbols, we use n to represent Tg++1, & to represent labeled sample æi and u to represent unlabeled sample un
log nt - (g|Z)= = log (y/ 7glg) 0t +0(100011000
Then, assuming the model updates its parameters using SGD calculated by an "updating labeled example" (z, y) or an "updating unlabeled example" u, we can rearrange the terms in the above equation get the following expression:
A log (v/ log (y) log '+1 (y/ = Vo log x ( (00)))))(0000))))00)00))))))
19

---

Published as a conference paper at ICLR 2026
To evaluate the leading term, we first take labeled sample example plug in the definition of SGD, and repeatedly use the chain rule:
Vo Volog 1' (z) (0 0²) (v= log N' (
(V log ग - -Vo (o='()) (33)
= - =,
= (1
C.5 MORE ABOUT ANALYZING THE DYNAMICS OF THE LOGITS DEBIASING ALGORITHM
C.5.1 PER-STEP DECOMPOSITION RESAMPLING
Resampling is another widely used strategy for mitigating class imbalance in long-tail semi- supervised learning. Instead of modifying the loss, resampling adjusts the data distribution by al- tering the frequency with which each class is drawn. Let IPss (I E c) = r denote the (possibly normalized) sampling ratio for class e, which determines the probability of selecting samples from that class during training. Then the per-step update of the log-posterior under resampling becomes
log Th L (y I; - T" (I) RI, (I, (e, v: O (v
(34)
(I, (2, yr Egare (G' v)] This decomposition highlights that resampling influences learning solely through changing the expectation measure. The modified kernel K;;, reshapes how training samples transfer influence to the test input, while the modified residual term G reweights the magnitude of each update. Increasing the sampling ratio of tail classes therefore amplifies their effective contribution at every step. accelerating their representation and decision boundary updates to match those of head classes, i.e. offering direct dynamical explanation for the effectiveness resampling in long-tail regimes.
whereR, (I; x; r")
C.5.2 PER-STEP DECOMPOSITION OF CDMAD
In this section, use the loss function of specific method in logits adjustment, CDMAD (Lee & Kim, 2024), as a case study and integrate it into the learning dynamics framework we propose. The consistency loss of CDMAD as:
B 1 1(max( (6) 2 TT) ()P((((l(((us
Lcon (w,, 0;) uB
(35)
where is cross-entropy loss, L = arg mwx max(mo(yla(u))) - no (yl Z)). Our framework reveals that CDMAD operates through two complementary dynamical mechanisms:
log Th(y Σ) -1T (T (I, (A(u)), axxx))) + O
(a(x)) 96)
(36)
According to the analysis of Xing et al. (2025), G using the baseline image enhances the balance of the base SSL model implicitly utilizing the integrated gradient flow VoLco
094. Σ. DL_ IntegratedGrads, (u,)) Vgb + E, 9A.6 20
We now place VoLcon directly into g
to capture the influence of the consistency loss on the model's update dynamics. The updated g is:
Lo a(xx)) = Σ
IntegratedGrads.
V + Σ qAo
dqA.
(37)
a0
The term G (A(u)) a(u)) explicitly includes the consistency loss gradient VoLcons which involves the Integrated Gradients over the perturbations us as well as the change in model output probabilities.
20

---

Published as a conference paper at ICLR 2026
Table 7: Comparison of bACC/GM on CIFAR-IO-LT under different baseline images.
TixMatchDDyTiin
CIFAR-IO-IT
Type of baseline
= 7/u =
100
100, 7u = 150
Noise
77.7 76.8
76.7 / 75.8
Batch Mcans
78.0/76.1
76.7 / 74.2
Red
83.5/83.2
82.2 81.7
Green
83.7 83.3
81.5/81.0
Blue
84.5 84.2
8 82.6
Gray
84.1 /3.7
82.3 81.9
White
84.2 / 83.8
82.4/ 82.0
Black
84.8 / 84.4
83.8 / 83.4
C.6 EFFECT OF THE BASELINE IMAGE FOR GUIDING DATA PRUNING
The training objective can be interpreted as the minimization of the empirical risk L. Assuming that all labeled samples zy from X' and unlabeled samples up from U are drawn from continuous distributions (x*) and pa (ul,") respectively, the training objective can be formulated
S. Couplag. (x))daz + S Lcon(ul" O)p (ul ddu."
arg min 9Ee
[C(a ui; ;]; ag agex, Eu
(38)
After applying a data pruning policy, we sample oi and uz," obtain the labeled pruned subset S and the unlabeled pruned subset Si", according to the labeled pruning probabilities P\ (z() and un- labeled pruning probabilities (u,"","", respectively. For the labeled samples, we directly optimize over the pruned subset S without reweighting the loss terms. Notably, the class-aware pruning probability r. No (I) inherently adjusts S toward an asymptotically balanced class distribution. By retaining more samples from minority classes (lower re) and pruning more samples from major- ity classes (higher r.), the pruned subset S naturally mitigates class imbalance, As result, even without explicit rescaling, the empirical risk over S approximates:
(x 0)) 1- (-P(())) o5 agest
J. [ O)p((z))dz}
arg min 8EB
(39)
di
where of = = mpi 1 - (=.)) The term 1-P((s) of acts as an implicit reweighting due to the class- aware pruning policy. For unlabeled samples, pruning with uniform probability and rescaling losses by Tu) I- (w) yields
1 [con(ul (uf ( (ul" ddu""
arg min
E
[ro(u])) (eon (u, .0] x ct
(40)
868
un ES"
(u Crucially, even with uniform pruning rates, the interplay of consistency regularization and confidence thresholding ensures S/' to be implicitly balanced, thus training on Sy with rescaled factor 7: (e(")) could achieve better result as training on the Ud
where oi = Eug
D MORE ABOUT THE BASELINE IMAGE
D.I MORE DETAIL ABOUT SELECTION OF BASELINE IMAGE
Sensitivity of different baseline images Z. We further examined the sensitivity of DyTrim to the choice of baseline image by conducting ablation studies on CIFAR-10-LT with different types of inputs, including noise, dataset means, and solid colors. Table 7 shows that solid-color images sistently outperform noise or mean-bel bean-based baselines. Among them, white and black images deliver the strongest results.
21

---

Effects of bias term. When the bias term /3 of the BN layer is frozen and equal to 0, h(T) becomes Y * ((w, k) [[, k/))/ /arr (w, (w, k)] which the same as the Eq.(7) except for bias term. Ignoring the running statistics strategy, the form of h(I) only depends on the 3. As result, ((T) becomes h(I) - 0 during training and l(Z) + -y * Emom [(w, / Varaomm [(w, ro)]] during testing. This shows that the gh operation has no effect the training phase and only eliminates the Published as a conference paper at ICLR 2026
Reference logits
Class-aware dynamic dataset pruning for class
Head
"cal"
class-eware pruning
"dog'
baselina image
Mn
Maxe
labeled pruned set S
biased degree
Tail
"aird"
more balaanedd
pruning
X1
X2
Labeled Dataset X
unlabeled pruned set go
Gradient rescaling
2?"
High confidence
labelirsennitive pruning
Unkown
"?"
SS
S
S
*?"
Low confidence
Unlabeled Dataset '/
Figure 4: Illustration of the proposed DyTrim framework. DyTrim mainly consists of two opera- tions, named labeled pruning and unlabeled pruning. H' and H," denote the adaptive thresholds of scores of labeled samples and unlabeled samples, with slight abuse of symbols. S denote the low confidence unlabeled sample which p" (u²") 2 т. Labeled pruning provides class-aware prun- ing policy for each sample from class Unlabeled pruning provides random pruning policy from the original unlabeled '/ and uses a gradient rescaling strategy (x1/(1 - for which samples from of is selected to pruning) to keep the approximately same gradient expectation.
D.2 DETAIL OF THE BIAS TERM AND RUNNING STATISTICS
impact of the unbalanced rumning means in the testing phase. This will affect the ability to benefit from gg. as shown in Table. 8.
Effects of running statistics. When we not keep running estimates, batch statistics are instead used during evaluation time as well. The form of h(()) becomes h(T) + 3 both training and testing. We can rewrite ø3 (z.) 7 ((w, x.) (w, s2)l)// / (Varr((w x1)] On the other hand, as h(I) - 0, the benefit of 93 is also vanishes, also shown Table.
We then extend our results to non-linear neural network, thus we have the following corollary:
Table 8: Comparison of bACC/GM CIFAR-1O-LT.
Metric
With original gō
96 without /
g without mom
96 without 3 & X
bACC
83.6 0.46
80.92 + (2.68
71.63 0.35
64.01 + 0. 19.55 54.48 + 0.36._8.62
GM
83.1 0.57
80.37 + 0.23
67.85 + 0.51
E MORE DETAILS ABOUT DYTRIM
E. MORE ABOUT LABELED PRUNING
Specifically, we exploit the pruning policy to prune samples based on their scores. Then, for the pruned labeled samples, their scores remain unmodified previously. For the remaining samples,
22

---

Published as a conference paper at ICLR 2026
their scores are updated by the losses in the current epoch. To ensure dynamic adaptation:
(x,)+ = =
H (a)( (x))
oo E XnS' Ig S
(41)
where St denotes the pruned subset formed for labeled datasets.
E.2 MORE ABOUT UNLABELED PRUNING
For a remaining sample with score Hf (ug") < Hi, whose corresponding pruning probability is r, its gradient is scaled to 1/(1 - r) times of the original, otherwise the gradient remains unchanged. The score 1 (u.² ( derived from the consistency regularization loss values Leon (ox(ug"). A(ug")) for unlabeled data points. To enhance pseudo-label reliability, we further apply confidence threshold т, where only samples with p" (uf") > т contribute to Leons where Leon Σ [(p"(u)") (ug") H(Po(llllA(u (6). Thus, we formulate the update of ( as:
H6 (4 -{
H (ug) Lcom (ug
" E uns"" " S
(42)
where S" denotes the pruned subset formed for labeled datasets. Initializatinn: at t = 0, scores H. and Hd all set to {1}, prior loss is available.
F PSEUDO CODE OF THE PROPOSED ALGORITHM
The pseudo-code that describes the DyTrrim is presented in Algorithm 1 and Algorithm 2.
Algorithm DyTrim for Labeled Data Selection
Input: Labeled set of N samples X' """) score set of the samples V number of classes nhe biased degree b Output: Labeled pruned set St (8) C X, |8| <= |XD
I: S 0
Initialize the labeled pruned set
2: for c == to n. do
3:
I, f- {i y/ =
4:
V ( lie Ic}
Select scores of class samples Compute target pruned set size of class based on biased degree & Select indices of top-k, scored samples
5:
ke - [1 be) -|eell
6:
Iep < TTp V., km)
7:
S + S' U Itop
8: end for
9: return S
G EXPERIMENTAL SETTINGS
G.1 1
MODELS
Unless otherwise specified, we adopt Wide ResNet (WRN) (Zagoruyko & Komodakis, 2016) as the default backbone following common practice in semi-supervised learning. Additionally, also evaluate Tiny Vision Transformers (TinyViT) (Wu et al., 2022) on CIFAR-10-LT and CIFAR-100- LT. For ImageNet-127] employ ResNet-50 (He et al., 2016) as the backbone to ensure scalability on large-scale datasets.
G.2 IMPLEMENTATION DETAILS
All experiments are trained for 500 epochs with 500 steps per epoch, resulting in a total of 250,000 iterations. We use Stochastie Gradient Descent (SGD) (Bottou, 2012) with fixed learning rate of 0.0015 and a batch size of 32. The pruning ratio of the unlabeled dataset is set to 0.7, and the pa- rameter 8 is aligned with InfoBatch (Qin et al., 2024), fixed 0.875. For CIFAR-10-LIT the largest
23

---

Published as a conference paper at ICLR 2026
Algorithm 2 DyTrim for Unlabeled Data Selection
Input: Unlabeled set of M samples U {(u")) ] score set of the samples V", pruning ratio r, weight of samples w Output: Unlabeled pruned set St (S' u |54| <= RT
1: S" + 0
Initialize the unlabeled pruned set
2: Ia + {i V. = 0}
> Select low confidence samples
3: Iza t ffil V/ VM + 0}
Select high confidence samples
4: S" 5 UIg
5: n Mean( (V E
6: Iwell + fi E Iyn
V < }u]
D Select well-learned samples
7: Ipcor + I4 Iwell
> Select poorly-learnn samples
8: S" + S" UIoor
9: Ixclect RRandomly select [(1 - r). [Teel]] samples from Twell
10: S" - Ineles
Wi 4 Vi El. (l..... M}
Reset weights D Rescaling
12: f I-
Vi E Inclect
13: return S"
labeled class contains 1,500 samples, while the largest unlabeled class contains 3,000 samples. For CIFAR-100-LT, the largest labeled and unlabeled classes contain 155 and 300 samples, respectively. For STL--1-LLT, the largest labeled class contains 450 samples. To assess classification performance, adopt balanced accuracy (bACC) (Huang et al., 2016) and geometric mean (GM) (Kubat, 1997) for CIFAR-IO-D 10-LT and STL-IO-LT. For CIFAR-IOR 100-LT and ImageNet- 127. evaluation is conducted solely using bACC. Each experiment is repeated three times on RTX 4090 GPUs to ensure repro- ducibility, and we report both the mean and the standard error.
H ADDITIONAL EXPERIMENTAL RESULTS
H.1 BASELINES
The classification performance of the DyTrim was compared with those of the following algo- rithms: 1. vanilla algorithm Deep CNN trained with cross-entropy loss, 2. CIL algorithms Resampling (JAPKOWICZ, 2000), LDAM-DRW (Cao et al., 2019), and cRT (Kang et al., 2020), 3. SSL algorithms FixMatch (Sohn et al., 2020), and 4. CISSL algorithms DARP, DARP+LA, DARP+cRT (Kim et al., 2020), CReST, CReST+LA (Wei & Gan, 2023), ABC (Lee et al., 2021), I), CoSSL (Fan et al., 2022), DASO (Oh et al., 2022), SAW, SAW+LA and SAW+cRT (Lai et al., 2022) combined with FixMatch. Adsh(Guo & Li, 2022), DebiasPL. (Wang et al., 2022), UDAL et al., 2023) and 1.2AC (Wang et al., 2023a) combined with FixMatch. We report the performance of the baseline algorithms reported in Tables of Lai et al. (2022) and Fan al. (Fan et al., 2022) when it is reproducible; the performance measured using the uploaded code was reported otherwise.
H.2 ADDITIONAL RESULTS ON CIFAR-IO-LT
Following prior works (Xing et al., 2025; Lee Kim, 2024; Guo et al., 2024), we evaluate under a more challenging scenario where the unlabeled set is imbalanced in the reverse direction of the la- beled set (Table 9). Across all settings, DyTrim delivers consistent gains by applying balanced prun- ing on the labeled data. Notably, when combined with FixMatch, DyTrim surpasses CDMAD by more than 1 1 in both bACC and GM. Similar benefits are observed for FlexMatch and FreeMatch: DyTrim improves Flex by approximately 1.1-1.3% and FreeMatch by around 0.9-1.5%.
We also compared the classification performance of CDMAD with ACR (Xiang et al., 2020) and BaCon, two recent CISSL algorithms. From Table. 10, we can observe that CDMAD outperforms both ACR and BaCon.
To further validate the balanced classification effect of DyTrim, visualized the dynamics of baseline image logits during training as shown in Figure. (a), (b) and (c). The results clearly showed that Dy Trim significantly reduced classifier bias induced by class imbalance.
24

---

Published as a conference paper at ICLR 2026
Table Comparison of bACC/GM on CIFAR-IO-LT 100, 7u (00
CIFAR-IO-LT, 100, 100(reeered)
Algorithm
ABC
SAW
SAW+LA
SAW+cRT
CDMAD
DyTrim
FixMatch+
69.5/66.8
72.3/68.7
74.1/72.0
75.5/73.9
77.1/75.4
78.2 76.7
FlexMatch+
67.2/65.1
66.4 69.4 67.9
FrecMatch+
68.5/66.4
Table 10: Comparison of bACC/GM on CIFAR-IO-LT
Algorithn/CIFAR 10-LT
Th. 100
7 %au
FiXMatch+ACCR
81.8/ 81.4
85.6 85.3
FixMatch+BaaCon
84.4 / 84.0
82.0 81.5
FixMatch+CDDMAD
83.6 83.1
87.5 87.1
DyyMatch+DyTrim
84.8 / 84.4
87.9 87.5
100
0.8
19
6.6
0.6
96
0.6
9.
0.1
0.2
02
60
598 ACD 200 Epach
Class
300
Clar
335
00 300
Cpoch
100
Methods
(a) FixMatch
(b) CDMAD
(c) DyTrim
(d) Metric
Figure 5: (a). (b) and (c) present the change of To (Z) for the baseline image on CIFAR-IO-LT with
N 7u = 100 across different methods. (d) present the bACC and GM on those methods.
H.3 RESULTS ON SMALL-IMAGENET--IE
ImageNet-127 is a naturally long-tailed dataset, widely used to evaluate class-imbalanced superisue learning (CISSL) algorithms at scale. Following standard protocol, we downsample images to resolutions of 32 and 64 x 64 using the box interpolation method from the Pillow library, and randomly select 10% the training samples as labeled data. Under such limited supervision and class imbalance, learning discriminative representations and a balanced clas- sifier is particularly challenging. As reported in Table. 11. DyTrim achieves the highest balanced accuracy (bACC) at both resolutions, outperforming the strongest baseline CD- MAD by 3.0% at 32 x 32 and 1.2% at 64 x 64. These im- provements demonstrate the robustness of our method, espe- cially under low-resolution and low-resource conditions. The performance gain lower resolutions suggests that DyTrim effectively handles the compounded difficulty of reduced vi- sual fidelity and severe label scarcity. This makes a promising solution for real-world applications where high-resolution data and abundant labels are often unavailable.
Table 11: Comparison of bACC on Small- Soall-ImaeNNet-io 127.
Small GeaalllmmgeNeet... 127
Algorithm
32 x 32
64 64
FixMatch
29.7
42.3
w/+DARP
30.5
42.5
w/+DARP+RT
39.7
51.0
w/CCReST
32.5
44.7
w/CReST+LA
40.9
55.9
w/AABC
46.9
56.1
W/+CoSSL
43.7
53.8
W/+CPE
47.8
58.2
w/CDDAD
48.4
59.3
w/+DyTrim
50.6
60.0
H.4 MORE RESULTS ON IMAGENET-LT
ImageNet-LT (Liu al., 2019) is long-tailed variant of ImageNet, constructed exhibit heavy class-imbalance that better reflects real-world data distributions. To assess the scalability our method large-resolution inputs (224 224), conducted experiments ImageNet-LT. Due to hardware constraints, we set the batch size to
As shown in Table 3, CDMAD yields substantial improvement over the FixMatch baseline, in- creasing bACC from 20.0% to 35.4%, which highlights the effectiveness of incorporating class- distribution modeling under long-tailed imbalance. Building upon the same baseline, method
25

---

Published as a conference paper at ICLR 2026
further pushes performance to 37.2%, achieving the best result among all compared approaches. Notably, the improvement over CDMAD remains consistent despite their strong performance, sug- gesting that our approach introduces complementary benefits rather than merely overlapping with prior re-balancing techniques.
H.5 RESULTS ON DYNAMIC DATA PRUNING EXPERIMENT
Recently, Infobatch (Qin et al., 2024) provides a no-bias dynamic data pruning method. In this section, compare it with DyTrim in the framework of CISSL The experiment is conducted on the CIFAR-10-LT dataset, comparing the settings of " = "hu and " * 7u- Specifically, we directly apply the pruning policy of InfoBatch labeled samples and unlabeled samples without distinction, and the results are shown in the Table. 12 and Table. 13. It can be seen that compared with the proposed method, the pruning policy directly combined with InfoBatch is not consistently effective in all settings, In particular, when " * Yus it will cause a decrease in accuracy, which is caused by the mismatch in the distribution of labeled samples and unlabeled samples.
Table 12: Comparison of bACC/GM on CIFAR-IO-LT.
CIFAR-IO-LT (7 = = Yu. Yo assumed be known)
Algorithm
?/1 50, 7v- 50
7 100, 7/2 100
7 150, 7v. 150
FixMatch
7 #0.36
71.5 0 66.8 11.51
68.4 59.9 +0.43
w/+CDMAD
87.3 87.0
83.6 13.6/ 83.1 =0.57
80.8 #0.86 79.9 1.07
W/tInfoBatch*
07.2 86.9 10.19
84.1 10.61 / 88.77 +0.69
81.6 #0.45 / 80.9 +0.59
w/+DyTrim
88.0 15.01/...88.88111 +0.32
84.8 +0.48 04..48/84.44 +0.51
82.0 +0.09 81.3 +0.03
Table 13: Comparison of bACC/GM on CIFAR- 10-LT (?1 * Yus Yu is assumed to be unknown).
CIFAR- I0-LT (t) 100, 7u Unknown)
Algorithm
7/ 50
%1 150
FixMatch
68.9 42.8
73.9 #0.25/ 70.5
69.6
w/+CDMAD
87.5 87.1 +0.50
85.7 10.7 85.3 10.38
82.3
81.8 +0.29
w/+InfoBatch*
86.4
85.9 +0.73
85.5 40.33 85.1 +0.37
83.3
82.8 +0.11
w/+DyTrim
88.9 -0.88
86.4 000.4338660000 +0.43
83.8 10.34/
/3.4 #0.33
H.6 ABLATION STUDY
Effectiveness of each component. We conducted ablation studies on CIFAR-IO-LT assess the contribution of each component in DyTrrm, varying the hyperparameter 7 100, and 150. As shown Table. 14, the best performance was achieved when both labeled and unlabeled pruning were combined with rescaling. Removing rescaling led to bACC drop 0.8- points across 7 values. Excluding either pruning component also reduced performance (e.g. -0.5 and -0.3 y 50 without unlabeled or labeled pruning, respectively). Removing both pruning strategies resulted the most significant degradation. These results highlighted the complementary benefits of pruning and rescaling.
across 50,
H.7 QUALITATIVE ANALYSES
Since the baseline image could implicitly reflect the bias of the classifier, argued that by tomizing dynamic data pruning methods for labeled and unlabeled data, DyTrrm significantly re- duced classifier bias while improving performance. To verify this claim, in Figure. 6 (a) and (b), analyzed the class probabilities predicted on the baseline image using FixMatch+DyTrinn on CIFAR-IO-LT under various settings. We observed that classifiers trained with DyTrim con- sistently produced more balanced predictions than CDMAD across all settings, with improved ac- curacy on tail classes. We defined as the probability of pruning an unlabeled sample ug." when H((u)) < Hm and max (Po(y) 2 In Figure. 7, evaluated different pruning ratios
trained
26

---

Published as a conference paper at ICLR 2026
Table 14: Ablation study for the proposed algorithm on CIFAR-IO-LT.
Labeled
Unlabeled
Rescaling
7/u
50
71 = 7v-
100
71 Ya
150
Pruning
Pruning
bACC
GM
bACC
GM
bACC
GM
87.3
87.0
83.6
83.1
80.8
79.9
87.5
87.2
84.4
84.0
81.3
80.6
87.7
87.4
84.0
83.6
81.4
80.6
87.2
86.9
83.6
83.1
79.9
79.0
88.0
87.8
84.8
84.4
82.0
81.3
11 100 103
130.
Probabilltes
1.7
Class Probabiiliicss
True Label
Class Index
Predicted Label
Predicted Label (d) F FixMatch+ DyTrim
(a) FixMatchCCDMAD
(b) FixMathh-DyTrie
(c) FiMMatch-CDMAD
Figure 6: (a) and (b) present the (Z) using the CDMAD and DyTrim. (c) and (d) present the
confusion matrices of the class predictions on test samples on CIFAR-IO-LT (? 7a 100).
for unlabeled samples on CIFAR-IO-LT. Results showed that setting r 2 0.1 yields higher perfor- mance across both architectures, indicating that DyTrim was relatively robust with respect to the hyperparameter with the best performance achieved when 0.3.
85
84
Performance
83
bACC
82
G M
Ratio (%)
Figure 7: Evaluation curves of hyper-parameter r on CIFAR-IO-LT under bACC and GM.
H.8 COMPARISON OF CLASS DISTRIBUTIONS BEFORE AND AFTER PRUNING
Figure 8 compares the class distributions before and after applying DyTrim on the labeled, unlabeled and full training sets. Across all three subsets, pruning consistently reduces the proportion of head classes while preserving or slightly increasing the relative proportion of tail classes. This produces a noticeably flatter long-tailed distribution. Unlike traditional pruning methods, which typically remove samples that contribute least to training progress, the behavior of Trim is different because the pruning decision is guided by baseline logits and the reliability pseudo- eliminate redundant head-class samples and low-quality unlabeled samples while rarely discarding the already scarce tail-class data. Consequently, the resulting effective training subset becomes more balanced without sacrificing essential information from tail classes.
This tends to
27

---

Published as a conference paper at ICLR 2026
40
40
40
Before pruning Aftor pouning
pruning After pruning
Before pruhing
35
36
35
Aftor puninng
Proportion (%)
30
Proportion (%)
30
Proportion (%)
30
25
25
25
20
20
20
15
15
15
10
10
10-
5
6-
5
0
0
0
3
4
5
8
0
1
2
5
6
2
3
6
7
8
Class
Class
Class
(a) Labeled dataset
(b) Unlabeled dataset
(c) Full dataset
Figure 8: Comparison of class distribution before and after pruning across three datasets: (a) Labeled
dataset, (b) Unlabeled dataset, (c) Full dataset.
H.9
ANALYSIS OF SAMPLE SELECTION FREQUENCY
1.0
w/o scaling
0.4
0.9
Maximum frequency
Class Probabilities
scaling
Average frequency
Frequency
0.8
Minimum frequency
0.3
0.7
0.6
0.2
0.5
0.1
0.4
0.3
0.0
8
9
1
2
Class
Class
Figure 9: Illustration of per-class maximum, average, and minimum sample selection fre- quencies during training.
Figure 10: Comparison of class- blass-probability distributions with and without scaling.
Figure 9 reports the maximum, average and minimum sample selection frequencies for each class. Three observations emerge clearly. First, the maximum frequency remains close to for all classes, which indicates that each class contains at least a subset of highly informative samples that almost always preserved dur- ing pruning. Second, the average frequency in- creases from head tail classes, showing that DyTrim removes a larger fraction of redundant samples from majority classes while retaining more samples in minority classes. This behavior matches the intended effect of mitigating class domi- nance through selective pruning. Third, the minimum frequency stays within narrow and rela- tively high range across all classes, suggesting that even least frequently selected samples are not entirely discarded. This prevents the severe under- inder-sampling of tail classes that often occurs in traditional pruming strategies.
Table 15: Comparison of bACC and GM CIFAR-10-LT on fixed and dynamic scaling fac- tors.
CIFAR- 10-I.
Algorithm
100, 7/ = 100
21 100, = 1/100
Fixed Scaling Dynamic Scaling
8 /8.4
76.7
84.9/ 84.4
7../78.1
H.10 EFFECT OF SCALING STRATEGIES ON CLASS-BIAS
Figure compares the class probability distributions obtained with and without the proposed scal- ing strategy. Although the curves differ for several head and mid-frequency classes, the overall
28

---

Published as a conference paper at ICLR 2026
decay pattern remains consistent, and the probabilities of head classes do not increase when scaling applied. This shows that the scaling mechanism does not intensify the influence of high confidence samples and preserves the ong-tailedd structure shaped by DyTrim.
Additionally, to provide each class with an adaptive scaling factor that assigns smaller scaling to head classes and larger scaling to tail classes, further compare fixed and dynamic scaling in Table 15. Dynamic scaling leads to higher bACC and GM under both matched and mismatched imbalance conditions, indicating that adapting the scaling factor to the current pruning state yields a more reliable correction for changes in the effective batch size. The dynamic scaling factor is computed as 1 Ty (I) + 1/(1 r). which stabilizes the loss magnitude during training and prevents undesirable shifts toward majority class predictions.
H.11 DYNAMICS OF SAMPLE SCORE ACROSS HEAD AND TAIL CLASSES
Head sample
4
Tail sample
3
Score
AMA AM
2
0
0 5000 10000 15000 20000 25000 30000 35000 40000 45000 Steps
Figure 11: Scores of a representative head class sample and a representative tail class sample over the first 50,000 training steps, recorded every 500 steps.
Figure 11 shows the dynamics of scores for one head class sample and one tail class sample over the first 50,000 training steps. The two trajectories exhibit a clear contrast. The tail class sample maintains consistently higher and more volatile scores throughout training, reflecting its larger con- tribution to reducing class bias and its higher utility for updating the classifier. In comparison, the head class sample quickly drops very low scores and remains close zero for most of training. This indicates that the head sample becomes saturated early and provides little additional informa- tion, which aligns with the design of DyTrim that aims to remove redundant head class samples.
H.12 PRUNING DYNAMICS ACROSS LABELED AND UNLABELED DATASETS
Class
Class
Closs
Class 7
Class 8
Number of pruned tamples
800-
of gruned samples
12
14
Number of pruned samples
2500
Number of pruned san samples
200
800
10
2000
200
1500
400
100
200
500
60-
500 1000 1500 2000
500 1000 1500 2000
500 1000 1500 Number of pruning
500 1000 1500 2000 Number pruning
Number of pruning
Number of pruning
(a) Head classes (labeled)
(b) Tail classes (labeled)
(c) Head classes (unlabeled)
(d) Tail classes (unlabeled)
Figure 12: Number of pruned samples for each class across training process CIFAR-IO-LIT (a) and (b) show the evolution for head and tail classes in the labeled set, and (c) and (d) show the corresponding results for the unlabeled Each curve indicates how many samples of given class have been removed up to each pruning step, recorded every 100 iterations.
29

---

Published as a conference paper at ICLR 2026
Faratth
CDMAD
InfoSatch
Dylri
10
[10
0.
0.
0.
103
04
first epoch
6.3
96
110
Banch
10
Batch
100
Batch
400
0.8
0.
9.5
0.6
0.6
0.
first 10 epoch
0.2
5.4
0.2
6.1
02
0.4
8.
8.0
8.
Class
Class
Claa
Epoch
Epach
Epoch
Spach
10
0.3
1.
full epoch
9.0
aM
up0
200
Cass
500 x0
Sporth
Cass
Epoch
Clasa
500
Cpcch
26
to
Figure 13: Comparison of the change of logits's ' probability distribution ne (T) for the baseline image Yu 100 across different CISSL methods.
on CIFAR-10-LT with "
Figure reports the number of pruned samples per class over the course of training. The results from both the labeled and unlabeled subsets exhibit consistent pattern. Head classes experience a rapid increase in pruned samples at the beginning of training and maintain high pruning counts throughout the process, which reflects the large amount of redundant information contained in these majority classes. contrast, tail classes show much slower growth curves with considerably lower pruning volumes, indicating that DyTrim preserves most of the scarce minority samples and avoids aggravating the long-tailed imbalance. The same trend appears in the unlabeled subset, where head classes accumulate substantially more pruned samples due the prevalence of high confidence but less informative pseudo-labeled instances. These observations confirm that Dy Dyim adaptively modulates pruning according to class frequency and sample utility, removing redundant head-class samples while retaining informative tail-class data.
I VISUALIZATION
1.1 DETAILS OF THE CHANGE LOGITS'S PROBABILITY DISTRIBUTION
In this section, conduct some visualization experiments to demonstrate the advantages of the DyTrim in debiasing and improving classifier performance. We first analyze the change of logits's 's probability distribution Sof toftmax(gg (T)) for the baseline image CIFAR-1O-LT with 7 = 7a 100 for fixmatch, CDMAD, and the DyTrim as shown in Figure. 13. It can be seen intuitively that the first epoch, the classifier has bias due to the imbalance of categories in the data. This situation increases significantly with the number of network training times, as shown in the second column of the figure. However, we can see that Dy Trim can effectively slow down the increase of this bias. Furthermore, after the model is fully trained for 500 epochs, it can be seen that after the 100th epoch, CDMAD starts use the baseline image for post-hoc debiasing, which significantly reduces the representation of the model. However, by dynamically pruning the data set, Dy Trim obtains more distinct debias effect shown in Figure. 14.
I.2 DETAILS OF THE CHANGE OF LOGITS'S S PROBABILITY DISTRIBUTION
Figure. 15 and Figure. 16 compare the confusion matrices of the class predictions on the test set of CIFAR-10 using (a) FixMatch, (b) FiMMatch+Infohatch (c) FiMMatch<CDMAD, and (d) Fix-
30

---

Published as a conference paper at ICLR 2026
16l,, 383
183 W
Pim 108
Probabiitoas
Class Probstalidiss
Cless Probebalitees
Class Probatbiitess
03
02-
02-
Class Index
Class Index
Class Index
Class Index
(a) FixMatch
[b] FisMalch+nnfoBllch
[c] FMM
(d) FixMatchee DyTrim
Figure 14: Class probabilities predicted on a baseline image using (a) FixMatch, (b) Fix-
Match+InfoBatch, (c) FiXMatch+CDMAD, (d) FixMatch-DyTrim.
True Label
True Label
True Label
Predicted Label
Predicted Label
Predicted Label
Predicted Label (d) FixMatch+ DyTrim
(a) FixMatch
(b) F FiMMathh
(c) F FiMMatch+CDMAD
Figure 15: Confusion matrices of the class predictions on the test set of CIFAR- 10 using (a) Fix- Match, (b) FxMMath--Inooatcch (c) FixMalchCCDMAD, and (d) FixMatch+DyTrin trained on
CIFAR- 1--R- under 7/1 100 and 7u 100.
True Label
True Label
Tine Label
Predicted Label
Predicted Label
Predicted Label
Predicted Label (d) FixMatch+ DyTrim
(a) FixMatch
(b) F ToMaachhinfoBatch
(c) FiMMatch+CDMAD
Figure 16:
Confusion matrices of the class predictions on the test set of CIFAR-10 using (a) Fix-
Match, (b) IiMMatch+Inffoatch, (c) FiMMath+CDDMAD, and (d) DixMatchDDTThin Trim trained on
CIFAR- 10-LT under 7/2 100 and %
Match+DyTrrim trained on CIFAR-10-LT under " = 100, Tu = 1, 100. FixMatch+DyTTrim made more balanced predictions across classes. Furthermore, we also conducted experiments under balanced setting (7 = 71 %h = 1), as shown in Figure. 17. The results show that even under a balanced data distribution, DyTrimcan still achieve better results on the pruned dataset than methods such as CDMAD trained on the full dataset.
Similar to confusion matrices, we also compare t-distributed stochastic neighbor embedding (t-SNE) of representations obtained for the test set of CIFAR-10 using FixMatch, FixMatch+CDDMAD FixMatch+InfoBacch and FixMatch+DyTrin trained on CIFAR-10 with Y = 100 and % 1, unk(uon N.), where different colors indicate different classes in CIFAR-10 Figure. 18, Fig- ure. 19. We can observe that the representations obtained using FixMatch+DyTrin are separated into classes with clearer boundaries compared the those from FixMatch and CDMAD. This is prob- ably because CDMAD appropriately refined the biased pseudo-labels and used them for training, whereas FixMatch failed to leamn the representations properly because they used the biased pseudo- labels for training. These results demonstrate that the quality of representations can be improved by using well-refined pseudo-labels for training.
31

---

Published as a conference paper at ICLR 2026
True Label
True Label
True Label
True
Predicted Label
Predicted Label
Predicted Label
Predicted Label (d) +ixMatch+ DyTrim
(a) FixMatch
(b) F FiMMatch-infoBatch
(c) FisMatch+CDMAD
Figure 17: Confusion matrices of the class predictions on the test set of CIFAR-10 using (a) Fix- Match, (b) FiMMathh-nnfoatcch (c) FixMatch+CDMAD, and (d) DixMatch+DyTrim trained on
1EAR--0-LT under ?/1 and 7u = 1.
(b) FiMMalch-hnoBBathh
FidMatch-DMMA
(6) FixMatch+ DyTrim
Figure 18: t-SNE of representations obtained for the test set of CIFAR- using (a) FixMatch, (b) IixMatch+InfoBachh (c) FixMatch+CDMAD, and (d) DixMatch+Dy Trim trained on CIFAR-10-LT
under ?/- 100 and 7/a = 100.
[b] FisMaichhhnoBBahhh
(c) FixMatch+-
DyTrim
Figure 19: t-SNE of representations obtained for the test set of CIFAR- using (a) FixMatch, (b) FixMatch+InfoBatch (c) FixMatch+CDMAA, and (d) FixMatch+DD Trim trained on CIFAR-10-LT under ?/1 100 and 7/u.
J LIMITATION
A key limitation of method is its reliance task-irrelevant baseline image bias indica- tor. If this baseline image is used as a training sample, it may no longer reflect the accumulated bias, reducing the effectiveness of our debiasing mechanism. Additionally, our framework does account for architectures with auxiliary classification heads or semi-supervised methods based on mixup-style (Zhang al., 2017) interpolations. limiting DyTrim's applicability to these models. Extending our approach to these settings is an interesting avenue for future work.
K
USE OF LLMS
Large language models (LLMs) were used solely to assist with minor language polishing during manuscript preparation. All scientific components of this work, including the design of experi- ments, data processing, analysis, and interpretation, were carried out entirely by the authors using established computational methods and human expertise, without reliance on automated reasoning or model-generated content.
32


Reviews:
[{"review_id": "4Fk74jaUH6", "reviewer": "Reviewer_hNP6", "summary": "This paper analyzes long-tailed SSL bias through learning dynamics. Key insight: predictions on baseline images (solid-color inputs) \\pi_\\theta(\\mathcal{I}) encode class bias via BatchNorm terms (Theorem 1). DyTrim uses these as pruning ratios for class-aware labeled data pruning and label-agnostic unlabeled pruning. Shows 1-3% improvements over CDMAD on CIFAR-LT/STL-LT/ImageNet-127.", "questions": "When does the method fail? What are its limitations?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Principled framework: decomposing learning dynamics with a formal link between baseline representations and class priors.\n\nConsistency: Works with FixMatch/FlexMatch/FreeMatch and WRN/ViT/ResNet.\n\nThorough validation: 15+ baselines, diverse settings (matched/mismatched \\gamma).", "weaknesses": "Incremental Gains: Performance improves by only 1–2 % over CDMAD, which is modest given their conceptual overlap—both methods rely on baseline-image statistics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yICJbpsuuG", "reviewer": "Reviewer_CLxv", "summary": "This paper tackles the problem of semi-supervised learning with heavy class imbalance. It proposes to dynamically prune the dataset according to the model confidence as well as the approximate marginal class distribution based on the logits derived from a baseline black image. The proposed approach performs well on small-scale image benchmarks (32x32).", "questions": "I am unsure why pruning of data is performed rather than simply reweighting the sampling frequency. Can the authors clarify this point?\n\nI am open to raising my score if my above issues are addressed; however, I will still keep low confidence even if I do so, as I am not familiar with this field of research.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed approach seems to intuitively make sense in solving the problem (for the most part)\n- The proposed approach seems to do well on the various benchmarks explored.\n- The presentation of the paper is clear and professional. Figure 2 is very helpful in understanding the proposed approach.", "weaknesses": "The paper load for ICLR this year has been large, and so I have not been able to spend as much time as I would like on reviewing. I encourage the authors to correct any errors/misunderstandings I may have with regards to the paper.  Moreover, I do not work in either semi-supervised or long-tailed learning, so my review confidence will be low, and I will defer to other more knowledgeable reviewers.\n\n1. **Motivation is hard to understand**\n    1. The learning dynamics theory doesn't feel particularly relevant or well-connected to the downstream approach. \n    1. Figure 1 is not very easy for the reader to interpret.\n1. **Baseline image design choice**\n    1. I am confused by the choice to use a black image as a baseline to extract marginal class probabilities. Why not just directly extract the bias vector from the final fully connected layer? This is usually accessible for most deep learning models in my experience. \n    2. Suppose we have a class-balanced dataset of images where a CNN learns to classify different solid colours (e.g. white, black, red). Wouldn't you expect the black baseline image to return a probability vector close to [0,1,0]?\n1. **Experiments only on small-scale data**\n    1. Although experimental results seem good, they are only on small-scale data (up to 64x64). I would be much more confident in the approach if the authors demonstrated strong performance on imagenet-scale (224x224) scale data such as https://github.com/zhmiao/OpenLongTailRecognition-OLTR. \n\n1. **Complexity of approach**\n    1. It seems like the proposed approach is quite complicated with many knobs for the practitioner to adjust. I am a little sceptical whether such an approach could be conveniently adopted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xam4aPgPlv", "reviewer": "Reviewer_yN6a", "summary": "In their manuscript, the authors address the problem of long-tailed semi-supervised learning. More specifically, the problem of estimating the accumulated biased is approached using techniques of learning dynamics and benchmark images. The proposed framework, DyTrim, uses these findings to guide data pruning. The approach is evaluated on three standard benchmarks.", "questions": "1. how does the proposed method goes beyond combining Ren&Sutherland, ICLR2025 and Xing et al., AAAI2025 for guiding sub-sampling?\n1. what is defined in Definition 1? In its current from, it says \"decompose\" which implies a proposition\n1. in which ways go propositions 1 and 3 beyond definition 1 (which might be a proposition)?\n1. how can the use of a bias-free first layer (line 173) be justified?\n1. what is the intuitive, non-trivial connection between proposition 2 and theorem 1?\n1. which results in the experiments were reproduced and which were taken from the literature?\n1. in case of deviations: why do the results differ?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The manuscript makes use of very recent work such as Learning Dynamics (Ren&Sutherland, ICLR2025) and solid-color input for optimal bias estimation (Xing et al., AAAI2025)\n- The writing is generally good, although the text-flow is sometimes confusing\n- Figure 1 is a good illustration, although it needs better connection with the main storyline.\n- The method has been compared in a comprehensive set of experiments with good results.", "weaknesses": "- Both soundness and contribution seem to be problematic, possibly cause by unclear presentation\n    - contribution: it remains unclear in which sense the current method goes beyond combining the works by Ren&Sutherland, ICLR2025 (btw: the reference should be updated from the pre-print) and Xing et al., AAAI2025 for guiding sub-sampling\n    - soundness: it remains unclear how the authors reflect about their use of definitions and propositions. The presentations of the theory leaves the reader with serious doubts that the construction based on definition 1, subsequent propositions, and theorem 1 is sound. The proofs in the appendix are not mentioned in the main text and seem to be not fully completed (e.g. missing references)\n    - writing: although generally good, it is sometimes unnecessary hard to interpret the writing, also caused by some language issues (e.g. line 040: \"the approach employ baseline image \") \n- Some relevant references are missing, e.g. mixing-based approaches (built on [1]) \n- The presentation of results is a bit confusing: which results are taken from the literature and which have been reproduced; why deviate reproduced values from the literature, etc.\n\n[1] Zhang et al. Mixup: Beyond empirical risk minimization. ICLR 2018.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PixZJmmy7r", "reviewer": "Reviewer_jDsT", "summary": "This paper proposes the DyTrim method, which eliminates bias by employing benchmark image-guided dynamic data pruning. The authors demonstrate that this approach implicitly implements risk reweighting, elucidate the mechanism by which benchmark images mitigate classifier bias in class-imbalanced scenarios, and reveal how class-imbalanced datasets influence a model's predictions on baseline images.", "questions": "As described in “Weaknesses”.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper demonstrates through a solid theoretical foundation that the DyTrim method can effectively suppress class bias. The DyTrim method provides an elegant solution to the class imbalance problem in semi-supervised learning without requiring complex network branches or de-biasing mechanisms. Extensive experiments on public benchmarks validate the effectiveness of the proposed method. This approach contributes to the advancement of the field.", "weaknesses": "[1] Conventional pruning strategies are typically employed to eliminate samples that contribute minimally to training, thereby accelerating the training process. However, this paper utilizes such strategies to obtain a relatively balanced training subset. Would this approach suffer from the undersampling problem, specifically for tail classes?\n\n[2] Figure 1 fails to provide sufficient evidence for the conclusions put forward in the paper. Especially in the bottom part, the predicted distributions derived from pseudo-labels—whether correct or incorrect—show no significant difference; in some cases, they are even completely identical.\n\n[3] In Equation 18, what is the distinction between H_t^l  and the subsequent H_(c,t)^l? Additionally, how are the scoring functions H_t^l and H_t^u specifically obtained?\n\n[4] In Eq (20), for unlabeled data, high pseudo-label confidence links to a random pruning probability r , where a larger r results in a larger gradient scaling factor. Could this intensify the model’s bias toward high-confidence samples (usually head classes)? Furthermore, the gradient scaling follows Qin et al.’s setting. Ablation studies demonstrate that it contributes more to model performance than other components proposed in this work, and removing it causes a notable decline in performance.\n\n[5] According to Equations (19) and (20), the scores corresponding to pruned samples will no longer be modified. Does this mean that these samples will not be used in subsequent training processes? If so, it would result in severe information waste.\n\n[6] The experimental section lacks comparisons with the latest literature. Additionally, it would be beneficial if the authors could provide information on the pruning amount of training samples in each iteration.\n\n[7] It seems that the predicted class probabilities obtained from Figure 3(a) and (b) both appear to be unsatisfactory. Additionally, some errors appear to exist in the titles and their corresponding descriptions for Figure 3, Figure 4, and Figures 6 to 10.\n\n[8] There are still some expression errors and unclear descriptions in the manuscript. For instance, the definition of the imbalance of the unlabeled dataset and the description of Figure 2 require clarification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)