A Square Peg in a Square Hole: Meta-Expert for Long-Tailed Semi-Supervised Learning
Authors: Yaxin Hou
Yuheng Jia
Venue: ICML 2025 poster (2025)
Keywords: Long tailed learning
semi-supervised learning
Primary area: general_machine_learning->unsupervised_and_semisupervised_learning
TL;DR: We propose a novel long-tailed semi-supervised learning with distribution mismatch method called Meta-Expert.

Abstract: This paper studies the long-tailed semi-supervised learning (LTSSL) with distribution mismatch, where the class distribution of the labeled training data follows a long-tailed distribution and mismatches with that of the unlabeled training data. Most existing methods introduce auxiliary classifiers (experts) to model various unlabeled data distributions and produce pseudo-labels, but the expertises of various experts are not fully utilized. We observe that different experts are good at predicting different intervals of samples, e.g., long-tailed expert is skilled in samples located in the head interval and uniform expert excels in samples located in the medium interval. Therefore, we propose a dynamic expert assignment module that can estimate the class membership (i.e., head, medium, or tail class) of samples, and dynamically assigns suitable expert to each sample based on the estimated membership to produce high-quality pseudo-label in the training phase and produce prediction in the testing phase. We also theoretically reveal that integrating different experts' strengths will lead to a smaller generalization error bound. Moreover, we find that the deeper features are more biased toward the head class but with more discriminative ability, while the shallower features are less biased but also with less discriminative ability. We, therefore, propose a multi-depth feature fusion module to utilize different depth features to mitigate the model bias. Our method demonstrates its effectiveness through comprehensive experiments on the CIFAR-10-LT, STL-10-LT, and SVHN-LT datasets across various settings.

A Square Peg in a Square Hole: Meta-Expert for Long-Tailed Semi-Supervised Learning
Yaxin Hou
Yuheng Jia
12
Abstract
Table I. Accuracy (%) of testing under three different unlabeled data distributions with varying experts. CPE (Ma et al., 2024), E2 denotes uniform expert, while E3 and Es denote long-tailed and inverse long-tailed experts, respectively. Our proposed method and Upper-E the proposed dynamic expert assignment (DEA) module and the ground-truth class membership to select specific expert, respectively. The dataset is CIFAR- 11--T with imbalance ratio " 200. f indicates our proposed method using the ground- truth class membership select specific expert.
This paper studies the mangttilled semi- learning (LTSSLL) with distribution mismatch, where the class distribution of the labeled training data follows a long-tailes distribution and mis- matches with that of the unlabeled training data. Most existing methods introduce auxiliary clas- sifiers (experts) model various unlabeled data distributions and produce pscudo-labels, but the expertises of various experts are not fully utilized. We observe that different experts are good at pre- dicting different intervals of samples, e.g., long- tailed expert is skilled in samples located the head interval and uniform expert excels sam- ples located in the medium interval. Therefore, propose a dynamic expert assignment mod- ule that can estimate the class membership (i.e., head, medium, or tail class) of samples, and dy- namically assigns suitable expert to each sample based on the estimated membership to produce high-quality pseudo-label in the training phase and produce prediction in the testing phase. We also theoretically reveal that integrating different experts' strengths will lead to a smaller gener- alization eтrоr bound. Moreover, we find that the deeper features are more biased toward the head class but with more discriminative ability, while the shallower features are less biased but also with less discriminative ability. We, there- fore, propose a multi-depth feature fusion mod- ule to utilize different depth features to mitigate the model bias. Our method demonstrates its ef- fectiveness through comprehensive experiments on the CIFAR- 10-LT, 10--1O-LT, and SVHN-LT datasets across various settings.
Distribution
Expert
Head
Medium
Overall
E;
94.67
74.10
38.73
69.66
E2
87.23
77.30
71.60
78.57
Es
3,57
72.35
68.47
50.55
Consistent
Ours
89.13
79.52
77.07
81.67
Upper-E
94.67
77.30
68.47
79.86
Upper-Ef
94.17
77.53
E
93.47
74.73
66.83
77.98
Eg
86.93
77.60
87.83
83.47
Es
0.53
74.55
90.20
57.04
Unifurm
Ours
89.40
78.35
86.00
83,96
Upper-E
93.47
77.60
90.20
Upper-Et
93.40
78.40
90.43
86.51
E
93.56
74.25
75.87
80.53
E2
83.
78.58
92.67
84.40
Ei
57.00
77.43
95.10
76.60
Inverse
Ours
88.60
78.90
92.03
85.75
Upper-E
93.56
78.58
95.10
88.03
Upper-Et
92.20
80.05
96.40
88.60
1. Introduction
Over the last decade, extensive high-quality labeled data have improved the performance of deep neural networks (DNNs). However, in specialized domains such as medi- cal diagnosis (Yuan et al., 2023; Zhang et al., 2023b), the scarcity and imbalance of labeled data can be a significant challenge due to the high costs associated with data collec- tion or annotation (Chen al., 2024). To solve this issue, semi cuni--sueeeiied learning (SSL) (Sohn al., 2020; Berth- elot et al., 2020; Zhang al., 2021) has been proposed and become popular method to utilize the easier and cheaper acquired unlabeled data to improve the performance of DNN models. Its core idea is generating pseudo-labele for unla- beled data and selecting high-confidence ones to train the model together with the labeled data, so as obtain a better
'School of Computer Science and Engineering, Southeast Uni- versity, Nanjing, China. *Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Appli- cations (Southeast University), Ministry Education, Nanjing, China. Correspondence to: Yuheng Jia <yhjia@@ sehjia@@su.cdu.co scu.cd.cn.>
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

A Square Peg in Square Hole: Meta-Expert for Loin-Taiildd Semi-Supervised I Learning
UP
Duta
Upper-E
Ours
Upparé
Upper-E
90
90
80
60
F1 scon
scooe
60
60-
50
50
30
Heee
Medium
lal
30
Feedd
Medium
Tail
Hlead
Medium
Tell
Class index
Class index
Cless index
(a) Consistent
(b) Uniform
(c) Inverse
Figure 1. Comparison Fl score (%) pscudo-label predictions hetween CPE (Ma et al., 2024) and our proposed method under three different cases. Upper-E E denotes the Fl score (%) of CPE with ground-truth class membership. The dataset is CIFAR- 10-LT with imbalance ratio " 200. Our proposed method can generate pseudo-labels with higher Fl scores than CPE on all the cases, indicating its effectiveness in the utilization of unlabeled samples.
model than using labeled data only. However, traditional SSL usually assumes that the class distributions of the la- beled and unlabeled data are balanced and consistent, which is easily violated in real-world applications. Specifically, data typically exhibit long-taile tailed distribution, and the class distribution of unlabeled data not always the same as that of the labeled data, i.c. unlabeled data may exhibit any one of the long-tailed, uniform, or inverse long-tailed dis- tribution, which further exacerbates the difficulty of model training (Zhang al., 2023c). This problem is known long-tailed semi-supervised learning (LTSSL).
generate pseudo-labels simultaneously in the training phase, which may introduce more error pseudo-labels. Second, in the testing phase, it only employs the uniform expert for prediction, ignoring the different characteristics of three experts.
We argue that each expert has its strength and weakness, e.g., the long-tailed expert is good at handling the samples the head classes but not the samples in the medium and tail classes, while the uniform expert excels in medium classes but not in tail and head classes, and should use different experts to process samples from differ- ent intervals, i.c., "a square peg in a square hole". To check this assumption, we first assume the class membership (i.e., head, medium, or tail class) of each sample known. Then, we construct the Upper-E, which uses the long-tailed expert in CPE produce pseudo-labels and predictions for head class samples in the training and testing phase, respectively, and the uniform and inverse long-tailed experts for medium and tail class samples. As shown in Fig. 1 and Table 1, this strategy can largely improve the quality of pseudo-labels in the training phase, and the prediction accuracy in the testing phase across all cases, indicating that each expert has its expertise. This observation motivates us design module to accurately estimate the class membership of each sample to realize the prior of "a square peg in square hole".
Motivation:
A motivating example. In the medical field, when collect- ing clinical data, we may obtain long-tailed dataset from hospitals, i.e., many common disease cases (head classes) accompanied by very few rare disease cases (tail classes). However, the clinical data collected from wide range of populations is unlabeled and characterized by abundance of non-diseased individuals and scarcity of diseased in- dividuals, especially those with rare diseases. Thus, the unlabeled data distribution is mismatched with the labeled data distribution.
To mitigate the model bias arising from long-tailes distri- bution, long-tailed learning techniques, such as logit adjust- ment (Menon et al., 2021) and re-sampling (Xu et al., 2022), have been utilized in LTSSL to produce unbiased and high- quality pseudo-labels. Despite their effectiveness, they still cannot effectively solve the model bias resulting from dis- tribution mismatch between labeled and unlabeled training data. Recently, ACR (Wei Gan, 2023) proposes handle the mismatched unlabeled data with various distributions by incorporating adaptive logit adjustment. While this method effective, it relies on pseudo-labels generated by single classifier (expert), limiting its performance. In response to this limitation, CPE (Ma et al., 2024) suggests training three classifiers (experts) to handle unlabeled data across various class distributions. However, CPE still suffers from the fol- lowing two drawbacks. First, employs three experts to
To this end, in this paper, we propose flexible and end-to- end LTSSI. algorithm, namely Meta-Fxpert. To estimate the class membership each sample, we propose a Dynamic Expert Assignment (DEA) module, which takes features from the encoder and logits from the three experts as in- put, to produce the probability (soft class membership) of assigning a specific expert to each sample. Subsequently, based on 2023; 2022), we integrate the expertises of these three ex- perts according to the estimated probabilities to construct an aggregator. The aggregator ensures the long-tailed expert dominates pseudo-label generation in the training phase and
fusion mechanism (Peng et al.,
2

---

A Square Peg in Square Hole: Meta-Expert for Long-Tailed Semi-Supervised Learning
expertises of different experts. Motivated by the empirical guide, we propose the Dynamic Expert Assignment (DEA) module to assign experts to different samples based on their specific expertises.
Table 2. Comparison of accuracy (%) on testing set using three dif- ferent depth features and the corresponding performance gap (%) between head and tail classes. We apply the K-means clustering on different depth features to produce the results, and we can clearly see that the feature more biased towards the head class but more discriminative as the depth increases.
2. We further theoretically show that leveraging different ex- perts' strengths efficiently will bring a lower generalization error bound.
Feature depth
Overall
Head
Medium
Tail
Gup
Shallow
30.30
26.00
35,50
27.67
1.67
Middle
38.10
36,67
41.25
35.33
1,33
3. We are the first to discover that shallow depth features are less biased than deep ones, and propose the Multi-depth Feature Fusion (MFF) module to help deal with the model bias towards the head class.
Deep
71.00
84.67
77.25
49.00
35.67
prediction in the testing phase when the sample belongs to the head class, and the uniform and inverse long-tailed ex- perts dominate when the sample belongs to the medium and tail classes, respectively. As shown in Fig. 1, the proposed method can produce significantly higher-quality pseudo- labels than CPE in the training phase. And as shown in Table 1, the proposed method can produce significantly higher prediction accuracies than CPE (i.e., employing the uniform expert for prediction) in the testing phase. Note that the proposed Meta- Deta-Expert integrates the logits from the three experts in a soft manner, pushing different experts to learn better. Thus, in several cases, it even outperforms CPE with the ground-truth class membership in pseudo- label generation in the training phase (Upper-E in Fig. 1) and prediction in the testing phase (Upper-E in Table 1).
4. We reach the new state-of-ther the--at art (SOTA) performances on the popular L.TSSI. benchmarks under various settings.
2. Related Work
Semi-supervised learning (SSL) uses both labeled and un- labeled training data to obtain a better model than using labeled training data only. Recent SSL methods are mostly based on consistency regularization, pseudo-labeling, or both. Consistency regularization methods (Miyato et al., 2019) are based on the manifold or smoothness assumption and apply consistency constraints to the final loss func- tion. Pseudo-labeling methods (Chen et al., 2018 8) produce pseudo-labels for unlabeled training data according to the model's high-confidence predictions and then use them to assist the model training. As representative method of combining both of these techniques, FixMatch (Sohn et al., 2020) encourages similar predictions between weak and strong augmentation views of an image, to improve model's performance and robustness. Afterward, many vari- ants based on FixMatch have been proposed, such as Flex- Match (Zhang et al., 2021), FlatMatch (Huang et al., 2023), SoftMatch (Chen et al., 2023), FreeMatch (Wang et al., 2023), (FL))² (Lee et al., 2024), and WiseOpen (Yang et al., 2024). Despite the superior performance of the above meth- ods, they cannot effectively handle the case where labeled data exhibit long-tailed distribution.
The proposed Meta-Expert can produce better pseudo-labels and predictions. More importantly, our theoretical analy- sis confirms that integrating different experts' expertises reduces the model's generalization error, thereby enhancing its overall performance. However, the model is still natu- rally biased towards the head classes due the scarcity of tail class samples. Fortunately, as shown in Table 2, we ob- served that shallow features are relatively balanced although less discriminative, and deep features improve the discrimi- native ability but are less balanced. This phenomenon aligns with the known behavior of deep networks: shallow layers capture local patterns while deep layers learn global seman- tics. For long-tailed learning, since head and tail classes may share similar local patterns, shallow features exhibit balanced discriminability across classes. Meanwhile, deep layers predominantly encode head class semantics due to their overwhelming sample dominance, thus biasing predic- tions toward head classes. Motivated by this observation, we further propose a Multi-depth Feature Fusion (MFF) module to mitigate the model bias towards the head class by fusing features across different depths to achieve both bal- anced and discriminative representation, which also echoes the wisdom of the proverb, "a square peg in a square hole".
Long-tailed tailed semi-supervised learning (LTSSL) has gained increased attention due to its high relevance to real-world ap- plications. It takes both the long-tailes distribution in long- tailed learning (LTLL) and the limited labeled training data in SSL into consideration, which makes more realistic and challenging. Existing LTSSL methods primarily improve the model performance by introducing L.TI. techniques (Li & Jia, 2025; Jia et al., 2024; Zhang et al., 2023a) to the ohf- shelf SSL methods like Fixh (Sohn et al., 2020). For instance, ABC (Lee et al., 2021). CReST (Wei et al., 2021), BMB (Peng et al., 2025), and RECD (Park et al., 2024) sample more tail class samples balance training bias to- wards the head class. SAW (Lai et al., 2022) introduces the class learning difficulty based weight to the consistency loss
In summary, our contributions are as follows:
We demonstrate that the pseudo-label quality and predic- tion accuracy can be notably improved by incorporating the
3

---

A Square Peg in Square Hole:
Mcta-Expert for Long-Tailed Semi-Supervised Learning
to enhance the model's robustness, INPL (Yu et al., 2023) proposes to select unlabeled data by the in-distribution prob- ability,, CDMAD (Lee & Kim, 2024) proposes to refine pseudo-labels by the estimated classifier bias, CoSSL (Fan et al., 2022) introduces feature enhancement strategies to refine classifier learning, and ACR (Wei & Gan, 2023) pro- poses to incorporate adaptive logit adjustment to handle unlabeled training data across various class distributions. Very recently, CPE (Ma et al., 2024) proposes to train mul- tiple classifiers (experts) to handle unlabeled data across various distributions and further enhances the pseudo-label quality through an ensemble strategy. Although effective, it lacks a comprehensive strategy to utilize the expertise of each expert in pseudo-label generation and unseen sample prediction, leading to sub-optimal performance.
we attach two auxiliary classifiers on typical SSL method like FixMatch (Sohn et al., 2020), and train each classifier (expert) with a specific logit adjustment intensity to realize that the first (long-tailed) expert is skilled in long-tailed distribution, and the second (uniform) and third (inverse long-tailed) experts are skilled in uniform and inverse ones, respectively. Similar to FixMatch, the loss Louse for the base LTSSL includes supervised classification loss on the labeled data and an unsupervised consistency regularization loss on the unlabeled data i.e..
Libac Ei 1
B, k= B i=
E (FF(((())) + Th log v.) E
(1)
>E k=1 B.
1 - E ((Ex())))) BTa)I,
Bn
where Q = 3 denotes the number of classifiers (experts), Bt and B, denote the batch sizes of labeled and unlabeled data, respectively, l denotes the cross-entropy loss, xr, and zi denote the 7t labeled and jth unlabeled samples in the current batch, respectively, g denotes the encoder, и de- notes the label frequency of the labeled data, Eg denotes the expert trained by cross-entropy loss with a specific logit adjustment intensity Tke 95,5 denotes the pscudo-label pre- dicted by E& on the 5th unlabeled sample in the current batch, and I denotes binary sample mask select samples with confidence larger than the threshold t. The first term in Eq. 1 is the supervised classification loss on the labeled data, and the second term defines the unsupervised consistency regularization loss on the unlabeled data.
More related literature and discussions are detailed in Ap- pendix B.
3. Method
3.1. Problem Statement
In long-tailed semi-supervised learning (LTSSL), we have a labeled training dataset DD = (a/, y N of size N and an unlabeled training dataset D., [ 5=1 of size M, where Di and D., share the same feature and label space, ry is the jth unlabeled sample, ar. is the 2th labeled sample with a ground-truth label nff E {I,..., C}. and C is the number of classes. Let N₂ denote the number of samples in the 5h class of labeled dataset, we assume that the C classes are sorted in descending order, i.e., N1 > No > > Ne. thus its imbalance ratio can be denoted as " = / As the label is inaccessible for unlabeled dataset, we denote the number of samples in its ct class by Mo. and define its imbalance ratio % in the same way as labeled dataset for theoretical illustration only. In this paper, we follow the previous works (Wei & Gan, 2023; Ma et al., 2024) to consider three cases of unlabeled data distribution, i.c.. consistent, uniform, and inverse. Specifically, i) for consistent setting, we have Mj > M2 > ...... > Mc and The = >; ii) for uniform setting, we have Mi- = M2 = Me and Yu 1; iii) for inverse setting, we have Mi < M2 <...< < Me and Tv 1/7g. The goal of LTSSL is to learn classifier F: : Rdd ......; CI parameterized by 0 on Dj and D,,, that generalizes well on all classes.
Dynamic expert assignment. As shown in Fig. 1 and Table 1, we observe that each expert has its strength and weakness, i.e., long-tailed expert is skilled in handling head class samples but not medium and tail class samples, while the uniform and inverse long-tailed experts are skilled in handling medium and tail class samples, respectively. Based on this observation, we propose to gather the strengths of different experts. To this end, we first propose a dynamic expert assignment (DEA) module to estimate the class mem- bership (i.e., head, medium, or tail class) of each sample.
As shown in Fig. 2, the DEA module adopts multilayer perceptron (MLP) architecture and takes the feature from the encoder and logits from the three experts as input and outputs the soft class membership for each sample, i.e., w = DEA((U, 21, 22, 23]), where w = [W1, W2, wa] denotes the probability of assigning each expert to produce labed and make prediction, v and z4 l2=1 denote the feature and logit generated by the encoder g and expert F 1 on the sample x, respectively. The parameters of the DEA module can be inferred by minimizing the following DEA loss Ldea.
3.2. Proposed Framework
Multi-expert based LTSSL. As the class distribution of the unlabeled training data may be inconsistent with that of the labeled ones, we follow the previous work (Ma et al., 2024) to train three experts to handle the unlabeled train- ing data across various class distributions, i.e., long-tailed, uniform, and inverse long-tailed distributions. Specifically,
Bi llw.. s:) + Lidea Bi 1
-
1
E E(wj. 8,)1, (2)
By (
B,
where wi and wzt denote the probabilities of assigning each
4

---

A Square Peg in Square Hole: Meta-Expert for Long-Tailed Tailed Semi-Supervised Learning
Encoder g
Feabure V
Dynamic Apserttnsignmmnt (DEA)
logit, logit
logits
E1
logits
Wi
A ggregator
T2
logita
Wia
Eg
11
MLP
logits
Wg
11
Eg
softmax
V2
V3
MLP
MLP
Which expert dominates?
MLP
MLP
Fusion Feature
MLP V.'
Head
Medium
Tail
MLP
(E.)
(E.)
(Eg)
Multi-depth Feature Fusion (MFF)
Figure Overview of our Meta- Meta-Expert algorithm. Meta Expert leverages DEA module to adaptively assigns suitable expert to each sample to generate pseudo-label in the training phase and make prediction in the testing phase, crucial for integrating expertises of different experts constructing the aggregator. Er, E2 and Es denote the long-tailed, uniform, and inverse long taiglt experts, respectively, тie denotes the logit adjustment intensity for expert Ex (k E 21 3}), and "4"" sign denotes adding different features.
expert the 1th labeled and jth unlabeled samples in the current batch, respectively, s; denotes the ground-truth class membership of the t labeled sample in the current batch, and šy denotes the pseudo class membership of the jth unlabeled sample in the current batch.
in the current batch, respectively, Úú denotes the pseudo- label produced by the aggregator the j"h unlabeled sam- ple in the current batch. As shown in Fig. and Table 1, our proposed method can integrate the expertises of all experts and generate high-quality pseudo-labels and predictions in the training and testing phase, respectively.
Aggregator. Subsequently, we construct an aggregator. The aggregator integrates the expertises three experts through weighted summation of their respective logits based the estimated class membership by the DEA module, i.e.,
Multi-depth feature fusion. Although the proposed method effective in integrating the expertise of each ex- pert, the model still naturally biased towards the head class due to the scarcity of tail class samples. Fortunately, as shown in Table 2, we observe that different depth fea- tures have different bias intensities, i.c., shallow features are relatively balanced although less discriminative, and deep features are more discriminative and more biased. Such a phenomenon motivates us use multiple depth features to learn representation with good trade-off between bias and discriminative ability.
Um = (E 2x2k))
where ym denotes the soft prediction produced by the aggre- gator and of-) denotes the softmax function. As the class membership estimated by the DEA module can reflect the class membership of each sample, aggregator in Eq. en- sures that the long-tailed expert dominates the pseudo-label generation in the training phase and prediction in the testing phase when the sample belongs to the head class, and the uniform and inverse long-tailed experts dominate when the sample belongs to the medium and tail classes, respectively.
Specifically, we propose a multi-depth feature fusion (MFF) module to fuse different depth features. As shown in Fig. 2, the MFF module involves several MLP layers and takes dif- ferent depth features from the encoder as input and outputs the fusion feature v, i.e., M F MFF(1 U2 U3) - v, where V1, U2 and v3 denote the shallow, medium and deep features, respectively. Specifically, for shallow depth features and medium ones U2, we first align their dimensions using an MLP and add them together. Subsequently, the resulting feature will added to the subsequent depth's feature set to perform similar operations. Beside the addition operation, we can also concatenate different depth features the MFF
Then, the META loss Lymeta for optimizing the overall net- work parameters based on the output of the aggregator is formulated as:
Limeta 1 B, 1 Plym..s 1y vi) 1 -, fV ú (4)
E.
where yin. and uh denote the soft prediction produced the aggregator on the 2h labeled and jth unlabeled samples
5

---

A Square Peg in Square Hole: Meta-Expert for Loig-Tailed Semi-Supervised Learning
module, which is investigated in Sec. 4.4. The parameters of the MFF module updated by the end-to-end training.
Algorithm 1 Training Process of the Proposed Method
1: Input: Labeled training dataset Di, unlabeled training dataset a.,, hyper-parameter t.
3.3. Model Training and Prediction
2: Output: Encoder g. long-tailed expert F1, uniform ex-
pert Eg, inverse long-tailed expert E3, and parameters of DEA module Wd and MFF module Wmfs-
In the training phase, we warm up the model by eighteen epochs with the base loss Liase in Eq. 1. Then, take the fusion feature from the MFF module and logit 2 from the expert E& (k E {1, 2, 3)) to calculate the DEA loss Ldea in Eq. 2 and META loss Emeta in Eq. 4, and together with the base loss Loose to jointly optimize the model. The overall loss Loverali is formulated as:
3: Initialize the parameters of g. E1, E2, Es. Waea and Wmf$ randomly.
4: for epoch=1, 2.... do for batch=1, 2. ... do
Get expert Es prediction 2 on batch of data B: Calculate the base loss by Eq. 1; if warm up ended then
7:
Loverall = Liase + Litea + Limeta-
(5)
8:
9:
Calculate the loss for DEA module by Eq. 2; Integrate the expertises of different experts by Eq. 3;
The whole framework of our method is shown in Fig. 2. The pseudo-code is summarized in Alg. 1. After training, we could obtain the predicted label of an unseen sample x² by selecting the label index with the highest confidence by ym in Eq. 3. Our code is made available'.
10:
11:
Calculate the loss for optimizing the overall net- work parameters based on the integrated exper-
tises by Eq. 4;
12:
Obtain the overall loss by Eq. 5;
13: end if
3.4. Theoretical Analysis
14: Update network parameters via gradient descent;
We provide the generalization error bound for Meta- Expert to analyze the factors that affect the model's general- ization ability. Before providing the main results, we first define the true risk with respect to the classification model fa; 0) as follows:
15: end for
16:
end for
Theorem 1 (Generalization Error Bound). Suppose that the loss function {(f v) is p-Lipschitz with respect to fxx for all y E {1,..., C} and upper-bounded by U. Given the class membership 17 E {l..... ,Q] and overall
R(f) = E( [e(f(x).))) v)]-
(6)
Definition! 1. We aim to learn a good classification model by minimizing the empirical risk R(f)) = R((f)++++ + Ru()), where Ri(f) = & &E {(f(z,), & vi) and (f) =
pseudo-labeling error >> 0, if M Ef==1 E I(īj =
[]](maz(fa(t)) > t) 1(;; = 13)l S for any O.
M ((f=))))) denote the empirical risk on labeled and unlabeled training data, respectively. In SSL, we cannot minimize the empirical risk Ra( directly, since the ground-truth label is inaccessible for unlabeled train- ing data. Therefore, we need to train the model with (,,()) = 1 Σド " {(f(z)), where M denotes the number of selected high-confidence unlabeled samples and 9 denotes the pseudo-label of unlabeled sample Zy.
with probability at least 1 - 8, we have:
log 2
RCA R()) & 2Ue 4V2 Σ Ro(Hy) + 20
(7)
20
The proof of Theorem is provided in Appendix D. It can be observed that the generalization performance of f mainly depends on two factors, i.e., the overall pseudo- labeling error e and the number of training samples O. As O 00, - 0, Theorem 1 shows that the empiri- cal risk minimizer f will get closer to the true risk min- imizer f". In CPE, the overall pseudo-labeling error is defined as ECPE = क Σe Σラ- Eij. where fij de- notes the pseudo-labeling error of the 2th expert on the unlabeled samples with class membership nl 1. While the counterpart of our Meta- Meta-EEppert is defined as cOurs @ Σe Σ li=jfti.j where Ti=; denotes binary expert mask to assign each expert to select high-confidence unla- beled samples located in its skilled interval. As illustrated in Table 3, compared with CPE in the consistent case, with the estimated class membership by the DEA module, our
Let ESSL = , e(f(z;). + M EME
be the loss for SSI., where NT denotes the number of selected high-confidence unlabeled samples. Let the function space Hy for the label y E (I, C) be (h: : + [x)] e F}. where fz) denotes the predicted probability of the 4tth class. Let Ro(H, be the expected Rademacher com- plexity (Mohri et al., 2018) of H,, with O = N + M train- ing samples (including N labeled and M unlabeled training samples). Let f argmin R(f) be the empirical risk minimizer, and f argmin fez R(f) be the true risk minimizer. Then we have the following theorem.
httpp::///t www.tcoooooooo.tttoooooootottttt wwwp//wwiooo/////occococooooott/ www.o/ht.uteeeeeee Met.a- Expert
6

---

A Square Peg in Square Hole: Meta-Expert for Long-Tailed Semi-Supervised Learning
the original unlabeled data.
Table 3. Pseudo-laheling error rate (i.e., e) (%) and utilization ratio (i.e, M/ /M) (%) under three different unlabeled data distributions with varying experts. In CPE (Ma et al., 2024), En denotes uniform expert, while Ej and Eg denote long-tailed and inverse long-tailed experts, respectively. Our proposed method uses the DEA module to select specific expert. The datasct is CIFAR-IOA with
Baseline. We compare our method with seven LTSSL algorithms published in top-conferences in the past few years, including SAW (Lai et al., 2022), Adsh (Guo & Li, 2022), DePL (Wang et al., 2022), ACR (Wei & Gan, 2023), BaCon (Feng et al., 2024), CPE (Ma et al., 2024), and SimPro (Du et al., 2024), which are all based on the typical SSL method FixMatch. For fair comparison, we test these baselines and our Meta-Expert on the widely-used codebase USB?. We use the same dataset splits with no overlap between labeled and unlabeled training data for all datasets.
imbalance ratio 72 = 200.
Distribution
Expert
Head
Medium
Tail
M
Ej
9.01
20.90
43.04
23.98
94,90
Fg
26.82
20.85
22.35
23.09
64.30
Consistent
Es
92.15
21.47
23.04
43.14
95.37
CPE
42.66
21.07
29.48
30.07
84,86
Ours
21.79
18.67
23.87
21.17
95.31
E1
7.56
24.50
25.22
19.63
95.73
Ez
13:67
24.58
11.00
17.23
96.57
Uniform
Ma
97.44
25.33
9.33
42.17
98.80
4.2. Implementation Details
CPE
39.56
24.81
15.19
26.34
97.03
Ours
12.89
24.92
15.11
18.37
98.44
We follow the default settings and hyper-parametee in USB, i.c., the batch size of labeled data B; is set to 64 and unla- beled data B, is set to 128, and the confidence threshold t is set to 0.95. Moreover, we use the WRN-28-2 (Zagoruyko & Komodakis, 2016) architecture, and the SGD optimizer with learning rate 3e-2, momentum 0.9, and weight decay 4 for training. We repeat each experiment over three different random seeds and report the mean performance and standard deviation. We conduct the experiments on a single GPU of NVIDIA A100 using PyTorch v2.3.1.
E
6.10
22.18
22.97
17.59
84,23
R2
15.79
21.13
9.03
15.90
93.39
Inverse
Eg
40.39
22.48
5.67
22.81
96.55
CPE
20.76
21.93
12.56
18.77
91.39
Ours
11.08
19.91
8.59
13.87
93.75
Meta- Meta-Expert achieves a smaller overall pseudo-labeling er- ror (from 30.07 percentage points (pp) reduced 21.17 pp) and higher unlabeled data utilization ratio (from 84.86 pp improved to 95.31 pp). and similar conclusions can be observed in uniform and inverse cases, which are beneficial for obtaining a smaller generalization error bound.
4.3. Main Result
In the case of consistent distribution (7 = %.) We initiate our investigation by conducting experiments in the scenario where N = Yu- The primary results for CIFAR- 10-LT are presented in Table 4. clear that across all different training dataset sizes and imbalance ratios, Meta- Expert achieves higher classification accuracy than all the previous baselines on LIFAR-1O-LY, For example, given (N1, M1,7) (500, 4000, 200). Meta-Expert surpasses all previous baselines by up to 2.47 pp. When moving to SVHN-LT dataset in Table 5, Meta-Expert performs comparable to the previous SOTA method BaCon, surpass- ing other baselines by up to 0.66 pp given (N1, M1 = 3000, 200).
4. Experiments
4.1. Experimental Setting
Dataset. We perform our experiments on three widely- used datasets for the LTSSL task, including CI- LT (Krizhevsky, 2009), SVHN-LT (Netzer et al., 2011), and STL-10-LT (Coates et al., 2011). We follow the dataset settings in ACR (Wei & Gan, 2023) and CPE (Ma et al., 2024), details are as below.
CI-AR-IO-LT: We test with four settings in the consis- tent case: (N1, M)) (1500, 3000) and (N1, M1) (500, 4000), with 7 {150, 200). In the uniform case, we test with (N,, M,) = (1500, 300). with E {150, 200), and 7u being 1. In the inverse case, we test with (N1, = (5500,3000, with ? E 200}. and Tu being 1/2u-
In the case of mismatched distribution (?1 # Ti) In practical applications, the distribution of unlabeled data might significantly differ from that of labeled data. There- fore, we investigate uniform and inverse class distribu- tions, such as setting 7u to I or 1/200 for CIFAR- For STL-IO-LT dataset, as the ground-truth labels of the unlabeled data are unknown, we only control the imbal- ance ratio of the labeled data. The results are presented in Tables 4 and 5, where we can find Meta-Expert can
SVHN-LT: We test our method under (N1, Mi) = (1500, 3000) setting. The imbalance ratio 7. set to 150 or 200. With a fixed ? 150, we also test our method under 7h E 1/150) for the uniform and inverse cases.
STL-1O-LT: Since the ground-truth labels of unlabeled data in STL-10-LT are unknown, we conduct experiments by controlling the imbalance ratio of labeled data only. We set N₁ as 150 or 450, with 7t E {15, 20), and directly use
Pnttps:/ /tta:///gittui githbb. crosoft/ Semi -spp-suipvviiii -ee-ieeddddededeeeeeeeeeee learning
7

---

A Square Peg in Square Hole: Mcta-Expert for Long-Tailed Semi-Supervised Learning
Table Comparison of accuracy (%) on CIFAR-IO-LT under 74 = Yhu and 7V o "/ settings. We set 72 to 150 and 200 for CIFAR-IO-J We use bold to mark the best results.
CIFAR-I0-LT
CIFAR-IO-LT
N1 500,
M1 4000
N; 1500, Mr 3000
N1 1500,
Mr 3000
N1 1500, M. 3000
Algorithm
"x 200
N = 150
3 = 200
7 150
"2 = 200
TV = 200
" = 150
" == 150
% 200
% 150
7/ 200
7/ 150
7
7/ 1/200
% 1
7/ 1/150
Supervised
41.15 1115
43.88 1: 1.1.61
56.83 11110
59.82 1.032
56.83 1.100
56.83 1:1.10
59.82 1.0.22
59.82 1.032
FixMatch (NIPS 20)
61.74 1:.81
65.68 100667
69.39 10.56
72.15 1..94
65,59 1:0.18
63.98
10036
69.07 1:074
65.24 1:0.63
W/ SAW (ICML 22)
61.22
68.51 +0.16
74.15 1.55
0.114
78.60 +0023
70.55 +0.48
80.02 +:050
73.67 +0,50
W/ Adsh (ICMI 22)
62.04 1131
66.55 12294
67.13 1.039
73.96 1.0.4
71.06 1.0.77
65.68 1..44
73.65 1:0.36
66.51 1.0.69
w/ DePeI. (CVPR 22)
69.21 40.62
71.95 4:254
73.23 +0.62
76.58 4-0.1
73.26 4:0.6
69.35 +-0.66
75.62 4-0.6
71.23 +0.54
W/ ACR (CVPR 23)
10.94
76.72:1..13
79,96
81.81
81.18 1:.73
81.23 1:0.9
83,46 1:0.42
84.63 10066
w/ WaBaCon (AAAI 24)
66.41 +0.31
60.31
71.33
70.44
81.63+00444
77.89 =0.97
31.87 10.16
82.05
83,30 +1.12
C/ (AAAI 24)
67.45 11.27
76.77 10.53
78.12
82,25 10.34
83,46 10003
84.07 10.40
84.50 10.40
85.52 10.02
w/ SimPro (ICML 24)
59.94 10.73
65.54 13.17
75.37
10.74
77.18 10.38
73.05 10.35
75.33 1:285
76.12
79.42
w/ Meta- Exeta-Expert (Ours)
74.39 1044
77.19 10.58
80.63
20.83
82.52
1:040
83,90 10.11
85.71 1:000
84,91 =0.14
86.78 1:033
Table Comparison of accuracy (%) on STL-I0-LT and SVHN-LT under x. = 7, and ' * 7h settings, We set 7V to 15 and 20 for STL-10-LT, and to 150 and 200 for SVHN-LT. We use bold to mark the best results. N/A denotes the unknown Yvu in STL-I0-LT since its inaccessible ground-truth truth label for unlabeled dataset.
STL-10-LT
SVHN-LT
N₂
150, M1 100k
N₂ = 450, M, 100k
N; = 1500, M, = 3000
N₁ = 1500, M, = 3000
Algorithm
? 20
"e- 15
71 = 20
1u = 15
'x = 200
7 = 150
" = 150
" 150
Tu = N/A
Tu = N/A
Tu = N/A
Tu = N/A
7/ = 200
% = 150
7/2 = 1
7/4
1/150
Supervised
40.44 1.442
42.31 1.0.2
56,56 1.107
59.81 100.5
84,10 1.0.05
86.14 1.0.50
86.14 10.50
86.14 1.0.50
PixMatch (NIPS 20)
56,12++1.38
60.63 +0.92
68.33 +0080
71,55 +0,74
91.36 +00.15
91. .9 +:0.18
93.94 +:0,79
90.25 +-2.45
w SAW (ICML 22)
66.62++0.44
67,00 +0079
74.59 +0.13
75.58+0.220
92.17 10
92.27 +0.01
94,41 +-038
91 .24 +0.41
/ Adsh (ICML 22)
66.56 1.0.6
66.72 1.0.2
72.95 10.45
74.28 10.24
90.87 1.0.32
91.68 1.0.25
94.04 10.68
88.71 10.52
w/ DePL (CVPR 22)
10.63
67.02 +0.89
73.43 10.11
00.14
92.16 10.16
92.85 20.04
94. +0.63
87.86 +0.50
W/ ACR (CVPR 23)
69.24 +0.95
68.74 95
78.13 +0029
78.555+0.50
92.90 +00.40
93.52 +0032
91.11 +.0.7
92.00 +0.34
w BaCon (AAAI 24)
66.68 +0.38
68,26
77.29 +0.23
77.73
93.30 93.30++0.. 10.15
93.94 +0.21
94,54 +:0.2
93.69 +0.41
w/CPE (AAAI 24)
68.01 10.65
67.07 11.72
78.02 10.14
78.71.10.22
85.79 100.4
86.31 10.05
94.14 10.24
93.06 10.34
w SimPro (ICML 24)
43.65 10.55
44.45
#0.98
57.23 11.43
00.59
92.51 10.71
93.94 10.10
94.59 1:028
94.76 10041
w Expert (Ours)
71.19 00.07
69.23 +-0.82
80.18
79.98
4-0.33
93.56 +-0.99
93.99 +-0.07
94.66
94.24 40.19
consistently and significantly outperform baseline algo- rithms on CIFAR- 10-LT, validating its effectiveness to cope with varying class distributions of unlabeled data. Con- cretely, Meta-Expert surpasses the previous SOTA method by 1.64 pp and other baselines by up to 4.48 pp with MN, 7 %a) = (1500, 3000, 200, 1/200). For STL- 10-LT dataset, Meta-Expert surpasses the previous SOTA method by 1.27 pp and other baselines by up to 1.43 pp with Ni = 450 and ア = 15. On SVHN-LT dataset, Meta-Expert achieves comparable performances with SimPro, surpassing other baselines by up to 0.55 pp.
experiments CIFAR- and SVHN-LT. According to Table 6, we can observe that both DEA and MFF bring sigmificant improvements. For example, on CIFAR-IO-LT with m = 7u = 200, DEA and MFF bring accuracy gains of 0.65 pp and 1.26 pp. respectively, and the accuracy is improved up to 3.10 pp when using them together. When evaluating the overall performance, the DEA module pro- vides an average 1.68 pp accuracy improvement across all imbalance ratios, showing relatively greater performance gains; while the MFF module delivers 0.76 pp average gain, which though comparatively smaller, remains statisti- cally significant. The combined DEA+MFF configuration achieves 2.42 pp improvement, confirming their comple- mentary effectiveness and synergistic interaction. These improvements conclusively validate the effectiveness of our proposed modules in enhancing model robustness across diverse imbalance ratios.
In summary, our method outperforms almost all previous baselines regardless of training dataset sizes, imbalance ratios, and unlabeled training data distributions. We also evaluate all methods on FreeMatch (Wang et al., 2023) in Appendix C, where we can get similar conclusion.
4.4. Ablation Study
Note that MFF improves the accuracy of all cases except the inverse case (ru = 1/200). This phenomenon can be attributed to two reasons. First, the overall imbalance ratio will be reduced as the inverse long-tailed unlabeled data (Yu = 1/200) complements the long-taile tailed labeled data
The effect of each module. In Table 6, we evaluate the contribution of each key component in Meta-Expert. Specif- ically, we set N1 to 1500 and M1 to 3000, and perform
8

---

A Square Peg in Square Hole: Meta-Expert for Long-Tailed Semi-Supervised Learning
Table Comparison of accuracy (%) on with and without the DEA and MFF modules. The datasets are CIFAR- 10-
Table 7. Comparison of accuracy (%) on utilizing different feature combination strategies in the MFF module. add and con denote using the addition and concatenation feature operation strategies, respectively. The dataset is CIFAR-IO-LT with (Ni, M1.7)) (1500, 3000, 200).
LT with (Ni, M1, 7) (1500, 3000, 200) and SVHN-LT with (Ni, Mi, m) = (1500, 3000, 150).
CPE
CIFAR LT
SVHN LT())
w/ DEA
w/ MFF
200
1/200
150
1/150
Strategy
T
78.57
83.47
84.40
86.26
03.82
92.62
200
1/200
79.22
83,61
84.58
93.90
94.30
93.60
79.83
83,89
83,10
90.49
92.89
93.47
con
80.32
83.38
84.65
81.67
83.96
85.75
93.89
94.34
94,05
add
81.67
83,96
85.75
0.6
Medium
Tall
W2
accurate estimation, our method can produce higher-quality pseudo-labels in the training phase, as can be seen in Fig. 1. Simultancously, according to Table 1, the accuracies of our method in testing phase are also the highest in all cases. The above observations verify that our method utilizes the expertises of all three experts effectively.
W3
Confidence
0.4
0.2
Computational overhead analysis. While our proposed modules (three experts, MFF, and DEA) introduce con- trolled parameter increase of 13.3 pp (1.5M - 1.7M), this design achieves a strategically balanced efficiency- performance trade-off. Experimental results on CIFAR- 10-LT demonstrate: a 6.4 pp increase in epoch time (234.5s + 249.5s). a 1.6s increase in inference time for evaluating 10,000 samples (7.1s + 8.8s), and substantial accuracy improvement of 3.5 pp (71.9 pp 74.4 pp). These results collectively indicate significant performance enhancement with modest computational overhead.
3
6
8 10
Class index
Figure 3. Confidence level for assigning specific expert the head, medium, and tail class samples. The datasets is CIFAR- 10- LT with (N1, M1, ?) = (1500, 3000, 200). We follow the setting in CPE to consider the first two of all classes as head classes. the last six as tail classes, and the remaining classes as the medium. We clearly see that when a sample helongs to the head class, is the largest, and 12 and wy are the largest when a sample belongs to the medium and tail classes, respectively.
(n 200). Second, without the DEA model, our method uses all experts to generate pseudo-labele in the training phase, which introduces more error pseudo-labels and lim- its the MFF module's effectiveness. When using the two modules (DEA+MFF) together, the DEA module reduces the quantity of error pscudo-labels thus, the effectiveness of the MFF module released completely.
5. Conclusion
In this work, we address the LTSSL problem from fresh perspective, i.e., automatically integrating the expertises of various experts to produce high-quality pseudo-labels and predictions. We also theoretically prove that effectively ex- ploiting different experts' expertises can reduce the general- ization error bound. Specifically, the proposed Meta-Exper algorithm comprises a dynamic expert assignment module and multi-depth feature fusion module, the former can assign a suitable expert to each sample based on the esti- mated class membership, and the latter relieves the biased training by fusing different depth features based on the ob- servation that deep feature is more biased towards the head class but more discriminative, which has not been observed before. We validate our proposed method's s effectiveness through extensive experiments across three widely-used LTSSL datasets with different imbalance ratios and unla- beled data distributions, reaching the new sfate-of-ther tet-hee-rr art performance. The ablation results show that both the DEA and MFF modules contribute to the performance improve- ment. Moreover, the DEA module can correctly estimate the class membership of given samples, which is essential to integrate the expertise of each expert.
Feature combination strategy. We evaluate the perfor- mance of using different feature combination strategies in the MFF module. As shown in Table 7, we observe that the addition operation outperforms concatenation, achiev- ing a 0.58 pp ~ 1.35 pp higher accuracy. Based on these empirical results, we adopt the addition operation for fusing multi-depth features.
How does the DEA module improve the performance? As previously mentioned, DEA can correctly estimate the class membership of given samples, which is essential to integrate the expertise of each expert and realize "a square peg in a square hole". To validate this assumption, we plot the soft class membership estimated by DEA in Fig. 3. As can be seen, DEA precisely assigns the head expert to the head class samples, and medium and tail experts to the medium and tail class samples, respectively. Based on this

---

A Square Peg in Square Hole: Meta-Expert for Long-Tailed Semi-Supervised Learning
Acknowledgements
Du, Y. and Wu, J. No one left behind: Improving the worst categories in long-tailed learning. In CVPR, 2023.
This work was supported by the National Natural Science Foundation of China under Grant U24A20322. This re- search work is also supported by the Big Data Computing Center of Southeast University.
Fan, Y., Dai, D., Kukleva, A., and Schiele, B. Cossl: Co- learning of representation and classifier for imbalanced semi-supervised learning. In CVPR, 2022.
Feng, Q. Xie, L., Fang. S., and Lin, T. Bacon: Boost- ing imbalanced semi-supervised learning via balanced feature-Ievel contrastive learning. AAAI, 2024.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning- Specifically, we propose a new long- tailed semi-supervised learning algorithm to improve the performance in distribution mismatched scenarios by inte- grating the expertises of various experts to produce high- quality pseudo-labels and predictions. We also theoretically prove that the model's generalization error can be reduced by integrating the expertises of different experts. There are many potential socictal consequences of our work, none which we feel must be specifically highlighted here.
Guo, I. and Li, Y. Class-inbalanced semi-supervised learn- ing with adaptive thresholding, In ICML, 2022.
Huang, 7.,, Shen, I.., Yu, J., Han, B., and Liu, T. Flat- match: Bridging labeled data and unlabeled data with cross-sharpness for supervupee learning. In NeurIPS, 2023.
Jia, Y., Peng, X., Wang, R., and Zhang, M. Long-tailed partial label learning by head classifier and tail classifier cooperation. In AAAI, 2024.
References
Kini, R., Paraskevas, O., Oymak, S., and Thrampoulidis, babblliimbalnnced and group-sensitive classification under overparameterizatton. In NeurlPS, 2021.
Aimar, E. S., Jonnarth, A., Felsberg, M., and Kuhlmann, M. Balanced product of calibrated experts for long-tailed recognition. In CVPR, 2023.
Krizhevsky, A. Learning multiple layers of features from
Bai, J., Liu, Z., Wang, H., Hao. J., Feng, Y., Chu, H., and Hu, H. On the effectiveness of out-of-distribution data in self-supervised long-tail learning. In ICLR, 2023.
tiny images. https://www.cs.tomm edu/ kriz/learning 200tu T...-TR..ffffffff.. 2009.
Lai, Z., Wang, C., Gunawan, H., Cheung, S. S., and Chuah, C. Smoothed adaptive weighting for imbalanced semi- supervised learning: Improve reliability against unknown distribution data. In ICML, 2022.
Berthelot, D., Carlini, N., Cubuk, E. D., Kurakin, A., Sohn, K., Zhang, H., and Raffel, C. Remixmatch: Semi- supervised learning with distribution matching and aug- mentation anchoring. In ICLR, 2020.
Lee, H. and Kim, H. CDMAD: class- wisttiibutionnmismmaiee aware debiasing for class-imbalanced semi-supervised learning. In CVPR, 2024.
Cao. K., Wei, C., Gaidon, A., Aréchiga, N., and Ma, T. Learning imbalanced datasets with label-distribution- aware margin loss. In NeurlPS, 2019.
Chen, D., Wang, W., Gao, W., and Zhou, Z. Tri-net for semi-suse comi-supervised deep learning. In IJCAI, 2018.
Lee, H., Shin, S., and Kim, H. ABC: auxiliary balanced classifier for class- imbalancedd semi-supervised learning. NeurlPS, 2021.
Chen, H., Tao, R., Fan, Y., Wang, Y., Wang, J., Schiele, B., Xie, X., Raj. B., and Savvides, M. Softmatch: Addressing the quantity-quality tradeoff in semi-supervised learning. In ICLR, 2023.
Lee, S., Le, T. V., Shin, J., and Lee, S. (A) Overcom- ing few labels in federated semi-supervised learning. In NeurIPS, 2024.
Chen, Y.. Mancini, M., Zhu, X., and Akata, Z. Semi- supervised and unsupervised deep visual learning: A survey. IEEE TPAMI, 46(3)
Li, Z. and Jia, Y. Conmix: Contrastive mixup at repre- sentation level for long-tailed deep clustering. In ICLR, 2025.
2024.
Coates, A., Ng, A. Y., and Lee, H. An analysis of single- layer networks in unsupervised feature learning. In AIS- TATS, 2011.
Liu, B., Li, H., Kang. H., Hua, G., and Vasconcelos, N. Breadcrumbs: Adversarial blas-bal sampling for long-tailed recognition. In ECCV, 2022.
Du, C., Han, Y., and Huang, G. Simpro: A simple prob- abilistic framework towards realistic long-tailed semi- supervised learning. In ICML, 2024.
Ma, C., Elezi, I., Deng, J., Dong, W., and Xu, C. Three heads are better than one: Complementary experts for long-tailed semi-supervised learning. In AAAI, 2024.
10

---

A Square Peg in Square Hole: Meta-Expert for Moine-Tilled Semi-Supervised Learning
Maurer, A. A vector-conttraction inequality for rademacher complexities. In ALT,, 2016.
Wei, C., Sohn, K., Mellina, C., Yuille, A. L., and Yang, F, Crest: A class-rebalancingg self-training framework for imbalanced semi-supervised learning. In CVPR, 2021.
Menon, A. K., Jayasumana, S., Rawat, A. S., Jain, H., Veit, A., and Kumar, S. Long-tail learning via logit adjustment. In ICLR,, 2021.
Wei, and Gan, K. Towards realistic long-tailed semi- supervised learning: Consistency is all you need. In CVPR, 2023.
Miyato, T., Maeda, S., Koyama, M., and Ishii, S. Virtual adversarial training: A regularization method for super- vised and remi-supervised learning. IEEE TPAMI, 41(8): 1979-1993, 2019.
Xu, Y., Li, Y., Li, J., and Lu, C. Constructing balance from imbalance for long-tailed image recognition. ECCV, 2022.
Yang, Y., Jiang, N., Xu, Y., and Zhan, D. Robust semi- supervised learning by wisely leveraging open-set data. IEEE TPAMI, 46( 12):3334 8347, 2024.
Mohri, M., Rostamizadeh, A., and Talwalkar, A. Founda- fions of Machine Learning. MIT Press, 2018.
Netzer, Y., Wang. T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y. Reading digits in natural images with unsupervised feature learning. In NewrIPS, 2011.
Yu, Z., Li, Y., and Lee, Y. Inpl: Pseudo-labeling the inliers first for imbalanced semi remi-supervised learning. In ICLR, 2023.
Pan, L., Zhang, Y., Yang, Q., Li, T., and Chen, Z. Combat long-tair tails in medical classification with relation-aware consistency and virtual features compensation. In MIC- CAI, 2023.
Yuan, Y., Wang. X., Yang, X., Li, R., and Heng, P. Semi- supervised class imbalanced deep learning for cardiac MRI segmentation. In MICCAI, 2023.
Zagoruyko, S. and Komodakis, N. Wide residual networks. In BMVC, 2016.
Park, T., Lee, H., and Kim, H. Rebalancing using esti- mated class distribution for imbalanced semi-supervised learning under class distribution mismatch. In ECCV, 2024.
Zhang, B., Wang, Y., Hou, W., Wu, H., Wang. J., Oku- mura, M., and Shinozaki, T. Flexmatch: Boosting semi- supervised learning with curriculum pseudo labeling. In NeurIPS, 2021.
Peng, W., Weng, Z., Li, H., Wu, Z., and Jiang, Y.GG. Bmb: Balanced memory bank for long-tailed semi-supervised learning. IEEE TMM, 2025.
Zhang, C., Hou, Y., Chen, K., Cao, S., Fan, G., and Liu, J. Quality- self-training on differentiable synthesis of rare relational data. AAAI, 2023a.
Peng, Z., Jia, Y., Liu, H., Hou, J., and Zhang, Q. Maximum entropy subspace clustering network. IEEE TCSV, 32(4): 2199-2210, 2022.
Zhang, Y., Hooi, B., Hong, I.., and Feng. Self-supervised aggregation of diverse experts for test-agnostic long- tailed recognition. In NeurlPS, 2022.
Peng. 7., Liu, H., Jia, Y., and Hou, J. Deep attention-guided graph clustering with dual self-supervision. IEEE TCSV, 33(7):3296-3300 2023.
Zhang, Y., Chen, J., Wang, K., and Xie, F. ECL: class- enhancement contrastive learning for long-tailed skin le- sion classification. In MICCAI, 2023b.
Sohn. K., Berthelot, D.. Carlini, N., Zhang, Z., Zhang, H., Raffel, C., Cubuk, E. D., Kurakin, A., and Li, C. Fix- match: Simplifying semi-supervised learning with con- sistency and confidence. NeurlPS, 2020.
Zhang, Y., Kang. B., Hooi, B., Yan, S., and Feng, J. Deep long-tailed learning: A survey. IEEE TPAMI, 45(9): 10795-10816, 2023c.
Wang, X., Lian, L., Miao, Z., Liu. Z., and Yu. S. X. Long- tailed recognition by routing diverse distribution-aware experts. In ICLR, 2021.
Wang, X., Wu, Z., Lian, L.. and Yu, S. X. Debiased learning from naturally imbalanced lseudo-labels. In CVPR, 2022.
Wang, Y., Chen, H., Heng, Q., Hou, W., Fan, Y., Wu, Z., Wang, J., Savvides, M., Shinozaki, T., Raj, B., Schiele, B., and Xie, X. Freematch: Self adaativee thresholding for supervupe learning. In ICLR, 2023.

---

A Square Peg in Square Hole: Meta-Expert for Long-Tailed Semi-Supervised Learning
Appendix
A. overview of the Appendix
Our appendix consists of three main sections:
More related literature and discussions.
Additional evaluation on FreeMatch (Wang et al., 2023).
Proof of the theorem in Sec. 3.4: theoretical analysis, i.e., Theorem 1 (generalization error bound).
B. More Related Literature and Discussions
B.1. Long-tailed learning (LTL)
Long-tailed learning (LTL) is tailored for the long-tailed distribution exhibiting in real-world applications (Pan et al., 2023), which aims to improve the performance of the tail class without compromising that of the head class. The existing methods can be roughly grouped into three categories: re-sampling, logit adjustment, and ensemble learning. Re-sampling (Bai et al., 2023; Liu al., 2022; Xu al., 2022) adjusts the number of samples for each class, i.e., under-sampling the head class or over-sampling the tail class. Logit adjustment (Cao al., 2019; Menon et al., 2021; Kini et al., 2021) seeks to resolve the class imbalance by adjusting the predicted logit of the classifier. Ensemble learning (Aimar et al., 2023; Du & Wu, 2023; Zhang et al., 2022) based methods combine multiple classifiers (experts) improve the performance and robustness of the model. While these methods have made significant progress in tailed learning, they often fail to achieve the expected performance gains when directly applied to long-tailed semi-supervised learning (LTSSL), particularly when the distributions are mismatched between labeled and unlabeled training data.
B.2. Connections and Differences Compared with Multiple Experts Based Methods
In the fields of long-tailed learning (LTL) and long-tailed semi-supervised learning (LTSSL), several works have employed multiple experts to enhance the model's performance. Nevertheless, they are significantly different from the proposed Meta-Expert.
In LTSSL, only CPE (Ma al., 2024) utilizes multiple experts to handle unlabeled data across various class distributions. CPE, however, lacks integrating the multiple experts, it only employs three experts to generate pscudo-label simultancously in the training phase and the uniform expert to make prediction in the testing phase. Consequently, CPE may introduce more error labels, thereby limiting its performance.
In contrast to LTSSL, numerous studies incorporate multiple experts in LTL, such as RIDE (Wang et al., 2021), SADE (Zhang et al., 2022), and BalPoE (Aimar al., 2023). All of these methods are designed for supervised learning and are incapable of handling sumisvipe learning (SSL). Extending them SSL requires extra effort, as exploiting unlabeled samples not trivial.
RIDE seeks to learn from multiple independent and diverse experts, making predictions through the averaging of their outputs. The router in RIDE primarily focuses on minimizing the number of experts to reduce the computational cost during the testing phase. SADE leverages self-supervision on testing set to aggregate the learned multiple experts and subsequently utilize the aggregated ones to make prediction in the testing phase. Although effective, it lacks efforts to guide different experts to learn better their expertises during the training phase. BalPoE refines the logit adjustment intensity across all three experts and averages their outputs minimize the balanced error while ensuring Fisher consistency.
Different from the above works, find that different experts excel predicting different intervals of samples, e.g., longttailleammiooonvveeeee long-tailed expert is skilled in samples located in the head/mediun/tail interval. Consequently, we propose the dynamic expert assignment (DEA) module to estimate the class membership of samples and dynamically assign suitable experts to each sample based on the estimated membership to produce high-quality pseudo-label in the training phase (Fig. !) and excellent predictions in the testing phase (Tables 4, 5, 8, and 9). Moreover, Fig. 3 proves the class membership estimated by the DEA module is accurate.
12

---

A Square Peg in Square Hole: Meta-Expert for Long-Tailed Semi-Supervised Learning
Finally, for the first time, we observe that shallow features are relatively balanced although less discriminative, and deep features improve the discriminative ability but are less balanced, thus proposing the multi-depth feature fusion (MFF) module to make the model both discriminative and balanced (Table 2).
In summary, the differences between our Meta-Expert and existing methods are substantial, even though they also utilize a multi-expert network architecture.
C. Evaluation on FreeMatch
To further illustrate method's universality, we reproduce our method and all the compared methods by taking FreeMatch (Wang et al., 2023) as the base SSL method and evaluate them CIFAR-IO-LT, SVHN-LT, and STL-10-LT with the same settings used in the main paper. Tables 8 and 9 present the main results, indicating that our Meta- Meta-Expert can produce higher prediction accuracies by incorporating the expertises of different experts. For example, on CIFAR-IO-LT, given (N1, M1,7) 7) = (500, 4000, 200). Meta-Expert surpasses all previous baselines by up to 7.05 pp in the case of 7 = Yu- More- over, in the case of y * YYu Meta-Expert surpasses the previous SOTA method by 1.02 pp and outperforms all other baselines by 1.90 pp given (Ni, n, Pw) (1500, 3000, 150, 1). When moving SVHN-LT dataset, Meta-Exper surpasses the previous SOTA method by 0.55 pp and other baselines by up to 3.29 pp given (N1, M1, 72 7u) (1500, 3000, 150, 1/150). On STL-10-LT, Meper- performs comparable to the previous SOTA method SimPro, surpassing other baselines by up to 0.62 pp. All of these results further illustrate our method's effectiveness.
Table 8. Comparison of accuracy (%) on CIFAR-IO-LT under 7/- = YTu and 'Y # 7u seltings. We set y to 150 and 200 for CIFAR- We use bold to mark the best results.
1IFAR-IO-L1
CIFAR-
N₂ 500, M₁- 4000
N1 1500, M₁ 3000
N1 1500, M. 3000
Ni 1500, M, 3000
Algorithm
TV 200
N = 150
11 = 200
Tt = 150
7 = 200
71 = 200
7: = 150
72 = 150
% = 200
Nu = 150
Tu = 200
Th. = 150
%/ =
Th = 1/200
Th = 1
2w = 1/150
Supervised
41.15
43.88
16.83
59.82 +0.32
56.83 1111
56.83 1.10
$9.82 #0032
$9.82 +0.32
FreeMatch (ICLR 23)
63.35 40.49
68.03 10.68
69.83
73.00 1:063
78.99 10053
71.33 0.33
79.60 =0.72
72.76 1:0.7
w SAW (ICML 22)
39.31+1.20
65.69
65.69:1..94 10.94
72.05 +0087
74.69
+0.85
80. 16.166611 +0.25
73.24 20.57
81.90 11000
73.45 +0024
w/ Adsh (ICML 22)
61.64
1..5
69.27 1.55
73.81
+0.53
71.88 +0.47
65.16 20006
73.14 +0.57
66. 29 #0076
w DcPL (CVPR 22)
65.04 11007
69.65 1..43
70.66 1.0.8
73.29
1:0.3
80.37 10.49
72.28 00.17
80.14 1:100
73.20 1:377
W ACR (CVPR 23)
58.36 +2.35
60.98 1.24
70.24 1.99
72.55
1.34
81.89 100.4
82.68 00.21
83.49 1:0.36
83.85 1.0.1
w/ BaCon (AAAI 24)
08.63:0.77
72.69:0.66
75.97 11.34
77.71
11.3
83.20
82.48 =0.08
83.91 #0.42
83.66 +1.25
/CP (AAAT 24)
66.82+1136
6..80+
77.28 +0.63
79.24
+0.30
83.59
40.17
80.37 1111
84.37 +-0.14
79. +0080
PimPro (ICML 24)
64,20 +0.39
17.1313.31
78.57 +0072
80.49
+0.39
80.82
+0.24
77.59 +0.72
81.30 +:0.8
79.67 +:0.99
Mcta- Mcta-Expert (Ours)
75.68 +0.28
78.26 +0.63
80.53 #0.71
82.69
00.51
84.30
+0.29
85.32 +0.40
85.39 +0.48
85.89 00.79
Table 9, Comparison of accuracy (%) on STL-10-LT and SVHN-LT under x = To and m * 7/u settings. We set "t to 15 and 20 for STL-10-LT, and n to 150 and 200 for SVHN-LT, We use bold mark the best results, N/A denotes the unknown Yo in STL-10-LT since its inaccessible ground-trume label for unlabeled datasct.
1TL-10-LT
SVHN-LT
N₂ 150, M2 100k
N₂ 450, M,- 100k
N₂ 1500, Mr, 3000
N1 1500, M. 3000
Algorithm
20
15
7 --=20 20
15
7x 200
TX = 150
7 = 150
NN 150
7u = N/A
7u = N/A
Th = N/A
7h = N/A
7u = 200
Ti = 150
7u 1
The = 1/150
Supervised
1..44
42.31 +00.42
1..5
59.81 +0.45
84. +0.05
86. +0.50
86. +0.50
86. 14 +-0.50
FreeMatch (ICLR 23)
70.67 40.83
70.58
76.66 +-0.3
77.40
90.87 1.00
91.66=0.22
94.66 +:0.1
88.01 +:0.87
/ SAW (ICML. 22)
71.27 10.69
70.91 0.054
78.07 4-0.66
78. 10.44
89.04 00.59
90.09 #0.16
94.57 1.0.24
89.22 1:1.18
w Adsh (ICML 22)
67.37
68. 18.100::.11 10
73.44
74.43 10014
90.22 +0.45
91.78 =0.22
94.21 +0.57
88.47 10.66
w/ DePL (CVPR 22)
70.89
70.47 :10.48
77.41 1:0.11
77.47 10.47
90.64 10.61
91.44 10.37
94.62 1:032
87.74 1:006
W/ ACR (CVPR 23)
69.89 69.89+0.55 +0.54
70.98+40.55
78.51 +-0.26
79.35+00.3
84.78 11.11
87.0
92.89 +-0.5
91.21 +:0.09
W BaCon (AAAI 24)
71.51 #0.22
71.69 1.069
78.13
79.34+0.77
11.10
92.45 10.18
94,82 +0024
92.85 +0044
w CPE (AAAI 24)
69.73 10.25
70.46 10.54
78.84 1:0.13
79.18 10.54
91.83 10.35
92.21 10.03
94.32 10.41
90.39 10.41
SimPro (ICML. 24)
43.30 13.18
46.05 1.22
66. 70.70.1 1.669
68.03
1.117
93.17 10.35
93.73 +0.06
94.86 +0.22
93.95 1.0.99
w Meta- Expert (Ours)
71.57 10.40
1..60
78.94 10.08
79.40
0.25
93.21 +0.09
93.67 =0.07
94. 80 +0.37
94.50 10.33
13

---

A Square Peg in Square Hole: Meta-Expert for Long-Tailed Semi-Supervised Learning
D. Proof of Theorem 1
We first copy the Theorem here.
Theorem Suppose that the loss function {(f(z). y) is p-Lipschitz with respect to f(x) for all y E {l...., O}
and upper-bounded by U. Given the class membership 7) {I,...,Q) .Q and overall pseudo-labeling error c > 0, if
M E==1 Σ M[1,, 1))[(maa((a(x))) > T(ū; Ui) S 6, for any 8 O, with probubility least we have:
log
R(f) R(F) 2 + 2v Σ Ro (HHH) +2U
2U
(8)
20
Proof. We first derive the uniform deviation bound between R(f) and R(f) by the following lemma.
Lemma 1. Suppose that the loss function {((()),)) v) is p-Lipschitz with respect to (x) for all y E {1..... C} and
upper-bounded by U. For any 8 > O, with probability at least 1 - 8, we have:
log
IR() - $
RN RN Σ y=1
+U
(9)
2(N + M)
where the function space H. for the label {I..... C} is {h: 1 fz)) f F}.
Proof. In order to prove this lemma, we define the Rademacher complexity of the composition of loss function & and model E F with N labeled and unlabeled training samples as follows:
(N+a((eoF)
Mi (((((r))))))) + Σ us (((((()))))))) j=1
((z
sup fEF
<v2p
RN+M(HH)
(10)
where denotes the function composition operator, B(z,,,)) denotes the expectation over x, y. and μ., μ denotes the Rademacher variable, sup denotes the supremum (or least upper bound) over the function set F of model f. The second line holds because of the Rademacher vector contraction inequality (Maurer, 2016).
Then, we proceed with the proof by showing that the one direction sup R(ʃ) - R(()) is bounded with probability at least 1 5/2, and the other direction suP R(ff) R(f) can be proved similarly. Note that replacing a sample (z,, 34) leads to change of sur R(f) R()) at most N due the fact that e is bounded by U. According the McDiarmid's inequality (Mohri et al., 2018), for any 5 > 0, with probability at least 1 - 8/2, we have:
U
log z
sup R(f) R(f) SE fEF
sup R(f) R(())
+U
(11)
JEF
2(N + M)"
JEF R(f) - P((S]] < 2R (F), and further considering
According to the result in (Mohri et al., 2018) that shows E
the other direction supjer R(f) R(ff), with probability least 8, we have:
log
sup IR()) - R(f) - 2v Σ fer
R RN+M + +U 2(N + M)
(12)
which completes the proof.
14

---

A Square Peg in Square Hole: Meta-Expert for Long-Tailed Semi-Supervised Learning
Then, can bound the difference between (J) and (f) as follows.
Lemma 2. Suppose that the loss function ((f(x),)) y) is p-Lipschitz with respect to f(z) for all y E {l..... CJ and upper-bounded by U. Given the class membership n E Q) and overall pseudo-labeling error > 0, if k= 17 = 1))[[maar (f((z,)) 1) T(ū; y3) S., c, we have:
プ M
M
((( Ru, Ue.
(13)
Proof. Let's first expand RT):)::
M
R, =
Tnz, = (fa((()))) > ()((((())).)))
(14)
k=
Then, we show its upper bound:
1
ΣΣ
Σ I(nza 1)((f((zz). + +I( = 1)I(0s # us, max(fe(z))) > =)((f((2)).0)
M
1 M
TmEAA ()((((((15)))))) + +c
M
k=
Q
s (f)
C = (f)+
(15)
k=1
and its lower bound:
1 M
Q
R,002
[n 1)((f((z))..)) 1( ((maa((f(zz)) ()))((((r)).))
M
k=
1 M
Q
I( =
- -c
MM
j=1 k=1
Q Ek, = R Ue.
U
(16)
Q
By combining these two bounds, we can obtain the following result:
(()(() - RR, S Ue,
(17)
which concludes the proof.
Based on the above lemmas, for any 5 0, with probability least 1 5, we have:
log z
R(F) < R())+ + 2/2p Σ
(Hy + UU
20
2=1
log 2
< Ri(l) + R..(.) + 2v2p Σ
Ro (Hly) + +U
20
C Σ
log 3P0
< Ra()) + RR + Ue +2
Ro (H,)) + +U
20
==1
15

---

A Square Peg in Square Hole: Meta-Expert for Long-Tailed Semi-Supervised Learning
log
$ + +U Σ
Ro (o(Hy) (H1))+UUy UV
(18)
20
y=1
log 2
R((f) R.((f) + 2UE+ + 2/2p Σ Ro(H.) + U
20
log 20
Ro(H)) + +U < R(N + 2Ue +VV22 Σ
y=1
log 2
R(f) 2 + 4v2p Σ Ro(H.) 2011
20
where the first and seventh lines are based on Lemma 1, and three and fifth lines are due to Lemma 2. The fourth line by the definition of f. At this point, we have proven the Theorem 1.
16


Reviews:
[{"review_id": "wZwyqwXP7c", "reviewer": "Reviewer_wN5j", "summary": "This paper introduces Meta-Expert, a framework designed for long-tailed semi-supervised learning. Specifically, Meta-Expert includes a Dynamic Expert Assignment module, which predicts the class membership of a sample. A Multi-Depth Feature Fusion module, which integrates features from different depths to mitigate bias. Through extensive experiments, Meta-Expert achieves new state-of-the-art performance. Additionally, the authors conduct ablation studies to analyze the importance of different modules.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### **References:**\nWe appreciate the reviewer's suggestion and confirm that the highlighted literature is relevant to our study, and we will include citations in the revised manuscript.\n#### **Properties of different layers (Question & Weakness1):**\nIn deep networks, shallow layers capture local patterns while deep layers learn global semantics. For long-tailed learning, since head and tail classes may share similar local patterns, shallow features exhibit balanced discriminability across classes. Meanwhile, deep layers predominantly encode head class semantics due to their overwhelming sample dominance, thus biasing predictions toward head classes. This is empirically supported by Table 2: **shallow features are relatively balanced although less discriminative, and deep features improve the discriminative ability but are less balanced**.\n\nOur backbone network employs WideResNet-28-2, which consists of three convolutional block groups. We use the outputs of these three block groups (the 10th, 19th, and 28th layers) to represent shallow, middle, and deep features respectively. While our primary analysis focuses on features from these three specified layers, the phenomenon observed in Table 2 is general applicability. To verify this, we conducted extended experiments using alternative intermediate layers (the 6th, 15th, and 24th layers), with supplementary results presented in the floowing table, which aligns with Table 2's observations, suggesting that the selection of different layers does not significantly impact the revealed observations.\n#### Table R1\n| Feature depth | Overall | Head   | Medium | Tail   | GAP    |\n|---------------|---------|--------|--------|--------|--------|\n| 6th layer       | 24.09   | 23.77  | 29.23  | 17.57  | 6.20   |\n| 15th layer       | 35.16   | 39.77  | 37.60  | 27.30  | 12.47  |\n| 24th layer       | 44.61   | 59.10  | 41.15  | 34.73  | 24.37  |\n#### **Computational overhead (Weakness 2):**\nWhile our proposed modules (three experts, MFF, and DEA) introduce a controlled parameter increase of 13.3% (1.5M → 1.7M), this design achieves strategically balanced efficiency-performance trade-offs. Experimental results on CIFAR-10-LT demonstrate: a 6.4% increase in epoch time (234.5s → 249.5s), a 1.6s increased in inference time for evaluating 10,000 samples (7.1s → 8.8s), and a substantial accuracy gain of +3.5% absolute improvement (71.9% → 74.4%), collectively indicating significant performance enhancement with modest computational overhead.", "claims_and_evidence": "N/A", "methods_and_evaluation_criteria": "The proposed method Meta-Expert makes sense for the LTSSL problem.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experiment is relatively extensive.", "relation_to_broader_scientific_literature": "Meta-Expert reveals that features from different layers exhibit distinct characteristics in terms of separability and bias distribution, which could be beneficial for future research.", "essential_references_not_discussed": "Although the authors discuss many related works, they still miss some highly relevant studies, such as:\n* BMB: Balanced Memory Bank for Long-Tailed Semi-Supervised Learning, TMM 2024\n* CoSSL: Co-Learning of Representation and Classifier for Imbalanced Semi-Supervised Learning, CVPR 2022", "other_strengths_and_weaknesses": "Strengths:\n* This paper is well-written and well-motivated.\n* The MFF and DEA modules proposed in this paper are effective and relatively novel.\n \nWeaknesses:\n* The paper does not analyze the reasons behind the different properties (such as separability and bias) of features from different layers.\n* Due to the introduction of new modules, such as the three experts, MFF, and DEA modules, an analysis of the additional computational overhead is needed.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Which specific layers do the shallow, medium, and deep layers correspond to? Does the choice of different layers have a significant impact?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w7MaMKzCEQ", "reviewer": "Reviewer_sReE", "summary": "This paper proposes Meta-Expert, a semi-supervised learning method tackling the long-trained problem. By investigating the effectiveness of assiging different experts regarding the class membership, the model applies a dynamic expert assignment model to learn to assign the soft weight for three expert models. The method is futher improved by a feature fusion module to balance the assignment. The experimental results prove the effectiveness of proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### **Questions 1 (Aligned experimental setting):**\nOur work focuses on advancing long-tailed semi-supervised learning, thus primarily investigating settings with higher imbalance ratios (the more challenging scenarios) compared to recent works. As suggested, we have conducted supplementary experiments on CIFAR-10-LT, CIFAR-100-LT, and STL-10-LT under lower imbalance ratios aligned with recent works. As empirically demonstrated in Tables R2 and R3, **our method achieves comparable or superior performance even in these standardized benchmarks**: attaining better results in 5/6 cases and comparable performance in 1/6 cases on CIFAR-10-LT (with maximum gains of +1.19%), showing +1.06% improvement on STL-10-LT and +0.58% average improvement on CIFAR-100-LT. Moreover, **our advantages become even more pronounced under higher imbalance ratios: on CIFAR-10-LT with (N_1, M_1) = (1500, 3000), our method achieves performance gains of +0.23% (γ_l = 100), +1.26% (γ_l = 150), and +1.64% (γ_l = 200) over previous SOTA methods.**\n#### Table R2: **CIFAR-10-LT** with (N1, M1, γl) = (500, 4000, 100) (left three columns) and (N1, M1, γl) = (1500, 3000, 100) (right three columns)\n|             | γu=100 | γu=1 | γu=1/100 | γu=100 | γu=1 | γu=1/100 |\n|-------------|--------|------|----------|--------|------|----------|\n| BaCon       | 80.82 | 79.79 | 79.61 | 83.13 | 82.66 | 85.94 |\n| CPE         | 80.68 | **82.32** | 83.88 | 84.44 | 85.86 | 87.09 |\n| SimPro      | 72.77 | 71.78 | 73.05 | 82.33 | 80.25 | 83.22 |\n| Ours        | **82.01** | 82.01 | **83.94** | **84.94** | **86.13** | **87.32** |\n\n#### Table R3: **STL-10-LT** with (N1, M1, γl) = (150, 100K, 10) (left one column) and **CIFAR-100-LT** with (N1, M1, γl) = (150, 300, 10) (right three columns)\n|STL-10-LT           | γu=N/A |CIFAR-100-LT       | γu=10 | γu=1 | γu=1/10 |\n|-----------|--------|-------|-------|------|---------|\n| BaCon     | 71.15 | BaCon | 60.05 | 50.21 | 60.30 |\n| CPE       | 73.07 | CPE   | 59.83 | 48.09 | 60.83 |\n| SimPro    | 50.91 | SimPro| 59.04 | 48.25 | 60.09 |\n| Ours      | **74.13** | Ours  | **60.21** | **50.73** | **61.88** |\n#### **Questions 2 (Upper-E):**\n**Upper-E denotes CPE directly use GT membership to select a specific expert to produce pseudo-labels and make predictions**, while our method utilizes the integration of the logits from the three experts to compute the loss in Eq. (4), which may push different experts learn better. Moreover, as shown in Eq. (3), the final prediction of our method is the soft ensemble of multiple experts, which further brings better performance.\n\nTo provide empirical validation, we replaced our DEA module with the GT membership and conduct experiment on CIFAR-10-LT. The experimental results in Table R4  demonstrate that: i) **The design motivation for employing the DEA module to orchestrate expert collaborations through learned membership relationships is empirically well-founded**, ii) The MFF module can helps experts toward more optimal direction by fusing different depth features.\n#### Table R4: **CIFAR-10-LT** with (N_1, M_1, γ_l) = (1500, 3000, 200)\n|        | γu=200 | γu=1 | γu=1/200 |\n|--------|------|--------|----------|\n| CPE    | 78.57 | 83.47 | 84.40 |\n| **Upper-E** (CPE + GT membership) | 79.86 | 86.14 | 88.03 |\n| Ours   | 81.67 | 83.96 | 85.75 |\n| **Upper-E+** (Ours + GT membership) | **85.04** | **86.51** | **88.60** |", "claims_and_evidence": "The claims are well supported by the evidence.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense.", "theoretical_claims": "I check the analysis of generalization error bound in Sec 3.4.", "experimental_designs_or_analyses": "The experimental designs is convincing. However, the data setting is not consist with recent LTSSL works. Please refer to the Question part.", "relation_to_broader_scientific_literature": "This work is related to the long-tailed semi-supervised learing task. Previous LTSSL methods tackled this problem by various techniques like using the fusion output from multi-head classifier. This work further propose a meta aggregation module to automatically assign the class to experts.", "essential_references_not_discussed": "The related works are referenced.", "other_strengths_and_weaknesses": "Please refer to the Question part.", "other_comments_or_suggestions": "Please refer to the Question part.", "questions_for_authors": "1. The data setting in Table.4 (e.g., the setting of $N_1$, $M_1$, $\\gamma_l$, $\\gamma_u$) is not consist with recent works [1,2]. More experiments on the aligned setting are expected to make the result convincing.\n2. As shown in Table. 1, Meta-Expert achieves a significant higher performance than Upper-E, which use the ground-truth membership to align the experts. Does this result mean that GT membership is not the best alignment target?\n\n[1] Three Heads Are Better Than One: Complementary Experts for Long-Tailed Semi-supervised Learning.\n\n[2] SimPro: A Simple Probabilistic Framework Towards Realistic Long-Tailed Semi-Supervised Learning.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WDmQFImscb", "reviewer": "Reviewer_65rT", "summary": "In response to the problem of distribution mismatch between labeled and unlabeled data in long-tail semi-supervised learning (LTSSL), current methods refer to experts to model various unlabeled data, but various experts cannot match long-tail pseudo-labeled data well. This paper proposes a dynamic expert allocation method to solve the problem of long-tail unlabeled data, while connecting multi-scale feature fusion to improve the classification accuracy of LTSSL. Through a large number of experiments, it has been proven that the method proposed in the paper is superior to the SOTA methods.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### **More experiments (Experimental Design 1):**\nFollowing recent works like CPE and BaCon, we conducted experiments across three datasets (CIFAR-10-LT, STL-10-LT, and SVHN-LT), each evaluated under two different imbalance ratios (150 and 200). As suggested, we have extended our evaluation to include CIFAR-100-LT, along with conducting evaluations on CIFAR-10-LT and STL-10-LT under lower imbalance ratios. The results presented in Tables R2 and R3 [refer to the responce to reviewer sReE's Questions 1 (Aligned experimental setting)] demonstrate that **our method achieves comparable or superior performance even on these standardized benchmarks, with more significant performance improvements observed under higher imbalance ratios.**\n#### **Results of CPE (Experimental Design 2 & Weakness 3 & Question 2):**\nWe rigorously evaluated baselines (like CPE) **using their official codes under the specified conditions**, re-implementing experiments with matched hyperparameters and three independent runs.\n#### **Effectivness of DEA and MFF (Experimental Design 3):**\nThe ablation results in Table 6 demonstrate that: the **DEA module provides an average 1.68% accuracy improvement** across all imbalance ratios (γ_u = 200/1/1/200), showing relatively greater performance gains; while the **MFF module delivers a 0.76% average gain**, which though comparatively smaller, remains statistically significant. The **combined DEA+MFF configuration achieves 2.42% improvement**, confirming their complementary effectiveness and synergistic interaction. These improvements conclusively validate the effectiveness of our proposed modules in enhancing model robustness across diverse imbalance ratios. \n#### **Multi-scale feature (Weaknesses 1 and Questions 5):**\nWe acknowledge existing discussions about multi-scale feature characteristics in deep learning. However, to the best of our knowledge, **no prior work has investigated how long-tailed distributions differentially affect shallow and deep features**. Our analysis reveals that shallow features are relatively balanced although less discriminative, and deep features improve the discriminative ability but are less balanced. We strategically leverage this complementary characteristic to achieve performance gains through fusion of different depth features. Specifically, the ablation results in Table 6 demonstrate that the MFF module **delivers a 0.76% average gain**, confirming its significant effectiveness.\n#### **Expert attention (Weaknesses 2 and Questions 4):**\nThe expert attention scope in our DEA module **follows the partitioning in CPE** while maintaining comparable performance to alternative scoping strategies. As shown in Table R5, we conducted additional experiments with alternative splits on CIFAR-10-LT. The average performance fluctuation of our method during partitioning changes is **only 0.37%**, which is smaller than CPE's 0.81%, verifying the DEA module's robustness regardless of specific partitions.\n#### Table R5: **CIFAR-10-LT** with (N_1, M_1, γ_l) = (1500, 3000, 200)\n||γu=200|γu=1|γu=1/200|\n|-|-|-|-|\n|CPE[2, 2, 6]|78.57|83.47|84.40|\n|CPE[3, 3, 4]|77.72|82.76|83.52|\n|Ours[2, 2, 6]|81.67|83.96|85.75|\n|Ours[3, 3, 4]|80.92|84.07|85.51|\n#### **DEA training (Questions 1):**\nTo address parameter optimization for the DEA module under poor initial pseudo-labels, we introduced a confidence threshold (t = 0.95) to filter unreliable pseudo-labels following CPE and ACR. To quantify its impact, we evaluated the pseudo-label accuracy with and without the threshold in the following Table R6, which suggests **incorporating threshold improves accuracy by 3.85% on average**.\n#### Table R6: **CIFAR-10-LT** with (N_1, M_, γ_l) = (1500, 3000, 200)\n||γu=200|γu=1|γu=1/200|\n|-|-|-|-|\n|Ours without threshold|84.39|81.80|91.65|\n|Ours (including threshold)|**91.71**|**82.87**|**94.81**|\n#### **Fusion strategy (Questions 3):**\nCPE does not use fusion strategy and employs three experts simultaneously to produce pseudo-labels for all samples, along with a uniform expert to make predictions. In contrast, our DEA module **learns class membership automatically** and utilizes it to select the most appropriate single expert for each sample. This strategy avoids error accumulation caused by conflicting predictions in CPE, thereby significantly improving pseudo-label quality. Our ablation study in Table 6 quantitatively demonstrates that DEA achieves an average 1.68% accuracy improvement across all imbalance ratios (γ_u = 200, 1, 1/200) compared to CPE.\n#### **Aggregator (Questions 6):**\nThe aggregator model processes both strong and weak augmentation views simultaneously, generating $y_{m, j}^{u}$ and $\\hat{y_j}$ respectively. The aggregator’s parameters are updated through a unified optimization process. Specifically, the model is trained to minimize the consistency loss between $y_{m, j}^{u}$ and $\\hat{y_j}$, ensuring that predictions align across augmentation views.", "claims_and_evidence": "Please provide specific examples regarding the mismatch between labeled and unlabeled data distributions in LTSSL.", "methods_and_evaluation_criteria": "The proposed method in the paper make sense for the problem or application at hand.", "theoretical_claims": "I have checked the theoretical proof, but I am not quite sure about its correctness.", "experimental_designs_or_analyses": "1. Compared with theBacon,  ACR and CPE methods, the experiments in this paper lack comparison on the CIFAR100-LT and ImageNet-127 datasets.\n2. When compared with baseline methods such as CPE, the accuracy values of the baseline methods mentioned in Table 4 of this paper were not found in the original text, such as when r_l=200,  r_u=200. I wonder how this paper obtained the accuracy values of various methods for this condition? Similar situations also appear in Table 5, such as STL-10-LT.\n3. In the ablation study, as shown in Table 6, when r_u=200, 1, 1/200, the accuracy using DEA module is similar to that without DEA module. Does this indicate that the role of DEA module is not significant?", "relation_to_broader_scientific_literature": "The method proposed in the paper is inspired by the published CPE method and improves the shortcomings of experts in CPE.", "essential_references_not_discussed": "Not found yet.", "other_strengths_and_weaknesses": "Strengths of the paper: \n1. The paper focuses on the attention range of different experts in LTSSL, and the accuracy of each expert's prediction of pseudo labels depends on the category of their respective training data. But the paper does not delve deeply into the issue, and the prediction accuracy of different experts is only related to the number of corresponding categories? \n2. The paper is well-organized, clearly expressed, and easy to read and understand. \n\nWeakness of the paper: \n1. The first discovery of the characteristics of features at different scales mentioned in the paper is a bit exaggerated. In deep learning, the characteristics of features at different scales have already been mentioned. \n2. The DEA module mentioned in the paper lacks a detailed introduction to the scope of expert attention. \n3. The experimental part of the paper has unclear data sources and lacks some of the datasets mentioned in the published papers.", "other_comments_or_suggestions": "Please refer to the question section.", "questions_for_authors": "1. At the beginning of model training, how to solve the parameter training of DEA module under poor pseudo-label conditions?  Because the parameter optimization of DEA module largely relies on labeled data and unlabeled data, and the pseudo-label accuracy of unlabeled data is poor. \n2. Does the experimental dataset fully reference the datasets of other long -tail semi-supervised training methods?\n3. What are the specific differences between the three expert fusion strategies proposed in this paper and the CPE method? \n4. How to quantitatively define the recognition scope of each expert in the fusion of the three experts proposed in this article? What is the recognition range of long tail experts? \n5. The advantages and disadvantages of multi-depth features have been discovered by researchers for a long time, and this paper insists on presenting them for the first time, which is a bit exaggerated. The MFF module's contribution is somewhat limited.  The ablation study also showed that the role of the MFF module was not significant.\n6. If the predicted labels y_ {m, j} ^ {u} and pseudo-labels \\hat {y}_j are obtained through an aggregator, are these two aggregators the same model or different models? How to update the parameters of two models to ensure optimal performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6EGIuOqktk", "reviewer": "Reviewer_LauN", "summary": "This paper addresses the challenge of long-tailed semi-supervised learning (SSL) by proposing a framework that automatically integrates multiple expert knowledge to generate high-quality pseudo-labels, thereby improving SSL performance in imbalanced data settings. The authors analyze three types of long-tailed distribution scenarios — consistent, uniform, and inverse data distributions — to simulate different real-world imbalance conditions. Leveraging the observation that different layers of features capture different levels of semantic information, the paper introduces a multi-layer feature fusion (DEA loss) mechanism that enables the aggregation of rich and diverse representations for better learning. The proposed loss combines three components: (1) a base SSL loss for standard semi-supervised objectives, (2) a meta-learning loss to aggregate the expertise of multiple models, and (3) the DEA loss to fuse multi-depth features. Additionally, a theoretical analysis is presented to establish a generalization error bound, which supports the robustness of the framework. Extensive quantitative and qualitative experiments on standard benchmarks demonstrate the superiority of the proposed approach over existing baselines.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### **References:**\nWe appreciate the reviewer's suggestion and confirm that the highlighted literature is relevant to our study, and we will include citations in the revised manuscript.\n#### **Realistic cases (Weaknesses 1):**\nIn the medical field, when collecting information from various patients, we may obtain a long-tailed dataset from non-specialized hospitals, i.e., a large number of common disease cases (head classes) accompanied by very few rare disease cases (tail classes). However, if we consider specialized hospitals focused on specific rare diseases, this scenario would yield an **inverse long-tailed dataset** characterized by abundant rare disease cases and scarce common disease cases. We will add this example in the final version.\n\nIn practical applications, where the **distribution of unlabeled data is unknown**, we follow recent works to investigate **three representative extreme distribution** scenarios. If a model can perform well on all those three extreme cases like ours, it is expected to fit different unlabeled data distributions. \n#### **Model design (Weaknesses 2):**\nTo effectively address three different unlabeled data distributions, we constructed three MLP-based experts with logit adjustment, specifically designed for long-tailed, uniform, and inverse long-tailed distributions, respectively. Furthermore, we observed that different depth features and differnt experts exhibit its own properties (as evidenced in Tables 1-3 and Fig. 1). To fully leverage the characteristics of multi-expert and multi-layer features, we employed DEA and MFF for respective expert and feature integration. Table 6 validates the effectiveness of DEA and MFF. Moreover, we investigated the performance differences between addition-based and concatenation-based feature operation strategies for feature fusion in Table 9, ultimately adopting the superior addition-based feature operation strategy.\n#### **Training Stability (Weakness 3):**\nOur empirical validation demonstrates consistent optimization behavior across multiple runs, as evidenced by the **lower standard deviations** reported in Tables 4-5 and 7-8 (which are comparable to or better than those of baselines). We further provide the accuracy curve in https://anonymous.4open.science/r/Acc-curve/Acc-curve.png, which shows that our method performs stable in the later stages of training. \n#### **Computational Complexity (Weankess 4):**\nPlease refer to the response to reviewer wN5j's Computational overhead (Weakness 2).", "claims_and_evidence": "Most of the claims are provided with evidence.", "methods_and_evaluation_criteria": "N/A", "theoretical_claims": "I didn't check the proof carefully.", "experimental_designs_or_analyses": "Yes, the experimental design is reasonable.", "relation_to_broader_scientific_literature": "This paper enhances the LTSSL with a novel training strategy which could be potentially impactful.", "essential_references_not_discussed": "Missing several references:\n- Huang et al., FlatMatch: Bridging Labeled Data and Unlabeled Data with Cross-Sharpness for Semi-Supervised Learning, in NeurIPS 2023.\n- Yang et al., Robust semi-supervised learning by wisely leveraging open-set data, in TPAMI 2024.\n- Lee et al., (FL)2: Overcoming Few Labels in Federated Semi-Supervised Learning, in NeurIPS 2024.", "other_strengths_and_weaknesses": "Strengths:\n- Clear and Well-Structured Presentation:\n  - The paper is well-written, logically organized, and easy to follow. The motivations, methodology, and results are clearly articulated, making it accessible to a broad research audience.\n- Comprehensive Experimental Evaluation:\n  - The experimental results show substantial improvements over a wide range of strong baseline methods, indicating that the proposed framework is effective and competitive.\n  - The inclusion of both quantitative and qualitative analysis enriches the experimental section, providing deeper insights into the behavior of the proposed method.\n- Theoretical Justification:\n  - The inclusion of a generalization error bound lends theoretical credibility to the method, which is often lacking in many applied SSL works. This analysis strengthens the contribution and demonstrates a solid understanding of the underlying learning dynamics.\n \nWeaknesses and Concerns:\n- Unclear Real-World Relevance of Distribution Scenarios:\n  - Although the paper defines three types of data distributions (consistent, uniform, inverse), their practical meaning and relevance to real-world scenarios are insufficiently justified.\n  - These settings appear to be artificial constructs, and while they provide controlled experimental environments, it is unclear how frequently such distributions occur in real applications.\n  - It would greatly improve the paper to discuss realistic cases where such distributions might naturally arise (e.g., medical diagnosis, fraud detection), or to propose alternative data splits grounded in real-world statistics.\n- Lack of Intuition and Justification for Framework Design:\n  - Although the overall methodology — aggregating multiple experts and multi-layer feature fusion — is intuitively reasonable, the specific architectural choices are not well-justified.\n  - The combination of multiple MLP-based experts and fusion mechanisms seems ad hoc. It remains unclear why these components are structured this way and what alternative designs were considered.\n  - More insight into the design rationale, possibly supported by ablation studies, would clarify why this particular form of integration is necessary or optimal.\n- Training Stability and Optimization Concerns:\n  - Given the complex interplay between multiple expert modules and feature fusion components, optimization stability becomes a natural concern.\n  - However, the paper does not provide a discussion on training stability, nor does it offer empirical evidence (e.g., loss curves, variance across runs) to assure readers of its robustness during training.\n  - Addressing how gradient conflicts, convergence difficulties, or sensitivity to hyperparameters are handled would greatly improve the trustworthiness of the method.\n- Computational Complexity and Efficiency Not Addressed:\n  - The proposed method introduces numerous additional components, including multiple expert models and cross-depth feature fusion layers, which likely result in significant computational overhead.\n  - Yet, no analysis or discussion on computational cost (e.g., training time, memory usage, inference latency) is provided, which is critical for assessing the practical deployability of the method.\n  - Providing comparisons of resource consumption with baseline methods would offer a more balanced view of the method's cost-benefit trade-off.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please check the weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper received three positive reviews and one negative review.  The strengths of the work include: (relatively) novel idea, effective algorithm, comprehensive evaluation, and some theoretical analysis. There are a few weaknesses, such as insufficient analysis of the property, lack of optimization stability, and computation cost, which, however, have been addressed by the rebuttal.

The negative rating was given by Reviewer 65rT. The weaknesses pointed out by the reviewer are:  1) The first discovery of the characteristics of features at different scales mentioned in the paper is a bit exaggerated; 2) The DEA module mentioned in the paper lacks a detailed introduction to the scope of expert attention; 3) The experimental part of the paper has unclear data sources and lacks some of the datasets mentioned in the published papers. Regarding these weaknesses and the concerns of the reviewer, the authors provided a lot of clarification. The reviewer didn't think the rebuttal addressed the concerns.

As far as I am concerned, the issues mentioned by Reviewer 65rT are relatively minor and at least partial of the concerns have been addressed.
Decision: Accept (poster)