SoLar: Sinkhorn Label Refinery for Imbalanced Partial-Label Learning
Authors: Haobo Wang
Mingxuan Xia
Yixuan Li
Yuren Mao
Lei Feng
Gang Chen
Junbo Zhao
Venue: NeurIPS 2022 Accept (2022)
Keywords: Partial-label learning
Long-tailed learning
Optimal Transport

Abstract: Partial-label learning (PLL) is a peculiar weakly-supervised learning task where the training samples are generally associated with a set of candidate labels instead of single ground truth. While a variety of label disambiguation methods have been proposed in this domain, they normally assume a class-balanced scenario that may not hold in many real-world applications. Empirically, we observe degenerated performance of the prior methods when facing the combinatorial challenge from the long-tailed distribution and partial-labeling. In this work, we first identify the major reasons that the prior work failed. We subsequently propose SoLar, a novel Optimal Transport-based framework that allows to refine the disambiguated labels towards matching the marginal class prior distribution. SoLar additionally incorporates a new and systematic mechanism for estimating the long-tailed class prior distribution under the PLL setup. Through extensive experiments, SoLar exhibits substantially superior results on standardized benchmarks compared to the previous state-of-the-art PLL methods. Code and data are available at: https://github.com/hbzju/SoLar.

SoLar: Sinkhorn Label Refinery for Imbalanced Partial-Label Learning
Haobo Wang'
Mingxuan Xia² Yixuan Li
Yuren Mao²
Lei Feng 45 Gang Chen' Junbo Zhao
Key Lab of Intelligent Computing based Big Data of Zhejiang Province, Zhejiang University
School of Software Technology, Zhejiang University
Department of Computer Sciences, University of Wisconsin-Madison
College of Computer Science, Chongqing University
Center for Advanced Intelligence Project, RIKEN
{wanghaobo, xi iaainnguuannuureeee mao, cg. zhao) (ajaaoeezzaa.. @zju.edu.
sharonli wisc.edu, ofeng@equ.edu. edu.
Abstract
Partial-label learning (PLL) a peculiar weakly-supervised learning task where the training samples generally associated with set of candidate labels instead single ground truth. While a variety of label disambiguation methods have been proposed in this domain, they normally assume a class- balanced scenario that may not hold in many real-world applications. Empirically, we observe degenerated performance of the prior methods when facing the combinatorial challenge from the long-tailed distribution and partial-labeling. In this work, we first identify the major reasons that the prior work failed. We subsequently propose SoLar, a novel Optimal Transport-baid based framework that allows to refine the disambiguated labels towards matching the marginal class prior distribution. SoLar additionally incorporates a new and systematic mechanism for estimating the long- tailed class prior distribution under the PLL setup. Through extensive experiments, SoLar exhibits substantially superior results standardized benchmarks compared the previous of-the- PLL methods. Code and data are available https:: //gi::///an thub. com hbz SoLar
Introduction
The remarkable success of deep learning typically requires collecting massive labeled data, which notoriously labor-intensive. Of particular concern, data labeling suffer from inherent and pervasive label ambiguity. Take Figure [ (a) as an example, a Mule can be visually similar to both Donkeys and Horses, which hinders non-expert annotators from recognizing the true breed. Similar cases also arise in many real-world applications, such as automatic image annotation [1 bird song recognition 2 and facial age estimation 33] To deal with ambiguous supervision, partial-label learning (PLL) [4] [}, which allows each training example to be annotated by set of candidate labels, has attracted significant attention from the community. A plethora of methods have been developed to tackle this problem, including pscudo-labeling methods 66 Z7 graph-baser methods [8] 9 [0], etc.
Despite the promise, existing PLL methods have been commonly driven by the assumption that training data consists of class-balanced distribution, which may not hold in practice. In many real- world scenarios, training data exhibits long-tailed label distribution [II.. That is, many labels occur infrequently in the training data. Concerningly, the imbalanced data can exacerbate the fundamental
"Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

40
100
Solar Cutimaaion PRODEN Estinntiinn
PR2300M
Radio of Exammla (9)
35
90
30
25
Accuracy
80
20
15
40
10
20
(Donieey, Mule, Horse)
5 6 Class Index
8
4
5 Class
6
10
(a) Example of label ambiguity
(b) Class prior estimation
(c) Class-wise accuracy
Figure 1: (a) An input image with three candidate labels, where the ground-truth is Mule. (b) The real/estimated class distribution (by counting real/predicted labels) on a long-tailed version of CIFARIO dataset with partial labels (more details in Section 4. The class prior estimated by SoLar is very close to the real one. while PRODEN fails to recognize some tail labels. (c) Class wise performance comparison of PRODEN and SoLar on the same dataset, where Sol performs much better data-scarce classes.
challenge of label disambiguation. i.e., identifying the ground-truth label from candidates. Indeed, we find that current best-performing PLL. methods display degenerated performance in the long-tailed setting. This happens because the predictions of pseudo-labeling methods rely on on-can be largely biased towards the head and majority classes. We exemplify the phenomenon in Figure (b), where a strong method PRODEN [6] rarely assigns probability mass to the tail classes (indexed by 9 and 10). As result, these tail classes suffer from almost zero accuracy, as shown in Figure [ (c). To date, few efforts have been made to resolve this.
core component that PLL.
Motivated by this, we propose a novel framework for long-tailed partial-label learning, called Sinkhorn Label refinery (dubbed SoLar). Our framework emphasizes the long-tailed nature of training data, while performing label disambiguation for partial-label learning. Our key idea is to enforce constraints on pseudo-lahels- the distribution of which should match the class prior distribution. The constraints can incentivize the label disambiguation process to select tail labels from candidate sets, instead of always selecting head classes. We formalize the idea as an optimal transport problem (12) that searches for proper label assignment subjected to constraints: (I) the probability mass distributed within candidate sets; (2) the distribution of (predicted) pseudo-labels matches the prior class distribution. The first constraint is natural in the PLL setup, and the latter one uniquely empowers SoLar to disambiguate tail labels from candidates. We show that our constrained optimization objective incurs only a few computational overheads. Thcoretically, we prove that our overall objective is statistically consistent, which ensures the optimality of the learned classifier at a population level.
be tractably solved using the Sinkhorn-Knon
algorithm [13,, which
As an integral part of our framework, SoL.ar tackles the challenge of estimating the class prior distribution, which is necessitated in our constrained optimization. Unlike conventional long-tailed learning (LTL) scenarios [4 [5). the PLL setup presented with label ambiguity in the proximity of candidate sets. This makes it intractable to estimate the marginal class prior distribution, especially by counting training samples grouped by class labels. We tackle this non-trivial problem by per- forming an iterative class prior estimation, which provides strong proxy for real prior (Section 3.2) Empirically in Figure (b), we show that the estimated class prior favorably matches the ground-trun truth prior on long-taiil maigtt version of the CIFAR10 10 dataset with partial labels.
We comprehensively evaluate SoLar on various benchmark datasets, where Sol establishes state- of-the-art performance. Compared to the best baseline, SoLar improves the accuracy on tail classes by 15.12% on the long-tailed version of the CIFARIO dataset. While our work primarily focuses on label disambiguation from the PLL perspective, we demonstrate that SoLar can be compatible with LTL techniques as well. For example, we equip SoLar with logit adjustment [14), a state-of-ther oftthe--rt LTL method and further improve SoLar's performance by 6.72% on long-tailed CIFARIO. We hope our work will inspire future works to tackle this important problem.
2 Problem Setup
Let X R.d denote the input space and y = (1...., T.) denote the label space. We receive a training datasct D- St)R with examples. Each tuple in D consists of image x, X' and
2

---

Algorithm 1: Pscudo-code of SoLar.
Training dataset D, classifier f. uniform marginal r; hyperparameters P: for epoch = 1, 1,2..... .
Input:
for step = 1....... do
Get classifier prediction P on mini-batch of data B of size Tt
Calculate M such that m.j = Pt₂ E St)
for - c./33)) B B-r./(M's.
1...., T do
// Sinkhorn-Knopp iteration
end Q diag(a) for j 1.... do
// Compute paeudo-labels
11
Generate a subset B, = [( q:) ) = arg maxy'ss, 4.y' from B Select reliable samples by small loss 4, and high-confidence e, // Llasswiise rel allable sample selection
12
13
14
Train the classifier by minimizing the cross-cntropy loss
end
Compute by 2j
Σ i=1
T(
arg max (a.)) yE.
// Empirical class prior estimation
17
pr + (1 11)
// Moving-average distribution updating
end
the candidate label set S, C y. Following previous works 6 we assume the true label ys E y of is concealed in S,, i.e., Vi E S,. A fundamental challenge in partial-labe learning is label disambiguation, i.e., identifying the ground-truth label y; from the candidate label set S;.
Our goal to train a classifier / X' + [0, 1)2, parameterized by 0, that can perform predictions on unseen testing data. Here f is the softmax output of neural network, and f-(-) denotes the j-th entry. Besides, denote the prediction matrix by P .p...... Pn] [p where Pij fx;)), To perform label disambiguation, we maintain pseudo-label matrix Q (q,, ..,,..., ..................... 2n [0. 1
and train the classifier with the cross-entropy loss (efff q:) = Σラ -9y (g(((((x;) where 4s is the j-th entry of qı
Proposed Method
In this section, we describe our novel Sinkhorn Label refinery (SoLar) framework for partial-label learning. As we reckon, the combinatorial challenge of partial-labeling and long-tail learning lies in matching between decent marginal prior distribution with drawing the pseudo labels. To cope with these problems, Sol ar comprises two components. On one hand, SoL.ar formulates an optimal transport objective to facilitate the refinement of pseudo-labels match the marginal class prior. The resulted constrained optimization objective encourages the model to properly draw the tail labels. On the other hand, the PLL setting manifests difficulty in straightforwardly estimating the class prior. SoLar henceforth incorporates two novel techniques-a mooving-averagg estimation approach with a complementary reliable set sampling mechanism. The pseudo-code is summarized in Algorithm 7
3.1 Sinkhorn Pseudo-Label Refinery
Optimal Transport Objective. We formalize an optimal transport problem for proper label assign- ments. At each training step, Sol ar searches for pseudo-labels Q close to the current classifier's predictions P. while subject to some constraints:
min E(Q, -Q, -Q.- -logg QEA
Σ
Σ
4i log Pij
==
s.t. A (1gglaxr 1 r. -
S.) C 10.
is an T-dimennional probability simplex that indicates the prior class distribution. other words, would like the summation of the j-th column in Q (i.e., the total probability mass predicted as

---

class j) to match the prior class probability. Note that, here we temporarily assume we have decent estimation of the class priors and we will describe the means of estimation in the following Section 3.2
The column vector c x In indicates that our training examples are sampled uniformly. In Eq. 1
with a slight abuse of notation, Q is essentially joint probability matrix post to the refinery. That is, each æ; is treated as supplier holding units of goods while the j-th label needs (ij units of goods. To obtain pscudo-labels, we take a re-scaled solution n2 bearing an assumption of a uniform
marginal A of æ,,
With further serutiny of the objective function above, this formulation complies with two constraints we posited, such that: (i)-it avoids biased assignment towards (only) the head labels in a long-tailed PLL setup. with Q 1, r constraining the distribution of pseudo-labeling to match the prior of the class labels. (ii)-the probability mass of Q in each row is driven to be distributed within the candidate label set of a sample Fz
In Eq. [T) we use (pgg)))) as cost such that predictions with higher confidence enjoy a lower cost. This can also be replaced by other cost functions h(pit) as long as is monotonically decreasing. But, we show that the negative log-likelihood cost results in a statistically consistent risk estimator.
Theorem 1. Denote Á the set of measures on X' x y that satisfies the constraints in Eq. be the distribution of tuple (x. 5). The population risk of our method is as follows,
7 Let Dr
R() = inf B(a.S)-D, =Eá
(2)
Let f and f denote the optimal classifiers trained on fully- fully-supervised data with cross-entropy loss and PLL data with R(f), respectively. Assume the small ambiguity condition (5) is satisfied. Then, under the deterministic scenario and ignoring the null set, we have =
We refer the readers Appendix A.1 for the proof. Theorem D iindicats that the optimal classifier f can be recovered by our algorithm at a population level. Compared with other theoretically consistent PLL risks 66 [6], risk preserves the marginal class distribution of the estimated pseudo-labels, and is reasonably more favorable in the long-tailed learning setup.
Fast Approximation. To resolve Eq. ( adapt the well-known Sinkhorn- Knopp algorithm [3] for efficient optimization. Formally, we define a matrix M such that m.y = p [[j E 8.),, where X is smoothing regularization coefficient. The pseudo-labels are obtained by,
Q = n diag(a) Mddag(())))
with iteratively updated a + c./(333), 3 B-r./(M a.).
Here // denotes dlement-wise division. a E R and 3 E TRL are known as scaling coefficients vectors. Detailed derivation are provided in Appendix A.2. Here, we set m, if j & S, which is equivalent to define log(0) = inf. Thus, Q will also be assigned Os outside the candidate sets.
Inspired by [17,, we also involve queue acceleration trick avoid traveling the whole training set. We found that Eq. (3) typically converges to a satisfactory result in < 50 rounds. With only simple algebra operations, this can be efficiently implemented on GPUs; see Appendix C.3.
3.2 Iterative Class Prior Estimation
Unlike conventional long-tailed learning scenarios 14 [5), the PLL setup is presented with label ambiguity in the proximity of candidate sets. This makes it intractable to estimate the marginal class prior distribution, especially by counting training samples grouped by class labels.
In SoLar, towards estimating the class marginals, our aim is to propose solution with (almost) full autonomy that can adapt to the data distribution and the training state of the model. Here, we propose two interdependent techniques to achieve this goal. The resulted estimation is utilized in Eq (
The core of our method is two-fold: (i)-we propose a moving-average style updating rule for class prior estimation, resulting in stable training dynamics; (ü)-we carefully devise a sample selection mechanism for robust training, while preserving the marginal label distribution as much as possible.
Moving-Average Distribution Updating. We propose using the model predicted results proxy for class prior estimation. In spite its simplicity, we cannot ignore the fact during the early stage of training, the predicted labels can be largely imprecise, not trustworthy, and biased towards

---

the head labels. Therefore, we propose an update mechanism in a moving moving-average average fashion. Since we hold no assumption on the true class prior, the updating scheme initiated from uniform class prior [/T...... .... [.]. Respectively at each cycle (often defined by epochs), Sol .ar refines the distribution as follows.
f- po + (1 22 where 2j - R
:=1 T(j = arg maxy'ss fy((:))
(4)
where la E (0, 1] is a preset scalar. As displayed, the class prior is progressively updated alongside stabler training dynamics through linear function. As the training proceeds, the model becomes more accurate and thus the estimated distribution becomes more reliable.
Class-wise Reliable Sample Selection. In the early stages of training, the class prior estimation can be inaccurate due to the poor pseudo label predictions. Thus, we posit that the lack of proper sample selection early stages may jeopardize the marginal prior estimation. As common as it is for PLL learning dynamics, we want to reduce the variance of the pseudo-labe quality, via a reliable sampling algorithm. This module works together with the moving-average class prior estimation.
foute-force but effective strategy is to employ the small-loss sampling criterion noisy-label learning [8] which assumes low-quality pscudo-labels can likely be characterized by large loss. However, in a long-tailed setup, the tail labels can also incur a large loss because of their low-shot nature, making them rarely selected. We empirically show this phenomenon in Section 4.3 implement the following procedures into the proposed SoLar framework. The goal of such a module to make sure the coverage of the tail labels well to keep the marginal prior unbiased.
We hence
To begin with, to make sure that all labels - particularly the tail labels are coverd, we split the batch of data B into L slices that are indexed by the argmax taken from the refined pseudo-labels: B₁ {(ar; q:)lj = arg res, 0iy' C B. In what follows, we derive two combinatorial rules on the selectivity of the training samples:
(i) Based on the aforementioned setup, we modify the small-loss criterion in the noisy-label learning [8] and adapt it to a long-tailed setup. Specifically, we first compute the instance- wise loss by marginalizing over the class index: 4 - E/= 95y log (Pt)) Then, within the provided set Bj, we anchor the total sample number to be min [rypl B[], [B,)), which is associated with ry for distribution-prrseerinn The samples are being selected with top-k smallest 4,
(ii) A parallel mechanism inspired by the application of pseudo-labeling schemes in semi- supervised learning 19]. To adopt such approaches in SoLar, we purposely select those samples having a very high confidence score which is measured by the cosine similarity: e, = q: Pi- As such, this second criteria selects samples that meet the incquality e; > This mechanism further ensures high utility on the (likely) dellbdisamb samples.
Notice that, p, > are two thresholding hyper-parametenn The two introduced mechanisms are established to work collaboratively. Simply put, samples that satisfy either criterion are used for training the classifier. We will show later in Section 4.3 the efficacy of our sample selection mechanisms.
Experiments
In this section, experimentally analyze the proposed SoLar method under various scenarios for the imbalanced PLL problem. More empirical results can be found Appendix C.
4.1 Setup
Datasets. First, we evaluate Sol.an Lorar on two long-tailed datasets CIFAR10-LT and CIFAR
introduced in 20 21. The training images are randomly removed class-wise to follow pre-defined imbalance ratio indices are sorted based on the class wise sample size, in descending order with .... 2 ny- We have n, = 5, 000 for CIFARIO-LT and n = 500 for CIFAR 100-LT. We use different imbalance ratios to evaluate the performance of SoLar, with 7 (50. 100, 200) for CIFAR 10-LT and 7 E (10, 20, 50) for CIFARIO0-ITT. We then generate partially labeled datasets by manually flipping negative labels ğ # y to false- false-positive labels with probability P( E Y g # y), which follows
di where n, is the image number of the j-th class. For convenience, class
5

---

Table I: Accuracy comparisons on CIFARIO-LT and CIFARI00-LT under various flipping probability & and imbalance ratio 7. Bold indicates superior results.
CIFARIO-LT
Methods
0.3
Φ = 0.5
50
100
200
Y = 50
100
Y 200
MSE
61.13
52.59 #0.48
48 +0.45
49.61 +:.4
43.90
39.52 -.700
EXP
52.93+344
43.59
42.56+0044
50.62+3.00
43.69 2.72
41.07 0.62
LWS
44.51
43.60=a.12
42.33
24.6229.57
27.33
28. 28.74-1.85
VALEN
58.34
50.20
46 1..4
40.02211.8
37.10=0.88
36.61 0.57
CC
78.76:2027
71.86
63.38
73.09
6..8
54.41 0.88
PRODEN
81.95+0.19
71.09
63.00
66.00
62.17
54.65 -165=1.000
PICO
75.42
67.73+0064
61.12
+2.33
63. 63.25 25.2255::00 00.64
53.92 1.64
SoLar (ours)
83.80
76.64
67.47
81.38
74.16
62.12
CIFARI00-LT
Methods
0.05
$ = 0.1
7 = 10
? = 20
Y == 50
Y =
Y = 20
? 50
MSE
49.92
43. 1..6
37.77
42.99
37.19
31.49=0.35
EXP
25.861091
2..8
23.58
24.82
21.27=1.24
19.88=0,43
LWS
48.85
35.88
19.2221.56
6,10=205
7.16=2.03
5.15
VALEN
49.12:10.8
42.05
35.6211043
33.39
30.67
24.93=0.87
CC
60.36+a.52
54.33
45.83+0031
57.91
51.09=0.48
41.74-0.11
PRODEN
60.31
50.39
42.29
47.32220.50
41.82=0055
35.1 0000
PiCO
54.05
46.93
38.7420.11
46.49
39.80=n.34
34.97
SoLar (ours)
64.75
56.47 10.76
46.18
61.82+00.1
53.03
40.96 1.01
The final candidate label set is composed of the ground- truth label and the flipped false-positive labels. We choose ó E {0.3, 0.5) for CIFARIO-LT and {0.05, 0.1} for CIFARIO0-LT. For all experiments, we report the mean and standard deviation based on 3 independent runs (with different random seeds).
the settings in previous works 6 22
Baselines. We compare SoLar with six state-of- sate-of-theeait the-trtt partial-label learning methods: PiCO leverages contrastive learning to disambiguate the candidate labels by updating the pseudo-labels with contrastive prototype labels. 2) PRODEN 6 is also a pseudo-labeling method that iteratively
updates the latent label distribution by re-normalized classifier prediction. 3) VALEN 23 assumes the candidate labels are instance-depen destance-dependett and recovers the latent label distributions by a Bayesian parametrization model. 4) LWS [22) also works in a pscudo-labeling style, which weights the risk function by considering the trade-off between losses on candidate labels and non-candidate ones. 5) CC 16] is classifier-consister method that assumes the candidate label set is uniformly sampled. 6) MSE and EXP 24 utilize mean square error and exponential loss as the risk estimators. All the hyper-parameters are searched according to the original papers.
Implementation details. We for 1000 epochs using a standard SGD optimizer with a momentum of 0.9. The initial learning rate is set 0.01, and decays by the cosine learning rate schedule. The batch size is 256. These configurations are applied for Sol.ar arr and all baselines for fair comparisons. For Sol Lar, we devise pre-estimation training stage, where respectively to obtain coarse- coarse-grained class prior. After that, we re- re-initialize the model weights and run with this class prior. For our Sinkhorn- Knopp algorithm, parameter as À 3 and the length of the queue for acceleration as 64 times batch size. The moving- average parameter μ for class prior estimation is set as 0.1/0.05 0 in the first stage and fixed as 0.01 later. For class-wise reliable sample selection, we linearly ramp up P from 0.2 to 0.5/0.6 in the first 50 epochs and the high-confidence selection threshold r as 0.99. To improve the representation ability of SoLaar, we further involve consistency regularization 119 and Mixup 25 techniques on reliable examples; see Appendix B.1 for more details. For fair comparisons, we equip all the baselines except PiCO with these two techniques as well.
18-layer ResNet as the feature backbone. The model trained
run model for 100/20 epochs (on (IFARR0/100-LT)
fix the smoothing regularization

---

Table 2: Different shots accuracy comparisons CIFARIO-LT ( = 0.5, 100) and CIFARTON
LT (Φ 0.1, 20). The best results are marked in bold and the second-best marked in underline.
CIFARIO-LT
CIFARI00-LT
Methods
Many
Medium
Few
Many
Medium
Few
MSE
81.11
42.03
9.18
57.46
37.57
16.53
EXP
92.64
39.75
0.00
60.35
3.99
0.00
LWS
89.09
1.52
0.00
20.80
0.86
0.00
VALEN
85.30
28.78
0.00
58.74
16.25
0.07
CC
94.56
65.61
34.22
73.03
52.15
28.05
PRODEN
96,83
72.18
14.17
76.86
43.14
5,43
PICO
93.33
66.14
29.30
70.75
42.42
6.14
SoLar (ours)
96.50
76.01
49.34
74.33
54.09
30.62
Table 3: Ablation results CIFARIO-LT (Ø 0.5, 100) and CIFARIO0-LT (Ø = 0.1, 20).
CIFARIO-LT
CIFARI00-LT
Ablation
All
Many
Med.
Few
All
Many
Med.
Few
SoLar
74.16
96.50
76.01
49.34
53.03
74.33
54.09
30.62
PRODEN S+CPE
62.83
96.80
69.79
19.58
41.69
75.70
42.29
7.06
SoLar w/o Pre-Est
63.97
79.97
70.21
39.67
50.51
68.09
54.15
29.18
SoLar Cand-Est
74.34
96.37
74.47
52.14
52.78
76.15
53.93
28.22
SoLar-Orack
74.11
96.81
77.10
47.43
54.16
77.91
56.94
27.55
SoLar w/o
29.61
86.10
6.95
3.34
35.80
66.85
32.06
8.61
SoLar w GS
53.01
96.58
60.10
0.00
33.62
72.03
23.91
5.21
SoLar w/o S-SL
47.49
96.46
46.37
0.00
12.80
30.42
7.97
0.15
4.2
Main Results
SoLar achieves SOTA results. As shown in Table 1 SoLar significantly outperforms all the rivals by notable margin under various settings of imbalance ratio 7 and label ambiguity degree ø. Specifically, on CIFARIO-LE dataset with Φ 0.5, we improve upon the best baseline by 8.29%, 9.28%, and 7.47% when the imbalance ratio y is set to 50, 100, and 200 respectively. Moreover, most baselines display significant performance degradation competitive. We also verify the effectiveness of SoLar on more challenging CIFAR100-L dataset with 10 times more classes and thus stronger label ambiguity. Notably, the performance gap between SoLar and baselines remains substantial. An interesting observation is that CC obtains impressive performance the CIFAR-100 dataset, but is still inferior SoLar in most These observations clearly validate the superiority of SoLar.
increases, whereas SoLar remains
Results on different groups of labels. We show that SoLar achieves overall strong performance on both head and tail classes. To see this, we report accuracy on three groups of classes with different sample sizes. Recall from Section that the class indices are sorted based on the sample size, in descending order. We divide the classes into three groups: many (f1, 2, 3)), medium ((4, 5, 6, 7}), and few ({8, 9, 10}) shots for CIFARIO-LT and many ((1,..., 33)), medium (f34, 67)), and few (168, 100}) shots for CIFARIO0-LT. Table T shows the accuracy of different groups on both CIFAR10-LT and CIFAR 100-LT. On few-shot classes, the gaps between SoLar and the best baseline are 15.12% and 2.57% on CIFAR10-LT and CIFAR 100-LT. Moreover, SoLar retains a competitive performance many-shot labels (i.e., head classes). In contrast, most baselines exhibit degenerated performance on tail classes, especially on the CIFAR 100-LT dataset. The comparisons highlight that SoLar poses good disambiguation ability in the long-tailed PLL setup.
4.3
Ablation Studies
In this section, we present our main ablation analysis to show the effectiveness SoLarr The main results of our ablation studies are shown in Table 3 more results can be found in Appendix C.4.

---

80
90
HANNGEIE OE
50
PCD
Accurscy
Accurecyy
40
60
30
20
50
All
Kany
Med
Few
Al1
Klaryy
Med
Few
Al1
Med Groups Labels
Mary
Few
Groups Labels
Groups Labels
(a) SoLaa with logit adjustment
(b) CUB200-LT
(c) CIFARIOO-H-LIT
Figure 2: (a) Performance comparisons SoLar and SoLar with logit adjustment (SoLar-LA) on CIFARIO-LE (o = 0.5, y = 100). ( is the temperature parameter of logit adjustment. (b) Performance comparisons SoLar and baselines on the fine tine-graiedd CUB200-LT LT dataset. (c) Performance comparisons of SoLar and baselines on the CIFARI00-LE dataset with hierarchical labels (CIFARIOO-H-LT),
Effect of Sinkhorn label refinery. It is of interest to see whether the performance improvement comes from our Sinkhorn label refinery procedure. To see this, we equip PRODEN with our sample selection and class prior estimation mechanism (dubbed PRODEN S+CPE). As shown in Table 3 PRODEN w S+CPE still fails to disambiguate labels on the tail, which validates the superiority of Sinkhorn label refinery procedure.
Effect of class prior estimation. To verify the effectiveness of class prior estimation in SoLar, we compare with three variants: 1) SoLar w/o Pre-Est which removes the pre-estimation stage of SoLar and trains from uniform class prior; 2) SoLar Cand- Est initializes the class prior distribution by counting per-class candidate numbers, i.e., Tj > 1 I[j E S:); 3) SoLar-Oracle also removes pre-estimation stage but trains from the real class prior. From Table 3 we draw three salient observations. First, SoLar outperforms SoLar w/o Pre-Est, which suggests that using coarsely- estimated class prior is more beneficial than using a uniform one. Secondly, SoLar w Cand-Est obtains competitive results, showing alternative way for initializing r. Finally, contrast with CoLar-Oracle which theoretically serves as an upper bound by using the real class prior. We observe that SoLar's ' performance favorably matches even outperforms SoLar-Orack on few-shot classes. The results are encouraging since SoLar does not require access to real class prior at all.
Effect of sample selection. Next, we experiment with different sample selection strategies, including: 1) SoLar w/o which regards all examples as clean samples and does not perform selection; 2) SoLar GS which performs selection in global manner rather than class-wise; 3) SoLar w/o S-SL which only selects high-confidence examples. From Table 3 we can observe that Sol.ar w/o S worsened the performance because many pseudo-labels are unreliable at the beginning. Solaar w/o GS works relatively well on data-rich labels but suffers on the tail. This happens because the majority classes dominate the sample selection. SoLar w/o S-SL performs similarly to SoLar w/o GS on CIFAR 10-LT, but diverges on CIFAR 100-LT. In the beginning, few examples are selected as the classifier less confident, resulting in degenerated solution. Our results indicate that our sample selection procedure indispensable to SoLar in handling long-tailed class distribution.
4.4 Further Analysis
Combining SoLar with logit adjustment. While our work mainly focuses label disambiguation from the PI.L. perspective, potential improvement is straightforward by applying long-tailed learning techniques well-disambiguaaee datasets. To this end, we equip SoLar with logit adjustment [14] (dubbed SoLar-LA), a state-of-the-ar LTL technique, for improved performance. Denote the logits of our model by 9(-), i.e., the output of the network prior to the softmax layer. Given a testing data æ, SoLar-LA made prediction by arg max (x) Ç log(r,). where Ç is temperature theoretically consistent for minimizing the balanced error [14]. Note that logit adjustment cannot be applied to the original PLL data directly, as it requires the true class prior as well as a classifier trained by minimizing the long-tailed empirical risk. Fortunately, SoLar achieves both goals. Hence, we implement Sol.r- LA by using the classifier as well as the estimated class prior of SoLar. Figure 2 (a) reports the
parameter for calibration. This post-hoc corrected prediction is known
performance of LA with varying Ç values. With proper SoLar-LA improves upon SoLar
8

---

by 6.72% on CIFARIO-LT. This suggests that SoLar is compatible with existing LTL methods and opens the door exploring more advanced LTL techniques.
Results on fine-grained partial-label learning. In practice, semantically similar classes can lead to significant label ambiguity, as exemplified in Figure I (a). To test the limit of SoLar, we follow PiCO
7 and evaluate on two fine-grained datasets: 1) CUB200-LT [26] dataset with 200 bird species; 2) CIFAR 100-LT with hierarchical labels (CIFARI00-H-LT)) where the candidate labels are generated within the same superclas4 We set & = 0.05, = 5 for CUB200-LT and Φ 0.5, = 20 for CIFARI00-H-LIT In Figure 2 (b) and (c), we compare SoLar with two strongest baselines PRODEN and PiCO, where SoLar improves the best baselines by large margin (4.37% % on CUB200-LT and 3.67% on CIFARI00-H-LIY These results clearly validate the effectiveness of SoLar, even when the dataset presents severe label ambiguity.
Results on real-world long-tailed learning data. It is of interest to verify the effectiveness of Sol.ar on real-world imbalanced datasets. To this end, we con- duct experiments on the large-scale SUN397 dataset 27 containing 754 RGB images and 397 scene classes; more results on real-world and imbalanced PLL datasets are shown Appendix C.6. For the
Table 4:
Performance comparisons the SUN397 dataset with e 0.05.
Methods
All
Many
Medium
Few
PRODEN
51.14
76.98
58.17
17.61
PiCO
29.54
57.91
21.27
9.33
SoLar
61.58
77.36
62.59
44.49
SUN397 dataset, we hold out 50 samples per class for testing and the resulting training set has an imbalanced ratio of 2311/50
46. Similarly, we synthesize a partial-label dataset set with an ambiguity degree of ø = 0.05; see Appendix C.1 for more details. From Table 4 we can observe that SoLar still outperforms the baselines by a substantial margin.
Related Work
Partial-label Learning (PLL) 4 allows each example to be equipped with a candidate label set with
the ground-truth label being concealed. A plethora of PLL methods (5]
28
29
30
31 have been
developed. For example, maximum margin-based methods (32 33 maximize the margins between
candidate labels and the remaining ones. Graph-base basph- approaches
10 35 typically leverage
nearest neighbors to smooth the logical label vectors. Recently, aseudo-labeling methods [66] 6 become increasingly prevalent due to their promising performance, e.g., PRODEN 6 re- re-normalizes classifier's ' outputs and PiCO 7 updates pseudo-targe by using contrastively
22 36 7
37
learned prototypes. However, most of these works overlook the fact that class distribution can be imbalanced. Our empirical studies demonstrate that state-of-the-art PI.L methods fail to disambiguate true labels on the minority classes, which motivates us to explore the optimal transport- -aase label disambiguation technique for PLL.
Long-tailed Learning (LTL) assumes that the training data follow Zipf's law that few labels pose a large number of instances, which is observed in a wide range of applications [.]. To cope with the LTL problem, sampling-bass
methods 39 re-balance different classes at a data level, egg. up- sampling on minority classes. Re-weighting methods [20] [4, 40
41 [5] typically adjust the weights of instances or classes that calibrates the classifier predictions to be balanced. Some works also explore transfer learning (42 43 or ensemble learning [44] 45 in L.T... A recent trend in L.TL
to decompose representation and classifier learning. Nevertheless, these methods assume a full superviservi of training data, which is unrealistic in many real-world applications. It has motivated researchers to study the LTL problem in suue-vuperviiion scenarios, such as semi-supervised learning
46 47
48 21 and noisy-label learning [49 (5 (1). In this work, we explore a new formulation of LTL where only candidate labels are provided.
Optimal Transport (OT) 12 is originally proposed to depict the distance between two measures. Recently, it has drawn huge attention from different fields in machine learning, including semi- supervised learning [52, domain adaptation [33] [4]. object detection [55], and generative models [56]. The most related one to our work is OTLM [5], which also applies OT to the supervised L.TI. problem. By assuming known test class distribution, OTLM performs OT- post-hoc correction on model predictions to obtain balanced results. In contrast, our work targets the long-tailed PI.I. problem and holds no assumption class priors.
"The CIFARI00 dataset comprises 20 superclasses, with classes in each superclass.

---

6 Conclusion
In this work, we present a novel Sinkhorn label refinery framework (SoLar) for a challenging imbalanced partial-label learning problem. Key to our method, we derive optimal transport objective that refines the pseudo-labels to match the true class prior. Additionally, we tackle the non-trivial class prior estimation problem by moving-average technique, which is further guarded by a sample selection mechanism for controllable model training. Empirically, SoLar can run without knowledge of the real class distribution. Comprehensive experiments show that SoLar improves baseline algorithmss by a sigmificant margin. As we collect the real-world applications, the partial- labeled tasks often arise together with a tainttaled class distribution. That is, from the perspective of human annotators, a long-tailed label much harder to tag and that may subsequently lead to a partial-labeled candidate set. In the future, we hope to explore this line of research in broader tasks. We hope our work will draw more attention from the community towards a broader view of tackling the imbalanced partial-label learning problem.
Acknowledgments and Disclosure of Funding
This work was largely supported by the Key R&D Program of Zhejiang Province (Grant No. 2020C01024), JZ was supported by the Fundamental Research Funds for the Central Universi- ties (No. 226-2022-00028). HW, MX were partially supported by the Key Research and Development Program of Zhejiang Province of China (No. 2021C01009) and Fundamental Research Funds for the Central Universities. LF was supported by the National Natural Science Foundation of China (Grant No. 62106028), Chongqing Overseas Chinese Entrepreneurship and Innovation Support Program.
References
[1]
Ching-Hui Chen, Vishal M. Patel, and Rama Chellappa. Learning from ambiguously labeled face images. IEEE Trans. Pattern Anal. Mach. Intell., 4067):1633-1667, 2018.
[2] Forrest Briggs, Xiaoli Z. Fern, and Raviv Raich. Rank-loss support instance machines for MIML instance annotation. In SIGKDD, pages 534-542. ACM, 2012.
[3] Gabriel Panis and Andreas Lanitis.
An overview of research activities in facial age estimation using the FG-NET aging database. In ECCV, volume 8926 of Lecture Notes in Computer Science, pages 737-750. Springer, 2014.
[4] Eyke Hüllermeier and Jürgen Beringer. Learning from ambiguously labeled examples. Intell. Data Anal., (055):419439) 2006.
[5] Timothée Cour, Benjamin Sapp. and Ben Taskar. Learning from partial labels. J. Mach. Learn. Res., 12:1501-1536, 2011.
[6] Jiaqi Lv, Miao Xu, Lei Feng, Gang Niu, Xin Geng, and Masashi Sugiyama. Progressive identification of true labels for partial-label learning. In ICML, volume 119 of Proceedings of Machine Learning Research, pages 6500-6510, PMLR, 2020.
[7] Haobo Wang, Ruixuan Xiao, Yixuan Li, Lei Feng, Gang Niu, Gang Chen, and Junbo Zhao Pico: Contrastive label disambiguation for partial label learning. ICLR, 2022.
[8] Deng-Bao Wang, Li Li, and Min-Ling Zhang. Adaptive graph guided disambiguation for partial label learning. In SIGKDD, pages 83-91. ACM, 2019.
[9] Ning Xu, Jiaqi Lv, and Xin Geng. Partial label learning via label enhancement. In AAAI, pages 5557-5564. AAAI Press, 2019.
[10] Gengyu Lyu, Songhe Feng, Tao Wang, Congyan Lang, and Yidong Li. GM-PLL: graph matching based partial label learning, IEEE Trans. Knowl. Data Eng., 33(2):521-535, 2021.
[11] Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey. CoRR, abs/2110.045% 2021.
10

---

[12] Gabriel Peyré and Marco Cuturi. Computational optimal transport. Found. Trends Mach. Learn., 11(5-6):355-677, 2019.
[13] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In NeurIPS, pages 2292-2300, 2013.
[14] Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In ICLR. OpenReview.nel 2021.
[15] Hanyu Peng, Mingming Sun, and Ping Li. Optimal transport for long-tailed recognition with learnable cost matrix. In International Conference on Learning Representations, 2022.
[16] Lei Feng, Jiaqi Lv, Bo Han, Miao Xu, Gang Niu, Xin Geng, Bo An, and Masashi Sugiyama. Provably consistent partial-label learning. In NeurIPS, 2020.
[17] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments, In NeurlPS, 2020.
[18] Bo Han, Quamming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor W. Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Neur/PS, pages 8536-85468 8846, 2018.
[19] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi- supervised learning with consistency and confidence. In NeurlPS, 2020.
[20] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Aréchiga, and Tengyu Ma. Learning imbalanced datasets with abbel-distribution-awaae margin loss. In NeurlPS, pages 1565-1576, 2019.
[21] Chen Wei, Kihyuk Sohn, Clayton Mellina, Alan L.. Yuille, and Fan Yang. Crest: A class- rebalancing self-training framework for imbalanced semi-supervised learning. In IEEE Con- ference Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 10857-10866. Computer Vision Foundation / IEEE, 2021.
[22] Hongwei Wen, Jingyi Cui, Hanyuan Hang, Jiabin Liu, Yisen Wang, and Zhouchen Lin. Lever- aged weighted loss for partial label learning. ICML, volume 139 of Proceedings of Machine Learning Research, pages 1 11091-11 11 100. PMLR, 2021.
[23] Ning Xu, Congyu Qiao, Xin Geng, and Min-Ling Zhang. Instance-dependent partial label learning. Advances in Neural Information Processing Systems, 34:27119-27130, 2021.
[24] Lei Feng, Takuo Kaneko, Bo Han, Gang Niu, Bo An, and Masashi Sugiyama. Learning with multiple complementary labels. In ICML, volume 119 of Proceedings of Machine Learning Research, pages 3072-3081. PMLR, 2020.
[25] Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. ICLR. OpenReview.net, 2018
[26] Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSI Birds 200. Technical Report CNS-TR-2010-001, California Institute of Technology, 2010.
[27] Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large-scale scene recognition from abbey to zoo. In CVPR, pages 3492. IEEE Computer Society, 2010.
[28] Li-Ping Liu and Thomas G. Dietterich. A conditional multinomial mixture model for superset label learning. In NeurlPS, pages 557-565, 2012.
[29] Yan Yan and Yuhong Guo. Partial label learning with batch label correction. In AAAL, pages 6575. AAAI Press, 2020.
[30] Changchun Li, Ximing Li, Jihong Ouyang, and Yiming Wang. Detecting the fake candidate instances: Ambiguous label learning with generative adversarial networks. In CIKM, pages 903-912. ACM, 2021.
11

---

[31] Yan Yan and Yuhong Guo. Multi-level generative models for partial label learning with non- random label noise. In IJCAI. pages 3264-3270. ijcai.org, 2021.
[32] Nam Nguyen and Rich Caruana. Classification with partial labels. In SIGKDD, pages 551-559. ACM, 2008.
[33] Haobo Wang, Yuzhou Qiang, Chen Chen, Weiwei Liu, Tianlei Hu, Zhao Li, and Gang Chen. Online partial label learning. ECML PKDD, volume 12458 of Lecture Notes in Computer Science, pages 455-470. Springer, 2020.
[34] Min-Ling Zhang, Bin- Bin Zhou, and Xu- Ying Liu. Partial label learning via feature-aware disambiguation. In SIGKDD, pages 1335-1344. ACM, 2016.
[35] Gengyu Lyu, Yanan Wu, and Songhe Feng. Deep graph matching for partial label learning. In IJCAI, pages 3306-3312. ijcai.org, 2022.
[36] Ning Xu, Congyu Qiao, Xin Geng, and Min-Ling Zhang. Instance-dependent partial label learning. In NeurlPS, pages 27119-27130, 2021.
[37] Fei Zhang, Lei Feng, Bo Han, Tongliang Liu, Gang Niu, Tao Qin, and Masashi Sugiyama. Exploiting class activation value for partial-lahel learning. In International Conference Learning Representations, 2022.
[38] Nitesh V. Chawla, Kevin W. Bowyer, Lawrence O. Hall, and W. Philip Kegelmeyer. SMOTE: synthetic minority over-sampling technique. J. Artif. Intell. Res., 16:321-357, 2002.
[39] Jonathon Byrd and Zachary Chase Lipton. What is the effect of importance weighting in deep learning? In ICML, volume 97 of Proceedings of Machine Learning Research, pages 372-881. PMLR, 2019.
[40] Seulki Park, Jongin Lim, Younghan Jeon, and Jin Young Choi. Influence- loss for imbalanced visual classification. ICCV, pages 715-724. IEEE, 2021.
[41] Mingchen Li, Xuechen Zhang, Christos Thrampoulidis, Jiasi Chen, and Samet Oymak. Au- tobalance: Optimized loss functions for imbalanced data. In NeurlPS, pages 3163-3177, 2021.
[42] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X. Yu. Large-scale long-tailed recognition in open world. In CVPR, pages 2537-2546. Computer Vision Foundation/ IEEE, 2019.
[43] Jaehyung Kim, Jongheon Jeong, and Jinwoo Shin. M2m: Imbalanced classification via major- to-minor translation. In CVPR, pages 13893-13900 Computer Vision Foundation / IEEE, 2020.
[44] Yu Li, Tao Wang, Bingyi Kang, Sheng Tang, Chunfeng Wang, Jintao Li, and Jiashi Feng. Overcoming classifier imbalance for long-tail object detection with balanced group softmax. In CVPR, pages 10988-10997. Computer Vision Foundation / IEEE, 2020.
[45] Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella X. Yu. Long-tailed recognition by routing diverse distribution-awar experts. In ICLR. OpenReview.net, 2021.
[46] Bingyi Kang. Saining Xie, Mareus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In ICLR. OpenReview.net, 2020.
[47] Kaihua Tang, Jianqiang Huang, and Hanwang Zhang. Long tailed classification by keeping the good and removing the bad momentum causal effect. In NeurIPS, 2020.
[48] Jachyung Kim, Youngbum Hur, Sejun Park, Eunho Yang, Sung Ju Hwang, and Jinwoo Shin. Distribution aligning refinery of pseudo-label for imbalanced supervise learning. In NewrlPS, 2020.
12

---

[49] Kaidi Cao, Yining Chen, Junwei Lu, Nikos Aréchiga, Adrien Gaidon, and Tengyu Ma. Het- eroskedastic and imbalanced deep learning with adaptive regularization. In ICLR. OpenRe- view.net, 2021.
[50] Tong Wei, Jiang-Xin Shi, Wei-Wei Tu, and Yu-Feng Li. Robust long-tailed learning under label noise. CoRR, abs/2 11669, 2021.
[51] Tong Wei, Jiang-Xin Xin Shi, Yu-Feng Li, and Min-Ling Zhang. Prototypical classifier for robust class-imbalanced learning. In PAKDD, volume 13281 of Lecture Notes in Computer Science, pages 44-57. Springer, 2022.
[52] Kai Sheng Tai, Peter Bailis, and Gregory Valiant. Sinkhorn label allocation: Semi-supervised classification via annealed self-training. In ICML, volume 139 of Proceedings of Machine Learning Research, pages 10065- PMLR, 2021.
[53] Nicolas Courty, Rémi Flamary, Devis Tuia, and Alain Rakotomamonjy. Optimal transport for domain adaptation. IEEE Trans. Puttern Anal. Mach. Intell., 39(9):1853-186 2017.
[54] levgen Redko, Nicolas Courty, Rémi Flamary, and Devis Tuia. Optimal transport for multi- source domain adaptation under target shift. In AISTATS, volume 89 of Proceedings of Machine Learning Research, pages 849-858, PMLR, 2019.
[55] Zheng Ge, Songtao Liu, Zeming Li, Osamu Yoshie, and Jian Sun. OTA: optimal transport assignment for object detection. In CVPR, pages 303-312. Computer Vision Foundation / IEEE, 2021.
[56] Martín Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In ICML, volume 70 of Proceedings of Machine Learning Research, pages 214 223. PMLR, 2017.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [N/A]
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Appendix D.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] See supplemen- tal materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] See Section 4
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix C.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes] See Section
(b) Did you mention the license of the assets? [Yes] See Appendix C.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you're you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "wes4DAKRNz", "reviewer": "Reviewer_HWkc", "summary": "This paper proposes an optimal transport framework Solar for imbalanced partial label learning where each data point has multiple candidate labels and the latent label distribution is skewed. The key of Solar is to refine the pseudo-labels by a constraint of within-candidate allocation and class prior matching. In parallel, a moving-average-based class prior algorithm is proposed to generate class prior for the aforementioned constraints. Experiment results on CIFAR-LT and CUB-200 datasets validate the effectiveness of Solar in improving few-shot accuracy.", "questions": "Please see my main review.", "limitations": "Societal Impacts: The main negative impact is lower annotation costs may decrease the requirement for annotator employment. \n\nLimitations: The experiments need to be further improved.\n", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1.The long-tailed learning problem is important but is generally overlooked in the PLL problem. This work systematically investigated the imbalanced PLL problem by identifying the failure of existing PLL methods and a new Solar framework, which can inspire future research in this field. \n\n2.Solid theoretical analysis is given. The overall objective guarantees to recover the true classifier at a population level. The detailed derivation of the Sinkhorn-Knopp algorithm is also provided.\n\n3.The experiments are sufficient. The ablation study strongly supports the effectiveness of the label refinery procedure. Reproducibility is ensured as the source code was submitted.\n\nWeaknesses:\n\n\n1.In experiments, the PRODEN method also uses mixup and consistency training techniques for fair comparisons. What about other competitive baselines? I'd like to see how much the strong CC method could benefit from the representation training technique.\n\n2.It is not clear why the proposed sample selection mechanism helps preserve the label distribution.\n\n3.In App. B.2, a relaxed solution of Sinkhorn-Knopp algorithm is proposed. Why the relaxed problem guarantees to converge?Does Solar always run this relaxed version of Sinkhorn-Knopp?\n\n4.How is gamma in the Sinknhorn-Knopp affect the performance?\n\n5.How does the class distribution estimate for PRODEN in Figure 1?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "eB_xcSc8Qz", "reviewer": "Reviewer_nPgC", "summary": "This paper concentrates on how to perform disambiguation in partial label learning with the long-tailed label distribution considered. The authors propose an Optimal Transport-based framework called SoLar that allows to refine the disambiguated labels towards matching the marginal class prior distribution. They conduct experiments on several benchmarks to demonstrate the superiority of the proposed method.", "questions": "The authors should compare more SOTA PLL methods except for some specific pseudo-labeling-based method to verify their point that the performance will degenerate in the long-tailed setting. The authors should also conduct experiments to show that their motivation works on benchmark PLL datasets, such as lost/BirdSong/Soccer Player/Yahoo!News. \n\n", "limitations": "The motivation is not convincing as stated in the weakness. \nThe novelty and the techquie seem to be limited.\nThe experiments have some limitations. For example, experiments on many PLL benchmark datasets (such as lost/BirdSong/Soccer Player/Yahoo!News ) are not conducted, and many SOTA methods are not compared, for example, Instance-Dependent Partial Label Learning, NeurIPS 2021.\n\n", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \nThe paper is well-written and easy to follow. The proposed method seems to be technically reasonable. \n\nWeaknesses: \nThe motivation is not convincing. On the one hand, the paper claims that \"existing PLL methods have been commonly driven by the assumption that training data consists of class-balanced distribution\", which is not true as many SOTA PLL methods do not have the assumption, and they can deal with the imbalanced-class problem natually embedded in the disambiguation. On the other hand, the paper claims that \"we find that current best-performing PLL methods display degenerated performance in the long-tailed setting. This happens because the predictions of pseudo-labeling—a core component that PLL methods rely on—can be largely biased towards the head and majority classes.\" However, many SOTA methods are not pseudo-labeling based methods, and the authors only present one PLL mehtod, i. e., PRODEN, to display the degenerated performance in the long-tailed setting, which is not convincing. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "YDpySj6_4Xp", "reviewer": "Reviewer_PJgg", "summary": "This work approaches the class-imbalanced partial-label learning problem where each training example is assigned a set of candidate labels. Efforts are made to address two technical challenges. First, even the best-performing PLL method fails to identify the true labels from the candidate labels, because the pseudo-labels are largely biased toward head labels. To address this problem, an optimal transport-based objective is proposed to enforce constraints on pseudo-labels to match the class prior. Second, this work proposes a new class prior estimation algorithm along with a sample selection mechanism for practical training. Experiments validate the effectiveness of the proposed method.", "questions": "I noticed that the class prior vector r is initialized as uniform. But, why can't we count the number of candidate labels as a rough estimation of the class prior? It is typically close to the true class prior and may perform better.", "limitations": "It would be appreciated if experimental results on large-scale LTL datasets are provided.\n\nPotential negative societal impacts are discussed in the Appendix and the PLL framework may cause potential employment destruction.\n", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\n(1) Most studies about PLL assume the actual label distribution is balanced, which may not hold in practice. This work reveals the leading cause of performance degradation of the SOTA PLL methods long-tailed PLL problem and proposes a principled OT-based framework to resolve this. It is a serious attempt that makes PLL more suitable for real-world applications. \n\n(2) The iterative class prior estimation technique is novel, as estimating the true class prior without an additional validation set is intractable for any weakly-supervised learning problems. Other imbalanced weakly-supervised learning algorithms can potentially benefit from this technique.\n\n(3) Theoretical results show the consistency of the proposed OT objective.\n\n(4) Experiments are thorough and the proposed SoLar algorithm shows impressive performance. \n\n\n\nCons:\n\n(1) The main weakness is the evaluation was conducted on CIFAR datasets. With long-tailed sampling, the resulting data examples are typically sparse. Can SoLar get similar performance improvement on real-world long-tailed datasets?\n\n(2) Typos: 'Goup of Labels' in Figures. \n\n(3) Wrong figure caption in Figure 3, CIFAR-10-LT rather than CIFAR100-H-LT.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept