Long-Tailed Partial Label Learning via Dynamic Rebalancing
Authors: Feng Hong
Jiangchao Yao
Zhihan Zhou
Ya Zhang
Yanfeng Wang
Venue: ICLR 2023 poster (2023)
Keywords: long-tailed learning
partial label learning

Abstract: Real-world data usually couples the label ambiguity and heavy imbalance, challenging the algorithmic robustness of partial label learning (PLL) and long-tailed learning (LT). The straightforward combination of LT and PLL, i.e., LT-PLL, suffers from a fundamental dilemma: LT methods build upon a given class distribution that is unavailable in PLL, and the performance of PLL is severely influenced in long-tailed context. We show that even with the auxiliary of an oracle class prior, the state-of-the-art methods underperform due to an adverse fact that the constant rebalancing in LT is harsh to the label disambiguation in PLL. To overcome this challenge, we thus propose a dynamic rebalancing method, termed as RECORDS, without assuming any prior knowledge about the class distribution. Based on a parametric decomposition of the biased output, our method constructs a dynamic adjustment that is benign to the label disambiguation process and theoretically converges to the oracle class prior. Extensive experiments on three benchmark datasets demonstrate the significant gain of RECORDS compared with a range of baselines. The code is publicly available.

Published as a conference paper at ICLR 2023
LONG-TAILES PARTIAL LABEL LEARNING VIA DYNAMIC REBALANCING
Feng Hong!
Jiangchao Y..
Zhihan Zhou'
Ya Zhang
Yanfeng Wang
'Cooperative Medianet Innovation Center, Shanghai Jiao Tong University
SShangaai AI Laboratory
[feng.hong, Sunarker, zhihanzhou, ya.zhang, wangyanfeng) @sjtu.edu.cn
ABSTRACT
Real-world data usually couples the label ambiguity and heavy imbalance, chal- lenging the algorithmic robustness of partial label learning (PLL.) and long-tailed learning (LTT). The straightforward combination of L.T and PLI.,, Le., LT-PLL,, suf- fers from a fundamental dilemma: LT methods build upon a given class distribu- tion that is unavailable in PLL, and the performance of PLL is severely influenced in long-tailed context. We show that even with the auxiliary of an oracle class prior, the state-of-the -hrt methods underperform due to an adverse fact that the constant rebalancing in LT is harsh to the label disambiguation in PLL. To over- come this challenge, we thus propose a dynamic rebalancing method, termed as RECORDS, without assuming any prior knowledge about the class distribution. Based on a parametric decomposition of the biased output, our method constructs dynamic adjustment that is benign to the label disambiguation process and theo- retically converges to the oracle class prior. Extensive experiments on three bench- mark datasets demonstrate the significant gain of RECORDS compared with range of baselines. The Theco is publicly available.
INTRODUCTION
Partial label learning (PLL) origins from the real-world scenarios, where the annotation for each sample is an ambiguous set containing the groundtruth and other confusing labels. This com- mon when we gather annotations of samples from news websites with several tags (Luo & Orabonal 2010), videos with several characters of interest (Chen al. 2018). or labels from multiple annota- tors (Gong et al. 20118). The ideal assumption behind PLL is that the collected data is approximately uniformly distributed regarding classes. However, natural distribution assumption in above real- world applications should be imbalance, especially follows the long-tailed law, which should be considered if we deploy the PLL methods into online systems. This thereby poses a new challenge about the robustness of algorithms to both category imbalance and label ambiguity in PLL studies.
Existing efforts, partial label learning and long-tailed learning, independently study the partial aspect of this problem in the past decades. The standard PLL requires the label disambiguation from candidate sets along with the training of an ordinary classifier (Feng et al.) 2020). The mainstream estimating label-wise confidence to implicitly or explicitly re-weight the
solve this problem classification loss, e.g., PRODEN (Lv et al. (020), LW (Wen et al. 2021), CAVL (Fei et al. 2022) and CORR (Wu et al. 2022). which have achieved the state-of of the- performance in PLL. When comes to the long-tailed learning, the core difficulty lies on diminishing the inherent bias induced by
state-of-the-arr
the heavy class imbalance (Chawla al.] 22002] Menon al., 2013). The simple but fairly effective
method is the logit adjustment (Menon et al.] 202 Ren et al. 2020). which has been demonstrated
very powerful range of recent studies (Cui et al. 2021 Narasimhan & Menon 2021).
Nevertheless, considering a more practical tailed partial label learning (LT-PLL) problem, sev- eral dilemma remains based on the above two paradigms. One straightforward concern that the skewed long-tailed distribution exacerbates the bias to the head classes in the label disambiguation,
"Corresponding to: Jiangehao Yao (Sunarker@ (wt.ekeur@si..euu.nn and Yanfeng Wang (wangyanfeng@ssjuu.ddd...

---

Published as a conference paper at ICLR 2023
disaniiggustoo confidence 0.2
disambiiguttion corficence 0.5
dieambiquation corfidence - 0.8
rrkiing stop
MUSOEN
P40009 PROSSP AEDORDS
PRODDM
0.00
PAOSEE RECDADS
1.00
0.00
PRODFi AFCORDS
Heator
tnitum
0.66-
5.00
0.06
2.85
0.04
0.00
0.
3.03
1.00
0.03
0.0
2:00
0.23
01-
0:01
0.91
100
<lbocinadie
Cboes
Figure 1: Average classifier prediction (on the CIFAR- 100 test set) of different methods during train- ing (on LT-PLL training set CIFAR-100-LT with imbalance ratio p = 100 and ambiguity q = 0.05). "PRODEN" (Lv et al. 2020) is popular PLL method. "PRODEN + Oracle-LA" denotes PRO- DEN with the state-of-the-at logit adjustment (Menon et al. 2021 Hong et al. 2021) in LT under the oracle prior. "PRODEN RECORDS" is PRODEN with our proposed calibration. "Uniform" characterizes the expected average confidence on different classes.
easily resulting in the trivial solution that are excessively confident to the head classes. More impor- tantly, most state-of-the-art long-tailed learning methods cannot be directly used in LT-PLL, since they require the class distribution available that is agnostic in PLL due to the label ambiguities. In addition, we discover that even after applying an oracle class distribution prior in the training, existing techniques underperform in LT-PLLL and even fail in some cases.
In F T we trace the average prediction of PL. model PRODEN (L al. 2020), on uniform test set. Normally, the backbone PLL method PRODEN exhibits the biased prediction towards head classes shown by the blue curve, and ideally, we expect with the intervention of the state-of-the-ar logit adjustment in LT, the prediction for all classes will be equally confident, namely, the purple curve. However, as can be seen, PRODEN calibrated by the oracle prior actually performs worse and is prone to over-adjusting towards the tail classes as shown in the orange curve. This is because logit adjustment in LT leverages a constant class distribution prior to rebalance the training and does not consider the dynamic of label disambiguation. Specially, at the early stage where the true label is very ambiguous from the candidate set, over-adjusting the logit only leads to the strong confusion of the classifier, which negative to the overall training. Thus, we can see in Figure the average prediction on the tail classes becomes too high along with the training.
Based on the above analysis, compared with the previous constant rebalancing methods in PLL, dynamic rebalancing mechanism friendly to the training dynamic will be more preferred. To this intuition, we propose novel method, termed as REbalanCing fOR Dynamic biaS (RECORDS) for LT-PLL. Specifically, we perform a parametric decomposition of the biased model output and im- plement dynamic adjustment by maintaining prototype feature with momentum updates during training. The empirical and theoretical analysis demonstrate that our dynamic parametric class dis- tribution is asymmetrically approaching to the statistical prior but benign to the overall training. A quick glance at the performance of RECORDS is the red curve in Figure I which approximately fits the expected purple curve in the whole training progress.
The contribution can be summarized as follows,
1. We delve into a more practical but under-explored LT-PLL scenario, and identify its several challenges in this task that cannot be addressed and even lead to failure by the straightfor- ward combination of the current long-tailed learning and partial label learning.
2. We propose a novel RECORDS for LT-PLL that conducts the dynamic adjustment to rebal- ance the training without requiring any prior about the class distribution. The theoretical and empirical analysis show that the dynamic parametric class distribution is asymmetri- cally approaching to the oracle class distribution but more friendly to label disambiguation.
3. Our method is orthogonal to existing PLL methods and can be easily plugged into the current PLL methods in an end-to-end manner. Extensive experiments on three bench- mark datasets under the long-tailed setting and range of PLL methods demonstrate the effectiveness of the proposed RECORDS. Specially, we show a 32.03% improvement in classification performance compared to the best CORR on the Pascal VOC dataset.
2

---

Published as a conference paper at ICLR 2023
2 RELATED WORK
Partial Label Learning (PIL.. PLL., each training sample is associated with candidate label set containing the ground-truth. Early explorations is mainly average-based based (Hüllermeier & Beringer 2006 Zhang & Yu 2015; Cour et al. 20111b. which treat all candidate labels equally during model training. Their drawback is that the model training is easily misled by the false positive labels that co-occur with the ground truth. Some other identification based methods consider the truth label as latent variable and optimize the objective under some criterions (Jin & Ghahramani| [2002 Liu
& Dietterich, 2014; Nguyen & Caruana 2008; Yu & Zhang 2015). Recently, self-training meth- ods (Feng et al.) 2020: Wen al. 2021
2020 Fei et al. 2022) that gradually disambiguate
the candidate label sets during training have achieved better results. PiCO (Wang et al. 2202bb) in-
troduces a contrastive learning branch that improves PLL performance by enhancing representation.
Long- Long-Tailed Learning (LT). In LT, several methods have been proposed to consider the extremely skewed distribution during training (Cui et al., 2019, Zhou et al.
2022 Chawla et al.) 2002). Re- sampling (Kubat & Matwin. 1997; Wallace et al. [001]] Han et al.
2005) is one of the most widely
used paradigm by down-sampling samples of head classes or up-sampling samples of tail classes.
Re-weighting (Morik et al., 1999. Menon et al., 2013) adjusts the sample weights in the loss function during training. Transfer learning (Chu et al. 2020] Wang et al.
2021 Kim et al.] 2020) seeks
to transfer knowledge from head classes to tail classes to obtain more balanced performance.
Recently, logit adjustment techniques (Menon et al.] 2021; Ren et al.
2020 Tian et al.] 2020) that
modify the output logits of the model by an offset term log Ptrain (3) have been the state-of-the-art,
Long-Taile Tailed Partial Label Learning (LT-PLL).. A few works approximately explore the LT-PLL problem. Liu et al. (202T) implicitly alleviates data imbalance in the non-deep PLL by constraining the parameter space, which is hard to apply in deep learning context due to the complex optimization. Concurrently, SoLar (Wang et al., 2022a) improves the label disambiguation process in LT-PLL through the optimal transport technique. However, it requires an extra outer-loop to refine the label prediction via sinkhorn- sinkhorn-knopp iteration, which increases the algorithm complexity. Different from these works, this paper tries to solve the LT-PLL problem from the perspective of rebalancing.
PRELIMINARIES
3.1 PROBLEM FORMULATION
Let X' be the input space and y {1, 2, .... C} be the class space. The candidate label set space is
the powerset of ) without the empty set: S = 2 - 0. A long-tailed partial label training set can be denoted as D Derain = {(xr y/: 5 E ( where any n, is associated with candidate label set S, C y and its ground truth y4 E S, is invisible. The sample number No of cach class c E y in the descending order exhibits a long-tailed distribution. Let fxx 0) denote a deep model parameterized by 0, transforming I into an embedding vector. Then, the final output logits of
given by 2(x) = (x; 0); W) = W fx:; where g is a linear classifier with the parameter matrix W. We leverage θ 10, W] denote all parameters of the deep network. For evaluation, class-balanced test set Duni = {(., }: is used (Brodersen et al. 2010). In a nutshell, the goal of LT-PLL learn O on Dirain that minimizes the following balanced error rate (BER) on D
min BER(O) Puns (y)= P(ma))EDnn (v F argmax 2² (æ)).
(1)
3.2
PLL METHODS
Self-training PLL methods maintain class-wise confidence weights w for each sample during train- ing and formulate the loss function weighted summation of the classification loss by w. w is updated in each training round based on the model output and gradually converges to the one-hot la- bels, converting PLL to ordinary classification. Specially, PRODEN (Lv et al. 2020) assigns higher confidence weights to labels with larger output logit; LW (Wen et al. 2021 additionally consid- ers the loss of non-partial labels and assigns the corresponding confidence weights to non-partial labels as well; CAVL (Fei et al. (002) designs a identification strategy based on the idea of CAM
and assigns hard confidence weights to labels; and CORR (Wu et al. 2022) applies consistency regularization in the disambiguation strategy, Please refer to Appendix B. for more details.
3.3 LOGIT ADJUSTMENT FOR LONG-TAILED LEARNING
We briefly review logit adjustment (LA) (Menon et al. 2021 Hong et al. 202I). a powerful tech- nique for supervised long-tailed learning. First, according the Bayes rule, have the underlying
3

---

Published as a conference paper at ICLR 2023
class- probabibbbbbbbyyyeoao (x) x (PPP)))))P())) Directly minimizing the cross-entropy loss to reach the op- timal has softmax(2"(x)) = train (1/ (v/=) Yu et al. allaal.2.000 (0118) However, for a opR-mmt output logit
Zuni. it is about the balanced data, which has softmax (mum
P uni (1/2z) oc P(æly) Puns
and P. (y) き (Menon et al. 2013 Then, we have the following relations
P uuni (y)e)) x (P(xll)) Prrain (1) / Ptrain(()
Preiin (u/²=) / Prrain (y) x softmax(="(x)) - log Prrin (v)).
That is to say, if we have the logit (²) by training on standard cross-entropy loss, BER-optimal logit z 2xxx) log I (rainn (y) can be obtained by subtracting an offset term log I' train (v). Using (m) as the test logit preserves the balanced part P(<xly) of the output and removes the imbalanced part IP (train(y) in a statistical sense. LA has demonstrated its effectiveness on range of recent long-tailed learning methods (Zhu et al. 2022
Cui et al. 202T.
METHOD
In LT-PLL, the model output is biased head classes, and using biased output to update confidence weights will result in biased re-weighting, namely, a tendency to identify more frequent label in the candidate set as the ground truth. In turn, biased re- weighting on the loss function will further lead to more severe imbalance on the model. In this section, we seek to propose universal rebalancing algorithm that obtains unbiased output Zuni from biased output z in self-training PLL methods.
4.1 MOTIVATION
Previous long-tailed learning techniques assume no ambiguity supervision and focus only on the model bias caused by the training set distribution, which we term as constant rebalancing. As shown in Figure T. the constant rebalancing (Oracle-LA) fails to rebalance model in LT-PLL. This is because in I.T-I.L., not only the skewed training set distribution, but also the ambiguous super- vision can affect the training, since the inferred label changes continuously along with the label disambiguation process. Specially, at the early stage, the prediction imbalance of PRODEN is not significant. Using the more skewed training set distribution instead leads to a model that is heavily biased towards the tail classes, inducing the difficulty in label disambiguation. Therefore, dy- namic rebalancing method that considers the label disambiguation process can be intuitively more effective, e.g., RECORDS in Figure [ In the following, we will concretely present our method.
4.2 RECORDS: REBALANCING FOR DYNAMIC BIAS
The above analysis empirically suggests that the constant calibration by LA cannot match the dy- namic of label disambiguation in LT-PLL. Formally, Equation fails because the label disambigua- tion dynamically affects model optimization during training. which induces the mismatch between the ground truth Ptrain (y) and the training dynamics of the label disambiguation. To mediate the conflict, we propose a parametric decomposition on the original rebalancing paradigm:
Pume ( x; θ) x PP ly: θ) Pirain (y)e) / Ptrasn (y)
x IP Ptrain (yl z;; O) /al//Praa (y)r))
x softmax (xx) log (y(ain(()()))
where Puni(ylar;; θ) is the parametric class-probabilittes under uniform class prior. Here, we start a perspective of dynamic rebalancing: a dynamic (yrras(y)))) adapted to the training process is re- quired instead of the constant prior P'train (y) to rebalance the model in LT-PLL. Existing PLL meth- ods work to achieve better disambiguu coaniiigutiinnnnoo results or improve the quality of the representation, i.e, learning P(ally: θ) that is closer P( ((=ly). Thus, our study is orthogonal to existing PLL methods and can be combined together to improve the performance (see Section 5.2). Also, the rebalanced output boost the accuracy of label disambiguation and better P( y:
be learned.
Here, our effective and lightweight design is the estimation of the dynamic class distribution Pirain (y) by the model via the training set, i.e., IP Ptraiin (y) () E, P Parain (y) aa;; +). First,
4

---

Published as a conference paper at ICLR 2023
feature
F
Dynamic (3
Constant
9
batch
EMA across butch
shore weight
training
Prrain (1)
(Gonslomi) Perei(()) does match the traening dymomin.
9
Prrai (2/ x (
debias
Puns (y/2;
updoting
PLL
disambiguation
LT-PLL
Dynamic Rebalancing
Constant Rebalancing
Figure 2: Illustration for RECORDS. The class-wise confidence weights are updating by the "dis- ambiguation" module for each sample and used as soft/hard pseudo labels in PLL Loss. 'he main differences between the PLL baselines are the Loss" and the "disambiguation" module (see Ta- 6le The "debias" module dynamically rebalances P Prrain (3/ ; Q) to obtain IP (yns(y/z:; θ). A bal- anced P. unn' Ly)a(yla: ②) helps tail samples disambiguate labels more accurately and avoid being over- whelmed by head classes. A poomentuu-updatee prototype feature is used to estimate Ptrain(y (wl@), which is benign label disambiguation and asymmetrically approaching the oracle prior IP Ptrein (t). In comparison, constant rebalancing does not consider the dynamic of the label disambiguation.
use the Normalized Weighted Geometric Mean (NWGM) approximation (Baldi & Sadowski 2013) to put the expectation operation inside the softmax as follows:
NWGM ~
IP trrain ()) Eeee Derrain softmax(2" (2" (x:))
softmax E Derain 2² (x.)) softmax (Beg: / ( 0); W)).
(4)
Note that, NWGM approximation is widely used in dropout understanding Baldi Sadowski 2014), caption generation (Xu al., 2015), and causal inference (Wang al. 2020). The intuition behind Equation[4ii to capture more stable feature statistics by means of NWGM and combine with the latest updated linear classifier to estimate P Prainn Oy) Nevertheless, with Equation we do not reach the final form, since directly estimating PPtrain (v) requires to consider the whole dataset via an EM alternation. To improve the efficiency, we design momentum mechanism to accumulatively compute the expectation of features along with the training. Concretely, we maintain prototype feature for the entire training set, using each batch's feature expectation for momentum updates:
F mf (1 Be: B (w,;
where E [0, 1) is momentum coefficient. Then, replacing Eg,EDrran (x; in Equation by F yields the final implementation of our method:
(x) =²(x)) log IP (tteiinnyyy (y)e) =2 (x) log softmax (F; W)).
Our RECORDS is lightweight and can be easily plugged into existing PLL methods in an end-to-end manner. As illustrated in Figure [ we insert "debias" module before label disambiguation of each training iteration, converting IP Previi (y xx; θ) to Puns Pyle(y/ θ) via Equation
4.3 RELATION BETWEEN DYNAMIC REBALANCING AND CONSTANT REBALANCING
In previous sections, design parametric class distribution to calibrate training in LT-PLL. How- ever, it is not clear about the relation Ptrain (y(O) and IPtrain (1)). In this section, we theoretically point out their connection. First, let PPtrain (u,) = ((a.))>(xx,)) (1 11), Mj y denote the ora- cle prior, where (-)) denotes the indicator function. Considering a hypothesis space H where each he H - y is multiclass classifier parameterized by (h = gof in Figure we define the parametric class distribution for he as Ptrain (y, ()) E(m.y) (1)
v,).20 y/ E V. As- sume the disambiguated label for (æ, y, 5) in LT-PLL is j(z, 5) E S during training. Then, the em- pirical risk basis of the disambiguated label under Dtrain is Reraai (0) = Ei (he *
5

---

Published as a conference paper at ICLR 2023
Training Set
Test
Set
Ground Truth
PRODEN
PRODEN + Oracle-LA
PRODEN + RECORDS
Figure 4: Visualization of the toy study. The first column illustrates the groud truth distribution of four classes on the training set and the test set, marked by color. The right three columns exhibit the label identification from the candidate set for training samples in the first row and the predictions on the test set in the second row W.E. three methods.
(a S. Motivated by the theoretical study the learnability of PLL (Liu & Dietterich 2014) we propose Proposition to discuss the relation between dynamic and constant rebalancing.
Proposition Let 7f = sup(a, X)X.x..))XXXXxx.px E.,,e...... vy (z) (y) E 5) denote the ambiguity degree, du be the Natarajan dimension of the hypothesis space H, h hg be the optimal classifier on the basis of the label disanbiguation, where è arg ming RPivain (9). If the small ambiguity degree condition (Cour et al. 20lla Liu & Dietterich 2014)) satisfies, namely, E [0, 1), then for 48 > 0, the L2 distance between IP Ptrain (2) and P train () given h is bounded as
L2 (h)) (In In(1 + 1)) (du (In 2N 21n - In
1n
with probability at least 1 - 6, where N is the sample number and C is the category number.
Proposition yields an important implication that alongside the label disambiguation, the dynamic estimation can progressively approach the oracle class distribution under small ambiguity degree. We kindly refer the readers to Appendix[Djor the complete proof. To further verify this, we trace the
L2 distance between the estimation of I t'triin (y)on)(y)))))) (y) and the oracle IP Fraai (y) during training each
epoch and visualize it in Figure 3(a) It can be observed that the distance between two distributions
are gradually minimized, i.e., our parametric class distribution gradu-
0244
prior
ally converges to the statistical or- acle class prior Besides, as shown in Figure (bb the final estimated class distribution is very close the oracle class prior. In total, Fig- indiiaaess that our dynamic re- balancing is not only benign to the label disambiguation at the early stages, but also finally approach to the constant rebalancing.
Dynamic estimation
L2 distance
Probabilty
28 Epoch
Class index
(a) T. distance during training
(b) Estimated class distribution
Figure 3: (a) L2 distance between the estimated class distri- bution and the oracle class prior during training. (b) The fi- nal estimated class distribution. Experiment is conducted by CORR RECORDS on CIFAR-100-LT with imbalance ratio
EXPERIMENTS
p = 100 and ambiguity q 0.03.
5.1 TOY STUDY
We simulate a four-class LT-PLL task where each class is distributed in different regions and their samples distribute uniformly in the corresponding regions. The label set of each sample consists of true label and negative labels having 0.6 probability flipped from other classes. In the first column of 4 we visualize the training set and the test set: the training set is highly imbalanced among classes, with the number of samples from each class being 30 (red), 100 (yellow), 500 (blue) and 1000 (green), while the test set is balanced. In the right three columns of Figure 4 we show the results of identifying true labels from the candidate sets on the training set and the prediction results on the test set for different methods respectively. As can be seen, PRODEN results in the minority category (red) being completely overwhelmed by the majority categories (blue and green), and the

---

Published as a conference paper at ICLR 2023
Table I: Top-1 accuracy three benchmark datasets. Bold indicates the superior results.
CIFAR- 10-LT
CIFAR- 100-LT
Imbalannce ratio
:0
100
50
100
PASCAL
VOC
Amboguity
0.3
0.5
0.7
0.3
0.5
0.7
0.02
0.05
0.07
0.03
0.05
0.07
CORR
76.12
56.45
41.56
66.35
50.09
38.11
42.20
38,03
36.59
38.39
54.09
31,05
24.43
OOaacle LA post-hoc
80.70
58.19
43.44
72.95
51.64
41.66
46.5 16.94
40.76
39.07
41.49
36.79
33.32
34,12
Onacle-LA
17.61
15,80
5.50
3.12
RECORDS
50.28
77.66
72,90
57.46
45.56
42.51
Vs. CORR
46.45
-23,3
+25.68
+11.28
+22.81
+19.35
47,53
+5.92
+3.95
46.40
+7.60
+32.03
PRODEN
73.12
54.45
41.3
63.55
47.37
38.06
30:23
35.45
13.90
34.52
$2.04
29.40
22.59
Oracle LA post hoc
77.41
57.14
42.91
70.71
18.79
41.38
63.40
38.64
35.82
38.
35.20
31,92
31.53
Onacle-La LA
22.18
16.97
11.52
19.51
14.11
11.17
12:37
4.09
2.64
6.79
2.73
1.58
48.33
RECORDS
79.48
76.73
65.31
72.15
65.22
52.26
44.56
41.31
39.36
39.13
37.23
35.26
52,65
PRODEN
28
70.11
39.67
22.73
39.57
23.54
29.50
27.85
31.58
28.09
Oracke LA post-hoe hoc
74.34
40:27
25.3 25.34
69.50
12.34
27.35
35:47
28.80
27.27
31,03
26.96
23.20
21,06
O.
41.90
21.36
15: 15.28
25.75
20.35
14:24
50:37
14.43
4.79
30.30
5.08
2.70
51,53
RECORDS
76,02
57.39
40.28
71.18
57.23
41.24
36.56
31.67
29.30
33,00
28.85
25.64
53.09
VS. L.W
+5.91
-19.72
+17.55
46.40
+37.66
+17.70
+1.02
+2,17
+1.55
+1.42
+0.76
40.99
+33.668
CAVE
36.73
40.27
18.52
56.28
38,97
17.26
29.63
17:31
8.34
28.29
25.39
8.20
17.25
Onacl-LA poss-hoe
39.76
18.3
37.28
14.86
5.76
28.
26.27
22:27
Onacle-LA
22.16
14.97
11.50
18.29
14.25
10:67
17.31
4.36
2,85
7:24
2.55
50.78
RECORDS
67.27
61.23
40.71
64.35
58.27
37.38
42.25
36,53
29.13
36.93
31,49
24.98
53,07
vs. CAVL
+10.54
-20.96
+22.19
+10.07
+19.30
+20.1
+12.62
+19.22
+14.27
+8.64
+5.10
+10.78
+35.82
Table 2: Fine-grained analysis on CIFAR-100-LT with 0 100 and q E {0.03, 0.05, 0.07).
Many/Mediun/Few corresponds to three partitions on the long-tailed data.
0.03
g =
0
== 0.07
Method
Many
Medium
Few
Overall
Many
Medium
Fow
Overall
Many
Medium
Few
Overall
CORR
68.43
37.40
4.50
38.39
67.51
29.60
0.33
34.09
68.86
19,80
0.07
31.05
+ OOrcle-LA post-hoc
70.37
41.89
7.33
41.49
70.46
33,40
1.47
36.79
69.77
24,86
0.67
33.32
+ Oracle-LA
11.03
11.37
0.34
4.46
5,47
0.00
0.71
5.77
1.98
+ RECORDS
66.37
42.54
13.77
42.25
68.49
40,20
8.50
40.59
69.97
36.71
4.37
38.65
CORR
-2.05
+5.14
+9.27
+3.86
+0.98
+10.60
+8.17
+6.50
+1.11
+16.91
+4.30
+7.60
yellow category being dominated mostly. After calibration with the oracle class prior, PRODEN Oracle-LA instead predicts all data in the test set as the minority class (red). This suggests that constant rebalancing is prone to being over-adjusted towards tail classes coupling with the label dis- ambiguation dynamic. In comparison, when using RECORDS as a dynamic rebalancing mechanism in LT-PLL, we achieve the desired results on both the training set and the test set.
5.2 BENCHMARK RESULTS
Long-tailed partial label datasets. We evaluate RECORDS on three datasets: CIFAR-10-LT LLiu
et al. 2019) CIFAR- 100-LT (Liu et al. 2019) and PASCAL VOC. For CIFAR-10-LT and CIFAR-
TT. we build the long-tailed version of of CIFAR-10/100 with imbalanced ratio p E (50, 100).
Following Lv et al. (2020); Wen et al. 2202I, we adopt the uniform setting in CIFAR-IO-LT and
CIFAR-100-LT. f.e., Pl E STT + y) & generate candidate label set of each sample. PASCAL VOC is a real-world LT-PLL dataset constructed from PASCAL VOC 2007 (Everingham et al.)). instances and all objects appearing the same original image are regarded as the labels of a candidate set and empirically we can observe the significant class imbalance. Note that, here
Specifically, we crop objects in images
the first conduct experiments based on deep models on real-world datasets, while previous PLL real-world datasets are tabular and only suitable for linear models or shallow MLPs (Fei al. 2022). Please refer to Appendix E. for more details.
Baselines. We consider the state-of-the-arr PLL algorithms including PRODEN (Lv et al. 2020). LW (Wen et al. (02TT. CAVL (Fei et al. 2022). and CORR (Wu et al. 2022) and their combination with the state- ffate-of-the-ars the rebalancing method logit adjustment (LA) (Menon et al. 2021 Hong et al., 202T]. We use two rebalancing variants: (1) Oracle-LA: rebalance PLL model during training with the oracle class prior by LA; (2) Oracle-LA post-hoc: rebalance pre-trained PL.L model with the oracle prior using LAA post-hoc way. Note that, for our methods, we directly apply RECORDS into PLL baselines but without requiring the oracle class prior. For comparisons with concurrent LT-PLL work SoLar (Wang et al. 2022a), please refer to Appendix
Implementation details. We use 1 1yylr ResNet as the backbone. The standard data augmenta- tions are applied as in Cubuk et al. 2020). The mini-batch size is set to 256 and all the methods trained using SGD with momentum of 0.9 and weight decay of 0.001 as the optimizer. The hyper-parameter m in Equation [5] is set to 0.9 constantly. The initial learning rate is set to 0.01. We train the model for 800 epochs with the cosine learning rate scheduling.
7

---

Published as a conference paper at ICLR 2023
1.0
1.2
68
2.8
666
6
6,2
6.0
Figure 5: The per-class accuracy on LIFAR-IO-LT with p 100 and 9 E {0.3, 0.5, 0.73.
Table 3: Fine-grained analysis on CIFAR- with p == 100 and q (0.03, 0.05, 0.07).
0.03
q 0.05
g 0.07
Method
Many
Medium
Fow
Overall
Many
Medium
Few
Overall
Many
Medium
Few
Overall
CORR
66.71
34.57
3.13
36.39
70.23
21.57
0.13
32.17
65,37
16.11
0.13
38.56
+ Dracle-LA post-hoc
69.80
40.43
4.57
39:95
71.20
28.77
1.30
35,38
68.71
20.17
1.10
31.44
+ Oracle-LA
3,26
9.97
13.10
8.56
0.86
5.34
12:27
5.85
0.00
0.25
16.87
5.92
RECORDS
65.60
41.26
11.77
66.40
38.71
8:57
39.36
62.09
37.40
7,47
37.06
vs. CORR
-1.11
+6.69
+8.44
+4.54
-3.83
+17.14
+8.44
+7.19
-3.28
+21.29
+7.34
+8.50
Table 4: Comparision with more baselines for imbalanced learning.
CIFAR-IO-LT
CIFAR- 100-LT
Imbalance ratio
50
100
50
100
ambiguity
0.3
0.5
0.7
0.3
0.5
0.7
0.03
0.05
0.07
0.03
0.05
0.07
CORR
76.12
56.45
41.56
66.38
50.09
38.11
42.29
38.03
36.59
38.39
34.09
31.05
+ Oracle-BCL
43.76
25.24
17.34
33,23
18.34
15.72
27.23
10.34
8.34
14.34
5.82
4.10
+ Oracle-PaCO
42.45
26.34
16.23
34.21
18.23
14.23
27.37
11.23
7.23
15.82
5.62
4.72
+ +LDD
75.23
58.02
42.14
67.59
51.38
39.28
42.91
39.05
36.98
39.20
34.87
32.15
SEN
75.98
57.13
41.77
66.91
50.23
38.71
43.14
38.62
62
37.03
39.31
33,91
31.67
RECORDS
82.57
80.28
67.24
77.66
72.90
57.46
48.06
45.56
42.51
42.25
40.59
38.65
Overall performance. In Table T we summarize the Top-1 accuracy on three benchmark LT-PLLL datasets. Our method clearly exhibits superior performance on all datasets with different experimen- tal settings under CORR, PRODEN, LW and CAVL. Specially, compared to the best PLL baseline CORR, RECORDS significantly improve the accuracy by 6. 16.45%-25.66 5.8% on CIFAR-1O-LT. 3.86% 7.60% on CIFAR- 100-LT, and 32.03% on PASCAL VOC. When Oracle-LA is applied into PLL baselines, it induces severe performance degradation CAL VOC. In comparison, Oracle-LA post-hoc can better alleviates the class imbalance on CIFAR. However, both of them cannot address the adverse effect of constant rehalancing label disam- biguation. In comparison, our RECORDS that solves this problem through dynamic rebalancing during training, achieves the consistent and the best improvements among all methods.
CIFAR but improves significantly on PAS-
Fine-grained analysis. In Figure 5 we visualize the per-class accuracy on CIFAR-I0-LT with p 100 under the best PLL baseline CORR. As expected, dominant classes generally exhibits higher accuracy in CORR. Oracle-LA post-hoc can improve CORR performance on medium classes, but accuracy remains poor on tail classes, especially when label ambiguity is high. Oracle-LA per- forms very poorly on head and medium classes due to over-adjustment towards tail classes. Our RECORDS systematically improves performance over CORR, particularly on rare classes. In Ta- accuracies of different methods on CIFAR-100-LE with P = 100. Our RECORDS shows the significant and consistent gains on Medium and Few classes when combined with CORR. As demonstrated in many long-tail tailed learning literatures (Kang al.) 2020 Menon et al.] 2021), there might be a head-to-tail accuracy tradeoff in LT-PLL, and our method achieve the best overall accuracy in this tradeoff. More results are summarized in Appendix EE4
ble 2 we show the Many-Medium-FFan
5.3 FURTHER ANALYSIS
Non-uniform candidates generation. Real-world annotation ambiguity often occurs among mantically close labels. To evaluate the performance of RECORDS in practical scenarios, we conduct experiments a more challenging dataset, namely CIFAR-100-LT-NE 100-LT-NU. To build CIFAR- 100-LT-NU, we generate candidates from the ground truth of CIFAR- 100-LT using a Non- con-uriform setting. Specifically, labels in the same superclass of the ground truth have a higher probability be selected into the candidate set, i.e., P( E SIT + y, D(T) * D(y)) = g. TP E STT F y, D(T) D(y))= = 8q, where D(y) denotes the superclass which y belongs. In Table 3 we show the fine- with p = 100 and q E {0.03, 0.05, 0.07}. Combined with
grained analysis on
MMan//Meiumm/Few corresponds classes with > 100, [20, 100), and <220 images (Kngg et al. 2020
8

---

Published as a conference paper at ICLR 2023
Table 5: Comparison with other dynamic strategies on CIFAR-IO-LT and CIFAR-I00-LT.
CIFAR- -0-LT
CIFAR- 100-LT
Imbalance ratio
50
100
50
100
Ambiguity y
0.3
0.5
0.7
0.3
0.5
0.7
0.03
0.05
0.07
0.03
0.05
0.07
CORR
76.12
56.45
41.36
66.38
50.09
38.11
42.29
38.03
36.59
38.39
34.09
31.05
TTmp Oracle-LA
81.37
43.62
18.10
76.09
25.88
16.11
47.44
43.46
29.75
41.78
39.19
33.69
Epoch RECORDS
75.43
70.27
59.50
69.38
63.12
47.85
46.54
43.07
38,28
41.58
37.14
34.38
RECORDS
82.57
80.28
67.24
77.66
72.90
57.46
48.06
45.56
42.51
42.25
40.59
38.65
0.55
3.2
6.4
use
0.5
-.radeeA AA
RROCORDS
3.55
0.50
6:25
3.3
0.0
0:15
0.00
0:15
Oueluation Serting
Salting
Linear Probing
(b) Ablation m
Figure 6: (a) Top-1 accuracy (left) and top-5 accuracy (right) under different shots of linear probing for different methods pretrained on CIFAR-100-LT (p = 100, 0.05). CORR Oracle-LA post-
hoc is same to PRODEN in terms of features and thus is not plotted. (b) Performance of CORR RECORDS with varying
CIFAR- 10AA- (p 100, q = 0.05).
our RECORDS, the best baseline CORR achieves significant gains, demonstrating the robustness of RECORDS in different scenarios. More results are summarized in Appendix E.3
Linear probing performance. Following the literatures of self-supervised learning (Chen et al., 2020 He et al.) 2020), we conduct linear probing on CIFAR-100- with p = 100 and y = 0.05 to quantitatively evaluate the representation quantity of different methods. To eliminate the class imbal- ance effect, the linear classifier is trained on a balanced dataset. From Figure [6(a]] our RECORDS consistently outperforms baselines in this evaluation under different shots. It indicates that our RECORDS that considers the label disambiguation dynamic can help extract better representation.
Other dynamic strategies. To verify the effectiveness of RECORD, we setup two other straight- forward dynamic strategies: (1) Temp Oracle-LA: temperature scaling Ptrasn (y) from 0 to during training; (2) Epoch RECORDS:
the prediction of the latest epoch to estimate the class distri-
bution and rebalance. In Table 5 we can find that although simper solutions might be effective, RECORDS outperforms them significantly, confirming the effectiveness of our design.
More baselines for imbalanced learning. We additionally compare our RECORDS with two contrastive-based SOTA long-tailed learning baselines, BCL (Zhu et al. 2022) and PaCO (Cui al. 2021), and two regularization methods for mitigating imbalance in PLL. LDD (Liu et al. 202 1 and SFN Liu al. 202I). Similar Oracle-LA, we use the oracle class distributions for BCL and PaCO, denoted as Oracle-BCL and Oracle-PaCO, In Table 4 RECORDS still significantly outper- forms these methods, showing the effectiveness of dynamic rebalancing.
Effect of the momentum coefticient m. In Figure 6(b) we explore the effect of the momentum update factor m on the performance of RECORDS. As can be seen, the best result achieved at m = 0.9 and the performance decreases when m takes smaller values or a larger value. Specially, when m = 0, still maintains a competitive result, showing the robustness of RECORDS. At the other extreme value, i.e,, m = 1.0, CORR RECORDS actually degenerates to CORR.
6 CONCLUSION
In this paper, we focus on more practical LT-PLL scenario and identify several critical challenges in this task based on previous independent research paradigms LT and PLI. To avoid the drawback of their combination, we propose a novel method for LT-PLL, RECORDS, without requiring the prior of the oracle class distribution. Both empirical and theoretical analysis show that our proposed parametric class distribution is asymmetrically approach the static oracle class prior during training and is more friendly to label disambiguation. Our method orthogonal to existing PLL methods and can be easily plugged into current PLL methods in end-to-ent manner. Extensive experiments demonstrate the effectiveness of our proposed RECORDS. In the future, the extension of RECORDS to other weakly supervised learning scenarios can be explored in more general scope.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
This paper does not raise any ethics concerns. This study does not involve any human subjects, practices to data releases, potentially harmful insights, methodologies and applications, poten- tial conflicts of interest and sponsorship, diserimination/iiasffaiinn concerns, privacy and security issues, legal compliance, and research integrity issues.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of experimental results, our code is available at htttts:: //tps://gtt thub. SJTUS
hom/Medi. i aBrai n-iiinnSJJJ---- S-TS/TTCOOOOS-LLPPPPPP We provide experimental setups and implemen-
lation details in Section 5 and Appendix E
The proof of Proposition i given in Appendix
ACKNOWLEDGGENNS
This work is supported by STCSM (No. 22511106101, No. 18DZ2270700, No. 21DZI 100100), 111 plan (No. BPO719010), and State Key Laboratory of UHD Video and Audio Production and Presentation.
REFERENCES
Pierre Baldi and Peter J. Sadowski. Understanding dropout. Christopher C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger (eds.), NewrlPS, pp. 2814 2822, 2013.
Pierre Baldi and Peter J. Sadowski. The dropout learning algorithm. Artif. Intell., 210:78- 2014.
Kay Henning Brodersen, Cheng Soon Ong, Klaas Enno Stephan, and Joachim M. Buhmann. The balanced accuracy and its posterior distribution. In ICPR, pp. 3121-3124. IEEE Computer Soci- ety, 2010.
Nitesh Chawla, Kevin W. Bowyer, Lawrence Hall, and W. Philip Kegelmeyer. SMOTE: synthetic minority over-sampling technique. J. Artif. Intell. Res., 16:321-357, 2002.
Ching-Hui Chen, Vishal M. Patel, and Rama Chellappa. Learning from ambiguously labeled face images. IEEE Trans. Pattern Anal. Mach. Intell., 4077):1533-6667, 2018.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In ICML, volume 119 of Proceedings of Machine Learning Research, pp. 1597-1607. PMLR, 2020.
Peng Chu, Xiao Bian, Shaopeng Liu, and Haibin Ling. Feature space augmentation for long-tailed data. In ECCV, volume 12374 of Lecture Notes in Computer Science, pp. 694-710. Springer, 2020.
Timothee Cour, Ben Sapp, and Ben Taskar. Learning from partial labels. The Journal of Machine Learning Research, 12:1501-1536, 201la.
Timothée Cour, Benjamin Sapp. and Ben Taskar. Learning from partial labels. J. Mach. Learn. Res., 12:1501-1536, 2b.
Ekin Dogus Cubuk, Barret Zoph, Jonathon Shlens, and Quoe Le. Randaugment: Practical automated data augmentation with reduced search space. In NeurlPS, 2020.
Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. Parametric contrastive learning. In
ICCV, pp. 695-704. IEFE, 2021.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge J. Belongie. Class-balanced loss based on effective number of samples. CVPR, pp. 0268-9277 Computer Vision Foundation IEEE, 2019.
10

---

Published as a conference paper at ICLR 2023
M. Everingham. L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The
PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results. http://www.pascal.
Zhang Fei, Feng Lei, Han Bo, Liu Tongliang, Niu Gang, Qin Tao, and Sugiyama Masashi. Exploit- ing class activation value for partial-label learning. In ICLR. OpenReview.net, 2022.
Lei Feng, Jiaqi Lv, Bo Han, Miao Xu, Gang Niu, Xin Geng, Bo An, and Masashi Sugiyama. Prov- ably consistent partial- label learning. In NeurIPS, 2020.
Chen Gong, Tongliang Liu, Yuanyan Tang, Jian Yang, Jie Yang. and Dacheng Tao. A regularization approach for massd superset label learning. IEEE Trans. Cybern., 48(3):667-973 2018.
Hui Han, Wenyuan Wang, and Binghuan Mao. Borderline-mmote: A new over-sampling method in imbalanced data sets learning. In ICIC, volume 3644 of Lecture Notes in Computer Science, pp. 878-887. Springer, 2005.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for un- supervised visual representation learning. In CVPR, pp. 9726 9765.997..22 Computer Vision Foundation / IEEE, 2020.
Youngkyu Hong, Seungju Han, Kwanghee Choi, Seokjun Seo, Beomsu Kim, and Buru Chang- Disentangling label distribution for long-tailed visual recognition. In CVPR, pp. 6626-6636. Computer Vision Foundation IEEE, 2021.
Eyke Hüllermeeir and Jürgen Beringer. Learning from ambiguously labeled examples. Intell. Data Anal., 10(5):419 2006.
Rong Jin and Zoubin Ghahramani. Learning with multiple labels. In NeurlPS, pp. 897-904. MIT Press, 2002.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yan- Kalantidis. Decoupling representation and classifier for long-tailed recognition. In ICLR. OpenReview.net, 2020.
Jachyung Kim, Jongheon Jeong, and Jinwoo Shin. M2m: Imbalanced classification via major-to- minor translation. In CVPR, pp. 13893-13902. Computer Vision Foundation IEEE, 2020.
Miroslav Kubat and Stan Matwin. Addressing the curse of imbalanced training sets: One-sided selection. ICML., pp. 179-186. Morgan Kaufmann, 1997.
Li-Ping Liu and Thomas G. Dietterich. A conditional multinomial mixture model for superset label learning. In NeurlPS, pp- 557-565, 2012.
Li-Ping Liu and Thomas G. Dietterich. Learnability of the superset label learning problem. In ICML, volume 32 of JMLR Workshop and Conference Proceedings, pp. 1629-1637. JMLR.org, 2014.
Wenpeng Liu, Li Wang, Jie Chen, Yu Zhou, Ruirui Zheng, and Jianjun He. A partial label met- ric learning algorithm for class imbalanced data. Vineeth Balasubramanian and Ivor Tsang C.ds.).AMM... volume 157 of Proceedings Machine Learning Research, pp. 1413-1428. PMLR, 2021.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang. Boqing Gong, and Stella X. Yu. Large- scale long-tailet recognition in an open world. CVPR, pp. 2537-2546. Computer Vision Foun- dation / IEFF, 2019.
Jie Luo and Francesco Orabona. Leaming from candidate labeling sets. In NewrlPS, pp. 1504-1512. Curran Associates, Inc., 2010.
Jiaqi Lv, Miao Xu, Lei Feng, Gang Niu, Xin Geng, and Masashi Sugiyama. Progressive identifi- cation of true labels for partial-label learning. ICML, volume 119 of Proceedings of Machine Learning Research, pp. 6500-6510. PMLR, 2020.

---

Published as a conference paper at ICLR 2023
Aditya Krishna Menon, Harikrishna Narasimhan, Shivani Agarwal, and Sanjay Chawla. On the statistical consistency of algorithms for binary classification under class imbalance. In JCML, volume 28 of JMLR Workshop and Conference Proceedings, pp. 603 903-611. JMLR.org. 2013.
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In ICLR. OpenReview.net, 2021.
Katharina Morik, Peter Brockhausen, and Thorsten Joachims. Combining statistical learning with a knowledge-baie approach- A case study in intensive care monitoring. ICML, pp- 268-277. Morgan Kaufmann, 1999.
Harikrishna Narasimhan and Aditya Krishna Menon. Training over-parameterized models with non- decomposable objectives. In NeurlPS, pp. 18165-18181, 2021.
B. K. Natarajan, learning sets and functions. Mach. Learn., 4:67-97, 1989.
Nam Nguyen and Rich Caruana. Classification with partial labels. In SIGKDD, pp. 551-559. ACM, 2008.
Jiawei Ren, Cunjun Yu, Shunan Sheng, Xiao Ma, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Bal- anced mota-softmax for long-tailed visual recognition. In NeurlPS, 2020.
Junjiao Tia,, Yen-Cheng Liu, Nathaniel Glaser, Yen-Chang Hsu, and Zsolt Kira. Posterior calibration for imbalanced datasets. In NeurlPS, 2020.
Laurens der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9866):557-26055 2008.
Byron C. Wallace, Kevin Small, Carla E. Brodley, and Thomas A. Trikalinos. Class imbalance, redux. In ICDM, pp. 754-763. IEEE Computer Society, 2011.
Haobo Wang, Mingxuan Xia, Yixuan Li, Yuren Mao, Lei Feng, Gang Chen, and Junbo Zhao. Solar: Sinkhorn label refinery for imbalanced partial-label learning. In NeurIPS, 2022a.
Haobo Wang, Ruixuan Xiao, Yixuan Li, Lei Feng, Gang Niu, Gang Chen, and Junbo Zhao. Pico: Contrastive label disambiguation for partial label learning. In ICLR. OpenReview.net, 2022b.
Jianfeng Wang, Thomas Lukasiewicz, Xiaolin Hu, Jianfei Cai, and Zhenghua Xu. RSG: simple but effective module for learning imbalanced datasets. In CVPR, pp. 3784-3793. Computer Vision Foundation/ / IEEE, 2021.
Tan Wang, Jianqiang Huang, Hanwang Zhang, and Qianru Sun. Visual commonsense R-CNN. In CVPR, pp. 10757-10767. Computer Vision Foundation/ IEEE, 2020.
Hongwei Wen, Jingyi Cui, Hanyuan Hang, Jiabin Liu, Yisen Wang. and Zhouchen Lin. Lever- aged weighted loss for partial label learning. In ICML, volume 139 of Proceedings of Machine Learning Research, pp. 11091-11100. PMLR, 2021.
Dong-Dong Wu, Deng-Bao Wang, and Min-Ling Zhang. Revisiting consistency regularization for deep partial label learning. In ICML, volume 162 of Proceedings of Machine Learning Research, pp. 1-22255, PMLR, 2022.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In Francis R. Bach and David Blei (eds.), ICML, volume 37 of JMLR Workshop and Conference Proceedings, pp. 2048-200 2057. JMLR.org, 2015.
Fei Yu and Min-Ling Zhang. Maximum margin partial label learning. In ACML, volume 45 JMLR Workshop and Conference Proceedings, pp. 96-161 JMLR. 2015.
Xiyu Yu, Tongliang Liu, Mingming Gong, and Dacheng Tao. Learning with biased complementary labels. In ECCV, volume 1 11005 of Lecture Notes in Computer Science, pp. 69-85. Springer, 2018
12

---

Published as a conference paper at ICLR 2023
Zinan Zeng, Shijie Xiao, Kui Jia, Tsung-Hai Haa Chan, Shenghua Gao, Dong Xu, and Yi Ma. Learning by associating ambiguously labeled images. In CVPR, pp. 708 -15. IEEE Computer Society, 2013.
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empiri- cal risk minimization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC. Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
Min-Ling Zhang and Fei Yu. Solving the partial label learning problem: An instance-based ap- proach. IJCAI, pp. 4048-4054 AAAI Press, 2015.
Zhihan Zhou, Jiangchao Yao, Yan-Feng Wang, Bo Han, and Ya Zhang. Contrastive learning with boosted memorization, In ICML, volume 162 of Proceedings of Machine Learning Research, pp- 27367-27377. PMLR, 2022.
Jianggang Zhu, Zheng Wang, Jingjing Chen, Yi-Ping Phoebe Chen, and Yu-Gang Jiang. Balanced contrastive learning for long-tailed visual recognition. CVPR, pp. 6898 6907. IEEE, 2022.
13

---

Published as a conference paper at ICLR 2023
Appendix
CONTENTS
Appendix A Pracatical Background of Long-Tailed Partial Label Learning.
Appendix
Preliminaries (Section 3).
Appendix
Pseudo-Code of RECORDS (Section 4.2)
Appendix
Proof of Proposition] | (Section 4.3)
Appendix
Detailed Supplement for Experiments (Section
A LONG-TAILED PARTIAL LABEL LEARNING: BACKGROUND
The remarkable success of deep learning is built on a large amount of labeled data. Data an- notation in real-world scenarios often suffers from annotation ambiguity. To address annotation ambiguity, partial label learning allows multiple candidate labels to he annotated for each train- ing instance, which can be widely used in web mining (Luo & Orabonal (0TO), automatic image annotations ZZng et al. 20133: Chen et al. 2018), ecoinformatics (Liu & Dietterich 2012), and crowdsourcing (Gong al.) 2018)).
For example, a movie clip may contain several characters talking to each other, with some of them appearing screenshot. Although we can obtain scripts and dialogues that indicate the names of the characters, we cannot directly confirm the real name of each face in the screenshot (see FigureF(a), similar scenario arises for recognizing faces from news images, where we can obtain the names of the people from the news descriptions but cannot establish one-to-one correspondence with the face images (see Fi) Partial label learning problem also appears crowdsoureing, where each instance may be given multiple labels by different annotators. However, some labels may be incorrect or biased due to differences in expertise or cultural background of different annotators, so it is necessary to find the most appropriate label for each instance from candidate labels (see Figure c))).
Meanwhile, in the real world, the data naturally exhibit long-tailed distribution. Corresponding to the three examples Figure 7 the main characters in movies tend to take up most of the time, while the supporting roles appear much less frequently; in sports news, superstars get most of the exposure, while many role players only get few appearances; and the number of different species in nature also shows clear long-tailed distribution. However, such common long-tailed distribution of real data has been ignored by most existing PLL methods. Further, the invisibility of ground truth in PLL makes it difficult to even manually balance the dataset. Therefore, we focus on the more difficult but common LT-PLL scenario where label ambiguity and category imbalance co-occur.
When facing LT-PLL, directly combining the paradigms of partial label learning and long-tailed learning poses some dilemmas. One immediate problem is that the skewed long-tailed distribution exacerbates the bias toward head classes label disambiguation and tends lead trivial solutions with overconfidence in head classes. More importantly, most state-of-the-art long-tailed learning methods cannot be directly used LT-PLL because they require available class distributions, which agnostic in PLL due to label ambiguity. Moreover, find that existing techniques perform poorly in LT-PLL and even fail in some cases even after applying an oracle class distribution prior in the training.
B PRELIMINARIES
B. SELF-TRAINING PLL METHODS
The self-trane self-training PLL methods (Feng et al. 2020 Wen et al. 2021 Lv et al. 2020 Fei et al. 2022) remove annotation ambiguities from model outputs and gradually identify true labels during training, achieving saate-o--the-ar results. These methods maintain class-wing wise confidence weights w for each sample during training and formulate the loss function weighted summation of the classification
14

---

Published as a conference paper at ICLR 2023
LeBron Jamaes Stephen Cuiry
Thompson
23
Korat Russian Blue
Momning, Flash. Thompson: Good morning, Parker.
Parker:
The History of LeBron James and Stephen
Annotator I: Kornt
Curry's Rivalrous Friendship
Annotator Russian Blue
(a)
(b)
Figure 7: Some practical applications of PLL. (a) A screenshot of the movie "The Amazing Spider- Man" and the corresponding dialogue script. (b) An image from NBA news with the caption "The History of LeBron James and Stephen Curry's Rivalrous Friendship". (c) an image of "Russian Blue", which is very similar to "Korat".
loss by w. They update the confidence weights from the output, then labels to learn classifiers from the data. Model training and weight updates are performed iteratively each round. In the late training period, converges stable one-hot labels, and PLL is converted to an ordinary classification problem that learns from the disambiguated labels. The differences among these self seff-training methods are the form of loss function and how the weights are updated.
soft/hard pseudo-
In Tabel 6 we summarize four state-of-the-arr self-training methods, PRODEN (Lv et al.) 2020).
LW (Wen et al., 2021), CAVL (Fei et al. 2022), and CORR (Wu et al.) 2022). Specially, PRODEN assigns higher confidence weights to labels with larger output logit; LW additionally considers the loss of non-partial labels and assigns the corresponding confidence weights to non-partial labels as
well; CAVL designs identification strategy based on the idea of CAM and assigns hard confidence
weights to labels; and CORR applies consistency regularization the disambiguation strategy.
Table The loss functions and confidence weights updating strategics for self self-training PLL meth- ods.
Methods Per-Sample Loss
Updating w
exp(e if S; wi. Exss (a.))
PRODEN
Ejey wi ((,((((a))))
wL, 0, Ef
Exes. (x)))+
wi Eies, expla (m,))) if S;
(x,))
LW
Eees, (m.(((=>(m.)))
wi Exgo (mr)) if
(xp()"(m.)))
w., if = argmax (m))) 1| (xx)
CAVL
Eney ((
w., 0, else
CORR
>Dku(e(x:)]lo:)+
w.,
Eayylloree exp(st(rr)
if E S
Ejes, (x=2((x)))
wi. 0, ifj
B.2 LOGIT ADIUSTMENT FOR LONG-TAILED LEARNING
Here give review the successful logit adjustment (LA) Menon et al. 2021 Hong et al. 202T) in long-tailed learning. LA uses the Bayes Rule to remove the adverse effect of class imbal- ance by performing offset term the output logit. In the long-tailed setting with highly skewed
15

---

Published as a conference paper at ICLR 2023
training category distribution (rain) (y), trivial classifier that classifies all instances majority la- bels can achieve high training accuracy. To cope with it, a class-balance test set Duni = {(x,, yt) used (Brodersen al., 2010). We donate P unnitl (v/ x P(xll)) the underlying class probability under uniform distribution (y) = c. The of long-tailed learning is to learn a model on D Dtrain that minimizes the following balanced error rate (BER) on Duni:
min BER(O) (y)
P(..)) (y + argmax 2d (æ)). y'Ey
For an optimal model trained by minimizing the traditional softmax cross cross-entropy loss, the output logit z satisfies softmax(2" (="((x P trrann (y) (y]=) PP(ally) Perain (v) Yu et al. 2018). Recall that the goal is to minimize BER. For a BER-optimal model, also known as Bayes-optimal model, its output logit Pnni satisfies softmax(=" (=". (æ)) (y)ar) (Menon et al. 2013). LA attempts
convert the output obtained from traditional cross-entropy training into Bayes-optimal output as follows:
softmax(=*(x)) Praain (y)==) P(mly) Ptrain (y) (u)ilylar) P( o soflmax(=*(x)) log Prrain
That is to say, if we have the logit (xx by training standard cross-entropy loss, Bayes-optimal
logit 22 (x) = 2 (x) log (rtain) can be obtained by subtracting an offset term log I' train (v)
Using 2mm as the test logit preserves the balanced part IP(ll)) of the output and removes the
imbalanced part I' Prrain (y) in a statistical sense. A number of recent studies (Zhu et al. 2022 Cui
et al. 2021 have demonstrated the powerful effects of logit adjustment in long-tailed learning.
C PSEUDO-CODE OF RECORDS
We summarize the complete procedure of our RECORDS in Algorithm I
Algorithm 1: Our proposed RECORDS.
Input:
Training dataset Dirain deep model f. classifier g. self-training PLL algorithm A. number of epochs
Output: Parameter / for f, parameter W for g.
Initialize uniform weights w;;
Initialize F with 0:
for / 1,,,,,..., do
7..,,To.
Shuffle training set Deroin into for & = 1, ......... B do
Compute output = for mini-batch DR: Update F according to Equation 5
Compute loss LA according to algorithm A:
Compute debiased output Zuni according to Equation 6
Update w using Zuni according to algorithm A:
10
Update 9 and W by minimizing L
12
end
13 end
D PROOF OF PROPOSITION
Lemma D.1. Let L (h) for h E H be L2 distance between Ptrain (y) and P Erain(y)1) where is parameterized by e. We have
(h) (x.) # y))-
16

---

Published as a conference paper at ICLR 2023
Proof.
(h)
(x. (1 vU) 1(h(x) -,,))}
$
E ((. 11 yy) ((((x) 9/51
For any instance (ær, y). if h(x) = y. ie., h correctly classifies æ, then for Vyg E (1,2,...,,C) we have [y Vj) (h(x)) y3)l 0. Otherwise if h((x)) + y, we have
1,
V, = y or y, = h(x)
(y = yj) - (M(()) = ,3)l
(8)
0, clse
Thus, we can bound (h) by
L2(h) S 2E (e.))) 9))
(9)
Lemma D.2. Define R as of all Dtroin for which there exists c-L2 Distance h with zero empirical risk: R = (Dtrain E (X, y, S) 3h, L2(h) 2 e, Rirrain (h)= Define M to be set of all pairs (Dtrains Dirain) for which there exists a h with zero empirical risk on Dirain that makes E-L2 Distance and make at least classification errors on Dirain i.e. M (Deroin D Perain) (K
= 0}. Introduce another
training set D Derain
(X, (X,Y,S)
3h, L2(h) Z
R. Riiii (h) = 0, E 1(h(x') +v) 2 1. If >
181n2 then we have
P(Dirain R) 21P((D D Dirain) M).
Proof. This lemma is used in many learnability proofs. Apply the chain rule of probability:
Pi(Drrain: Dirain) E M) P(((Draiin Dirain) M (Dtrain R) -P(PPrrriin R)
N
eN
P(hh, L2(h) 2 Dimen (h) = 0,
(h((x') + 4/) 2
Dtrain E R) -PPPraDr E R)
4
==1
1(h(x') +ッ)) eN (2(h) 2 Dirain (h) =0= 0) PP R)
2P1
(10)
Given (h) 2 6, we can get (( 1 (h(z)) * v) 2 & (Lemma D.I
Using the Chernoff
bound, when N
16ln can get
N
1((x')) +1) 2 eN
P
4
=1 PP
(h()') H5) e.V
)
(11)
1
> P
1()(2e)) / (1 5
Bl (X))))))
1(h(2') +v)1
21 (y
Ne 1
21 c
2
17

---

Published as a conference paper at ICLR 2023
Thus bound P(Dtrain R)) by
P(Dtrain E R) < 2P(((Drrinns Draai) E M)
(12)
Lemma D.3. (Liu Dietterich 2014) If the hypothesis space H has Natarajannimmension dæ and < 1, then
P((Dtroin: Derain) E M) < (2N) a2dx 1 1+ 7 2x
Proof. Expand Drrain and D brain as Drrain (x' ,NNNN E (X, S) We can get
(a N E (K,2, Drrain
P((D Derain) E M)
E ((((( Dirain) E M| ,yy x. y
(13)
Where the expectation E is taken with respect to (x: y x N) and the probability P comes from (S S given (x>,, y x." v/²).
Let H( )N") be the set of hypothesis making different classifications for Dirain and D D Diraino we can get
trrtiii Pirain) _M .y
,,,,""" xx', .y²N)
P(RPerain (h) 0, Σ ((h(r') # 2 cN
(14)
N y/
NEH (o))))
(((Drrain D'rain) E M|xx ,y" ,x', ,2¹N
,
cN
(15)
P(R (h) =0 1(h('') * サ/
x N
Randomly match the instances in Drrain and Di train into n pairs. Following Liu & Dietterich 2014 4) define group G of swaps. A swap
G swaps the instances in pairs indexed by Ja (D Dirain). Let a1 0g be the number for
{1. .1......,11) G 2 (Drrain D Dirrin) which h classifies Dirain incorrectly, and D Prrain incorrectly. Let b1, b2 be the number for which classifies both incorrectly, and only one incorrectly.
N
Ptraat (h) 0, 1(((2') +v) 2 eN
Σ
|x yy x. )
N
eN N
1(((2') 33
P(h(x.) = 5(x; (1) S:) NV N. N ///
(16)
1=1
cN
=1 (a2
Il P(h(z.)) = 5o., St)l N NN NNNN
We then give the bound for 11 P(h(x;;) = 5( 2. V x'N )g
N
N
(h(()))) = ((2,. N,,,""""" nNN) < II (((())) E S V. N NN s 1 (17)
18

---

Published as a conference paper at ICLR 2023
Combining Equation 131 17
can get
P((Drrains Dirain) E M)
=E ((Prrain D Plrasn) E M |x Vy 2' )/A))
=E 2N
Σ
P(o Dirainl E M N y xN 1.N))
oEG
eN
Σ
P(R (h) =0 I(h(x')) # v)
x
y
2 (
2N
BEG (o
(18)
cN
<B (2N) 2dd
1 z
2N
4
oEG
<E (2N) dx 2dd
1(b1 + b2 2
e\V
Cai
4
=b
=B (2N) 2 (b1 + b2
cN )n
+ 17 2
where the third line uses the inequality [< x S (2N) du C2dn Natarajan 1989. When
0, b2 <N the right side reaches the maximum of (2N)du C2dy
For ease of reading, here we restate Proposition D in the main text. Proposition D.I. Let 1 s) w., uuuu Ps((
E ES denote the ambiguity degree, du be the Natarajan dimension of the hypothesis space H, h h6 be the optimal classifier on the basis of the label disambiguation where θ arg arg ming RPPraan (0). If the small ambiguity degree condition (Cour et al. 200 201la Liu & Dietterich 2014)) satisfies, namely, E [0, 1), then for V8 > 0, the L2 distance between F Firain(y) and P trrain ()){) for h is bounded as
L2(h)- < (In - In(1 + N))N (4n (nn 2N 2ln In 1n2)
with probability at least where is the sample number and C is the category number.
Proof. Recall the definition of R:
R (Dtrain E (X, Eh, L2(h) 2 e, Perain (h) = =0}
(19)
Here we need to prove the sufficient condition for P(Dergin P(Dtrain E R) by P(Dtrain E R) < 2TP((Drrain) Dtrain) E M). Lemma
R) < 8. Lemma D.2
bounds
D.3
bounds
P((Dirain; Di Drrain) E M) by (2N)au C2du ( + Takinng it into Lemma
D.2 get
P(Perain R) 2P((Drrain, D'rasni E M) S 2(2N))H 2d0
1 + y x
(20)
2
Then we can prove the sufficient condition for P(Drrann E R) S 8:
eN
+ "
P(Drrain R) 28 x In + (In In 2dx In C
In
In
4
2
(21)
4
[ln -n + n)) (d (In 2N + 2 2mn - ln In 2) 2
That is, when c (la2 [In2-mn1+77jJ (dH (In + 2 C) In 8 + In 2), we have L2 hh) < c with
probability at least 1 1-61 6.
19

---

Published as a conference paper at ICLR 2023
Table 7: Characteristics of PASCAL VOC
#Classes
#Train
#Test
Imbalance ratio
Avg #Candidate Labels
20
11706
4000
118.8
2.46
Table 8: Sample size for each class of PASCAL VOC
Class Number
person 5702
chair
car
bottle
sofa
bicycle
horse 333
pottedplant 319
diningtable 304
motorbike 300
1285
884
452
364
354
Class
tvmonitor 277
dog 251
bus
Cow
hoat
hird
cat
train 87
sheep 66
aeroplanc 48
Number
193
134
125
117
111
E ADDITIONAL EXPERIMENTAL SETUP AND RESULTS
E.1 DATASETS
CIFAR-10-LT and CIFAR-100-LT. The original versions of CIFAR-10 and CIFAR- 100 contain 50,000 training images and 10,000 validation images in 32 x 32 size with 10 and 100 categories, re- spectively. The 100 categories in CIFAR-100 form 20 superclasses, each with 5 classes. Following Liu et al. (2019), we conduct the long-tailed version of CIFAR-10 and CIFAR- by an exponen- tial decay across sample sizes of different classes, namely CIFAR-1O-LT and CIFAR- imbalance rate denotes the ratio of the sample sizes of the most frequent and least frequent classes, ie., Nc. where the sample number N₂ of each class c E y is in the descending order.
The
Ny
Candidate label sets generation. Following Lv et al. 2020); Wen et al. 202I). we adopt the uniform setting in CIFAR-1O-LT and CIFAR- i.e., Ply E Sly 7 y) q to generate candidate label set of each sample. That is, all the C - negative labels have the same probability of to be selected into the candidate set along with the ground truth. Besides, for CIFAR-100-LT, challenging non-umiform setting where other labels in the same superclass of the ground truth have higher probability be selected into the candidate set, i.e., IP E S + y, D(T) + D(y)) q. PT E S + y, Dī)) = Dgy) 8q. where D denotes the superclass to which y belongs. The non-uniform setting is more practical and challenging for the LT-PLL algorithms because the semantics of the categories in the same superclass are closer. We refer to CIFAR-100-LT with candidates generated by the non-uniform setting as CIFAR-IO0-LT--N
also
more
Dataset partitioning. To demonstrate the performance of the algorithm categories with different frequencies, we partition the dataset according to the sample size. Following Kang et al. (2020), We split the dataset into three partitions: Many-shot (classes with more than 100 images), Medium-shot (classes with 20-100 images), and Few-shot (classes with less than 20 images).
PASCAL VOC. We conduct the real-world LT-PLL dataset PASCAL VOC from PASCAL VOC 2007 Everingham et al.). Specifically, we crop objects in images as instances and all objects ap- pearing in the same original image regarded the labels of candidate set. Note that, here we are the first to conduct experiments based on deep models on real-world datasets, while previ- PLL real-world datasets are tabular and only suitable for linear models or shallow MLPs (Fei) (et al., 2022). Empirically we can observe the significant class imbalance as shown in Table g We manually balanced the test for fairness of the evaluation. In Table 7 we summarize the basic characteristics of PASCAL VOC.
F.2 ADDITIONAL IMPLEMENTATIO DETAILS
Toy study. (Figure 4 We simulated four-class LT-PLL task with each class distributed in a dif- ferent region and their samples uniformly distributed in the corresponding region. The four uniform distributions are U( -1,, [0, 0)). U([0, -1. [1, 0)). U([-1,0], [0, 1)). and U([0, 0j, [1, I)), re- spectively. The candidate label set for each sample consists of the ground truth and negative labels with 0.6 probability of being flipped from the other classes e.g., P STy # y) 0.6. The sample size of each class in the training set is 30 (red), 100 (yellow). 500 (blue) and 1000 (green), respec- tively. The test set is balanced, with 100 samples per class. We use two-layer ML.P with 10 hidden
20

---

Published as a conference paper at ICLR 2023
Table Top-1 accuracy on CIFAR-I00-LT-NE with imbalance ratio p E (50, 100} and ambiguity q E {0.03, 0.05, 0.07]. Bold indicates superior results.
Imbalance ratio
50
100
ambiguity q
0.05
CORR
40.73
35.76
32.21
36.39
32.17
28.56
Oracle-LA post-hoe
45.57
39.07
34.75
39.95
38
31.44
Oracle-LA
16.52
10.25
8.61
8.56
5.85
5.92
RECORDS
44.93
vs. CORR
+6.25
+9.17
+11.16
+4,54
+7.19
+8.50
PRODEN
Oracle-LA post-hoe
42,69
35.86 .6
30.42
36.83
32.72
29.07
Oracle-LA
7.28
5.23
RECORDS
43.83
41.39
39.88
37.99
35.76
34.19
PRODEN
+5.83
+8.36
+11.59
+5.02
+5.78
+7.74
LW
33.76
28.73
26, 11
30.10
25. 56
22.58
Oracle-LA post-hoe
33.53
28.39
25.83
29.31
24.56
21.84
Oracle-LA
28.22
12.13
11.11
14.36
7.73
6.85
RECORDS
34,93
30.78
27.79
31,51
26,65
25,066
vs.
+1.17
+2.05
+1.68
+1.41
+1.09
+2.48
CAVL
27.30
20.85
13.18
27.26
20.53
7.34
Oracle-LA post-hoe
20.56
14.96
13.56
Oracle LA
11.66
7.11
6.47
6,86
5.16
4.80
RECORDS
31,02
vs. CAVL
+12.41
+10.17
+3.24
+7.51
+5.83
+13.11
Table 10: Fine-grained analysis on CIFAR-100-LT with p = 100 and q E f0.03, 0.05, 0.07).
Many/Medium/Feww corresponds to three partitions the long-tailed data. Bold indicates supe- rior results.
0.03
0.05
==
0,07
Method
Many
Medium
Few
Overal
Many
Medium
Few
Overall
Many
Medium
Few
Overall
CORR
68.43
37.40
4,50
38.39
67.51
29.60
0.33
34.09
68.86
19.80
0.07
31.05
+ Oracle-LA post-hoc
70.37
41.89
7.33
41.49
70:46
33.40
1.47
36.79
69.77
24.86
0.67
33.32
11.37
3.32
1.98
+ RECORDS
66.37
42,54
13.77
42.25
68.49
40,20
8,50
40.59
69.97
36.71
4,37
38.65
CORR
-2.66
+5.14
+9.27
+3.86
+0.98
+10.60
+8.17
+6.50
+1,11
+16.91
+4.30
+7.60
PRODEN
64.23
32.49
2.23
34.52
62.60
28.66
0.33
32.04
65.49
18.51
0.00
29.40
+ Oracle-LA post-hoc
66.63
38.03
5.93
38.40
66.86
31.80
2:23
35,20
66.60
24.17
0.50
31.92
Oracle-LA
6.79
2.73
0.03
+ RECORDS
64.91
38.40
9.90
39.13
65.94
36.54
4.53
37.23
66.14
33,03
1.83
35.26
PRODEN
+0.68
+5.91
+7.67
+4.661
+3.34
+7.88
+4.20
+5.119
+0.65
+14.52
+1.83
+5.86
+ Oracle-LAA LA post-hoe
59.89
25.57
3.73
31.03
58.03
16.26
3.20
26.96
55.14
8.49
3.10
23.20
Oracle-LA
50.37
29.60
7,70
30.30
4,29
4,63
6.53
5.08
0.00
1.54
7.20
2.70
+ RECORDS
61.66
29.29
33.00
63.17
28.85
58.51
11.74
3:25
25.64
V3. LW
-2.71
+3.43
+4,42
+1.42
-0.68
-0.14
+3.45
+0.76
-3.80
+3.63
+3.25
+0.99
CAVL
8.20
+ Oracle- post-hoc hoc
51.51
27.26
2.57
28.34
57.85
16.51
0.80
26.27
7.60
5.54
3.97
5.80
4 Omalle-LA
5.29
9.00
7.47
7.24
(100)
2,03
6.13
2.55
0.00
0.29
6.43
2,03
RECORDS
vs. CAVI.
43.12
+14.63
+8.10
+8.64
-4.37
+18.46
+5.07
46.10
+27.95
+18.84
+1.30
+16.78
neurons as the model for the toy study. The batch size is set to 512. The toy models are trained using SGD with momentum of 0.9. We train the models for 50 epochs with initial learning rate 2.0.
Linear Probing. (Figure 6(a)) We train linear classifier on a frozen pre-trained backbone and mea- sure the quality of the representation through the test accuracy. To eliminate the effect of the long- tailed distribution in the fine-tuning phase, the classifier is trained on balanced dataset. Specifically, the performance of the classifier is reported on the basis of pre-trained representations for different amounts of data, including full-shot, 100-shot, and 50-shot. In the fine-tuning phase, we train the linear classifier for 500 epochs with SGD of momentum 0.7 and weight decay 0.0005. The batch size is set to 1000. The learning rate decays exponentially from 10 to 10 The loss function is set to the ordinary cross-entropy loss.
E.3 ADDITIONAL RESULTS FOR NON-UNIFORM CANDIDATES GENERATION.
In Table 9 we show the complete experimental results on CIFAR-IO0-LT--N Under a more prac- tical and challenging candidate generation setting, our RECORDS achieves consistent and signifi-
21

---

Published as a conference paper at ICLR 2023
Table 11: Fine-grained analysis on CIFAR-I00-LT-NO with p 100 and g E {0.03, 0.05, 0.07). Mann/MMeium/Few corresponds to three partitions on the long-tailed data. Bold indicates superior results.
== 0.03
x= 0.05
q = 0.07
Method
Many
Medium
Few
Overall
Many
Medium
Few
Overall
Many
Medium
Few
Overall
CORR
66.71
34.57
3.13
36.39
70.23
21.57
0.13
32.17
65.37
16.11
0.13
28.56
+ Oracle-LA LA post hoc
69.80
40.43
4.57
39.95
71.20
28.77
1.30
35.38
68.71
20.17
1.10
31.44
Oracle-LAA
3.26
13,10
8.56
0.86
5.34
12:27
5.85
0.00
16.87
5.92
+ RECORDS
65.60
41.26
11.77
40.93
66.40
38.71
8.57
39.36
62.09
37.40
7.47
37.06
CORR
-1.11
+6.69
+8.44
+4.54
-3.83
+17.14
+8.44
+7.19
-3.28
+21.29
+7.34
+8.50
PRODEN
+ Oracle-LA post-hoe
66.09
36.37
3.23
36.83
65.94
26.83
0.83
32.72
04.17
18.89
0.00
29.07
Oracle-LA
1.91
8.11
10.47
6.65
0.20
2.57
14.20
5.23
0.00
2,43
13.70
4,96
RECORDS
36.89
35.76
57.29
6.97
34.19
Vs. PRODEN
-1.03
+8.03
+8.57
+5.02
-6.34
+16.39 6.80
+7.07
+5.78
-5.60
+21.74
+6.97
+7,74
LWV
63.69
62.94
0.00
56
0.00
22.58
+ Oracle-LA LA post-hoc
58.60
25.51
3,07
29.31
57.89
9.37
3.40
24.56
53.40
6.37
3.06
21.84
+ Oracle-L.A
14.03
17.37
11,23
14.36
1.34
6.74
16.33
7.73
0.00
6.57
5.17
6.85
RECORDS
4.33
26.65
3.26
25.06
Vs. L.W
-2.28
+2.08
+4.45
+1.41
-1 1.0
+0.51
14.33
+1.09
+0.71
+3.31
+3.26
+2.48
CAVL
54.94
21.29
1.93
27.26
42.17
14.74
2.03
20.53
18.37
2,06
0.63
7.34
+ Oracle-LA post-hoc
25.03
25.06
9.97
20.52
15.37
10.00
13.56
8.49
5.83
4.93
+ Oracle-LA
2.20
6.63
12.57
6.86
0.00
2.77
13.97
5.16
0.00
1,20
14.60
4.80
RECORDS
56.89
35.09
8.60
34.77
43.26
25.34
7,83
26.36
32.40
19.23
7.87
20.45
CAVL
+1.95
+13.80
+6.67
+7.51
+1.09
+10.60
+5.80
+5.83
+14.03
+17.17
47.24
+13.11
60
60
40
E-NN Almentioons
E-SNE menssinn
20
-20
-20
-40-
40
-60
-60
-80
-50
-50
50
-50
50
1:5E dimension
dimension
dimension
(a) PRODEN
(b) PRODEN Oracle-LA
(c) PRODEN RECORDS
Figure 8: T-SNE of image representation on CIFAR- 10-LT with p = 100 and = 0.5. Note that,
PRODEN + Oracle-LA post-hoe does not alter feature and shares the same figure PRODEN.
cant improvement over the PLL baselines for varying imbalance ratio E {50, 100} and ambiguity q E (0.03, 0.05, 0.07). It demonstrates the stable enhancement by our RECORDS under different potential candidate generation settings.
E.4 FINE-GRAINED ANALYSIS ON MORE PLL BASELINES
In Table 10 and we show the additional results of fine-grained analysis on CIFAR-IO0-LT (p = 100. E (0.03, .0.05, 0.07)) and CIFAR-I00-LT-NU (p = 100, E f0.03, .0.05, 0.07)). Our RECORDS shows consistent gains on Medium and Few classes when conbined with all four PLL baselines. As result of head-to-tail tradeoff, the accuracies of Many classes improve less or decrease slightly, which can be observed in many long-tailed learning literatures (Kang al. 2020 Menon al.) 202T). Our RECORDS consistently improves the overall accuracies in this tradeoff.
E.5 FEATURE VISUALIZATION
In Figure g We visualize the image representation produced by the feature encoder f using t- SNE (van der Maaten & Hinton 2008) on CIFAR-10-LT with p 100 and q 0.5. We can observe that there is some overlap in the representations of different classes of PRODEN due to the heavy class imbalance and the label ambiguity. PRODEN Oracle-LA basically fails to learn discriminative representations due to the tough constant rebalancing. In contrast, our dynamic re- balancing method RECORDS produces more distinguishable representations.
22

---

Published as a conference paper at ICLR 2023
Table 12: Top-1 accuracy on three benchmark datasets. Bold indicates the superior results.
CIFAR-IO-ET
CIFAR-IOE-ITT
Imbalance rato
:0
100
50
100
PASCAL
VOC
Ambiguit
0.3
0.5
0.7
0.3
0.5
0.7
0.00
0.05
0.07
0.03
0.05
0.07
CORR
76.12
56.45
41.56
66.38
50.09
38.11
38.03
36.59
38.39
34.09
31,05
24.43
OmacleLLA posi-hoe
80.70
54.64
41.66
39.07
33.30
Onalle-IAA.
36.27
17.61
12:77
29.97
15.80
11.75
5.50
3.32
1.98
Oracke BCL
43.76
25:24
17.34
33.23
18.34
15.72
27.23
10.34
8.34
14.34
5.82
4.10
53.23
Oracle PACO
42.45
26.34
16:23
34.21
18.25
14.25
27:37
11.23
7.25
15.82
5.02
4.72
53.19
LDD
75.23
58.02
42.14
67.59
$1.38
29,
42,91
39.05
36.95
29.
34.87
32.15
25.45
SFN
75:98
57.13
41.77
66.91
110.25
38.71
43.14
38.62
37.03
39.3
53.01
31,67
25.63
RECORDS
82,57
$0.28
67.24
77.66
$7.46
45.56
42.51
42.25
38.65
56,46
+25.68
PRODEN
73.12
54.45
47.57
38.06
33.90
34.52
52.04
Onaile-1 post-hoe
77,41
57.14
42.91
70.71
48.79
41.38
43,40
38.64
35.82
35.20
31,92
31.53
Onall--LA
27.18
16.97
11.52
19.51
14.11
11.17
12:37
4.09
2.64
6.79
2.73
1.98
48.53 53
Oracle BCL
31,66
21.92
14.97
26,14
18.51
13.85
16.12
8.A5
7,65
10.28
7.58
6.01
49.71
Onacle-PaCO
31,48
21.09
14.58
23,87
19,00
15.76
16 34
8.05
7,06
11.
7.52
7,40
50.04
LDD
72.89
54.89
41.
47.73
38,06
39.68
35.36
34,03
24.49
32.64
29.83
22.79
73:00
54.73
41.
34.19
35.13
$2.60
22.94
79.48
76.73
72.15
65.22
52.26
41.31
39.36
39.13
37.23
Vs. PRODEN
+6.36
+23.04
+8.60
+17.85
+14.3
+5.35
+5.86
+5.36
61
+5.19
+5.86
+30.26
LW
70.11
32.67
22.73
64.78
39,57
23.54
35.54
29.50
27.86
31.38
28.09
24.65
19,41
Oracle LA post hoc
74,34
40.27
25.34
69.60
12.34
27.35
35.47
28.50
27.27
31.03
26.96
25.20
21.06
Onacle-LA
41.90
21. 36
15. 28
25.75
20.33
14.24
50.37
14.43
4.79
30.30
5.08
2,70
51.53
OnacleBCL
45.89
38
18.46
28.77
23.95
15551
33.84
17.75
8.04
33.55
8.83
6.18
$2.06
OnaclePPCOO
7.91
$1.54
LDD
37.87
23.44
29.05
27.20
31.
21.07
64.00
25.87
27.14
31.19
18.68
RECORUS
76.02
57.39
40.28
71.18
$7,23
41.24
36.56
31.67
29.39
33.00
28.85
25.64
$3.09
Vs. I.W
+5.91
-19.72
+17:55
+6.40
+17.66
+17.70
+1.02
+2.17
+1.55
+1 142
+0.76
+0.99
+33..68
CAVE
36.73
40.27
18.52
54.28
38.97
17.28
29.63
17:31
8.34
28,
25.29
8.20
17:25
Oracle LA post hoc
55:23
38.76
18.34
51.37
37.28
14.58
29.65
14,86
5.76
28.34
26.27
5.80
22:27
Onalle-AA. -LA
22.16
14.97
14.25
10.67
4.36
2.83
7.34
50.78
Onacle-BCT.
17.82
12.71
15,45
5.79
OmiClePPCOO
24.60
13.51
21.55
16.04
12.02
17:97
6.33
6.80
9.64
4.12
LDD
56.35
40.28
18.40
54.33
39.15
16.49
29.06
17.26
8.11
28. 19
25.14
7.50
16.65
SFN
56.74
29
18.3 33
$4.87
39.80
16.89
29.11
17.46
9.33
28. 34
25.39
9.00
18.31
RECORDS
67.27
61.23
40.71
64.35
$8.27
37.38
42.25
36.53
29.13
36.93
31.49
24.95
53.07
vs. CAVI.
+10.54
+20.96
+22.19 19
+10.07
+39.
+20.1
+12.62
419.22
+14.27
+8.64
46.10
+16.78
433.82
Table 13: Results with the addition of Mixup on benchmark datasets.
The best and second best
results are respectively in bold and underline.
CIFAR-IO-LT
CIFAR-100-11
PASCAL.
Imbulance ratio P
100
50
100
VOC
Ambiguity y
0.3
0.5
0.7
0.3
0.5
0.7
0.03
0.05
0.07
0.03
0.05
0.07
LW
71.31
39.69
25.
66.08
41.2
26.59
37.34
31.59
30.6
33.78
30.69
27.91
20.73
CAVL PRODEN
57.77
41.68
21.32
54.28
39.47
20.67
23.01
21.29
14.23
31.65
28.77
13.65
18.33
74.22
56.25
45.11
65.72
49.25
42.26
43.02
40.15
39.1
38.66
36.23
35.52
24.47
CORR
77.53
58.35
45.60
69.12
50.96
42.87
45.95
42.83
41.72
41.79
38.47
36.96
26.72
Sol.ar
83.88
76.55
54.61
75.38
70.63
53.15
47.93
46.85
45.1
42.51
41.71
39.15
56.49
LW WW-RECORDS
77.22
43.98
72.2
38.16
33.6
31,53
35.33
55.19
CAVL-RECORDS
68.93
63.59
44.61
66.65
59.62
41.88
46.13
41.93
34,01
41.33
35.79
31.29
55.03
PRODEN-RECORDS
81.23
78.82
46.01
42.98
54.78
CORR-RECORDS
84.25
82.5
71.24
79.79
74.07
62.25
52.08
50.58
47.91
46.57
45.22
44.73
58.45
E.6 COMPARISON WITH MORE BASELINES FOR IMBALANCED LEARNING
We additionally compare our RECORDS with two contrastive- based state-of-the-an long-tailed learning baselines, BCL Zhu et al.) 2022) and PaCO (Cui et al. (021). and two regularization meth- ods for mitigating imbalance in PLL, LDD (Luu et al. 2021) and SFN (Liu et al. 2021). l. Similar to Oracle-LA, we use the otherwise invisible oracle category distributions for BCL and PaCO, denoted as Oracle-BCL and Oracle-PaCO. We use pseudo-labels to construct positive and negative sample pairs for the contrastive learning part of BCL and PaCO, because the true labels are not visible dur- ing training. In Table 12] we show the top-1 accuracy on LIFAR-10-L7 and CIFAR- 100-LT under the best PL.. baseline CORR. Both BCL, and PaCO use an LA-like form for the supervised part. Despite the performance gains from contrastive learning (compared to Oracle-LA). they also experi- ence performance degradation similar to that of LA. LDD and SFN implicitly mitigate the imbalance in PLL by regularization constraints on the parameter space, and achieve slight improvement over the PLL baseline CORR. However, they do not explicitly address the lem of LT-PLL. Our method achieves significant improvement over all basclines. We should also note that, in these baselines except LDD and SFN, they use the oracle class distribution, which our method also does not use. Therefore, our method is more general than them.
distribution prob-
23

---

Published as a conference paper at ICLR 2023
E.7 COMPARISON WITH THE CONCURRENT LT-PLL WORK SOLAR
SoLar (Wang et al., 2022a) is a concurrent LT-PLL work published in NeulPS 2022. At the time of our submission, did not have access this work and make comparisons with it. It improves the label disambiguation process in LT-PLL through the optimal transport technique. However, requires extra outer-loop to refine the label prediction via sinkhorn- -nopp iteration, which increases the computational complexity. Different from SoLar, this paper tries to solve the LT-PLL lightweight and effective manner.
problem from the perspective of rebalancing
Note that compared to other PLL baselines and previous experiments in this paper, SoLar addition- ally applies Mixup (Zhang et al.) 2018) in the experiments. To align the experimental setup, show the results with the addition of Mixup in Table All other settings remain the same as previ- ous sections. Compared to SoLar, which designed for the LT-PLL problem, our method still offers consistent improvements. The code implementation of comparisons with Solar can be found here.
24


Reviews:
[{"review_id": "lFa26ogVoeN", "reviewer": "Reviewer_sVPa", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper describes a new method for rebalancing of class probabilities for longtailed learning under the setting of partial label learning. No others followed the idea of logit adjustment, which factories P(y|x) as P(x|y) P(y). The second component is estimated on a per class basis in the training algorithm.\n\nThe authors show their experimental results on CIFAR-100 LT and CIFAR 10 LT data sets. ", "summary_of_the_review": "Overall the paper is well written and dwells on topical problems. However the current problem is poorly motivated and seems almost like a combination of two other problems. The proposed method is logical and simple. However the exposition of the proposed methods in my opinion needs to be more lucid. Finally while lot of experiments have been performed showing improvement over existing PLL techniques, the baselines for long tail learning have not been incorporated.", "strength_and_weaknesses": "Strength:\n- Proposed dynamic re-balancing technique is interesting.\n- The problem formulation is theoretically interesting.\n\nWeaknesses:\n- The experimental results are not convincing. The main issue with this paper is that it does not compare its performance with the state of the art methods for longtailed learning. For example the following paper reports in excess of 52% accuracy on CIFAR-100 LT. \n\nJianggang Zhu, Zheng Wang, Jingjing Chen, Yi-Ping Phoebe Chen, Yu-Gang Jiang. Balanced Contrastive Learning for Long-Tailed Visual Recognition. CVPR 2022\n\n- Another major problem is about the practical motivation for the proposed problem. Authors almost assume that LT+PLL is an important problem setting to learning with without giving any motivating examples. This is also reflected in the experimental set up on benchmark data sets where for the PLL setting the authors simply use uniform probability to generate partial label sets from CIFAR 100 LT. This might generate more partial labels for the tail classes.\n\n\n- Finally, the paper is very difficult to read. This is because, the exposition in the paper assumes that the author is familiar with the literature on long-tailed learning and partial label learning. However many of these settings are not mainstream and the approaches should be introduced properly.\n\nFor example, prior to equation 2 the authors do not define p_uni(y) but refer to a previous paper. It is hard for a reader not familiar with the literature to understand the definition from the previous paper.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jkbkKaifuM", "reviewer": "Reviewer_dMZw", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors consider a problem where data exhibit both a long-tail (LT), where there is major class imbalance that we do not expect at test time, and PLL (partially label learning), where there exists a true label but at training time we only observe a superset that contains the true label. The authors show that standard LT adjustment interact poorly with PLL, even if the oracle class distribution is known. The authors propose a solution that applies a dynamic LT adjustment that is more consistent with the model's current predictions at each step of training. This dynamic adjustment can be used with any PLL algorithm. The authors show that this scheme will lead to an LT adjustment that converges to the oracle, but argue that the dynamic nature of the adjustment allows the model to train better. This last claim is substantiated with several empirical experiments that consider how different LT rebalancing schemes compare with the same PLL algorithm, as well as some studies exploring and probing representations, and a study examining the effect of the momentum tuning parameter that controls the dynamics of the class adjustment.", "summary_of_the_review": "This is a nice, thorough treatment of this problem, with a method that appears to be quite successful. The authors do a nice job of motivating their work, and showing how their method fits modularly with current PLL methods. The experimental comparisons are generally compelling. The theory could stand to be better-modularized, but that is a minor concern. The paper could potentially be improved by showcasing a \"killer app\" where the LT-PLL problem arises.", "strength_and_weaknesses": "Strengths:\n + The paper does a nice job setting up the LT + PLL problem, and showing that naive combinations of existing methods tend to fail.\n + Each of the LT and PLL problems are well-motivated, and prior work seems to be well-cited.\n + The method is explained well and empirical results are compelling, with appropriate ablations and comparisons.\n + The authors construct a new benchmark dataset, CIFAR-100-LT-NU\n\nWeaknesses (and questions):\n- It is still somewhat mysterious why using the oracle class probabilities fails so badly with PLL methods. Is the main pathology that the model simply doesn't train well at all? E.g., if one had an identically distributed test set with the same class distribution as training, would PLL methods using constant rebalancing behave poorly there too? Would \"tempering\" in the oracle logit adjustment (i.e., temperature scaling P(y) so that you start close to uniform and eventually adjusting for P(y)) lead to a similar result as the dynamic rebalancing? This question doesn't undercut the method in the paper (since it requires no access to the oracle), but it would be useful to shed a bit more light on how the oracle LA is disrupting training dynamics. The linear probing analysis does some of this.\n- Most of Theorem 1 seems to be a restatement of results from Liu and Dietterich. Specifying exactly what is added here would be useful. For example, it seems like one could assume that the Liu and Dietterich error bound holds, and Theorem 1 would be a relatively straightforward corollary (Note: the straightforwardness would not be a weakness!). Perhaps this is wrong, but some explicit discussion would be useful to understand what is novel.\n - Switching the softmax and expectation in (4) seems questionable as an approximation, especially in the long-tailed context where we expect some rare or common class logits to lie outside of the region where the softmax is approximately linear. The justification given in the text could at least use a citation if not some elaboration (say in an appendix). Alternatively, could this be formulated as a form of stabilization or regularization via Jensen's inequality?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "FIxL60AOYk", "reviewer": "Reviewer_MogU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on a more challenging but practically real-world data scenario, which couples the label ambiguity and heavy imbalance. This paper first demonstrates the infeasibility of the straightforward combination of the current long-tailed learning and partial label learning. Then it proposes a dynamic rebalancing method, termed RECORDS, to rebalance the training process. Experimental results on several benchmark datasets show the significant gain of RECORDS compared with a range of baselines. ", "summary_of_the_review": "The proposed method is well-motivated and seems to be effective, and experimental results and analysis are quite adequate. However, The experimental evaluation requires improvement, and code should be submitted along with the paper or made publicly available. ", "strength_and_weaknesses": "Strength:\n1. Well-motivated. The study of complex label distributions of data is a hot research area. This paper introduces a more difficult but common LT_PLL scenario by combining long-tailed and partial label learning.\n2. Well-written. This paper is clearly organized and easy to follow.\n3. The proposed method is reasonable and effective, as shown in experimental results.\n\nWeakness:\n1. There are other work focusing on similar scenarios, e.g. [1], so this paper is not the first attempt. It would be better to include some discussion and experimental comparison.\n2. There are some grammars. For example, in the last paragraph of Page 1 , “In Figure 1, we trace the average prediction of a PLL model PRODEN Lv et al. (2020) “, \\citep should be used here, i.e., “PRODEN Lv et al. (2020)” should be PRODEN (Lv et al. , 2020). There are many such mis-uses of \\cite in this paper.\n“Works” should be replaced by studies\n3. Please explain how the approximation in Equation (4) is obtained and what is the gap between these two items. Also explain how the second equation is obtained.\n4. The toy data are not a good testbed for the long-tailed and partial labeling setting, as there are no partial labels there.\n\nRef:\n[1] Liu W, Wang L, Chen J, et al. A Partial Label Metric Learning Algorithm for Class Imbalanced Data[C]//Asian Conference on Machine Learning. PMLR, 2021: 1413-1428.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "7aAqW984Vor", "reviewer": "Reviewer_djhm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the combination of long-tailed learning and partial label learning.\n\nApplying existing methods from long-tailed learning to this regime result in poor performance because of ambiguous labels. This paper proposes dynamic rebalancing as a solution.", "summary_of_the_review": "The paper combines the long-tailed learning and partial label learning settings. Dynamic rebalancing improves upon the baselines, but mostly combines existing techniques together.", "strength_and_weaknesses": "Strengths\n\n - The motivation behind the setting and the problems with existing solutions are described well.\n - Dynamic rebalancing is clearly explained and the toy example is a clear example of where it works well.\n - Dynamic rebalancing can be applied to many methods as a drop-in replacement.\n\nWeaknesses\n\n - The baselines are primarily focused on long-tailed learning, so it is natural that the baselines would not handle the partial labels properly.\n - Dynamic rebalancing is a relatively obvious solution (possibly the only simpler solution is to just use the previous epoch to estimate the class distribution, and it's not clear that momentum to average would outperform this).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster