Maximising the Utility of Validation Sets for Imbalanced Noisy-label Meta-learning
Authors: Hoang Anh Dung
Cuong C. Nguyen
Vasileios Belagiannis
Thanh-Toan Do
Gustavo Carneiro
Venue: Accepted by TMLR ()

Abstract: Meta-learning is an effective method to handle imbalanced and noisy-label learning, but it generally depends on a clean validation set. Unfortunately, this validation set has poor scalability when the number of classes increases, as traditionally these samples need to be randomly selected, manually labelled and balanced-distributed. This problem therefore has motivated the development of meta-learning methods to automatically select validation samples that are likely to have clean labels and balanced class distribution. Unfortunately, a common missing point of existing meta-learning methods for noisy label learning is the lack of consideration for data informativeness when constructing the validation set. The construction of an informative validation set requires hard samples, i.e., samples that the model has low confident prediction, but these samples are more likely to be noisy, which can degrade the meta reweighting process. Therefore, the balance between sample informativeness and cleanness is an important criteria for validation set optimization. In this paper, we propose new criteria to characterise the utility of such meta-learning validation sets, based on: 1) sample informativeness; 2) balanced class distribution; and 3) label cleanliness. We also introduce a new imbalanced noisy-label meta-learning (INOLML) algorithm that auto- matically builds a validation set by maximising such utility criteria. The proposed method shows state-of-the-art (SOTA) results compared to previous meta-learning and noisy-label learning approaches on several noisy-label learning benchmarks.

Published in Transactions on Machine Learning Research (03/2025)
Maximising the Utility of Validation Sets for Imbalanced Noisy-label Meta-learning
Dung Anh Hoang Department of Data Science and AT Monash University
hoang.dung@monnshh.co
Cuong Nguyen
cangggen@@surreg.ccc.uk
Centre for Vision, Speech and Signal Processing University of Surrey
Vasileios Belagiannis
vasileios. hellliios.bbeggiaannni@oooo.dd
Aridrich-AAezanddrr Universität Syrmbeen-Nürubery
Thanh-Toae Toan Do
toan. wown.do@@mmmshh.cii
Department of Data Science and AI Monash University
G. Carneiro
g cwwrueirrosurrrry.ooom
Centre for Vision, Speech and Signal Processing University of Surrey
Reviewed on OpenReview: https: // openreview. net/ forum? id- SBM9yeNZa5
Abstract
Meta-learning an effective method to handle imbalanced and noisy-label learning, but it generally depends clean validation set. Unfortunately, this validation set has poor scalability when the number of classes increases, traditionally these samples need to be randomly selected, manually labelled and balanced-disstrbutted. This problem therefore has motivated the development. of meta-learning methods to automatically select validation samples that are likely to have clean labels and balanced class distribution. Unfortunately, a common missing point of existing meta-learning methods for noisy label lcarning the lack of consideration for data informativeness when constructing the validation set. The construction of informative validation set requires hard samples, i.e., samples that the model has low confident prediction, but these samples more likely to be noisy, which can degrade the meta reweighting process. Therefore, the balance between sample informative- ness and cleanness is an important criteria for validation set optimization. In this paper, we propose new criteria to characterise the utility of such meta-learning validation sets, based on: 1) sample informativeness; 2) balanced class distribution; and 3) label cleanliness. We also introduce new imbalanced noisy-labe label meta-learning (LNOLML) algorithm that auto- matically builds validation set by maximising such utility criteria. The proposed method shows state-of-the-art (SOTA) results compared to previous meta-learning and noisy-label learning approaches on several noisy-label learning benchmarks.
1 Introduction
The development of new deep learning methods will gradually depend more on poorly curated datasets (Xiao et al., 2015; Li al., 2017). Such datasets tend have class-imbalanced distributions and to contain large amounts nobsy-label samples. The problems of class-imbalanced learning and noisy-label learning have, however, been addressed separately. For instance, while noisy-label methods are based robust

---

Published in Transactions on Machine Learning Research (03/2025)
loss functions (Wang et al., 2023; 2019), label cleaning (Jachwan et al.; 2019; Yuan et al., 2018), meta- learning (Ren et al., 2018; Han et. al., 2019), ensemble learning (Miao et al., 2015), and semi-superviset learning (Li et al., 2020), imbalanced learning approaches often rely on meta-learning (Ren et al., 2018; Han et al., 2019; Zhang & Pfister, 2021), transfer learning (Chu et al., 2020; Wang et al., 2017), classifier design (Wu et al., 2021; Liu al., 2020), and re-sampling (Wang el al., 2020). Notice from the methods above that meta-learning methods have the potential to address both noisy-label and imbalanced learning problems (Ren et al., 2018; Zhang et al., 2020; Zhang & Pfister, 2021; Xu et al., 2021; Shu et al., 2019).
Meta- Meta-learning is often formulated as bi-level optimization, where the upper-level optimization estimates the meta-parameters of interest using validation set, while the lower-level optimization trains model using a training set. In meta-learning, the validation set is commonly built by randomly selecting and manually labelling a class-balanced set of samples. However, the process of building these validation sets scales poorly when the number of classos escalates. Such issue, therefore, motivates the design of methods that automatically build validation sets containing pscudo-clean and class-bllanced samples (Zhang & Pfister, 2021; Xu et al., 2021). However, the prediction accuracy results of such methods are lower than the accuracy from conventional approaches that rely manually- that this poor performance might be attributed to the low informativeness of the validation sets based on pseudo-clean samples (Zhang & Pfister, 2021; Xu et al., 2021), comparing to the potentially more informative randomly selected samples. The reason is that these methods tend to prioritize only high-confidence samples, as they are more likely to be clean, but such samples are often less informative to the model. As a result, defining sample informativeness within a meta reweighting framework emerges as a crucial challenge. At the same time, intuitively, informalive samples are more likely to be hard sample (samples with low confident prediction, high gradient magnitude from the model), thus also more likely be noisy, which can contaminate the validation set and degrade the performance of the meta learning framework. This issue has highlighted the need for a framework that can automatically construct and optimize the balance between informativeness and the integrity of the validation set.
validation sets (Zhang et al,, 2020). We argue
Motivated by the bi-level optimiiaation of the meta-learning algorithm, we propose novel criteria to charac- terise the utility of validation sets used in meta-learning. In particular, the proposed criteria are based on: 1) sample informativeness: 2) class-balanced distribution; and 3) label cleanliness. We also introduce new imbalanced noisy-label meta-learning (INOLML) method that automatically builds validation set by max- imising its utility according to our proposed criteria. The proposed method, depicted in Figure 1, consists of 3 iterative steps: 1) detecting pseudo-clean samples from the noisy training set and labelling them; 2) forming the validation set from the robustly-labelled pscudo-clean set in step (1), using the proposed utility criteria; and 3) leaanng using the validation set. from step (2). The main contributions of our paper can be summarised as follows:
A new critcria set to form the meta-learning validation set based on sample informativeness, class- balanced distribution, and label cleanliness.
An innovative meta-learning algorithm (Figure 1), which automatically builds a validation set by
maximising its utility according to our criteria, comprising the following steps: 1) detection and robust labelling of pseudo-clean samples from the noisy training set; 2) formation of the validation set using the proposed utility criteria; and 3) meta-learning using the validation set from step (2).
A state-of-the-art meta-learning method to form validation sets for meta-learning that outperforms prior approaches that rely on random selecting or manually labelling.
In addition, the proposed meta-learning method INOLML shows competitive results with respect to previous meta-learning and noisy-label learning approaches on noisy-label learning benchmarks.
2

---

Published in Transactions on Machine Learning Research (03/2025)
2 Related Work
2.1 Noisy-label learning
Noisy-label learning methods rely on many strategies, namely: robust loss functions (Charoenphakdee et al., 2019; Zhang & Sabuncu, 2018; Ghosh et al., 2017), label cleaning (Yuan et al., 2018; Jachwan et al., 2019), co-teaching (Li et al., 2020; Jiang et al., 2018; Han et al., 2018), iterative label correction (Chen al., 2021b; Arazo et al., 2019; Tu et al., 2023), semi-supervised learning (Ortego et al., 2021a; Li et al., 2020; Ortego et al., 2021b), contrastive learning (Huang et al., 2023), meta-learning (Ren et al., 2018; Zhang el al., 2020; Zhang & Pfister, 2021; Xu et al., 2021; Shu et al., 2019), and hybrid methods (Nguyen et al., 2020; Jiang et al., 2020). Except for the meta-learning approaches (Ren et al., 2018; Zhang et al., 2020; Zhang & Pfister, 2021; Xu et al., 2021; Shu et al., 2019), the majority of these methods assume a bass-balanced training dataset.
For meta-learning approaches (Ren et al., 2018; Zhang al., 2020; Zhang & Pfister, 2021; Xu et al., 2021; Shu et al., 2019), pertamarassttrss are introduced to automatically weigtweight the losses of noisy samples and up-weight the losses of clean samples (Dehghani et al., 2017b;a). Such meta-parameters are then learnt by optimising the one-step-ahead model on a validation set.
To deal with noisy labels effectively, most of existing meta-learning approaches require a clean validation set. Such a set is, however, expensive to acquire and does not scale well with the number of classes. Recent studies relax such assumption by sclocting the validation set from the noisy training set (Zhang & Pfister, 2021; Xu et al., 2021). Unfortunately, their sample selection process demonstrate lower performance compared to their counterparts using real validation data. This motivates to develop an approach that can automatically build a clean and balanced validation sct, but unlike them, our approach is motivated by the meta-learning optimization that also takes into consideration the informativeness of the validation samples.
2.2 Imbalanced learning
Imbalanced learning can result a biased model with good accuracy for majority classes, but poor perfor- mance for the minority ones (Zhang et al., 2023). Methods to address imbalanced learning are based on transfer learning (Chu et al., 2020; Wang et al., 2017), classifier design (Liu et al., 2020; Wu et al., 2021), cost-sensitive learning (Zhou& & Liu, 2006; Sun et al., 2007; Elkan, 2001), data augmentation (Zang et al., 2021; Chou et al., 2020), logit adjustment (Menon et al., 2021; Provost, 2008) and representation learn- ing (Zhang et al., 2017; Huang et al., 2016). These existing methods often assume that training labels are clean, which does not hold for noisy-label datasets.
2.3 Noisy-label and imbalanced learning
Most of the methods mentioned in Sections 2.1 and 2.2 treat noisy-label and imbalance learning as two separate problems. Label noise in imbalanced datasets has also been considered by non mela-learning ap- proaches (Cao et al., 2021; Wei et al., 2021; Karthik et al., 2021), but they either have different setups or achieve sub-par results. For meta-learning, the validation set is crucial to allow good model performance, where we empirically observe that different randomly selected clean validation sets can lead to substantially different performances. For example, the validation set only contains "easy samples" that lie far from clas- sification boundaries, meta-learning tends to produce poor classification accuracy. Unfortunately, previous approaches have not studied this issue. For instance, classic meta-learning approaches (Ren et al., 2018; Shu et al., 2019) rely random sample selection and manual labelling, potentially producing uninformative validation sets. Furthermore, the fact that such manually-curated validation set is fixed for the whole train- ing process may hinder model generalisation because the meta-learning optimization can quickly overfit to this validation set. Recent meta-learning approaches try to automatically build clean validation sets with "casy samples", obtained from low-loss samples (Xu et al., 2021) or well-optimised samples (Zhang & Pfister, 2021). Instead of selecting random samples or "casy samples", we study how select informative samples for meta-learning.
3

---

Published in Transactions on Machine Learning Research (03/2025)
3
Background
This section presents an overview of the label noise problem and two mela-learning methods widely-used when dealing with noisy-label datasets.
3.1 Noisy-label learning
In the conventional supervised learning, we are given a training Delean [(,, [.)] Patea [ X C R represents D-dimensional input data, and t; E ft (0,1)-A1'E 1] is C-dimensional one-hot clean label. The aim is to find a classification model Se X - Ac-1, parameterised by 0 E θ that maps the input data to its corresponding label, where Ac-1 = {p p E [0, A Alp 1} denotes the
where x; e
(C (C- ----mmmmnsoonnl probability simplex.
In practice, instead of observing the correctly- training set D Delean: we are given a noisy training set D = ((x,, (x), where the noisy label y; E y might or might not be the same as the clean label t,. The aim is to exploit the information in such noisily-annotated training set to still learn a good model fo that can accurately predict the clean label for an instance x.
3.2 Learning to reweight
Learning to reweight (Ren et al, 2018) is meta-learning approach proposed to address the label noise problem defined in Section 3.1. Given the nature of meta-learning, the original training set is split into two non-overlapping subsets: training (or support) set D(t) and validation (or query) set Div), The main idea is to employ the validation subset D(e) to learn meta-parameter w E Dioj-1 (in the form of a probability vector) that weights the cross-entropy loss & of each training sample in the training subset D(t), The process obtaining is based on maximising some utility criteria, e.g,, informativeness. class-balanced distribution and label cleanliness (Ren et al., 2018). Formally, the objective function of this meta-learning approach is a bi-level optimization defined as:
min E (xs [e (fe
s.t.: 0" (w) = arg min (x.Y))DDO (ull()(()))
(1)
where the notation E denotes the expectation and can be expressed Eneu [g(u]] | 1/1 Em.uu
Intuitively, the lower-level (or the constraint) in (1) learns a classifier with weighted cross-entropy loss on the training subset D(1), while the upper-level evaluate the performance of the classifier on the training validation subset D(v) and optimises that performance w.r.t. the meta-parameter w.
The bi-level optimization in (1) can be solved by iterating the following steps. In the first step, the parameter of the meta-learning model, (*(w,, is estimated from the stochastic gradient descent (SGD) the training subset D(O,, with each step defined by:
("(w) = 0o - no V x. (x.. [[f((x))
(2)
where is the step size or lcarning rate.
In the second step, the meta-parane meta-parameter w in the upper-level is updated by applying one SGD step on the loss evaluated on the validation subset (v)) using the classifier's parameter 0 (w) obtained in (2). The update is defined as:
= wa Yhe V T B [e (fo-cs)
where Nu is the step size to update w.
The meta-paramoter wn obtained in Eq. (3) projected to the ([D(t) 1)-dimensional probability simplex, A -1 before being used to train the classifier of interest with the weighted loss defined in the lover-level of (1).
4

---

Published in Transactions on Machine Learning Research (03/2025)
3.3 Meta-relabelling
Based on the learning lo reweight approach presented in Section 3.2, the Distill method (Zhang el. al., 2020)
proposes a slight modification of the objective function in (1) as follows:
min E D))ED()) [e (fo- (x))())) s.t.; 0* (w, X) = argmin I8 (.)DDD bor (fe()))
(4)
in which À 1 | is considered another meta-parameter representing the tradcoff between the original label and model prediction. For each training sample in the training subset D(C), Distill uses two labels, (Ao) for sample reweighting and (A,) for sample relabeling. The modified label yi given the (noisy) label y; and trade-off variable A is defined as:
S:(A,) - A.y;
+ + Ax)s((x;),
(5)
where A, = 0.9 Vi = =,2,.... ID(0). For label (t(A,,, instead of optimizing the parameter λo its value is fixed at 0.9 (close to 1) to ensure that label (A,) retains information from the original label. Supervised learning with label (A,) and the optimized weight w; effectively corresponds upweighting clean samples and downweighting noisy samples.
On the other hand, the parameter X for the new label (A5) is optimized with Eq.6 to infer the correct label for the samples, enabling uniform supervised learning. The pseudo-label ył is defined as:
yys
x
y
fo()))
otherwise,
where A is the gradient w.r.t. X; of the final loss over the validation set D(v):
x = sign (-E -E
Oi, (for
(6)
where (sig (sign()g 0) denotes the rectification operator, and sign represents the sign function.
The estimated w", (A,) and y are then used to train a classifier of interest (Zhang et al., 2020). Given the pseudo label yi, the Distill method further employs supervised learning with images obtained via the minup operator (Zhang et al., 2018) using the training and validation sets. Additionally, they adopt a KL-divergence loss between the model's outputs on the original and augmented inputs to enhance the consistency of the pseudo-label distribution. The final objective of Distill framework is defined as:
L= (x..y.))D (F(((f(x)))((()))))) ((6())) pl +ep (vx.))(x))) + k KL Ifo (xx) llfo (x:]] B
(7)
@ y and x are obtained via the mirup operator (Zhang et al., 2018) using the training and validation sets, KTL[-II-] denotes the Kullback-Leibler (KT.) divergence, &; is an augmented sample of xi, and p and k are byper-parameters, B is the batch size.
4 Methodology
The meta-leariing Distill model (Zhang et al., 2020) presented in Section 3.3 optimisos meta-parametors X and w under the guidance of a validation subset Dív), and we adopt a similar approach for our framework. However, instead of selecting a fixed validation subset at the beginning of training as the Distill model, we propose definition for sample informativeness within meta reweighting framework, as well an itcrative mechanism to automatically select "high-utility" validation subset the start of each epoch. This section presents the utility criteria to select and the mechanism to update such validation set.
5

---

Published in Transactions on Machine Learning Research (03/2025)
shaze classifier feature extractor
Erain sidt
Dig
2) ntility: Informativvness. cleanliness and class-balance validation set
Noisy-label
Pseudo- clean dotector
pecudo-clom
Pseudo-clean ses Di1
3) Meta- lourner
training
samples
validiation iou seg
samples likedy with noisy labels
Z)
Pscudo-noisy
Robust labels sumples in D
Dis
Figure 1: Main stages INOLML: 1) classify the noisy-label samples from D into D(c) (samples that are likely to have clean labels) and D(n) (samples likely to have noisy labels); 2) build validation set D(v) containing samples that are informative (from a meta-learning perspective). balancod and with a high likelihood of containing clean labels, and 3) train the meta-learning classifier with D(1) D(e) D(o) and D(v),
Specifically, we propose 2-step process (corresponding to steps and 2 Figure/ 1 to select high-utility validation set to train meta-learning model presented in (4). In the first step, split the original noisy training set DD into pseudo-clean set D(e) and pseudo-noisy set D(n), In the second step, we further split the pseudo-clean set D(e) into training subset D(t) and a validation subset D(e) based on some criteria, such as cleanliness, informativeness and class-balanced distribution.
4.1 Detecting noisy and clean label subsets from
Formally, the original training set split into two subsets: clean and noisy, following a certain criterion:
((c) PseudoCleanDetectoo (D) A D( D D(e).
(8)
We first calculate the cross-entropy losses between noisy labels and the prediction fa(.) of all samples the training set D. We then apply the small loss hypothesis (Han al., 2018;
al., 2020) through the notation PseudoCClenDeetcttor]] in (8) to select samples having small losses to form pseudo-clean set D(e), because they are more likely to be clean. Note that the pseudo-clean set D(c) is independent from the target model since D(e) is initialised using warm-up model (i.e., a model trained on D for few epochs), while the target model is trained from scratch. Hence, the small-loss samples in the pseudo-cleat set D(e) are likely informative to the target model. The remaining samples form the pscudo-noisy set D(n), as shown in (8). These two sets, D(e) and Din), are regularly updated during training.
4.2 Maximising the utility of the validation set
The pseudo-clean D(e) obtained in (8) is then divided into a validation set D(e) and training D(t) The validation set ((e) must have the following properties:
class-balanced: containing the same number of samples per class.
informative: being useful to guide the meta-learning framework. We define validation samples with high informativeness samples that can assign larger weight wi for clean samples, and lower weight for noisy samples.
clean: most likely having clean labels.

---

Published in Transactions on Machine Learning Research (03/2025)
To enforce such properties, propose the objective function of the validation set as follows:
D D((o)
D opt
( = argmax
Clean (D(1), D(o)
opt
D cD |MxC
8.b.: D
argmax
Info ((()) Dle)),
(9)
DDn) CD())
Dio) =KxC
where K and M are the number of samples per class with M < K to guarantee a class-balanced distribution for D(v), and the functions Clean() and Info() denote the cleanliness and informativeness of samples. D and D respectively denote the optimized D(v) and D(v).
opt
In other words, the lower-level in (9) means to determine "coarse" validation set D(")) that is class-balanced (containing samples per class) and informative from the pseudo-clean set D(e), The upper-level in (9) refines the coarse validation set D(v)) further to form "fine" subset D(v) containing most likely clean samples. The need for this bi-level optimization is because in practice, as the training progress, we find that informative samples need to become harder samples (samples with low confident, prediction from the model) over time, hence more likely to be noisy. Note that while clean and informative samples are beneficial for the mela reweighling framework, informalive--ut-noiiyy validation samples will make the model prone overfitting. Our optimization for the validation set D(#) hence aim find the balance between class- balance, informativeness and cleanness. The details of Clean(-) and Info(.), are presented in the following subsubsections.
4.2.1 Informativeness
As described above, informative samples for the validation set of a meta reweighting framework should be able to upweight clean samples, and downweight noisy samples effectively. Based on this motivation, our Info(.) function in the lower-level of (9) is motivated by learning to reweight (Ren et al., 2018) (similar to (1)). According to (Ren et al., 2018, Eq. (12)), the gradient w.r.t. can be written follows:
B(xY)))DD
e (10)())))))))))5)) aws
x -K 1,
(1) 2 2/1-11
83,1 Bi!) B..).
(10)
W==
1=1
where 2 is the feature of the validation sample Xj processed by a deep model at the layer 1-th (similar definition is applied for the training feature =(1-1), and gj, is the gradient from layer I for the validation sample X2 (similar definition is applied for the training gradient g.()
The re- weightnht factor Wi of a training sample x; is, therefore, high if its feature and gradient are similar to the validation samples' feature and gradient; else, the weight low. Hence, a validation set that maximises the weight w training samples also maximises its meta-learning optimization utility. This observation is crucial to our validation sample selection. Intuitively, we can define the sample informativeness function Info(.) that want to maximize below:
(D(o) (()))====
Σ
Σ
h(x,,,,);
(11)
(x..)) ((te)) (E)
where h is analogous to a measure of similarity between samples:
xxx,,x)= = 2,, -1
(12)
1=1

---

Published in Transactions on Machine Learning Research (03/2025)
15
44
44
(a) Naive utility (Eq. (11)).
(b) Our utility (Fq. (13)).
Figure Comparison between 2-dimensional t-SNE representations the samples selected (samples for each class have different colours, and the selected validation samples per class are highlighted with a blue dot with black outline) by (a) naive utility in Eq. (11), and (b) our utility in Eq. (13). Note that this the t-SNE representation for CIFAR1O dataset with uniform noise rate of 40%.
with zj,-1 being the feature xf x, the input of layer 1 (same for feature Z of sample x;). and gj, being the validation gradient of layer from xj (same for gi, from x.). Note that h(...) in (12) is part of the weight w; for the training sample ys) (sce (10)), and h(z;, represents how much the validation sample (x),y;) can contribute to the weight of the clean training sample (x), consists of a gradient matching term gi, and a feature matching term 2J,1-1 2,,11 between the two samples. Ideally, we want the weight of the clean samples remain high throughout the training process. Because h(,,, .ry) depends on samples gradient fgu {g.:)= that may vary greatly during the training process, maximizing the function from Eq. (11) may lead to biased validation set that is optimal only for the current. iteration, not. throughout the training process. If validation sample (z) has high contribution to the weight of clean training sample yz), then they may share similar characteristics. Consequently, their gradient matching B,, and feature matching 2 will remain high and robust during the training process. Therefore, we need to modify the function Info(.), that for cach clean training sample (x,, there must be at least validation sample (y) that has high weight contribution to it. Such observation allows us to re-define the function Info(.) in the lower-level of (9) as follows:
In(o (())) =
Σ
max (1
h(x;,
(13)
(x.)))(DDO) ((e) y,=y;
In order to observe the effectiveness of this approach, we compare our modified Info(.) in Eq. (13) with the naive utility from Eq. (11). Figure? shows a visual comparison between the naive utility in Eq. (11) and our utility in Eq. (13) for the selection of validation samples from the pseudo-clean set D(e) using t-SNE projection 2-dimensional space for CIFAR10 dataset with uniform noise rate of 40%. It is noticeable that the selected validation samples using proposed utility are diverse and distributed around the clusters of their respective classes, while the validation samples selected with the naive utility in Eq. (11) tend to be located closer to each other and inside cach class cluster. This is because the naive utility favors low- confidence samples at the selection time. However, as training progresses and the prediction confidence of these samples increases, other samples should become the target for upweighting instead, making the selected validation set no longer optimal. Another observation that our utility maximization results in a validation that constructs boundary between each class in feature space, helping our method to identify the classification boundary between different classes.
8

---

Published in Transactions on Machine Learning Research (03/2025)
Algorithm 1 Training procedure of the proposed INOLML.
1: procedure TRAIN(D, 7, T, T, T(",, 7, R, N, M, K, C?)
2: > D: noisy training set, {n learning rates
> T: tolal number of ileralions
b T: minimum raumber of iteralions before updating the robust labels
T(w); interval between updates
> K, N, M, K, C: huper-parameters
7:
Warm-up fo(.) with (.F(()) on 'D
8:
D(e) t PseudoCleanDetector (D)
D. Eq. (8)
Initialise robust label {5r)]=m || of samples in ((c)
9:
10:
Initialise (( and D(t) from D(c)
b Eq. (9)
11:
Reinitialize fe(.)
12:
for t= to T do
13:
Meta-learn and A if (t T) then
D. Eq. (4)
14:
15:
Update 1D1 [D(0]
of samples in D(c)
Eg. (15)
16:
i mod T(w) =0 then
17:
Construct Die)
Eg. (16) > Eq. (9)
Select new D(e) from D(e)
18:
19:
return the trained model paramcter 0
4.2.2 Cleanliness
Note that our focus on the informativenoss of the validation set should not compromise the cleanliness of samples, meta-learning under the guidance of informative-but-noisy validation samples will make the model prone to overfitting, Indeed, the samples in the have noisy labels since D(s) is not completely clean. In addition, the function Info(z;,, 2z) prioritizes samples in D(v) with large gradient magnitude, as has gradient matching component g (8.1. Consequently, the selected validation samples usually "hard samples" and more likely to be noisy. It is, therefore, important to balance the tradc-off between sample informativeness and cleanness. However, one of our main factors to identily informative samples, we do not want to minimize their gradient magnitude
validation set D(e) can still
gradient magnitude is
for cleanliness metric. By observing that the samples from D(o) are more likely to be clean when they have high similarity with other samples in the same class, we propose a heuristics to identify the samples of interest with high chance being clean. Specifically, we employ the cosine similarity between the samples of interest and other samples the same class in D(e), defined as follows:
Clean (D(+), (D())
Σ (x)t))) (x)))) yi=y,
Σ 1=1
(14)
Both combinatorial optimizations in (9), and in particular Eqs. (13) and (14), are solved with greedy strate- gies. We initially loop through each class and select K samples per class maximise the lower-level objective function. We then sequentially select M samples among the previous set of K samples per each class to opti- mise the upper-level objective function. We simplify the calculation of gradient in (10) and the optimization in (9) by using only the features and gradients in the last layer of the deep neural network of interest. This simplification is rcasonable since according to (Zhang & Pfister, 2021), the weights of training samples in meta-learning depend mostly on the last layer of the model, and for datasets with a large number of classes, the memory complexity intractable. For instance, we need around K = 200 candidate samples per class for D(") before selecting M 10 samples per class for the validation set D(e) (c.g., for CIFAR100, we nood 20,000 samples for D(w),, where for each sample, we have to store its gradient and feature embedding for every layer. For example, for Resnet18 that has 1 10" parameters, if needed to store all features and

---

Published in Transactions on Machine Learning Research (03/2025)
gradients from all layers for the whole D(v), we would need to store (2 x 104) x (1.1 x 10?) floating point numbers (i.e., se 1012 bytes), which is intractable.
4.3 Dynamic Pseudo Clean Set Refinement
To further refine the pseudo clean set D(o), we adopt the robust moving average label of each sample which is defined follows:
1 Σ fo(x:),
y; ky. + (1 R)
(15)
E
with x = 0.9. The robust label is updated regularly, and becomes more accurate as the model improves. At the start of each data selection step, eliminate polentially noisy samples from the pseudo clean set that were previously overlooked by the PssudoCleanDetecto using the following method:
Die) = (xx.))) (x) E D(e), A argmax yı(k) = argmax Vi(k:)
(16)
kE( (E).....))
We utilize the proxy set Die) for our data selection algorithm instcad of the full set D(e)
4.4 Training procedure
Our training follows the 3-step iterative approach in Figure 1, where step 1 (pseudo-clean label detector) and step (utility maximisation of the validation set) have been explained above, and step 3 (meta-leariing) is based on (4) with the loss L defined in Eq. (7). The details of the training process are shown Algorithm
5
Experiments
5.1
Datasets
The proposed method INOLML is evaluated on several datasets, inchuding CIFAR1O, CIFAR100, mini- WebVision and Controlled Noisy Web Labels (CNWL). Both CIFAR10 and CIFAR100 datasets (Krizhevsky & Hinton, 2009) contain 50k and 10k images used for training and testing, respectively. Each image has a size of 32x32 pixels and is labelled as one of 10 or 100 classes. WebVision (Li et al., 2017) dataset of 2.4 million images crawled from Google and Flickr based the 1,000 ImageNet classes (Deng et al., 2009). The dataset is more challenging than CIFAR since it is class-imbalan inass-imbalanced and contains real-world noisy labels. Following (Zhang & Pfister, 2021), we extract a subset that contains the first 50 classes to create the mini- WebVision dataset (Jiang et al., 2018). CNWL (Jiang et al., 2020) is a benchmark of controlled real-world label noise that contains noise rates from to 0.8. Following recent studies (Xu et al., 2021), we evaluate the proposod method on the Red mini-ImageNNet dataset consisting of 50k training images from 100 classes for training and 5k images for testing. Note that we use the image size of 32x32 pixels for fair comparison with FAMUS (Xu et al., 2021) and other related methods (Xu et al., 2021).
5.2 Implementation details
For all experiments on CIFAR datasets, except long-tail imbalance, we use the same hyperparameters and network architectures as the Distill model (Zhang et al., 2020). We adopt the cosine learning rate decay with warm restarting (Loshchilov & Hutter, 2017) and SGD optimiser. For CIFAR, with 100k iterations and batch size of 100. We also train smaller network (Resnet29) to fairly com- pare with (Zhang et al., 2020). For mini-WebVisioon we follow FSR (Zhang & Pfister, 2021) and train a single Resnet50 nelwork with million iterations and a batch size of 16. For Red mini-ImageNNt, we run experiments with 150k iterations and a batch size of 100. For CNWL, we use a single PreAct Resnet18 net- work to be consistent with previous works (Cordeiro et al., 2021; Ortego et al., 2021b) on this benchmark. For the class imbalance problems, we use Resnet32 to fairly compare with FaMUS (Xu al., 2021) and FSR (Zhang& & Pfister, 2021). We report the prediction accuracy of each experiment on their corresponding
train WideResnet28-10 10
10

---

Published in Transactions on Machine Learning Research (03/2025)
Table 1: 'ess accuracy (%) INOLML and previous methods for symmetric noise; methods with represent meta-learning methods that need clean validation scts; the lower block contains meta-learning methods, while the upper block shows SOTA methods.
CIFAR10
CIFAR100
Method
0.2
0.4
0.8
0.2
0.4
0.8
GJS
95.3 0.2
93.6 0.2
79.1 + 0.3
78.1 + 0.3
75.7 0.3
44.5 + 0.5
DivideMix
95.7 + 0.0
92.9 + 0.0
76.9 + 0.0
59.6 + 0.0
CRUST
91.1 + 0.2
89.2 + 0.2
58.3 + 1.8
PENCIL
73.9 + 0.3
69.1 + 0.6
UNICON
96.0 = 0.0
93.9 0.0
78.9 + 0.0
63.9 + 0.0
CausaINT. + NPC
81.2 + 0.0
18.8 + 0.0
DMLPT
94.2 = 0.0
+..0+000
72.3 + 0.0
63.2 0.0
DMLP-DivideMix
96.2 + 0.0
94.3 0.0
79.4 0.0
68.5 0.0
TLC
95.0 = 0.1
92.5 0.2
78.0 0.2
65.0 0.3
PSDC
96.2 = 0.0
94.0 + 0.0
79.4 + 0.0
64.3 + 0.0
Bayesian DivideMix+
96.13 = 0.07
94.97 = 0.02
80.02 + 0.03
70.01 = 0.23
DistilIT
96.2 = 0.2
95.9 + 0.2
93.7 = 0.5
81.2 = 0.7
80.2 0.3
75.5 0.2
MentorNet
92.0 0.0
89.0 0.0
49.0 + 0.0
73.0 + 0.0
68.0 0.0
35.0 + 0.0
L2RT
90.0 = 0.4
8.2
73.0 + 0.8
67.1 + 0.1
61.3 2.0
35.1 + 1.2
MWNT
90.3 + 0.6
87.5 0.2
64.2 + 0.3
58.6 0.5
GDWT
88.1
59.8 #1.6
FaMUS
95.3 0.2
76.0 + 0.2
FSR
95.1 - 0.1
93.7 + 0.1
82.8 + 0.3
78.7 + 0.2
74.2 = 0.4
46.7 + 0.8
INOLML
96.9 0.1
96.6 0.1
95.0 0.2
82.0 0.2
81.3 1.2
74.710.11
lesting sets. We compare method with recently published state-of-the-art (SOTA) meta-learning ap- proaches, including FaMUS (Xu et al., 2021), FSR (Zhang & Pfister, 2021), Meta Weight Net (Shu et al., 2019), Distill (Zhang et al., 2020), GDW (Chen et al., 2021a), L2R(Ren et al., MLLL), & Ulusoy, 2021). Furthermore, some SOTA noisy labels learning approaches are also being compared with, such as DivideMix (Li et al., 2020), CausaINT. (Yao et al., 2021), NPC (Bae al., 2022), MentorMix (Jiang et al., 2020), MentorNet (Jiang et al., 2018), MOIT (Ortego et al., 2021b), GJS (Englesson & Azizpour, 2021), CRUST (Mirzasoleiman et al., 2020), Co-teaching (Han et al., 2018), Iterative-CV (Chen et al., 2019), HAR. (Cao et al., 2021), UNICON (Karim al., 2022), NCR (Iscen et al., 2022), C2D (Zheltonozhski et al., 2022), BtR (Smart & Carneiro, 2022). CC (Zhao et al., 2022), SSR (Feng et al., 2021), DMLP (Tu et al., 2023), Twin Contrast (Huang et al., 2023). For imbalance learning mixed with noisy label experi- ments, we compare with recent noisy-label imbalanced learning methods, including ROLT (Wei et al., 2021), FSR (Zhang Pfister, 2021), LDAM (Cao et al., 2019), BBN (Zhou et al., 2020), HAR (Cao al., 2021). and CRUST (Mirzasolciman et al., 2020). The supplementary material shows more implementation details.
5.3 Symmetric noise
Table shows the test accuracy results on symmetric noise rates varying from 20% 80% for different meta- learning approaches that require clean validation set (indicated with T) and others that automatically build validation sets. INOLML outperforms all previous methods in most cases. The slightly lower performance than Distill on CIFAR100 at 80% noise rate can be explained by Distill's large manually-curated clean validation set with 10 clean samples per class. In addition, as shown in Figure 3a, at 80% symmetric noise rate, a significant proportion (20% to 45%) our clean validation set D(") contains noisy samples at the final training stages. This, however, deteriorates the efficacy of our approach. Note that is reasonable that the noise rate increases in the validation set as training progresses because our method prioritises the more informative samples (or harder samples) over clean ones to be included in the validation set at each
11

---

Published in Transactions on Machine Learning Research (03/2025)
ImgNet
Cifar100
Accuracy on D(v)
Both
0.9
Noise rate 0.2
Noise 0.4
0.88
Accuracy
0.8
0.6
0.7
2
G
8
2
G
Iteaaiio (x10,000)
Iteration [x 10,000
(a) Symmetric noise
(b) Open-set noise
Figure 3: Accuracy of the clean validation set D(v) training progresses evaluated on different noise benchmarks.
Test accuracy (%) INOLML on CIFAR10 with 0.4 asymmetric noise, in comparison with Distill using a validation set D(o) of sizes 1, 5 and 10 samples per class on Resnet29 and WideRcsnet28-100 10. The superscript. indicates the need for clean validation sets.
Table 2:
Method
Size of validation set DD())
Resnet29
WRN28-10
Distill
76.8 2.9
93.2 + 0.2
1xC
INOLML
86.8 + 0.9
93.6 + 0.3
Distil 5 C
86.7 + 0.5
94.5 + 0.2
INOLML
89.3 0.2
94.1 + 0.1
Distillf
88.6 + 0.7
94.9 + 0.1
10 C
INOLML
89.8 + 0.3
94.2 + 0.1
epoch. Such prioritisation results a docrease the clcanliness our validation set as the model gets more accurate. We also carry out additional experiments with different validation set sizes to fairly compare with Distill in Appendix C. In particular, our method outperforms Distill by 1% 3% in most scenarios. Overall, these results show that psoudo-clean, balanced, and informative validation set, can outperform randomly-collected clean validation set in symmetric noise scenarios. Our results also set new SOTA results the symmetric label noise benchmarks for methods without a clean validation set.
5.4 Asymmetric noise
Table shows comparison between INOLML and Distill (Zhang al., 2020) with different validation set sizes: 1, 5 and 10 samples per class. Although proposed method does not rely
hamually-labellle validation set, it performs better than Distill, especially with small network architectures (Resnet29) and small validation set sizes (1 sample per class). Our method has slightly lower accuracy than Distill with larger clean validation sel sizes (at least. 5 random clean samples per classes) on larger network archilectures (WideResnet28). This might be caused by the confirmation bias of asymmetric noise our pseudo-clean validation subset and the high capacity of larger models, c.g., WideResnet28-10, which are more prone overfit label noise when trained on small dataset, like CIFAR10. In addition, INOMML also shows better performance than SOTA methods, like FSR, Mix and UNICON (see Table 3).
12

---

Published in Transactions on Machine Learning Research (03/2025)
Table 3: Test accuracy (%) of INOLML and previous methods on CIFARIO with 0.4 asymmetric noise (meta-learning approaches are in the bottom part of the table). The superseript indicates the need for clean validation sets.
Method
Accuracy
GJS
89.7 + 0.4
R-Correction
83.6 + 0.3
UNICON
94.1 + 0.0
PENCIL
91.2 = 0.0
DivideMix
92.1 0.0
CausaINL
74.8 0.0
TLC
92.6 0.0
TLC+
93.7 0.0
PSDC
9.0
DMLPT
0.0
L2RT
90.8 0.3
FSR
93.6 + 0.3
INOLML
94.2 + 0.1
Table 4: Test accuracy (%) of INOLML and other SOTA meta-learning approaches evaluated on the CIEAR imbalanced learning (long-tailed) recognition task. The reported results are from (Zhang Pfister, 2021) and (Xu et al., 2021).
CIFAR10
CIFAR100
Imbalance ratio
200
50
10
200
50
10
Softmax
65.68
74.81
86.39
34.84
43.85
55.71
CB-Focal
65.29
76.71
86.66
32.62
44.32
55.78
CB-Best
68.89
79.27
87.49
36.23
45.32
57.99
L2R
66.51
78.93
85.19
33.38
44.44
53.73
MWN
68.91
80.06
87.84
37.91
46.74
58.46
GDW
86.8
56.8
FaMUS
83.32
87.90
49.93
59.03
FSR-DF
66.15
79.78
88.15
36.74
44.43
55.60
FSR
67.76
79.17
87.40
35.44
42.57
55.45
INOLML
74.91
84.43
90.81
41.52
51.35
62.07
5.5
Imbalanced learning
We evaluate INOLML on the imbalanced CIFAR datasets following the setting in (Zhang & Pfister, 2021). The prediction accuracy in Table 4 shows that INOLML considcrably surpasses other meta-lcarning ap- proaches.
5.6 Imbalanced noisy-label learning
We evaluate our proposed method in the setting that combines class imbalance and label noise from (Zhang & Pfister, 2021). We follow the same experimental configuration by adding 20% and 40% symmetric noise to the CIFAR10 imbalanced datasets with different imbalance ratios (10, 50 and 200). The results in Table 5 show that INOLML outperforms other approaches by a large margin. This result even more remarkable when studying the noise rate of 40%. For CIFAR100, we do not show results with imbalance ratio > since for larger imbalance ratios, was impossible build validation sets with 10 samples per class for the minority classes. Nevertheless, for the two CIFAR100 problems, our method shows substantially better
13

---

Published in Transactions on Machine Learning Research (03/2025)
Table 5: Test accuracy (%) INOLML and other SOTA methods on CLEARIO and CIEAR100 imbalanced learning mixed with symmetric noise. The reported results are from (Zhang & Pfister, 2021) and (Wei al., 2021).
Dataset
CIFARI1
CIFARI00
Noise ratio
0.2
0.4
0.2
0.4
Imbalance ratio
10
50
200
10
50
200
10
10
CRUST
65.7
41.5
34.3
59.5
32.4
28.8
LDAM
82.4
71.4
48.1
36.7
LDAM-DRW
83.7
74.9
50.4
39.4
BBN
80.4
70.0
47.9
35.2
HAR-DRW
82.4
77.4
46.2
37.4
ROLT-DRW
85.5
82.0
52.4
46.3
FSR
85.7
77.4
65.5
81.6
69.8
49.5
INOLML
90.1
80.1
66.6
89.1
78.1
61.6
59.8
56.1
Table 6: Test accuracy (%) of INOLML and previous methods in open-set noise using WideResnet28-10 with 10 samples per class for validation.
Method
ImageNet CIFAR100 BOTH
RoG (Patrini al., 2017)
83.4
87.1
84.4
L2R (Ren et al., 2018)
81.8
81.8
85.0
Distill (Zhang et al., 2020)
94.0
92.3
93.0
INOLML
94.5 + 0.1
93.6 : 0.0
93.6 + 0.1
results than previous SOTA methods. Our method can, therefore, be considered the new SOTA for this imbalanced noisy-label learning benchmark with Resnet32.
5.7 Open-set noise
Open-set noise occurs when training images may belong to classes outside the classes D. We consider benchmarks using CIFAR10 contaminated with images from CIFAR100 and ImageNet (Lee el al., 2019b). Table shows results from INOLML, Distill (Zhang al., 2020) and other meta-learning methods (Patrini et al., 2017; Ren al., 2018; Zhang & Pfister, 2021), where INOLML achieves the best performance in all cases. Comparing to Distill, INOLML is 0.5% to 1% better, despite the selected validation set D(") being largely contaminated with noisy samples (up to 40%), as shown in Figure 3b. However, such performance on open noise contrasts with our observation in the 80% symmetric noise settings, where just 30% noise rate in D(") degrades the performance of INOLML, compared to Distill model (Table 1). Such difference might be attributed to the samples that do not belong to the set of known classes, such samples might help regularise the lcarning on mislabelled data, reducing the effect of confirmation bias, resulting in better performance.
characteristic of open-set noise. As open-set noisy-label datasets contain
5.8 Real-world datasets
Tables 7 and 8 show the results of INOLML and other SOTA approaches real-world datasets. Table 7 shows the performance on mini-Web WiiVisonn with 2 popular models: Resnet50 (RN50) and InceptionResnet V2 (InRN), while Table 8 shows results on four different noise ratios evaluated on Red mini-ImageNNe with 32 x 32 images with 2 popular models: Resnet18 (RN18) and InceptionResnet V2 (InRN). INOLML is the new SOTA on mini- Web
with Resnet50 model and Red mini-ImageeNet. We note that our method is more efficient in terms of memory footprint than most of Co-training based approaches (Li al., 2020; Cordeiro al., 2021; Xu al., 2021) evaluated Red Mini-ImageNNet since use only single PreAct Resnet18
14

---

Published in Transactions on Machine Learning Research (03/2025)
Table 7: Prediction accuracy (%) on the real-world dataset mini- Vision with Resnet50,, evaluated on Webvision and ImageNet test sets; the results of other methods are from (Zhang & Pfister, 2021; Cordeiro et al., 2021) or from their original papers.
Num. Params
Method
Base
WebVision ImageNet
HAR
InRN
56M
75.5
57.4
GJS
RN50
23M
78.0
74.4
MW-Net
InRN
56M
74.5
72.6
UNICON
2xInRN
112M
77.6
75.3
MOIT
RN18
11M
78.8
SSR
TnRN
56M
80.9
75.8
C2D
2x RN50
46M
79.4
78.6
Bayesian DivideMix+
2x RN50
23M
80.12
78.51
NCR
RN50
23M
80.5
BtR
InRN
56M
80.9
76.0
CC
InRN
56M
79.4
76.1
TLC
RN50
23M
79.1
75.4
FSR
RN50
23M
74.9
72.3
INOLML
RN50
23M
81.7
78.1
Table 8: Prediction accuracy (%) on the real-world dataset Red mini-ImageNNt datasel. The results of other methods are from (Zhang & Pfister, 2021; Cordeiro et al., 2021) or from their original papers.
Noise ratio
Method
Backbone N parameters
(millions)
0.2
0.4
0.6
0.8
CE
2x RN18
22
47.36
42.70
37.30
29.76
Mix Up
2x RN18
22
49.10
46.40
40.58
33.58
DivideMix
2x RN18
22
50.96
46.72
43.14
34.50
MentorMix
InRN
56
51.02
47.14
43.80
33.46
PropMix
2x RN18
22
61.24
56.22
52.84
43.42
FaMUS
2x RN18
22
51.42
48.06
45.10
35.50
InstanceGM
2x RN18
22
58.38
52.24
47.96
39.62
InstanceGM-SS
2x RN18
22
60.89
48.06
45.10
35.50
INOLML
RN18
11
63.23
58.21
53.39
45.32
with meta-learning instead of two separate PreAct Resnet18. On mini- Viii-on benchmark, our method can also achicve significantly better performance despite using a smaller architecture in terms of the number of parameters compared to previous methods.
6 Ablation studies
This section studies the factors affecting the optimization in (9). In the lower-lever optimization of (9), the function Info(.) selects samples that maximise the training sample weight w from (10), well samples that maximise the maximum "information content". We, therefore, carry out ablation study about the importance of this factor by replacing Info(.) in (9) with the MaxWeight(.) in Eq. (11) and show the results in Table 9. We also study the role of the Clean(.) utility function in (9) by optimising only the lower- level of (9) (see Info(.) only). This ablation study conducted on CIFAR10 and CIFAR100 under 40% asymmetric noise and 20% symmetric noise with imbalanced data. Overall, each component improves model performance compared to naively optimising the average weight in (10). Naively selecting samples based (10) facilitates the overfiting of the noisy-label samples, leading to confirmation bias. The framework
15

---

Published in Transactions on Machine Learning Research (03/2025)
Table 9: Test accuracy (%) on CIFARIO and CLEAR100 under asymmetric and imbalanced noisy-label problems, where IR denotes the imbalance ratio. The 1st row shows the results of the optimization of the average of weight (col. Average Weight in (10)) instead of (9). The 2nd row shows the results of optimising the lower part of (9) (col. Info(.) Only) without the upper part of (9) Clean(.). The last row (Whole (9)) shows our final model result.
Asymmetric CIFARIO
Symmetric
Replace Info Info(.)
Equation
CIFAR10
with MaxWeight Only
WRN
RN29
RN32
RN32
RNN2
IR=1
IR=1
IR=10
IR=50
IR=200 200
91.0
56.6
68.8
37.6
23.4
92.1
89.3
89.0
79.1
65.9
91.1
89.8
90.1
80.1
66.6
mitigates this problem by using Clean(.) limiting the noise in the clean validation set, while Info(-) prevents the gradient to go toward single wrong direction.
As shown in Table 9, the impact of Clean(.) varies across different settings. Consoquently, applying a uniform optimization framework for all noise lypes, noise rate and imbalanced ratio may result in unnecessary overhead and potentially suboptimal performance. A promising future direction could involve designing an adaptive framework that automatically determines the optimal training strategy by estimating the class imbalance and noise rate, thereby minimizing framework overhead and optimizing performance.
To demonstrate the effectiveness of INOLML, measure the distributions of the sample weights during the training process compared to other meta reweighting methods, demonstrated in Figure 4. Recall that INOLML targets the maximisation of w for training samples of high utility, so it is important to measure how w progresses during training. The evaluation is conducted on CTFAR-100 with 80% symmetric noise setting, and we compare against FSR (Zhang & Pfister, 2021) and Distill (Zhang et al., 2020), as they are the most closely related frameworks to our approach. The results in Figure 4 show that overall, our method provides higher weight for clean samples and lower weight for noisy samples, compared to other meta reweighting methods such as FSR (Zhang & Pfister, 2021) or Distill (Zhang et al., 2020), demonstrating the effectiveness of our method.
We also compare the validation set produced from (9) with sets built with random sampling and most confident sampling based on the highest confidence scores. Figure 5 shows that the most confident sampling has inferior results compared to random sampling. Nevertheless, INOLML with built-in high utility validation set formation shows the best results.
Traditional meta-lcarning approaches (Dehghani et al., 2017b; Shu et al., 2019) always keep the clean vali- dation set separate from the training set, while our method iteratively extracts ((v) from the training set. Tt can be argued that this non-separation of the training and validation sets can cause confirmation bias to happen during training. Hence, we evaluate our approach in a scenario where the candidate samples to form the validation set is always separate from the training set during training. However, our results show that such separate validation set causes a 2% drop in accuracy, on average. This can be explained by the smaller size of the training set and the restriction in potential choices for validation samples.
A final discussion point is the time needed for the INOLML training. Using CIFAR10 with uniform noise, the Distill model takes around 5 and 29 hours to train the Resnet29 and WideResnet28-U1 10 models, respectively. When integrating our method with Distill, training slightly increases to 3.5 hours on Resnet29 and 31 hours on WideRessne(2-11 Hence, our algorithm adds around 10% to the training time for optimising the validation set, which happens once per epoch. The experiment above was conducted on a single NVIDIA V100 GPU, We also compare the training time of our approach with other recently proposed methods using the PreAct Resnet18 model on CIFAR100 dataset with 40% uniform noise using a single V100 GPU. While our approach takes 10.5h, DivideMix (Li et al., 2020) takes 8.25 hours, Causa al., 2021) takes around 12.5 hours.
16

---

Published in Transactions on Machine Learning Research (03/2025)
Clean Samples Weight
Noisy Samples Weight
FSR
Ours
Ours
Empinical pdr
Empiriaal
Value
Value
Figure 4: Weight distribution of samples from different data reweighting methods, under the setting CI- FAR100 with 0.8 uniform noise ratio.
100
Most confidence
88.6 89.2
91.0
87.0 87.9
Random clean
INOLML
Accuracy
59.2
50.7
CIFARIO 0.4
CIFARIO 0.8
CIFAR100 0.8
Figure 5: Accuracy (%) of our INOLML using different sample selection methods under uniform label noises.
17

---

Published in Transactions on Machine Learning Research (03/2025)
and MOIT (Ortego et al., 2021b) takes 8 hours, indicating that our approach has competitive training time compared to recently proposed methods.
7
Conclusion
We presented novel meta-learning approach, called INOLML, that automatically and progressively selects pseudo-clean validation set from a noisily-labelled training set. This selection is based on our proposed validation set utility criteria that take into account sample informativeness, class-balanced distribution, and label cleanliness. Our proposed method is more effective and practical than prior meta-learning approaches since we do not require mamally-labelled samples to be included in the validation set. Compared with other meta-lcarning approaches that do not require a manually labelled validation set (c.g. FSR or FaMUS), INOLML has demonstrated to be more robust high noise rale problems and able to achieve SOTA results on several synthetic and real-world label noise benchmarks. In fact, INOLML has SOTA results in mini- mini-Web WebVision, Red mini- ImageNt,,,, open-set noise, long-tailed + symmetric noise for CLFAR-10/-100, symmetric and asymmetric CIFAR-10/-100, and imbalanced learning, with substantial improvements (c.g.. CIFAR-10 80% symmetric, CIFAR-100 40% symmetric, imbalanced and noisy-label imbalanced benchmarks, and mini- Web
A limitation of our approach is that the model can suffer from confirmation bias as it is based on a single network. As future work, will tackle this problem by incorporating co-teaching in our meta-learning algorithm. Another limitation is the greody and complex bi-level optimization to form the validation set in (9), which can be improved in two ways: 1) the complexity can be reduced by replacing the bi-level optimization with a single-level optimization, and 2) the greedy strategy can be replaced by a relaxation method to solve the combinatorial optimization problem. Additionally, optimising the clean validation set once per epoch is not ideal since the validation set can be outdated by the end of epoch. This issue will be addressed by updating the clean validation set more regularly. Finally, another point missing from this paper is a theoretical analysis of the proposed criteria to characterise the utility of the meta-learning validation set. In particular, we plan to study why the relaxation of the assumption of clean and balanced validation set made by Ren et al. (2018) in (10) still works for our validation set that has pseudo-clean, balanced and informative samples.
Acknowledgments
G.C. and C.N. supported by the Engineering and Physical Sciences Rescarch Council (EPSRC) through grant EP/Y018036/1.
References
G. Algan and I. Ulusoy, Meta soft label generation for noisy labels. In International Conference on Pattern Recognition, pp. 7142-7148, 2021.
Eric Arazo, Diego Ortego, Paul Albert, Noel O'Connor, and Kevin Mcguinnoss. Unsupervised label noise modeling and loss correction. In International Conference on Machine Learning, pp. 312 321, 2019.
HeeSun Bac, Seung-Jae Shin, Joonho Jang, Byeonghu Na, Kyungwoo Song, and II-hhul Moon. From noisy prediction to true label: Noisy prediction calibration via generative model. In International Conference on Machine Learning, 2022.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Aréchiga, and Tengyu Ma. Learning imbalanced datascts with labelldiistrbbution-awaa margin loss. In Advances in Neural Information Processing Systems, 2019.
Kaidi Cao, Yining Chen, Junwei Lu, Nikos Aréchiga, Adrien Gaidon, and Tengyu Ma. Hoteroskodastic and imbalanced deep learning with adaptive regularization. In International Conference on Learning Representations, 2021.
18

---

Published in Transactions on Machine Learning Research (03/2025)
Nontawat Charoenphakloe, Jongyeong Lee, and Masashi Sugiyama. On symmetric losses for learning from corrupted labels. In International Conference on Machine Learning, volume 97, pp. 961-970. PMLR, 2019. URL https://proceediing mlr. /
Can Chen, Shuhao Zheng, Xi Chen, Erqun Dong, Xue Liu, Hao Liu, and Dejing Dou. Generalized data weighting via class-level gradient manipulation. In Advances in Neural Information Processing Systems, 2021a.
Pengfei Chen, B. Liao, Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. In International Conference on Machine Learnäng, 2019.
Pengfei Chen, Junjie Ye, Guangyong Chen, Jingwei Zhao, and Pheng-Ann Heng. Beyond class-conditional assumption: A primary attempt combat instance-dependent label noise. In AAAI Conference on Artificial Intelligence, 2021b.
Hsin-Pine Ping Chou, Shih-Chieh Chang, Jia- Pan, Wei Wei, and Da- Da-Cheng Juan. Remix: Rebalanced mixup. Tn European Conference on Computer Vision, 2020.
Peng Chu, Xiao Bian, Shaopeng Liu, and Haibin Ling. Feature space augmentation for long-tailed data. In European Conference on Computer Vision, 2020.
Filipe R. Cordciro, Vasilcios Belagiannis, Ian D. Reid, and G. Carnciro. Propmix: Hard sample filtering and proportional mixup for learning with noisy labels. In Brilish Machine and Vision Conference, 2021.
M. Dehghani, Aliaksei Severyn, Sascha Rothe, and J. Kamps. Avoiding your teacher's mistakes: Training neural networks with controlled weak supervision. ArXin, abs/1711.00% 100333, 2017a.
M. Dehghani, Aliaksei Severyn, Sascha Rothe, and J. Kamps. Learning to learn from weak supervision by full supervision. In NIPS Workshop on Meta-Learning, 2017b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hicrarchical image database. In Conference on Computer Vision and Pattern Recognition, pp. 248 255, 2009.
Charles Peter Elkan. The foundations of cost-sensitive learning. In International Joint Conference
Artificial Intelligence, 2001.
Erik Englesson and Hossein Azizpour. Generalized Jensen- Shannon divergence loss for learning with noisy labels. In Advances in Neural Information Processing Systems, 2021.
Chen Feng, Georgios Tzimiropoulos, and 1. Patras. SSR: An efficient and robust framework for learning with unknown label noise. In British Machine Vision Conference, 2021.
Aritra Ghosh, Himanshu Kumar, and P Shanti Sastry. Robust loss functions under label noise for deep neural networks. In AAAI Conference on Artificial Intelligence, volume 31, 2017.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in Newral Information Processing Systems, pp. 8527-8537, 2018.
Bo Han, Gang Niu, Jiangchao Yao, Xingrui Yu, Miao Xu, Ivor Tsang, and Masashi Sugiyama. Pumpout: A meta approach for robustly training deep neural networks with noisy labels. In ICML Workshop on Uncertainty and Robustness in Deep Learning, 2019.
Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Learning deep representation for imbalanced classification. In Conference on Computer Vision and Pottern Recognition, pp. 5375 5384, 2016.
Zhizhong Huang, Junping Zhang, and Hongming Shan. Twin contrastive learning with noisy labels. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11661-11670, 2023. doi: D0.119/CVVR527200
19

---

Published in Transactions on Machine Learning Research (03/2025)
Ahmet Iscen, Jack Valmadre, Anurag Arnab, and Cordelia Schmid. Learning with neighbor consistency for noisy labels. Conference on Compuler Vision and Pallern Recognition, pp. 4662-4671, 2022.
Lee Jaehwan, Yoo Donggeun, and Kim Hyo-Eun. Photometric transformer networks and label adjustment for breast density prediction. In International Conference on Computer Vision Workshops, pp. 0-0, 2019.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International Conference on Machine Learning, 2018.
Lu Jiang, Di Huang, Mason Liu, and Weilong Yang. Beyond synthetic noise: Deep learning on controlled noisy labels. In International Conference on Machine Learning, 2020.
Nazmul Karim, Mamshad Nayeem Rizve, Nazanin Rahnavard, Ajmal S. Mian, and Mubarak Shah. UNICON: Combating label noise through uniform selection and contrastive learning. Conference on Computer Vision and Paltern Recogrition, pp. 9666 9676, 2022.
Shyamgopal Karthik, Jérôme Revaud, and Chidlovskii Boris. Learning from long-tailed data with noisy la- bels. In ICCV Workshop: Self-superoised Learning for Nent-Generalion Induslry- Aulonomous Driv- ing, 2021.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario, 2009.
Kimin Lee, Sukmin Yun, Kibok Lee, Honglak Lee, Bo Li, and Jinwoo Shin. Robust inference via generative classifiess for handling noisy labels. In International Conference on Machine Learning, pp. 3763-3772.
PMLR, 2019a.
Kimin Lee, Sukmin Yun, Kibok Lee, Honglak Lee, Bo Li, and Jinwoo Shin. Robust inference via generative classifiers for handling noisy labels. In International Conference on Machine Learning, pp. 3763-3772. PMLR, 2019b.
Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. In International Conference on Learning Representations, 2020.
Wen Li, Limin Wang, Wei Li, E. Agustsson, and L. Gool. WebVision database: Visual learning and under- standing from web data. ArXiv, abs/1706 02280662, 2017.
Jialun Liu, Yifan Sun, Chuchu Han, Zhaopeng Dou, and Wenhui Li. Deep representation learning on long- tailed data: A learnable embedding augmentation perspective. In Conference on Computer Vision and Pallern Recognition, pp. 2967 2976, 2020.
I. Loshchilov and F. Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017.
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In International Conference on Learning Representations, 2021.
Qiguang Miao, Ying Cao, Ge Xia, Maoguo Gong, Jiachen Liu, and Jianfeng Song. RBoost: Label noise- robust boosting algorithm based on a nonconvex loss function and the numerically stable base learners. IEEE Transactions on Neural Networks and Learning Systems, 27(11):2216 22222 2015.
Baharan Mirzasoleiman, Kaidi Cao, and Jure Leskovec. Coresets for robust training of deep neural networks against noisy labels. In Advances in Neural Information Processing Systems, volume 33, pp. 11465-11477, 2020.
Tam Nguyen, C Mummadi, T Ngo, L Beggel, and Thomas Brox. SELF: learning to filter noisy labels with self-ensembling. In International Conference on Learning Representations, 2020.
20

---

Published in Transactions on Machine Learning Research (03/2025)
Diego Ortego, Eric Arazo, Paul Albert, Noel E O'Connor, and Kevin McGuinness. Towards robust learning with different label noise distributions. In International Conference on Pollern Recognilion, pp. 700-00007000 LEEE, 2021a.
Diego Ortego, Eric Arazo, Paul Albert, Noel E O'Connor, and Kevin McGuinness. Multi-objective interpo- lation training for robustness to label noise. In Conference on Computer Vision and Pullern Recognition, pp. 6606 6615, 2021b.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In Conference Computer Vision and Pallern Recognilion, pp. 2233 2241, 2017.
Foster J. Provost. Machine learning from imbalanced data sets 101. In AAAI Workshop on Imbolanced Dala Sets, 2008.
Mengye Ren, Wenyuan Zeng, Binh Yang, and R. Urtasun. Learning to reweight examples for robust deep learning. In International Conference on Machine Learning, 2018.
Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meia-weightt-el: Learning an explicit mapping for sample weighting. In Advances in Neurol Information Processing Systems, 2019.
Brandon Smart and G. Carneiro. Bootstrapping the relationship between images and their clean and noisy labels. Winter Conference on Applications of Computer Vision, pp. 5333-5343, 2022.
Yanmin Sun, Mohamed S. Kamel, Andrew Wong, and Yang Wang. Cost-sensitive boosting for classification of imbalanced data. Pattern Recognition, 40:3358 3378, 12 2007.
Yuanpeng Tu, Boshen Zhang, Yuxi Li, Liang Liu, Jian Li, Yabiao Wang, Chengjie Wang, and Cai Rong Zhao. Learning from noisy labels with decoupled meta label purifier. In IEEE/CVP Conference on Computer Vision and Pallern Recognalion, pp. 19934 19943, 2023.
Tao Wang, Yu Li, Bingyi Kang, Junnan Li, Jun Hao Liew, Sheng Tang, Steven C. H. Hoi, and Jiashi Feng. The devil is in classification: A simple framework for long-tail instance segmentation. In European Conference on Computer Vision, 2020.
Xinshao Wang, Yang Hua, Elyor Kodirov, and N. Robertson. IMAE for noise-robust learning: Mean absolute error does not treat examples equally and gradient magnitude's variance matters. ICLR 2023 Workshop on Trustworthy and Reliable Large-Scale Machine Learning Models, 2023.
Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In International Conference on Computer Vision, pp. 322 330, 2019.
Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Learning to model the tail. In Advances on Neural Information Processing Systems, 2017.
Tong Wei, Jiang-
Main-XXi &e Xin Shi, Wei- Wei Tu, and Yu-Feng Li. Robust long-tailes tailed learning under label noise, ArXiv, abs/2108.1 11s62 2021.
Tong Wu, Ziwei Liu, Qingqiu Huang, Yu Wang, and Dahua Lin. Adversarial robustness under longetailed distribution. In Conference Compuler Vision and Paltern Recognilion, pp. 8655-8664, 2021.
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Conference on Computer Vision and Pattern Recognition, pp. 2691-2699, 2015.
Youjiang Xu, Linchao Zhu, Lu Jiang, and Yi Yang. Faster meta update strategy for noise-robust deep learning. In Conference Compuler Vision and Pottern Recognition, 2021.
21

---

Published in Transactions on Machine Learning Research (03/2025)
Yu Yao, Tongliang Liu, Mingming Gong, Bo Han, Gang Niu, and Kun Zhang. Instance-dependent label- noise learning under structural causal model. Advances in Neural Information Processing Systems, 34: 4409 4420, 2021.
Bodi Yuan, Jianyu Chen, Weidong Zhang, Hung-Shuo Tai, and Sara McMains. Tterative cross learning on noisy labels. In Winter Conference Applications of Computer Vision, pp. 757-765, 2018.
Yuhang Zang, Chen Huang, and Chen Change Loy. FASA: Feature augmentation and sampling adaptation for long-tailed instance segmentation. In International Conference on Computer Vision, 2021.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
Xiao Zhang, Zhiyuan Fang, Yandong Wen, Zhifeng Li, and Yu Qiao. Range loss for deep face recognition with long-tailed training data. In International Conference on Compuler Vision, pp. 1428, 2017.
Yilan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey. IEEE Transactions on Pottern Analysis and Machine Intelligence, 45(9):10795 10816, 2023.
Zhilu Zhang and Mert Sabuncu. Generalized cross ent loss for training deep neural networks with noisy labels. In Advances in Neural Information Processing Systems, pp. 8778 8788, 2018.
Zizhao Zhang and Tomas Pfister. Learning fast sample re-weighting without reward data. In International Conference on Computer Vision, 2021.
Zizhao Zhang, Han Zhang, Sercan Ö. Arik, Honglak Lee, and Tomas Plister, Distilling effective supervision from severe label noisc. In Conference Computer Vision and Pattern Recognition, pp. 9291-9300, 2020.
Ganlong Zhao, Guanbin Li, Yipeng Qin, Feng Liu, and Yizhou Yu. Centrality and consistency: two-stage clean samples identification for learning with instance-dependent noisy labels. European Conference on Computer Vision, pp. 21 21-37. Springer, 2022.
Evgenii Zheltonozhskii, Chaim Baskin, Avi Mendelson, Alex M Bronstein, and Or Litany. Contrast to divide: Self-supervised pre-training for learning with noisy labels. In Winter Conference on Applications of Computer Vision, pp. 1657 1667, 2022.
Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. BBN: bilateral-branch network with cumulative learning for long-tailed visual recognition. Conference on Computer Vision and Pattern Recognation, pp. 9716-9723) 2020.
Zhi- Hua Zhou and Xu- Ying Liu. Training cost-sensitive neural networks with methods addressing the class imbalance problem. IEFE Transactions on Knowledge and Data Engineering, 18(1 (8(1):63-77, 2006.
22

---

Published in Transactions on Machine Learning Research (03/2025)
A Real world noise results
Table 10: Prediction accuracy (%) on real-world datasets. (eft):: WebVision with Resnet50, evaluated on Webvision and ImageNet test sets; and (right): Red Mini-ImageNet. The results of other methods are from (Zhang & Pfister, 2021; Cordciro et al., 2021) or from original papers.
WibVision
ImageNett
Method
top-1
top-5
top-1
top-5
Noise ratio
Method
HAR
75.5
90.7
57.4
82.4
0.2
0.4
0.6
0.8
Co-teaching
63.6
85.2
61.5
84.7
Iterative-CV
65.2
85.3
61.6
85.0
CE
47.36
42.70
37.30
29.76
MentorNet
63.0
81.4
63.8
85.8
Mix Up
49.10
46.40
40.58
33.58
CRUST
72.4
89.6
67.4
87.8
DivideMix
50.96
46.72
43.14
34.50
GJS
78.0
90.6
74.4
91.2
MentorMix
51.02
47.14
43.80
33.46
MW-Net
74.5
88.9
72.6
88.8
PropMix
61.24
56.22
52.84
43.42
UNICON
77.6
93.4
75.3
93.7
FaMUS
51.42
48.06
45.10
35.50
MOIT
78.8
INOLML
63.23
58.21
53.39
45.32
FSR
74.9
88.2
72.3
87.2
INOLML
81.7
93.8
78.1
92.9
Implementation Details
All CIFAR experiments use batches size 100, which are trained on a single GPU. Similar to the Distill
noise model (Zhang et al., 2020), we use p = 5, = 20 for CIFAR experiments, except the ones with the imbalance setting.
For Red Mini-ImageNNt experiments, trained the model single GPU with batches of size 100, with 5, 10.
For the WebVision experiment, we use 4. = with 4 NVIDIA V100 GPU and batches of size 16. All experiments use N 200, = 50, x = 0.9.
practice, to reduce the computational cost of the optimisation in (9), we replace the pseudo-clean set D(e) with the following subset:
DO = -{ (x..))): (x.yy.) D(e) argmax Vi(k) argmax Vi(kk) )
kE(1....C) kE.,...,C)
In summary, we aim to progressively refine the pseudo clean set D(e), making it cleaner time. D(e) is a subset of D(e), containing N randomly-selected samples (z) of each class in D(c) that have their observed labels y, consistent with the corresponding moving average robust label computed with the average prediction over the last E epochs, as in:
y, k.. + (1
fo(x:),
with x E [] being a hyper-parameter.
C Additional Results of Symmetric Noise CIFAR Datasets
We provide additional symmetric noise results of our proposed method and the Distill model (Zhang al., 2020) in Table 11. Note that method is markedly better than Distill, particularly for the simpler model
23

---

Published in Transactions on Machine Learning Research (03/2025)
(RN29) with few samples per class (1 and in the validation set. For the more complex model (WRN) and large validation set (10 samples per class), our method is still better than Distill, except for CIFARI00 at 0.8 symmetric noise rate.
As shown in Table 11, our method achieves only moderate improvements over Distill when the validation set size is large (10 samples per class). However, demonstrates significant improvements when the validation set size smaller, or the noise rate is higher (e.g., 11.5% improvement for CIFAR-100 with 0.8 uniform noise and 4% improvement for CIFAR-100 with 0.2 uniform noise, using 5 samples per class in the validation set). Similarily, as shown in Table 5. This property highlights that our method performs better in scenarios with an extreme scarcity of clean data, especially when the dataset is both noisy and imbalanced. A possible explanation for this phenomenon is that when the number clean training samples limited, the informativeness each sample becomes more critical for training compared scenarios with abundant clean training data.
method demonstrates significant improvements imbalanced and noisy benchmarks,
Table 11: Test accuracy (in %) comparison between our method ("INOLML") and the Distill noise ("DN") on symmetric noise using 1, 5 and 10 samples per class in the validation set on two backbone models: Resnet29 (*RN29") and 11deresnet28-10 ("WRN"). The results the Distill model with WideResnet28-- are collected from (Zhang et al., 2020). Recall that the Distill needs clean set, while INOLML works with pseudo-clean set.
Dataset
Val. Set size
Method
CIFAR10
CIFAR100
0.2
0.4
0.8
0.2
0.4
0.8
DN-RN29
87.0 + 0.5
85.3 + 0.5
FAIL
58.9
0.5
55.8 = 0.7
FAIL
1
INOLLLRR229
90.3 + 0.2
89.1 + 0.5
79.1 0.3
65.9
0.2
61.5 - 0.4
55.1 - 0.6
DN-RN29
90.7 + 0.3
89.0 + 0.3
83.5 + 0.2
62.6 + 0.4
58.8 + 0.5
48.5 - 0.5
5
INOLMI-RN29
90.9 + 0.2
90.9 + 0.1
87.4 + 0.2
66.6 + 0.1
65.7 = 0.1
59.0 = 0.5
DN-RN29
91.0 + 0.2
89.2 = 0.1
87.0
0.1
63.7
0.2
60.5 = 0.2
57.5 = 0.5
10
INOTMI-RN29
92.2 + 0.1
91.0 + 0.1
87.9 + 0.2
67.1
0.1
66.3 + 0.1
59.2 - 0.2
DN-WRN
95.4 + 0.6
94.5 +
87.9
77.4
0.4
75.5 + 1.1
62.1 - 1.2
INOLML- WRN
96.0 + 0.2
95.9 + 0.2
94.3 0.2
81.6
0.2
79.5 = 0.2
73.6 = 0.3
DN-WRN
96.4 + 0.0
95.5 + 0.6
91.8 + 3.0
80.4 + 0.5
79.6 = 0.3
73.6 = 1.5
5
INOLML-WIR
96.4 + 0.1
96.2 + 0.1
94.6 + 0.2
82.2 + 0.2
81.5 - 0.2
74.5 - 0.2
DN-WRN
96.2 + 0.2
95.9 + 0.2 93.7 + 0.5
81.2
0.7
80.2 + 0.3
75.5 + 0.2
10
LNOL WRN
96.9 - 0.1
96.6 - 0.1
95.0
82.0
81.3 + 0.2
74.7 = 0.1
24

---

Published in Transactions on Machine Learning Research (03/2025)
D Additional Results of Semantic Noise on CIFAR100 Dataset
We provide additional results for semantic noise introduced by RoG (Lee et al., 2019a) comparing our proposed method and the Distill model (Zhang et al., 2020) in Table 12. Both methods utilize the ResNet29 architecture, with validation set sizes ranging from 1, 5, to 10 samples per class.
Table 12: Test accuracy (%) of INOLML CIFAR100 with the semantic noise introduced by RoG (Lee et al., 2019b), in comparison with Distill using validation set D(e) of sizes 1, and 10 samples per class on Resnet29 model. The superscript indicates the need for clean validation sets.
Method
[D(n)]
Accuracy
Distill
58.1 + 0.1 61.5 + 0.2
x C
INOLML
Distil
61.3 + 0.3
INOLML 5 C'
63.1 + 0.3
Distill
61.5 + 0.3
C
INOLML
63.5 0.1
25


Reviews:
[{"review_id": "EiPOnSWQMB", "reviewer": "Reviewer_n416", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "(S1) The experiment results are impressive. \n\n(S2) The approach is simple, intuitive, and well-explained, with some design choices supported by ablation studies.\n\n(S3) The paper is generally easy to follow. \n\n(W1) Not all three utility criteria (informativeness, class balance, and label cleanliness) appear strictly necessary for all scenarios. For example, incorporating the label cleanliness objective may not be essential in imbalanced settings without noisy labels, potentially leading to unnecessary computational overhead. In practical applications, it may not be clear whether the dataset is imbalanced, noisy, or both. How do the authors propose handling such situations?\n\n(W2) There is a consistent use of \"optimisation\" throughout the paper, which appears to be a typographical preference.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a novel meta-learning approach, INOLML, designed for validation set selection in the context of imbalanced and noisy-label datasets. INOLML is guided by three core principles: sample informativeness, class balance, and label cleanliness. Extensive experimental results on noisy-label and class-imbalanced benchmarks highlight the effectiveness of the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weakness", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2ZxEnZZWDS", "reviewer": "Reviewer_Mm2x", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n* The proposed method for automatically curating high-quality validation data for meta-learning is relevant to TMLR readerships, and the proposed criteria are, at least to me, interesting.\n* The results demonstrate that SoTA performance is achieved in many situations.\n\n## Weakness\n* [Section3.3] The explanation of the existing method,  which the proposed method relies on, is too concise, making it difficult to understand the ideas behind each component. \n* [section6] The validation of the proposed method's idea is somewhat insufficient. In particular, while the proposed method significantly outperforms other methods when the class-imbalance ratio is high, the mechanism behind this advantage has not been adequately examined.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, a meta-learning (ML) method for training classifiers using noisy and class-imbalanced labels. The previous method ``Distill'' proposed by Zhang et al. has provided an ML framework that meta-learns both sample importance weights and NN-based label correction for noisy samples. However, this method relies on a clean manually crafted validation set, which is costly in large-scale training. \n\nTo address this issue, this study introduces a novel method to automatically construct a high-quality validation set based on three criteria: (i) sample informativeness, (ii) class-balanced distribution, and (iii) label cleanliness. Notably, the informativeness and label cleanliness criteria are original contributions introduced by the authors. The performance of the proposed method is comprehensively evaluated using standard experimental setups similar to existing research. The results demonstrate that the proposed method outperforms previous methods in most cases. Additionally, section 6 presents experimental evidence supporting the validity of the proposed method's key ideas.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* [Section 3.3] Please extend the explanation of _Distill_. Although the key equations are shown in the text, the idea behind each content is not clearly explained. What is the idea behind introducing $\\lambda$ and pseudo-labels? Of course, consulting Zhang et al. solves this, however, those should be explained in this paper as the proposed method heavily relies on the ``Distill'' framework.\n - Also, ** where $\\lambda_i=0.9$** below equation (5) is weird as it should be optimized.\n* [Section 5-6] The proposed method seems to be mostly effective when the class-imbalance ratio is large, while the performance improvement over ``Distill'' is moderate in class-balanced cases. Please add an explanation on this point. Why the proposed method is so effective in class-imbalanced cases (or equivalently the previous methods are ineffective)? Could you conduct some ablation studies that directly validate the mechanism by which the proposed method operates in label-imbalanced cases?", "broader_impact_concerns": "No broader concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kdVA0n4jcp", "reviewer": "Reviewer_YQ3e", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nThe method is intuitive and not too complicated. \n\nThe experiment is rather extensive (in terms of baselines).\n\nWeaknesses:\n\nThe overall approach sometimes looks a little brute-force/naive, in particular the \"greedy strategies\" mentioned right after Eq(14). This is not an issue by itself, but I'd like to see some experiments by replacing these with simple variants. \n\nSome baselines look a little old and the proposed method is not always better.\n\nI worry that the proposed method encodes some implicit bias about how the noisy dataset is constructed. For example, the cleanliness heuristic may not work if the label is noisy in a different manner, or is similar in a manifold not captured by $z$. I understand that this is, however, very hard to test.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new automatic meta-learning pipeline that involves choosing samples from the validation set using a utility function that balances 3 objectives (sample informativeness, label cleanliness and balanced class distribution). Experiment results show that the proposed method", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The writing of argmax equations (e.g. Eq (9)) is nonstandard. It is unusual to use the same symbol for the left-hand-side as well as the quantify being argmax'd. Something like $x^* = argmax_{x} f(x)$ would make more sense. \n\nAdditional experiments on some implementation choices would be good.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Initially, the reviewers expressed concerns about two main aspects of the submission. After the rebuttal, the AE confirmed that the paper now meets expectations. The major concerns of the reviewers have been addressed.  Specifically,

- *Explanation of Baseline Components.*
The reviewers noted that the description of the baseline framework was too concise. For instance, Reviewer Mm2x pointed out that while the paper included key equations, the underlying rationale such as the role of pseudo-labels and the mechanism for differentiating between the original and corrected labels was not adequately explained. Reviewer n416 similarly suggested that extending the discussion on how pseudo-labels contribute to handling noisy data would help readers understand the baseline method's strengths and limitations. In response, the authors have substantially expanded Section 3.3, providing a detailed breakdown of the Distill framework, and clarifying why each component (including the fixed label for supervision and the optimized parameter for label correction) is crucial for achieving robust performance.

- *Optimization Strategy for Validation Set Selection.*
The second area of concern was the use of a greedy optimization strategy for selecting validation samples. Both Reviewer YQ3e and Reviewer Mm2x expressed that the “greedy” approach, particularly as presented immediately after Equation (14), appeared somewhat naïve and warranted a deeper justification. The reviewers recommended including additional experiments such as comparisons with alternative strategies (e.g., a simple min-loss selection) to validate the effectiveness of the proposed optimization mechanism. The authors addressed this by incorporating additional experiments and ablation studies that compare the original greedy approach with other methods.
Decision: Accept (poster)