Mitigating the Impact of Labeling Errors on Training via Rockafellian Relaxation

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 02:34:36

Abstract: Labeling errors in datasets are common, if not systematic, in practice. They naturally arise in a variety of contexts—human labeling, noisy labeling, and weak labeling (i.e., image classification), for example. This presents a persistent and pervasive stress on machine learning practice. In particular, neural network (NN) architectures can withstand minor amounts of dataset imperfection with traditional countermeasures such as regularization, data augmentation, and batch normalization. However, major dataset imperfections often prove insurmountable. We propose and study the implementation of Rockafellian Relaxation (RR), a new loss re-weighting, architecture-independent methodology, for neural network training. Experiments indicate RR can enhance standard neural network methods to achieve robust performance across classification tasks in computer vision and natural language processing (sentiment analysis). We find that RR can mitigate the effects of dataset corruption due to both (heavy) labeling error and/or adversarial perturbation, demonstrating effectiveness across a variety of data domains and machine learning tasks.

Main paper: 
 introduction: 
 Labeling errors are systematic in practice, stemming from various sources. For example, the reliability of human-generated labels can be negatively impacted by incomplete information, or the subjectivity of the labeling task -as is commonly seen in medical contexts, in which experts can often disagree on matters such as the location of electrocardiogram signal boundaries [8] , prostate tumor region delineation, and tumor grading [20] . As well, labeling systems, such as Mechanical Turk1 often find expert labelers being replaced with unreliable non-experts [27] . For all these reasons, it would be advisable for any practitioner to operate under the assumption that their dataset is corrupted with labeling errors, and possibly to a large degree.In this paper, we propose a loss-reweighting methodology for the task of training a classifier on data having higher levels of labeling errors. We show that our method relates to optimistic and robust distributional optimization formulations aimed at addressing adversarial training (AT). These findings underscore our numerical experiments on NNs that suggest this method of training can provide test performance robust to high levels of labeling error, and to some extent, feature perturbation. Overall, we tackle the prevalent challenges of label corruption and class imbalance in training datasets, which are critical obstacles for deploying robust machine learning models. Our proposed approach implements Rockafellian Relaxations [23] to address corrupted labels and automatically manage class imbalances without the need for clean validation sets or sophisticated hyper-parameters -common constraints of current methodologies. This distinct capability represents our key contribution, making our approach more practical for handling large industrial datasets. We proceed to discuss related works in section 2, and our specific contributions to the literature.In section 3 we discuss our methodology in detail and provide some theoretical justifications that motivate the effectiveness of our methodology. The datasets and NN model architectures upon which our experimental results are based are discussed in sections 4 and 5, respectively. We then conclude with numerical experiments and results in section 6.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: Sections 6.1, 6.2, 6.3, 6.4Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Corrupted datasets are of concern, as they potentially pose severe threats to classification performance of numerous machine-learning approaches [36] , including, most notably, NNs [15, 33] . Naturally, there have been numerous efforts to mitigate this effect [28, 8] . These efforts can be categorized into robust architectures, robust regularization, robust loss function, loss adjustment, and sample selection [28] . Robust architecture methods focus on developing custom NN layers and dedicated NN architectures. This differs from our approach, which is architecture agnostic and could potentially "wrap around" these methods. While robust regularization methods like data augmentation [26] , weight decay [16] , dropout [29] , and batch normalization [14] can help to bolster performance, they generally do so under lower levels of dataset corruption. Our approach, on the other hand, is capable of handling high levels of corruption, and can seamlessly incorporate methods such as these. In label corruption settings, it has been shown that loss functions, such as robust mean absolute error (MAE) [10] and generalized cross entropy (GCE) [35] are more robust than categorical cross entropy (CCE). Again, our method is not dependent on a particular loss function, and it is possible that arbitrary loss functions, including robust MAE and GCE, can be swapped into our methodology with ease. Our approach resembles the loss adjustment methods most closely, where the overall loss is adjusted based on a (re)weighting scheme applied to training examples.In loss adjustment methods, individual training example losses are typically adjusted multiple times throughout the training process prior to NN updates. These methods can be further grouped into loss correction, loss reweighting, label refurbishment, and meta-learning [28] . Our approach most closely resembles the loss reweighting methods. Under this scheme each training example is assigned a unique weight, where smaller weights are assigned to examples that have likely been corrupted. This reduces the influence of corrupted examples. A training example can be completely removed if its corresponding weight becomes zero. Indeed, a number of loss reweighting methods are similar to our approach. For example, Ren et al., [22] learn sample weights through the use of a noise-free validation set. Chang et al. [5] assign sample weights based on prediction variances, and Zhang et al. [34] examine the structural relationship among labels to assign sample weights.However, we view the need for a clean dataset, or at least one with sufficient class balance, by these methods as a shortcoming, and our method, in contrast, makes no assumption on the availability of such a dataset. Satoshi et al. [12] propose a two-phased approach to noise cleaning. The first phase trains a standard neural network to determine the top-m most influential training instances that influence the decision boundary; these are subsequently removed from the training set to create a cleaner dataset. In the second phase, the neural network is retrained using the cleansed training set. Their method demonstrates superior validation accuracy for various values of m on MNIST and CIFAR-10. Although impressive, their method does not address the fact that most industrial datasets have a reasonably large amount of label corruption [28] which, upon complete cleansing, could also remove informative examples that lie close to the decision boundary. Additionally, the value of m is an additional hyper-parameter that could require significant tuning on different datasets and sources. c) Finally, compute the exemplar weights that temper the training loss computed in step two with validation loss. This approach is algorithmically the most similar to ours, with some key differences.The major difference is that it treats noise and class imbalance similarly. Our approach deals with noisy labels explicitly and can cope with almost any amount of class imbalance automatically, as tested in our experiments with the open-source Hate-Speech dataset, where we experimented with different prevalence levels of Hate-Speech text. The biggest drawback of the method proposed by Mengye et al. is that it requires a clean validation set, which in practice is almost impossible to obtain; if it were possible, it would not be very prohibitive to clean the entire dataset. Noise, typically, is an artifact of the generative distribution which cannot be cherry-picked as easily in practice. Our approach does not require a clean dataset to be operational or effective.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: The paper has focused more on label corruption, rather than feature perturbation settings.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
mislabeling: 
 Let X denote a feature space, with Y a corresponding label space. Then Z := X × Y will be a collection of feature-label pairs, with an unknown probability distribution D. Throughout the forthcoming discussions, {(x i , y i )} N i=1 will denote a sample of N feature-label pairs, for which some pairs will have a mislabeling. More precisely, we begin with a collection (x i , ỹi ) drawn i.i.d. from D, but there is some unknown set C {1, . . . , N } denoting (corrupted) indices for whichy i = ỹi if and only if i / ∈ C.For those i ∈ C, y i is some incorrect label, selected uniformly at random, following the Noise Completely at Random (NCAR) model [8] also known as uniform label noise.We adopt the empirical risk minimization (ERM) [31] problem formulation:EQUATIONas a baseline against which our method is measured. Given an NN architecture with (learned) parameter setting θ that takes as input any feature x and outputs a prediction ŷ, J(θ; x, y) is the loss with which we evaluate the prediction ŷ with respect to y. Finally, r(θ) denotes a regularization term.In ERM it is common practice to assign each training observation i a probability p i = 1/N . However, when given a corrupted dataset, we may desire to remove those samples that are affected; in other words, if C {1, ..., N } is the set of corrupted training observations, then we would desire to set the probabilities in the following alternative way:EQUATIONwhere |C| is the cardinality of the unknown set C. In this work, we provide a procedure -the Rockafellian Relaxation Method (RRM) -with the intention of aligning the p i values closer to the desired (but unknown) p of (2) in self-guided, automated fashion. It does so by adopting the Rockafellian Relaxation approach of [23] . More precisely, we consider the problemEQUATIONwhereU := {u ∈ R N : N i=1 u i = 0, 1 N + u i ≥ 0 ∀i = 1, .. . , N }, and some γ > 0.We proceed to comment on this problem that is nonconvex in general, before providing an algorithm.Although problem (3) is nonconvex in general, the computation of v(θ) for any fixed θ amounts to a linear program. The following result characterizes the complete set of solutions to this linear program, and in doing so, provides an interpretation of the role that γ plays in the loss-reweighting action of RRM. EQUATIONThe theorem explains that the construction of any optimal solution u * essentially reduces to categorizing each of the losses among {c i = J(θ; x i , y i )} N i=1 as "small" or "big", according to theirposition in the partitioning of [c min , ∞) = [c min , c min + 2γ) ∪ [c min + 2γ, ∞).For losses that occur at the break points of c min and c min + 2γ, this classification can be arbitrary -hence, the use of S 1 and S 2 set configurations to capture this degree of freedom.In particular, those points with losses c i exceeding c min + 2γ are down-weighted to zero and effectively removed from the dataset. And in the event that c max -c min < 2γ, no loss reweighting occurs. In this manner, while lasso produces sparse solutions in the model parameter space, RRM produces sparse weight vectors by assigning zero weight to data points with high losses.Consequently, if χ := {i : c i ∈ (c min + 2γ, ∞)} converges over the course of any algorithmic scheme, e.g., Algorithm 1, to some set C, then we can conclude that these data points are effectively removed from the dataset even if the training of θ might proceed. This convergence was observed in the experiments of Section 6. It is hence of possible consideration to tune γ for consistency with an estimate α ∈ [0, 1] of labeling error in the dataset {(x i , y i )} N i=1 . More precisely, we may tune γ so that |χ| N ≈ α.In this section, we discuss RRM's relation to distributionally robust and optimistic optimization formulations. Indeed, (3)'s formulation as a min-min problem bears resemblance to optimistic formulations of recent works, e.g., [19] . We will see as well that the minimization in u, as considered in Theorem 3.1, relates to an approximation of a data-driven Wasserstein Distributionally Robust Optimization (DRO) formulation [30] .For this discussion, as it relates to reweighting, we will lift the feature-label space Z = X × Y.More precisely, we let W := R + denote a space of weights. Next, we say W × Z has an unknownprobability distribution D such that π Z D = D and Π W D({1}) = 1.In words, all possible (w.r.t.D) feature-label pairs have a weight of 1. Finally, we define an auxiliary loss ℓ :W × Z × Θ by ℓ(w, z; θ) := w • J(x, y; θ), for any z = (x, y) ∈ Z. Given a sample {(1, x i , y i )} N i=1, just as in Section 3.2, we can opt not to take as granted the resulting empirical distribution D N because of the possibility that |C|-many have incorrect labels (i.e., y i = ỹi ). Instead, we will admit alternative distributions obtained by shifting the D N 's probability mass off "corrupted" tuples (1, x i , y i ) i∈C to possibly (0, x i , y i ), (1, x i , ỹi ), or even some other tuple(1, x j , ỹj ) with j / ∈ C for example -equivalently, eliminating, correcting, or replacing the sample, respectively. In order to admit such favorable corrections to D N , we can consider the optimistic [19, 30] data-driven problemEQUATIONin which for each parameter tuning θ, v N (θ) measures the expected auxiliary loss with respect to the most favorable distribution within an ǫ -prescribed W 1 (1-Wasserstein) distance of D N . It turns out that a budgeted deviation of the weights alone (and not the feature-label pairs) can approximate (up to an error diminishing in N ) v N (θ). More precisely, we derive the following approximation along similar lines to [30] .Proposition 3.2. Let ǫ > 0, and suppose for any θ, max (x,y)∈Z |J(θ; x, y)| < ∞. Then there exists κ ≥ 0 such that for any θ, the following problemv M IX N (θ) := min u1,...,uN N i=1 ( 1 N + u i ) • J(θ; x i , y i ) + γ θ N i=1 |u i | s.t. u i + 1 N ≥ 0 i = 1, . . . , N satisfies v N (θ) + κ N ≥ v M IX N (θ) ≥ v N (θ).In particular, -γ θ ≤ min i J(θ; x i , y i ), and {i : J(θ; x i , y i ) > γ θ } are all down-weighted to zero, i.e., u* i = -1 N for any u * solving v M IX N (θ).In summary, while the optimistic Wasserstein formulation would permit correction to D N with a combination of reweighting and/or feature-label revision, the above indicates that a process focused on reweighting alone could accomplish a reasonable approximation; further, upon comparison to(3), we see that RRM is a constrained version of this approximating problem, that is,v(θ) ≥ v M IX N (θ) ≥ v N (θ).Hence, in some sense, we can confirm that RRM is an optimistic methodology but that it is less optimistic than the data-driven Wasserstein approach.Towards solving problem (3) in the two decisions θ and u, we proceed iteratively with a blockcoordinate descent heuristic outlined in Algorithm 1, whereby we update the two separately in cyclical fashion. In other words, we update θ while holding u fixed, and we update u whilst holding θ fixed. The update of θ is an SGD step on a batch of smany samples. The update of u reduces to a linear program. In light of the discussion in 3.4, we also outline an Adversarial Rockafellian Relaxation method (A-RRM), an execution of RRM that includes a perturbation (parameterized by ǫ ≥ 0) to the feature x of a sample (x, y), for the purposes of adversarial training.Algorithm 1 (Adversarial) Rockafellian Relaxation Algorithm (A-RRM/RRM) Require: Perturbation Multiplier ǫ ∈ [0, 1], Number of epochs σ, Batch size s ≥ 1, learning rate η > 0, regularization parameter γ > 0, reweighting step µ ∈ (0, 1).u ← 0 ∈ R N repeat for e = 1, . . . , σ do for b = 1, . . . , ⌈ N s ⌉ do {(x b i , y b i )} s i=1 ← Draw Batch of size s from {(x i , y i )} N i=1 for i = 1, . . . , s do x b i ← x b i + ǫ • sign ∇ x J(θ; (x b i , y b i )) end for θ ← θ -η s i=1 1 N + u i • ∇ θ J(θ; (x b i , y b i )) end for end for u * ← min u∈U N i=1 1 N + u i • J(θ; x i , y i ) + γ u 1 u ← µu * + (1 -µ)u until Desired Validation Accuracy or LossThe stepsize parameters µ, η and the regularization parameter γ are hyper-parameters that may be tuned, or guided by the general discussions above in Section 3.3.The RRM algorithm, in which ǫ = 0, is meant for contexts in which only label corruption and no feature corruption occurs. The A-RRM algorithm, for which ǫ > 0, is intended for contexts in which both label and feature corruption is anticipated.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes]Answer: [No] Justification:The datasets are open-source, and the code will be made available pending conference review of this work Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
datasets: 
 We select several datasets to evaluate RRM. In some cases, the selected dataset is nearly pristine. In these cases we perturb the dataset to achieve various types and levels of corruption. Other datasets consist of weakly labeled examples, which we maintain unaltered. The varied data domains and regimes of corruption enable a robust evaluation of RRM.MNIST [17] : A multi-class classification dataset consisting of 70000 images of digits zero through nine. 60000 digits are set aside for training and 10000 for testing. 0%, 5%, 10%, 20%, and 30% of the training labels are swapped for different, randomly selected digits. The test set labels are unmodified.Toxic Comments [6] : A multi-label classification problem from JIGSAW that consists of Wikipedia comments labeled by humans for toxic behavior. Comments can be any number (including zero) of six categories: toxic, severe toxic, obscene, threat, insult, and identity hate. We convert this into a binary classification problem by treating the label as either none of the six categories or at least one of the six categories. This dataset is a public dataset used as part of the Kaggle Toxic Comment Classification Challenge.IMDb [18] : A binary classification dataset consisting of 50000 movie reviews each assigned a positive or negative sentiment label. 25000 reviews are selected randomly for training and the remaining are used for testing. 25%, 30%, 40%, and 45% of the labels of the training reviews are randomly selected and swapped from positive sentiment to negative sentiment, and vice versa, to achieve four training datasets of desired levels of label corruption. The test set labels are unmodified. Tissue Necrosis: A binary classification dataset consisting of 7874 256x256-pixel hematoxylin and eosin (H&E) stained RGB images derived from [2] . The training dataset consists of 3156 images labeled non-necrotic, as well as 3156 images labeled necrotic. The training images labeled nonnecrotic contain no necrosis. However, only 25% of the images labeled necrotic contain necrotic tissue. This type of label error can be expected in cases of weakly-labeled Whole Slide Imagery (WSI). Here, an expert pathologist will provide a slide-level label for a potentially massive slide consisting of gigapixels, but they lack time or resources to provide granular, segmentation-level annotations of the location of the pathology in question. Also, the diseased tissue often occupies a small portion of the WSI, with the remainder consisting of normal tissue. When the gigapixelsized WSI is subsequently divided into sub-images of manageable size for typical machine-learning workflows, many of the sub-images will contain no disease, but will be assigned the "weak" label chosen by the expert for the WSI. The test dataset consists of 718 necrosis and 781 non-necrosis 256x256-pixel H&E images, which were also derived from [2] . For both the training and test images, [2] provide segmentation-level necrosis annotations, so we are able to ensure a pristine test set, and, in the case of the training set, we were able to identify the corrupted images for the purpose of algorithm evaluation. 
architectures: 
 We do not strive to develop a novel NN architectures capable of defeating current state-of-the-art (SOA) performance in each data domain. Nor do we focus on developing robust architectures as described in [28] . Rather, we select a reasonable NN architecture and measure model performance with and without the application of RRM. This approach enables us to demonstrate the general superiority of RRM under varied data domains and NN architectures. We discuss the underlying NN architectures that we employ in this section.The MNIST dataset has been studied extensively and harnessed to investigate novel machine-learning methods, including CNNs [4] . We adopt a basic CNN architecture with a few convolutional layers. The first layer has a depth of 32, and the next two layers have a depth of 64.Each convolutional layer employs a kernel of size three and the ReLU activation function followed by a max-pooling layer employing a kernal of size 2. The last convolutional layer is connected to a classification head consisting of a 100-unit dense layer with ReLU activation, followed by a 10-unit dense layer with softmax activation. In total, there are 159254 trainable parameters. Categorical cross-entropy is employed for the loss function.We use a simple model with only a single convolutional layer. A pretrained embedding from FastText is first used to map the comments into a 300 dimension embedding space, followed by a single convolutional layer with a kernel size of two with a ReLU activation layer followed by a max-pooling layer. We then apply a 36-unit dense layer, followed by a 6 unit dense layer with sigmoid activation. Binary cross-entropy is used for the loss function.IMDb: Transformer architectures have achieved SOA performance on the IMDb dataset sentiment analysis task [7, 32] . As such, we a adopt a reasonable transformer architecture to assess RRM. We utilize the DistilBERT [25] architecture with low-rank adaptation (LoRA) [13] for large language models, which reduces the number of trainable weights from 67584004 to 628994. In this manner, we reduce the computational burden, while maintaining excellent sentiment analysis performance.Binary cross-entropy is employed for the loss function.Tissue Necrosis: Consistent with the computational histopathology literature [21] , we employ a convolutional neural network (CNN) architecture for this classification task. In particular, a ResNet-50 architecture with pre-trained ImageNet weights is harnessed. The classification head is removed and replaced with a dense layer of 512 units and ReLU activation function, followed by an output layer with a single unit using a sigmoid activation function. All weights, with the exception of the new classification head are frozen, resulting in 1050114 trainable parameters out of 24637826.Binary cross-entropy is employed for the loss function. 
experiments and results: 
 In this work, we have discussed errors/perturbations/corruption to features and labels. We now perform experiments to see how RRM performs under one or the other, or both. The MNIST experiments are performed under a setting of both adversarial perturbation, as well as label corruption.The Toxic Comments experiments are performed under settings of label corruption only. All experiments are performed using a combination of GPU resources, both cloud-base, as well as access to an on-premise high-performance computing (HPC) facility. We refer the reader to the Appendix (Sections 6.3 and 6.4) for the experiments on IMDb and Tissue Necrosis.Twenty percent of the training data is set aside for validation purposes. Using Tensorflow 2.10 [1], 50 iterations of RRM are executed with σ = 10 epochs per iteration for a total of 500 epochs for a given hyperparameter setting. For RRM, the hyperparameter settings of µ and γ at 0.5 and 2.0, respectively, are based on a search to optimize validation set accuracy. For contrast, we perform a comparable 500 epochs using ERM. Both ERM and RRM employ stochastic gradient descent (SGD) with a learning rate (η) of 0.1. Each time a batch is drawn, each training image is perturbed using the Fast Gradient Sign Method (FGSM) [11] adversarial attack:adv x = x + ǫ • sign(∇ x J(θ, x, y)),where adv x is the resulting perturbed image, x is the original image, y is the image label, ǫ is a multiplier controlling the magnitude of the image perturbation, θ are the model parameters, and J is the loss. An ǫ = 1.0 is used for all training image perturbations.For each of the 0%, 5%, 10%, 20%, and 30% training label corruption levels, we compare adversarial training (AT) and adversarial RRM (A-RRM) performance under varios regimes of test set perturbation (ǫ test ∈ 0.0, 0.1, 0.25, 0.5, 1.0). In Table 1 we show the test set accuracy achieved when validation set accuracy peaks. We can see that training with an ǫ train = 1.0 and testing with lower ǫ test levels of 0.00, 0.10, and 0.25, results in a drastic degradation in accuracy for AT for corruption levels greater than 0%. This performance collapse is not observed when using A-RRM. Given that it may be difficult to anticipate the adversarial regime in production environments, A-RRM seems to confer a greater benefit than AT.We examine the u i -value associated with each training observation, i, from iteration-to-iteration of the heuristic algorithm. Table 2 summarizes the progression of the u i -vector across its 49 updates for the dataset corruption level of 20%. Column "1. iteration" shows the distribution of u i -values following the first u-optimization for both the 9600 corrupted training observations and the 38400 clean training observations. Initially, all u i -values are approximately equal to 0.0. It is once again observed that, over the course of iterations, the u i -values noticeably change. In column "10. iteration" it can be seen that a significant number of the u i -values of the corrupted training observations We use the Toxic Comment dataset to test the efficacy of RRM on low prevalence text data. The positive (toxic) comments consist of only 3% of the data and we corrupt anywhere from 1% to 20% of the labels. There are a total of 148,000 samples, and we set aside 80% for training and 20% for test. σ = 2 with 3 iterations of the heuristic algorithm results in a total of 6 epochs, and ERM is run for a total of 6 epochs to make the results comparable. Since the data is highly imbalanced, we look at the area under the curve of the precision/recall curve to assess the performance of the models. Unsurprisingly, as the noise increase, the model performance decreases. We note that RRM outperforms ERM across all noise levels tested, though as the noise increase, the gap between RRM and ERM decreases. 30 iterations of RRM are executed, with σ = 10 epochs per iteration for a total of 300 epochs for a given hyperparameter setting. For RRM, the hyperparameter settings of µ and γ at 0.5 and 0.4, respectively, are based on a search to optimize validation set accuracy. For contrast, we perform a comparable 300 epochs using ERM. Both ERM and RRM employ stochastic gradient descent (SGD) with a learning rate (η) of 0.001. In Table 4 we record both the test set accuracy achieved when validation set accuracy peaks, as well as the maximum test set accuracy. At these high levels of corruption RRM consistently achieves a better maximum test set accuracy. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Sections 4, 5, 6Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 In this study, we demonstrate the robustness of the A-RRM algorithm in a variety of data domains, data corruption schemes, model architectures and machine learning applications. In the MNIST example we show that conducting training in preparation for deployment environments with varied levels of adversarial attacks, one can benefit from implementation of the A-RRM algorithm. This can lead to a model more robust across levels of both feature perturbation and high levels of label corruption. We also demonstrate the mechanism by which A-RRM operates and confers superior results: by automatically identifying and removing the corrupted training observations at training time execution.The Toxic Comment example presents another challenging classification problem, characterized by a low prevalence target class amidst label noise. Our experiments demonstrate that as the amount of label noise increases, standard methods become increasingly ineffective. However, RRM remains reasonably robust under varying degrees of label corruption. Therefore, RRM could be a valuable addition to the set of tools being developed to enhance the robustness of AI-based decision engines.In the IMDb example we demonstrate that RRM can confer benefits to the sentiment analysis classification task using pre-trained large models under conditions of high label corruption. The success In the Tissue Necrosis example, we demonstrate that RRM also confers accuracy benefits to the necrosis identification task provided weakly labeled WSIs. Again, RRM can mitigate the need for expert-curated, detailed pathology annotations, which are costly and time-consuming to generate.Upon introducing the change of variable u i = wi N -1 N , and applying a Lagrange multiplier γ θ to the ǫbudget constraint (any convex dual optimal multiplier), we recovermin u1,...,uN N i=1 ℓ(u i + 1 N , z i ; θ) + γ θ N i=1 |u i | s.t. u i + 1 N ≥ 0 i = 1, . . . , NNeurIPS Paper ChecklistQuestion: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Error bars are not reported because it would be too computationally expensive.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: See section 6Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [No] Justification: The work in the paper is foundational research and is not tied to a particular application or deployment.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [No] Justification: No models are released as part of this work, and the datasets are publicly available.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Citations for publicly available datasets and code are provided.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: http://mturk.com Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS2024). Do not distribute. 


Decision: This paper studies learning from noisy labels. The idea here is to learn the weights on the datapoints (i.e., a loss reweighting method) to handle corrupted data. The authors draw some connections between this approach and existing frameworks. They do a set of experiments showing that their technique holds up better to large proportions of noisy labels compared to adversarial training and ERM.  

The paper’s main idea is reasonable, but there’s a number of improvements that are needed before the paper reaches the bar. The reviewers asked a bunch of reasonable questions that should be addressed, especially around computational complexity. Similarly, the experiments are a bit limited (and there’s really only a very small number of baselines). For example, loss reweighting dates back to Natarajan’s 2013 paper and perhaps even earlier, and follow-up work for that technique shows how to do reweighting in a very simple way with minimal or no hyperparameter sweeps. How do  these various techniques compare? One argument the authors make is that their method can act as a wrapper and therefore be used in concert with these other noisy label techniques. I can believe this, but it is good to include the evidence as part of the next version of the paper.

Reviews: review: paper_summary: paper_summary: The paper presents Rockafellian Relaxation (RR), a new method to address labeling errors in machine learning datasets. RR is a loss reweighting technique that enhances neural network robustness against labeling errors and adversarial attacks, working across various data domains and model architectures. The key contribution is an approach that mitigates label corruption and class imbalance without needing clean validation sets, offering a practical solution for training robust models.
strength_weakness: strength_weakness: - While not explicitly mentioned, the iterative nature of the RR algorithm could potentially be computationally intensive, especially for large datasets.

- The method assumes a specific model of label noise (e.g., uniform label noise), which may not hold in all real-world scenarios. 

- The paper could benefit from a more comprehensive comparison with other state-of-the-art methods for handling noisy labels, such as GCE [1], ELR[2], to better position RR in the existing literature.

[R1] Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels

[R2] Early-Learning Regularization Prevents Memorization of Noisy Labels
questions: questions: - Could the authors provide insights into the computational complexity of the RR algorithm, particularly in the context of large-scale datasets such as clothing1m?

- How does the performance of RR compare under different models of label noise, especially those that deviate from the assumed uniform label noise model such as asymmetric/instance-dependent label noise?

- How does RR perform relative to other state-of-the-art methods like Generalized Cross Entropy (GCE) and Early-Learning Regularization (ELR) in terms of handling noisy labels?

- How sensitive is the performance of RR to the choice of hyperparameters, and are there any techniques to optimize these selections effectively?
limitations: limitations: Authors have adequately addressed the limitations.
paper_summary: paper_summary: This work proposes a loss reweighting scheme to train models in the presence of label errors. When training an NN with empirical risk minimization in this setting, one would want to assign a weight of zero to all datapoints that are mislabeled and a weight of one to all datapoints that are correctly labeled. This paper presents an automated method for accomplishing this weighting, called the Rockafellian Relaxation Method (RRM). It is noted in Theorem 3.1 that the inner minimization objective of RRM reduces to a linear programming problem, despite RRM being non convex in general. After relating RRM to distributionally robust optimization techniques, the adversarial variant of RRM is introduced (A-RRM), which includes adversarial perturbations to induce adversarial training as well as loss reweighting. Experiments on four datasets show that RRM and A-RRM outperform other methods in both adversarial settings and settings with high proportions of noisy labels.
strength_weakness: strength_weakness: The experimental section lacks a relevant baseline for comparison. As it stands, it is unclear how this compares to other noise-reduction techniques. The relationship to other techniques is discussed in the related work section, it would be nice if the purported benefits of this approach were borne out empirically.

The introduction of adversarial training in section 3.5 is under-motivated. Based on the earlier sections, it is unclear how label and adversarial feature corruptions are related to each other, why we would want to achieve robustness to both, and whether previous approaches have attempted this before. I would suggest explicitly motivating this earlier in the paper.
questions: questions: - Could you expand on what you mean by RRM producing sparse weight vectors? Does assigning zero weight to data points with high losses result in sparsity in the parameter space?
- What is the reason for choosing the FGSM attack in A-RRM rather than a different attack, like PGD? Could other attacks be used in the place of FGSM?
- In Table 1, is the epsilon in the fourth row supposed to be 0.50 instead of 50?
- Do you have a hypothesis for why the test accuracy increases for models trained with AT and tested with $\epsilon_{test} = 0.50$ or $1.0$ when the level of corruption is increased (i.e. the AT columns in the bottom two rows of Table 1)? That seems like an unexpected result that may warrant further study?
limitations: limitations: The limitations are briefly discussed in the paper. As noted above, one main limitation is that it only studies $\ell_\infty$ bounded FGSM attacks. Furthermore, this paper only considers the uniform label noise model, and does not consider the case when label corruption might be correlated with features.
paper_summary: paper_summary: This paper proposes a methodology called Rockafellian Relaxation (RR) to mitigate the impact of labeling errors in neural network training. The method is architecture-independent and integrates concepts from adversarial training to address dataset imperfections robustly. Through theoretical justifications and a series of experiments on standard datasets like MNIST and Toxic Comments, the paper demonstrates that RR can significantly improve the performance of neural networks trained under various corruption levels. The paper’s contributions are particularly valuable as they provide a new tool for improving training accuracy in the presence of label noise, enhancing the robustness and applicability of machine learning models in diverse and error-prone real-world settings.
strength_weakness: strength_weakness: Computational Complexity: The added complexity might limit the practical application of the method in scenarios with constrained computational resources.
questions: questions: Scalability: How does the RRM scale in terms of computational cost and effectiveness with larger and more complex datasets?
limitations: limitations: Generalization to Different Noise Types: While the method is tested against uniform label noise, its effectiveness against other types of noise is not thoroughly investigated.

Dependence on Hyperparameter Tuning: The effectiveness of RRM is likely sensitive to the choice of hyperparameters, such as the regularization term and the parameters controlling the adversarial component. The paper does not provide extensive guidance on hyperparameter selection, which could affect the reproducibility and ease of application in different scenarios.

Comments: title: On the Matter of ELR versus RRM-wrapped ELR
comment: Thank you to the authors for addressing my questions. I have decided to maintain my score, which leans toward acceptance.

I suggest that in the revised version, the authors include additional experiments beyond MNIST, such as CIFAR, CIFAR-N, and Clothing-1M, to further support the efficacy of the proposed method.
Towards addressing your concern over the inconclusiveness of our comparison, due to a lack of hyperparameter search, we have endeavored to revisit the experiments, now with a hyperparameter search, to show how RRM-wrapping a loss methodology can enhance performance. The table below reports the test accuracy results obtained upon revisiting the experiments on MNIST3 with a hyperparameter search for all methods. 

|   |  |  |  |
|----------|----------------------|----------------------|----------------------|
|  **Method** \ **Contamination Level**         | 0.55                 | 0.60                 | 0.65                 |
| ERM      |  0.90  |  0.77   |  0.46  |
| RRM(ERM) |  **0.98**  |  **0.96**   |  **0.67**  |
| ELR      |  0.98  |  0.97   |  0.82  |
| RRM(ELR) |  **0.99**  |  **0.98**   |  **0.87**  |

We highlight that the RRM-wrapped ERM method was uniformly better than ERM across all contamination levels examined. We observed this uniform enhancement for the case of ELR as well. Of particular note, once we tuned **ELR**'s hyperparameters, those same hyperparameter settings were maintained in **RRM(ELR)** (along with the RRM-specific hyperparameters tuned). 

**These results suggest that even after hyperparameter tuning of a method like ERM or ELR, there are further enhancements to be obtained with RRM-wrapping.**
Thank you for your response to my questions. I appreciate the empirical comparison to ELR in the response to reviewer UZpK. However, I still believe that a more thorough comparison is needed, as these results are fairly inconclusive and you state that a more thorough hyperparameter search is necessary. I will therefore be keeping my current score.