Title: Learning from Noisy Labels via Conditional Distributionally Robust Optimization
Venue: NeurIPS 2024
Accepted: yes
Keywords: noisy label

conditional distributionally robust optimization (CDRO)

crowdsourcing
Abstract: While crowdsourcing has emerged as a practical solution for labeling large datasets, it presents a significant challenge in learning accurate models due to noisy labels from annotators with varying levels of expertise. Existing methods typically estimate the true label posterior, conditioned on the instance and noisy annotations, to infer true labels or adjust loss functions. These estimates, however, often overlook potential misspecification in the true label posterior, which can degrade model performances, especially in high-noise scenarios. To address this issue, we investigate learning from noisy annotations with an estimated true label posterior through the framework of conditional distributionally robust optimization (CDRO). We propose formulating the problem as minimizing the worst-case risk within a distance-based ambiguity set centered around a reference distribution. By examining the strong duality of the formulation, we derive upper bounds for the worst-case risk and develop an analytical solution for the dual robust risk for each data point. This leads to a novel robust pseudo-labeling algorithm that leverages the likelihood ratio test to construct a pseudo-empirical distribution, providing a robust reference probability distribution in CDRO. Moreover, to devise an efficient algorithm for CDRO, we derive a closed-form expression for the empirical robust risk and the optimal Lagrange multiplier of the dual problem, facilitating a principled balance between robustness and model fitting. Our experimental results on both synthetic and real-world datasets demonstrate the superiority of our method.

Learning from Noisy Labels via Conditional Distributionally Robust Optimization

Abstract

While crowdsourcing has emerged as a practical solution for labeling large datasets, it presents a significant challenge in learning accurate models due to noisy labels from annotators with varying levels of expertise. Existing methods typically estimate the true label posterior, conditioned on the instance and noisy annotations, to infer true labels or adjust loss functions. These estimates, however, often overlook potential misspecification in the true label posterior, which can degrade model performances, especially in high-noise scenarios. To address this issue, we investigate learning from noisy annotations with an estimated true label posterior through the framework of conditional distributionally robust optimization (CDRO). We propose formulating the problem as minimizing the worst-case risk within a distance-based ambiguity set centered around a reference distribution. By examining the strong duality of the formulation, we derive upper bounds for the worst-case risk and develop an analytical solution for the dual robust risk for each data point. This leads to a novel robust pseudo-labeling algorithm that leverages the likelihood ratio test to construct a pseudo-empirical distribution, providing a robust reference probability distribution in CDRO. Moreover, to devise an efficient algorithm for CDRO, we derive a closed-form expression for the empirical robust risk and the optimal Lagrange multiplier of the dual problem, facilitating a principled balance between robustness and model fitting. Our experimental results on both synthetic and real-world datasets demonstrate the superiority of our method.

1 Introduction

Recent advancements in supervised learning have spurred a growing demand for large labeled datasets [1, 2]. However, acquiring accurately annotated datasets is typically costly and time-consuming, often requiring a pool of annotators with adequate domain expertise to manually label the data. Crowdsourcing has emerged as an efficient and cost-effective solution for annotating large datasets. On crowdsourcing platforms, multiple annotators with varying levels of labeling skills are employed to gather extensive labeled data. However, this approach introduces a significant challenge: the labels collected through crowdsourcing are often subject to unavoidable noise, especially in fields

requiring substantial domain knowledge, such as medical imaging. Consequently, models trained on noisy labels are prone to error, including overfitting, since deep models can memorize vast amounts of data [3]. In addition to statistical research on label noise (often termed response measurement error, e.g., [4-6]), a growing body of recent machine learning literature has focused on developing effective algorithms capable of training accurate classifiers using noisy data, e.g., [7-10]. Many of these methods seek to approximate the posterior distribution of the underlying true labels using the observed data.

Let X be an instance, Y denote the unobserved true label for X , and ˜ Y represent a vector of crowdsourced noisy labels for X . The data-generating distribution, P ∗ x , y , ˜ y , can be factorized in two ways: P ∗ x P ∗ y | x P ∗ ˜ y | x , y or P ∗ x , ˜ y P ∗ y | x , ˜ y , with P ∗ denoting the (conditional) distribution for the variables indicated by the corresponding subscripts. These factorizations have inspired research that trains models by estimating the posterior distribution of the true labels, P ∗ y | x , y , in the latter factorization.

˜

Previous work [9-12] introduced various algorithms for estimating the annotator confusions , also known as noise transition probabilities , which yield an approximated conditional distribution of ˜ Y , given Y and X , denoted P ˜ y | y , x . For ease of reference, we use P ∗ and P to denote the true distribution and an approximate distribution for the variables indicated by the corresponding subscripts. Given the observed data X and ˜ Y , along with an approximated conditional distribution P ˜ y | y , x and a prior for Y given X , denoted P y | x , the true label posterior is then computed as P y | x , ˜ y ∝ P y | x · P ˜ y | y , x by Bayes's theorem [10, 13]. This estimated true label posterior is often used to infer the underlying true labels or to weight the loss functions [7, 9, 10, 14].

However, accurately computing the posterior of the true label is challenging, and the estimated posterior P y | x , ˜ y may deviate from the underlying true distribution P ∗ y | x , ˜ y due to potential misspecifications in the prior belief and the conditional noise transition probabilities [15]. To address this issue, we introduce a robust scheme for handling crowdsourced noisy labels through conditional distributionally robust optimization (CDRO), as discussed in [16]. Specifically, we frame the problem as minimizing the worst-case risk within a distance-based ambiguity set , which constrains the degree of conditional distributional uncertainty around a reference distribution . By leveraging the strong duality in linear programming, we derive the dual form of the robust risk and establish informative upper bounds for the worst-case risk. Additionally, for each data point, we develop an analytical solution to the robust risk minimization problem, which encompasses existing approaches as special cases [9]. This solution is presented in a likelihood ratio format and inspires a robust approach that assigns pseudo-labels only to instances with high confidence, with uncertain data filtered out. These pseudo-labels also enable us to construct a pseudo-empirical distribution that serves as a robust reference probability distribution in CDRO under potential model misspecifications. Moreover, we derive a closed-form expression for the empirical robust risk by identifying the optimal Lagrange multiplier in the dual form. Building on this, we ultimately develop an algorithm for learning from noisy labels via conditional distributionally robust true label posterior with an adaptive Lagrange multiplier (AdaptCDRP).

Our contributions are summarized as follows: (1) We formulate learning with noisy labels as a CDRO problem and develop its dual form to tackle the challenge of potential misspecification in estimating the true label posterior from noisy data. (2) We derive an analytical solution to the dual problem for each data point, and propose a novel algorithm that constructs a robust reference distribution for this problem. (3) By deriving the optimal Lagrange multiplier for the empirical robust risk, we develop an efficient one-step update method for the Lagrange multiplier, allowing for a principled balance between robustness and model fitting. Code is available at https://github.com/hguo1728/AdaptCDRP.

Notations. We use [ k ] to denote { 1 , . . . , k } for any positive integer k , and 1 ( · ) to denote the indicator function. For a vector v , v j stands for its j th element, and v ⊤ denotes its transpose. For v = ( v 1 , ..., v p ) ⊤ and q ∈ [1 , + ∞ ] , the L q norm is defined as ∥ v ∥ q = ( ∑ p j =1 | v j | q ) 1 /q if 1 ≤ q < ∞ , and ∥ v ∥ ∞ = max j | v j | if q = + ∞ . For a matrix V , we use V i,j to represent its ( i, j ) element. Furthermore, let (Ω , G , µ ) denote the measure space under consideration, where Ω is a set, G is the σ -field of subsets of Ω , and µ is the associated measure. For q > 0 , let L q ( µ ) represent the collection of Borel-measurable functions f : Ω → R such that ∫ | f | q dµ < ∞ . Let d ( · , · ) denote a metric on Ω . We call f L -Lipschitz with respect to d ( · , · ) if | f ( u 1 ) -f ( u 2 ) | ≤ L · d ( u 1 , u 2 ) for all u 1 , u 2 ∈ Ω , where L is a positive constant.

2 Proposed Framework

2.1 Problem Formulation

Consider a classification task with feature space X ⊂ R d and label space Y , where d is the feature dimension. Here Y is taken as { 0 , 1 } for binary classification and [ K ] for multi-class classification with K > 2 . Let X ∈ X denote an instance and Y ∈ Y denote its true label. Let Ψ denote the considered hypothesis class consisting of functions ψ defined over X , which, for example, can be neural networks that output predicted label probabilities for each x ∈ X . Specifically, for binary classification, ψ : X → [0 , 1] , with ψ ( x ) representing P (Y = 1 | X = x ) , and the classified value is given by 1 ( ψ ( x ) > 0 . 5) . For multi-class classification, ψ : X → ∆ K -1 with K > 2 and ∆ K -1 denoting the K -simplex, where the j th component of ψ ( x ) , denoted ψ ( x ) j , represents the conditional probability P (Y = j | X = x ) for j ∈ [ K ] , with the classified value defined as arg max j ∈ [ K ] ψ ( x ) j .

In applications, the true label Y is often unobserved, and instead, a set of crowdsourced noisy labels ˜ Y ≜ { ˜ Y ( r ) } R r =1 is collected, where ˜ Y ( r ) ∈ Y , denoting the label provided by annotator r out of R annotators. Let D ≜ { X i , ˜ Y i } n i =1 denote the observed data of size n , where ˜ Y i contains noisy labels provided by R annotators for instance X i , which may differ from the true label Y i for each i ∈ [ n ] . Our goal is to train a classifier using D to accurately predict the true label for a future instance.

A common assumption in supervised learning is that the data points { X i , Y i , ˜ Y i } for i ∈ [ n ] are independently drawn from a probability measure P ∗ x , y , ˜ y for { X , Y , ˜ Y } , defined over the space Z ≜ X × Y × Y R . Under this assumption, many existing methods aim to approximate the posterior distribution of the underlying true label Y , given the observed data X and ˜ Y [7, 9-11]. The estimated true label posterior, denoted P y | x , ˜ y , is then applied to either infer the true labels or to weight the loss functions. For example, [9] utilized P y | x , ˜ y as a weight in the loss functions, without considering potential misspecification of the associated model. However, such strategies typically ignore the variability induced in estimating the true label posterior.

To mitigate the effects of potential misspecifications, we propose a conditional distributionally robust risk optimization problem:

where ℓ ( · , · ) is a loss function, the expectation E x , ˜ y is taken with respect to the joint distribution of the observed data X and ˜ Y , and the expectation E Q y | x , ˜ y is evaluated under the conditional distribution model, denoted Q y | x , ˜ y , of the true label Y , given X and ˜ Y . Here, Γ ϵ ( P y | x , ˜ y ) is an ambiguity set of probability measures centered around the reference probability distribution P y | x , ˜ y , indexed by ϵ > 0 [16-18]. For instance, Γ ϵ ( P y | x , ˜ y ) can be conceptualized as a 'ball' with P y | x , ˜ y at its center and ϵ as the radius, where elements in the ball represent possible distribution models for P ∗ y | x , ˜ y , and the distance between two points is measured using a standard metric for distributions. Specifically,

where P ( Y ) denotes all Borel probability measures on Y , and /u1D4B9 is a discrepancy metric of probability measures. In this paper, we employ the Wasserstein distance in Definition 2.1 to define the ambiguity set. By taking the supremum in (1) over the ambiguity set (2), we aim to minimize the worst-case risk around the reference distribution, thereby mitigating the impact of potential model misspecifications.

One main obstacle in solving (1) is constructing a reliable reference distribution P y | x , ˜ y , which typically depends on an empirical distribution that requires true labels in conventional distributionally robust optimization (DRO). We address this issue by investigating the dual form of the robust risk presented in (1), which enables us to create a robust pseudo-empirical distribution using a likelihood ratio test, as detailed in Section 3.1. An additional advantage of our approach is that it provides informative upper bounds for the worst-case risk in Section 2.3 via the dual formulation.

Remark 2.1. For simplicity in theoretical presentation, we assume access to all annotations from all R annotators. However, the theoretical framework presented in this paper is applicable to both single-annotator ( R = 1 ) and multiple-annotator ( R > 1 ) scenarios. In our experiments in Section 4, we also consider the scenario of sparse labeling, where we generate a total of R annotators and then randomly select one annotation per instance from these R annotators. We also conduct experiments with varying numbers of annotators for a comprehensive analysis.

2.2 Duality Result and Relaxed Problem

To derive the pseudo-label generation algorithm and establish a reliable reference distribution, we analyze the dual form of (1). We first define the Wasserstein distance of order p for p ∈ [1 , + ∞ ) .

Definition 2.1 ( p -Wasserstein distance, [17]) . For a Polish space S (i.e., a complete separable metric space) endowed with a metric c : S × S → R ≥ 0 , also called a cost function, let P ( S ) represent the set of all Borel probability measures on S , where R ≥ 0 represents the set of all nonnegative real values. For p ≥ 1 , let P p ( S ) stand for the subset of P ( S ) with finite p th moments. Then, for P 1 , P 2 ∈ P p ( S ) , the Wasserstein distance of order p is defined as

where Cpl ( P 1 , P 2 ) comprises all probability measures on the product space S × S such that their marginal measures are P 1 ( · ) and P 2 ( · ) . Here, c p ( · , · ) represents { c ( · , · ) } p .

In (2), we set /u1D4B9 ( · , · ) as the p -Wasserstein distance and incorporate the constraint /u1D4B9 ( Q y | x , ˜ y , P y | x , ˜ y ) ≤ ϵ using the Lagrange formulation, and then establish the strong duality result for (1) as follows.

Proposition 2.1 (dual problem) . Assume that for every given x ∈ X , ˜ y ∈ Y R and ψ ∈ Ψ , ℓ ( ψ ( x ) , · ) ∈ L 1 ( P y | x , ˜ y ) , where L 1 ( · ) is defined in Section 1. Consider /u1D4B9 ( · , · ) in (2) as the Wasserstein distance of order p . Then, for any ϵ > 0 , R ϵ ( ψ ; P y | x , ˜ y ) in (1) becomes:

To avoid solving nested optimization problems, we consider an alternative formulation by swapping the infimum and the first expectation operations:

which is an upper bound of R ϵ ( ψ ; P y | x , ˜ y ) according to Proposition 2.1, and hence, (4) can be regarded as an relaxation of (3). The empirical counterpart of R ϵ ( ψ ; P y | x , ˜ y ) is given by

where P ( n ) x , ˜ y ≜ 1 n ∑ n i =1 δ X i , ˜ Y i is the empirical distribution of ( X , ˜ Y ) based on the dataset D defined in Section 2.1. Here, for any v ∈ X × Y R , δ v represents the Dirac measure on X × Y R , defined as δ v ( A ) ≜ 1 { v ∈ A } for any A ⊂ X × Y R .

Remark 2.2. The Lagrange multiplier γ in (4) and (5) captures the trade-off between robustness and model fitting in the presence of label noise and potential model misspecifications. When the solution in γ is large, the inner supremum tends to favor y ′ = Y , thus encouraging the minimization of the natural risk using the reference distribution directly. In contrast, a small solution in γ introduces perturbations to the data, pushing the classifier away from the sample instances weighted by the reference distribution.

Remark 2.3. When p = 1 , (5) represents the dual form of the following problem:

where Γ ϵ ( P y | x , ˜ y ) = { Q y | x , ˜ y ∈ P ( Y ) : E P ( n ) x , ˜ y /u1D4B9 ( Q y | x , ˜ y , P y | x , ˜ y ) ≤ ϵ } . The proof of this statement is deferred to Appendix A.3. This result indicates that the empirical robust risk in the relaxed problem (5) corresponds to the worst-case risk within an ambiguity set that constrains the size of the average conditional distributional uncertainty.

2.3 Generalization Bounds

With the duality result in Proposition 2.1 and the derivations of the alternative formulations (4) and (5), we now characterize the difference between ̂ R ϵ ( ψ ; P y | x , ˜ y ) and its population counterpart R ϵ ( ψ ; P y | x , ˜ y ) .

̸

Theorem 2.2. Consider the loss function ℓ ( · , · ) in Proposition 2.1, and let the cost function c ( y, y ′ ) = κ 1 ( y = y ′ ) for y, y ′ ∈ Y , where κ is a positive constant. Assume that there exists a positive constant M such that ℓ ( ψ ( x ) , y) ∈ [0 , M ] for all x ∈ X , y ∈ Y , and ψ ∈ Ψ , and that ℓ ( ψ ( x ) , y) is L -Lipschitz in the second argument with respect to the cost function c ( · , · ) . Then, there exists a positive constant C 1 such that for any given ϵ > 0 , ψ ∈ Ψ , and 0 < η < 1 , with probability at least 1 -η :

Theorem 2.2 suggests that the empirical counterpart, ̂ R ϵ ( ψ ; P y | x , ˜ y ) , is a useful approximation for the risk function R ϵ ( ψ ; P y | x , ˜ y ) , as their disparity is bounded and cannot grow indefinitely large. For a finite sample size n , this disparity is upper bounded by a finite value depending on the characteristics of the cost and loss functions, as reflected by κ , L , and M . As the sample size n →∞ , the difference tends to zero with high probability, and specifically, the difference is of order O ( n -1 / 2 ) .

Next, we establish an informative bound for the empirical robust risk minimizer. For ψ 1 , ψ 2 ∈ Ψ and for any given norm ∥ · ∥ , let ∥ ψ 1 -ψ 2 ∥ ∞ ≜ sup x ∈X ∥ ψ 1 ( x ) -ψ 2 ( x ) ∥ . Here, ∥ · ∥ can be taken as any specific norms, including the L q norm with q ≥ 1 that is defined in Section 1.

Corollary 2.3 (Empirical Robust Minimizer) . Let ̂ ψ ϵ,n ∈ inf ψ ∈ Ψ ̂ R ϵ ( ψ ; P y | x , ˜ y ) . Under the assumptions in Theorem 2.2, if we further assume that the loss function ℓ ( · , · ) is L ′ -Lipschitz in terms of the first argument with respect to the supremum metric ∥ · ∥ ∞ , then there exists a positive constant C 2 such that for any ϵ > 0 and 0 < η < 1 , with probability at least 1 -η , the empirical robust risk minimizer ̂ ψ ϵ,n satisfies:

where N ( s ; Ψ , ∥ · ∥ ∞ ) denotes the s -covering number of Ψ with respect to the supremum metric.

3 Implementation Algorithm

In Section 3.1, we derive the analytical solution to the dual robust risk minimization problem in (5), which leads to the development of a novel approach for assigning pseudo-labels using the likelihood ratio test. These pseudo-labels facilitate the construction of a pseudo-empirical distribution , serving as a robust reference distribution in using (5). In Section 3.2, we derive the optimal value in γ for the empirical robust risk (5) and establish its closed-form expression. This analysis provides a principled framework for balancing the trade-off between robustness and model fitting and also motivates an efficient one-step update technique in solving the robust empirical risk minimization problem.

3.1 Optimal Solution for Single Data Point

In this subsection, we determine the optimal value of ψ ( x ) in (5) for a single data point ( x , ˜ y ) . To simplify the analysis, we first focus on the binary classification problem with Y = { 0 , 1 } and consider a broad family of loss functions of the form:

where ψ ( x ) represents the conditional distribution P (Y = 1 | X = x ) as described in Section 2.1, T : [0 , 1] →I is a bounded, decreasing, and twice differentiable function, and I is a compact subset of R .

For any given x ∈ X and ˜ y ∈ Y R , let P j ( x , ˜ y ) ≜ P (Y = j | X = x , ˜ Y = ˜ y ) for j = 0 , 1 . With the loss function in (7) and the metric c ( · , · ) considered in Theorem 2.2, minimizing (5) with respect to ψ ∈ Ψ becomes:

and let ψ ⋆ denote the solution of (8). For ϵ and κ described in Theorem 2.2, let ϱ ( ϵ ) ≜ ϵ p /κ p . The following theorem shows that (8) has a closed-form solution, with its form varying based on whether T is concave or convex.

Theorem 3.1 (Optimal Action for Single Data Point: Binary Case) . Let x ∈ X and ˜ y ∈ Y R be given. Then, for a concave function T , the optimal solution for (8) is given by:

with ϖ 1 = {T (0) -T (1 / 2) } / {T (0) -T (1) } ∈ (0 , 1 / 2] ; and for a convex function T , the optimal solution of (8) is given by:

where ϖ 2 = {T ′ (0) } / {T ′ (0) + T ′ (1) } ∈ [1 / 2 , 1) , t ∗ 0 is the unique solution of { P 0 ( x , ˜ y ) -ϱ ( ϵ ) }T ′ (1 -t ) = { P 1 ( x , ˜ y )+ ϱ ( ϵ ) }T ′ ( t ) for t ∈ (0 , 1 2 ) , and t ∗ 1 is the unique solution of { P 0 ( x , ˜ y )+ ϱ ( ϵ ) }T ′ (1 -t ) = { P 1 ( x , ˜ y ) -ϱ ( ϵ ) }T ′ ( t ) for t ∈ ( 1 2 , 1) .

̸

Remark 3.1. The optimal solution ψ ⋆ ( x ) in Theorem 3.1 can also be expressed in a likelihood ratio format, which naturally leads to a novel algorithm for assigning robust pseudo-labels . Specifically, when T is concave, the optimal solution can be expressed as: ψ ⋆ ( x ) = 0 if P 0 ( x , ˜ y ) /P 1 ( x , ˜ y ) ≥ C 1 ; ψ ⋆ ( x ) = 1 if P 1 ( x , ˜ y ) /P 0 ( x , ˜ y ) ≥ C 1 ; and ψ ⋆ ( x ) = 1 / 2 otherwise, where C 1 ≜ ( ϱ ( ϵ ) + ϖ 1 ) / { 1 -( ϱ ( ϵ ) + ϖ 1 ) } > 1 serves as a threshold for the likelihood ratio test. Consequently, for a data point ( x , ˜ y ) , if P 1 ( x , ˜ y ) /P 0 ( x , ˜ y ) ≥ C 1 , we assign a robust pseudo-label y ⋆ = 1 ; if P 0 ( x , ˜ y ) /P 1 ( x , ˜ y ) ≥ C 1 , we assign y ⋆ = 0 . Leveraging the likelihood ratio format also facilitates extending the robust pseudo-label selection method to the multi-class case by considering pairwise comparisons. Specifically, if P k ⋆ ( x , ˜ y ) / max j = k ⋆ P j ( x , ˜ y ) ≥ C 1 , we assign the pseudo-label y ⋆ = k ⋆ to the instance.

Remark 3.2. Existing pseudo-labeling methods [9, 19] typically identify the underlying true label as the one with the highest probability in the approximated true label posterior. In contrast, the proposed approach in Remark 3.1 considers both the highest and second-highest predicted probabilities. A pseudo-label is assigned only if the ratio of these probabilities exceeds a specified threshold. This strategy ensures that pseudo-labels are assigned to instances with high confidence, effectively filtering out uncertain data.

̸

Remark 3.3. In the special case where P j ( x , ˜ y ) ∝ τ j ( ˜ y ; x ) P j ( x ) , with τ j ( ˜ y ; x ) = P ∗ ( ˜ Y = ˜ y | Y = j, X = x ) denoting the noisy label transition probability and P j ( x ) representing a proper prior for Y = j conditional on x for j = 0 , 1 , previous studies have indicated the existence of a Chernoff information-type bound on the probability of error for robust pseudo-label selection, as described in Remark 3.1 [10, 20]. Specifically, for a fixed instance x , let a pseudo-label Y ⋆ be generated as described in Remark 3.1, which depends on x and the corresponding noisy label vector ˜ Y . Consider the Bayes error, defined as ℜ Bayes ≜ ∑ j =0 , 1 P j ( x ) P ∗ (Y ⋆ = j | Y = j, x ) . According to Section 11.9 of [20], ℜ Bayes ≤ exp {-C ( τ 0 ( · ; x ) , τ 1 ( · ; x )) } , where C ( · , · ) represents the Chernoff information between two distributions.

Remark 3.4. In practice, one can use either uninformative priors, such as a uniform prior for each class, or informative priors derived from pre-trained or concurrently trained models for P j ( x ) as discussed in Remark 3.3 [10, 13]. Moreover, the estimation of P j ( x , ˜ y ) is not limited to Bayes's rule. For example, [11] proposed aggregating data and noisy label information by maximizing the f -mutual information gain.

Remark 3.5. Theorem 3.1 is developed based on the assumption that the function T is convex or concave. In our experiments, we use the cross-entropy loss for ℓ , meaning T ( t ) = -log t for t > 0 . To meet the required conditions, we clip its input to [0 . 01 , 1 -0 . 01] to ensure T ( · ) remains bounded.

Next, we extend the preceding development for binary classification to multi-class scenarios with K > 2 . Letting T ( · ) in (7) be specified as T ( t ) = 1 -t , we extend loss function form (7) to facilitate the worst-case misclassification probability in multi-class scenarios: ℓ ( ψ ( x ) , y) = ∑ K j =1 1 (y = j ) { 1 -ψ ( x ) j } . For ease of presentation, we sometimes omit the dependence on x and ˜ y in the

notation. Specifically, for j ∈ [ K ] , we let P j ≜ P j ( x , ˜ y ) ≜ P (Y = j | x , ˜ y ) and ψ j ≜ ψ ( x ) j . In a manner similar to deriving (8), given x , minimizing (5) with respect to ψ ( x ) can be expressed as:

Theorem 3.2 (Optimal Action for Single Data Point: Multi-class Case) . Let { P 1 , . . . , P K } be arranged in decreasing order, denoted P (1) ≥ . . . ≥ P ( K ) , with the associated indexes denoted χ (1) , . . . , χ ( K ) . Let ψ ⋆ denote the solution of the outer optimization problem in (9). For j ∈ [ K ] , let ψ ⋆ ( j ) denote the χ ( j ) -th component of ψ ( x ) corresponding to P ( j ) . Then, the elements of ψ ⋆ are given as follows:

Remark 3.6. The robust pseudo-labeling method described in Remark 3.1 can also be extended from Theorem 3.2. Specifically, by Theorem 3.2, if P (1) ≥ max { 1 K + ϱ ( ϵ ) , P (2) + ϱ ( ϵ ) } , then the optimal solution is: ψ ⋆ (1) = 1 and ψ ⋆ ( j ) = 0 for j = 2 , . . . , K , which can also be expressed in a likelihood ratio format and applied to assign robust pseudo-labels.

3.2 Closed-Form Robust Risk

We investigate the empirical robust risk (5) by examining its closed form expression. For i ∈ [ K ] and j ∈ [ K ] , we let P i,j ≜ P j ( x i , ˜ y i ) ≜ P (Y = j | X = x i , ˜ Y = ˜ y i ) and ψ i,j ≜ ψ ( x i ) j . For simplicity, we denote the Wasserstein robust loss in (5) and the nominal loss respectively as:

For given x i , we sort { ψ i, 1 , . . . , ψ i,K } in decreasing order, denoted as ψ (1) i ≥ . . . ≥ ψ ( K ) i . Let α i,j ≜ T ( ψ ( K ) i ) - T ( ψ i,j ) for i ∈ [ n ] and j ∈ [ K ] , and sort { α i,j : i ∈ [ n ] , j ∈ [ K ] } in decreasing order, denoted as α (1) ≥ . . . ≥ α ( nK ) . Correspondingly, the P i,j values with the associated indexes are denoted as P (1) , . . . , P ( nK ) . For any ϱ ( ϵ ) , define an associated positive integer s ∗ ∈ [ nK +1] as follows: if 1 n P (1) < ϱ ( ϵ ) < 1 n ∑ nK t =1 P ( t ) , then there exists s ∗ ∈ { 2 , . . . , nK } such that 1 n ∑ s t =1 P ( t ) < ϱ ( ϵ ) for s < s ∗ , and 1 n ∑ s t =1 P ( t ) ≥ ϱ ( ϵ ) for s ≥ s ∗ ; if ϱ ( ϵ ) ≤ 1 n P (1) , then s ∗ is set as 1; if ϱ ( ϵ ) ≥ 1 n ∑ nK t =1 P ( t ) , then s ∗ is set as nK +1 .

Let γ ⋆ ψ denote the optimal value of γ in ̂ R ϵ in (10), where its dependence on ϵ is implicit, but its dependence on ψ is explicit. The following theorem presents this value, based on which we demonstrate that the Wasserstein robust loss ̂ R ϵ can be expressed as the nominal loss plus an additional term 1 n ∑ s ∗ -1 t =1 P ( t ) α ( t ) 1 ( s ∗ > 1) that prevents the classifier from becoming overly certain on the data.

Theorem 3.3 (Closed-Form Robust Risk) . The optimal value of γ in (10) is given by γ ⋆ ψ ≜ α ( s ∗ ) /κ p , and the resulting robust risk is expressed as

Remark 3.7. Theorem 3.3 shows that minimizing the Wasserstein robust loss ̂ R ϵ in (10) effectively minimizes the nominal loss ̂ R in (11) while simultaneously penalizing terms associated with | α i | values exceeding a certain threshold | α s ∗ | , weighted by the corresponding reference probability values. This minimization prevents the classifier from becoming overly confident in certain data points, particularly when there are potential misspecifications in the approximated true label posterior.

Remark 3.8. As suggested by Remark 2.2, Theorem 3.3 provides a guideline for balancing robustness and model fitting by deriving the optimal value for γ in (10). In Section 3.3, we develop a one-step update method for determining γ ⋆ ψ .

3.3 Training using Conditional Distributionally Robust True Label Posterior

In this subsection, we outline the steps for approximating the true label posterior, constructing the pseudo-empirical distribution as the reference distribution for solving the robust risk minimization problem (5), and subsequently training classifiers robustly. The pseudo code for the training process is provided in Algorithm 1 in Appendix B.1. Here we elaborate on the details.

Approximating noise transitions probabilities. We begin by warming up the classifiers on the noisy training data, denoted ˇ D = { x i , ˇ y i } n i =1 , where ˇ y i represents the majority vote label for instance x i , determined by the label that receives the highest number of votes from the annotators. After warming up the classifiers for 20-30 epochs, we sort the dataset by the cross-entropy loss values and collect a subset of size m with the smallest m losses, denoted as D ⋆ 0 = { x i , ˇ y i } i m =1 , where m ≪ n , and the ratio of m to n is set to 1 minus the estimated noise rate. Next, we estimate the noise transition probabilities by ̂ τ j ( ˜ y ) = ∑ m i =1 1 ( ˜ y = ˜ y i , ˇ y i = j ) / ∑ m i =1 1 (ˇ y i = j ) for ˜ y ∈ [ K ] R and j ∈ [ K ] (Line 1 of Algorithm 1). With ̂ τ j ( ˜ y ) , we then iteratively update the approximated true label posterior, construct the pseudo-empirical distribution, and robustly train the classifiers (Lines 2-13 of Algorithm 1). Here, we employ the straightforward frequency-counting method for noise transition estimation for simplicity. However, our approach is versatile and can be integrated with various methods for estimating the noise transition matrices or the true label posterior. Additional experimental results using more advanced transition matrix estimation methods are provided in Appendix B.

̸

Constructing a pseudo-empirical distribution. We train two classifiers, ψ (1) and ψ (2) , in parallel each serving as an informative prior for the other. In the t th epoch, the approximated true label posterior with prior ψ ( ι ) is updated as ̂ P ( ι ) j ( x , ˜ y ) ≜ ̂ P ( ι ) (Y = j | ˜ Y = ˜ y , X = x ) ∝ ψ ( ι ) j ( x ) · ̂ τ j ( ˜ y ) (Line 4 of Algorithm 1), where ψ ( ι ) j ( x ) denotes the j th element of the vector-valued function ψ ( ι ) ( x ) for j ∈ [ K ] and ι = 1 , 2 . As described in Remark 3.1, for i ∈ [ n ] , if ̂ P ( ι ) k ⋆ ( x i , ˜ y i ) / max j = k ⋆ ̂ P ( ι ) j ( x , ˜ y ) ≥ C for a pre-specified threshold C > 1 , we assign the robust pseudo-label y ⋆ i = k ⋆ to the instance and collect it into D ⋆ t,ι (Lines 5-7 of Algorithm 1). The pseudo-empirical distribution P ⋆ t,ι is updated based on D ⋆ t,ι (Line 8 of Algorithm 1).

Robustly training the classifiers. For ι = 1 , 2 , let \ ι ≜ 1 if ι = 2 ; and \ ι ≜ 2 if ι = 1 . With the updated pseudo-empirical distribution, the classifier ψ ( ι ) is then trained by minimizing the empirical robust risk (5) with the reference distribution P ⋆ t, \ ι (Line 9 of Algorithm 1). After updating the classifier ψ ( ι ) with γ ( ι ) t -1 from the previous iteration, we take one step to update the γ value γ ( ι ) t . In particular, as suggested by Theorem 3.3, we use γ 0 ,t = | α ( s ∗ ) | /κ p as a reference value for γ (Lines 11-12 f Algorithm 1). We then update γ ( ι ) t by minimizing [ γ { ϵ p -E P ⋆ t, \ ι c p ( y ′ , Y) } + λ 2 ( γ -γ 0 ) 2 ] (Line 13 of Algorithm 1) with respect to γ , where y ′ is determined by (5) after updating ψ ( ι ) , and λ > 0 is a positive constant that determines the learning rate of γ .

4 Experimental Results

Datasets and model architectures. We evaluate the performance of the proposed AdaptCDRP on two datasets, CIFAR-10 and CIFAR-100 [21], by generating synthetic noisy labels (details provided below), as well as four datasets, CIFAR-10N [22], CIFAR-100N [22], LabelMe [23, 24], and Animal10N [25], which contain human annotations. For all datasets except LabelMe, we set aside 10% of the original data, together with the corresponding synthetic or human annotated noisy labels, to validate the model selection procedure. We use the ResNet-18 architecture [26] for CIFAR-10 and

CIFAR-10N, and the ResNet-34 architecture [26] for CIFAR-100 and CIFAR-100N. Following [27], we employ a pretrained VGG-16 model with a 50% dropout rate for the LabelMe dataset. In line with [25], the VGG19-BN architecture [28] is used for the Animal-10N dataset. Further details on the datasets and experimental setup are provided in Appendix B.1.

Noise generation. We generate synthetic annotations on the CIFAR-10 and CIFAR-100 datasets using Algorithm 2 from [29]. Three groups of annotators, labeled as IDN-LOW, IDN-MID, and IDN-HIGH, are considered, with average labeling error rates of approximately 20%, 35%, and 50%, respectively, representing low, intermediate, and high error rates. Each group consists of R = 5 annotators. To assess the algorithms in an incomplete labeling setting, we randomly select only one annotation per instance from the R annotators for the training dataset rather than using all available annotations [10]. Further details on noise generation are provided in Appendix B.1.

Comparison with SOTA methods. We compare our method with a comprehensive set of state-ofthe-art approaches, including: (1) CE (Clean) with clean labels; (2) CE (MV) with majority vote labels; (3) CE (EM) [7]; (4) Co-teaching [30]; (5) Co-teaching+ [31]; (6) CoDis [32]; (7) LogitClip [33]; (8) DoctorNet [34]; (9) MBEM [9]; (10) CrowdLayer [27]; (11) TraceReg [12]; (12) Max-MIG [11]; (13) CoNAL [35]; and (14) CCC [36]. We report the average test accuracy over five repeated experiments, each with a different random seed, on synthetic datasets, CIFAR-10 and CIFAR-100, with instance-dependent label noise introduced at low, intermediate, and high error rates. Standard errors are shown following the plus/minus sign ( ± ), and the two highest accuraries are highlighted in bold. Table 2 presents evaluation results on four real-world datasets. As shown, our AdaptCDRP consistently outperforms competing methods across all scenarios. To further explore the impact of annotation sparsity, we conduct additional experiments with the number of annotators ranging from 5 to 100, with each instance labeled only once. Figure 1 illustrates the average accuracy across different numbers of annotators on CIFAR-10, highlighting the advantages of the proposed method under diverse settings. The results for the CIFAR-100 dataset are shown in Figure 3 in Appendix B.2.

Hyper-parameter analysis. We investigate the impact of the hyperparameter ϵ in the empirical robust risk (5). Under our experiment setup, ϵ should be chosen within (0 , 1 /K ) for a K -class classification problem with K ≥ 2 as demonstrated in the proof of Theorem 3.3 in Appendix A.8. Hence, we take ϵ ∈ (0 , 0 . 1) for CIFAR-10 and ϵ ∈ (0 , 0 . 01) for CIFAR-100, with the results presented in Figure 2. The results suggest that setting ϵ near zero leads to relatively low test accuracies, highlighting the importance of CDRO under model specification when handling noisy labels. Furthermore, continually increasing ϵ eventually results in a drop in accuracy due to excessive noise injection into the data.

Additional experimental results. To further evaluate the performance of the proposed method across various scenarios, we conducted additional experiments, detailed in Appendix B.2. Specifically, we compare different annotation aggregation methods, present average test accuracies and robust pseudo-label accuracies during training, assess sensitivity to the number of warm-up epochs, explore different noise transition estimation methods, and examine the impact of sparse annotation.

0.001

0.005

0.01

0.02

0.03

0.04

0.05

0.06

hyperparameter

(CIFAR-10)

85

86

87

88

CIFAR10-LOW

CIFAR10-MID

CIFAR10-HIGH

0.0001 0.0005

0.001

0.002

0.003

0.004

0.005

0.006

hyperparameter

(CIFAR-100)

50

52

54

56

58

60

CIFAR100-LOW

CIFAR100-MID

CIFAR100-HIGH

Test Accuracy

5

10

30

50

100

Number of Annotators

74

76

78

80

82

84

86

88

Test Accuracy

CE(Clean)

CE(MV)

CE(EM)

Co-teaching

Co-teaching+

CoDis

LogitClip

DoctorNet

MBEM

CrowdLayer

TraceReg

CoNAL

CCC

AdaptCDRP (Ours)

(a) IDN-LOW

5

10

30

50

100

Number of Annotators

65

70

75

80

85

Test Accuracy

CE(Clean)

CE(MV)

CE(EM)

Co-teaching

Co-teaching+

CoDis

LogitClip

DoctorNet

MBEM

CrowdLayer

TraceReg

CoNAL

CCC

AdaptCDRP (Ours)

(b) IDN-MID

5

10

30

50

100

Number of Annotators

50

55

60

65

70

75

80

85

Test Accuracy

CE(Clean)

CE(MV)

CE(EM)

Co-teaching

Co-teaching+

CoDis

LogitClip

DoctorNet

MBEM

CrowdLayer

TraceReg

CoNAL

CCC

AdaptCDRP (Ours)

(c) IDN-HIGH

5 Conclusion

In this paper, we address the challenge of learning from noisy annotations by estimating true label posteriors using the CDRO framework. We formulate the problem as minimizing the worst-case risk within a distance-based ambiguity set, which constrains the conditional distributional uncertainty around a reference distribution. By deriving the dual form of the worst-case risk and finding the analytical solution to the robust risk minimization problem for each data point, we propose a novel approach for determining robust pseudo-labels using the likelihood ratio test. This approach further leads to the construction of a pseudo-empirical distribution that serves as a robust reference probability distribution in CDRO. We also derive a closed-form expression of the empirical robust risk and identify the optimal Lagrange multiplier for the dual problem. This leads to a guideline for balancing robustness and model fitting in a principled way and inspires an efficient one-step update method for the Lagrange multiplier.

Limitations and Extensions. Our development here does not focus on precisely estimating the noise transition matrix or the true label posterior. Further research may be conducted to address the sparse annotation problem and improve estimates of the true label posterior. This can be accomplished through several approaches: (1) employing regularization techniques to mitigate the impact of small sample sizes by smoothing estimates and reducing sensitivity to outliers; (2) leveraging subgroup structures among annotators to capture additional nuances; and (3) directly modeling the true label posterior by integrating both data and noisy label information, moving beyond the limitation of purely applying Bayes's rule.

References

Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning . MIT press, 2016.

Jürgen Schmidhuber. Deep learning in neural networks: An overview. Neural Networks , 61:85-117, 2015.

Devansh Arpit, Stanisław Jastrz˛ ebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In Proceedings of the 34th International Conference on Machine Learning , volume 70, pages 233-242, 2017.

Raymond J Carroll, David Ruppert, Leonard A Stefanski, and Ciprian M Crainiceanu. Measurement Error in Nonlinear Models: A Modern Perspective . Chapman and Hall/CRC, 2006.

Grace Y Yi. Statistical Analysis with Measurement Error or Misclassification . Springer, 2017.

Grace Y Yi, Aurore Delaigle, and Paul Gustafson. Handbook of Measurement Error Models . CRC Press, 2021.

Alexander Philip Dawid and Allan M Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society: Series C , 28(1):20-28, 1979.

Jacob Whitehill, Ting-fan Wu, Jacob Bergsma, Javier Movellan, and Paul Ruvolo. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in Neural Information Processing Systems , volume 22, pages 2035-2043, 2009.

Ashish Khetan, Zachary C Lipton, and Anima Anandkumar. Learning from noisy singly-labeled data. arXiv preprint arXiv:1712.04577 , 2017.

Hui Guo, Boyu Wang, and Grace Y Yi. Label correction of crowdsourced noisy annotations with an instance-dependent noise transition model. In Advances in Neural Information Processing Systems , volume 36, pages 347-386, 2023.

Peng Cao, Yilun Xu, Yuqing Kong, and Yizhou Wang. Max-mig: an information theoretic approach for joint learning from crowds. arXiv preprint arXiv:1905.13436 , 2019.

Ryutaro Tanno, Ardavan Saeedi, Swami Sankaranarayanan, Daniel C Alexander, and Nathan Silberman. Learning from noisy labels by regularized estimation of annotator confusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11244-11253, 2019.

Ravid Shwartz-Ziv, Micah Goldblum, Hossein Souri, Sanyam Kapoor, Chen Zhu, Yann LeCun, and Andrew G Wilson. Pre-train your loss: Easy bayesian transfer learning with informative priors. In Advances in Neural Information Processing Systems , volume 35, pages 27706-27715, 2022.

Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? In Advances in Neural Information Processing Systems , volume 32, pages 6838-6849, 2019.

Hisham Husain and Jeremias Knoblauch. Adversarial interpretation of bayesian inference. In Proceedings of The 33rd International Conference on Algorithmic Learning Theory , volume 167, pages 553-572. Proceedings of Machine Learning Research, 2022.

Alexander Shapiro and Alois Pichler. Conditional distributionally robust functionals. Operations Research , 2023.

Jose Blanchet and Karthyek Murthy. Quantifying distributional model risk via optimal transport. Mathematics of Operations Research , 44(2):565-600, 2019.

Rui Gao and Anton Kleywegt. Distributionally robust stochastic optimization with Wasserstein distance. Mathematics of Operations Research , 48(2):603-655, 2023.

Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 5552-5560, 2018.

Thomas M. Cover and Joy A. Thomas. Elements of Information Theory . Wiley-Interscience, 2006.

Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.

Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. In International Conference on Learning Representations , 2022.

Filipe Rodrigues, Mariana Lourenco, Bernardete Ribeiro, and Francisco C Pereira. Learning supervised topic models for classification and regression from crowds. IEEE Transactions on Pattern Analysis and Machine Intelligence , 39(12):2409-2422, 2017.

Antonio Torralba, Bryan C. Russell, and Jenny Yuen. Labelme: Online image annotation and applications. Proceedings of the IEEE , 98(8):1467-1484, 2010.

Hwanjun Song, Minseok Kim, and Jae-Gil Lee. SELFIE: Refurbishing unclean samples for robust deep learning. In Proceedings of the 36th International Conference on Machine Learning , volume 97, pages 5907-5915, 2019.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 770-778, 2016.

Filipe Rodrigues and Francisco Pereira. Deep learning from crowds. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, pages 1611-1618, 2018.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations , 2015.

Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. In Advances in Neural Information Processing Systems , volume 33, pages 7597-7610, 2020.

Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in Neural Information Processing Systems , volume 31, pages 8527-8537, 2018.

Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? In Proceedings of the 36th International Conference on Machine Learning , volume 97, pages 7164-7173, 2019.

Xiaobo Xia, Bo Han, Yibing Zhan, Jun Yu, Mingming Gong, Chen Gong, and Tongliang Liu. Combating noisy labels with sample selection by mining high-discrepancy examples. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1833-1843, 2023.

Hongxin Wei, Huiping Zhuang, Renchunzi Xie, Lei Feng, Gang Niu, Bo An, and Yixuan Li. Mitigating memorization of noisy labels by clipping the model prediction. In Proceedings of the 40th International Conference on Machine Learning , volume 202, pages 36868-36886, 2023.

Melody Guan, Varun Gulshan, Andrew Dai, and Geoffrey Hinton. Who said what: Modeling individual labelers improves classification. In Proceedings of the AAAI conference on artificial intelligence , volume 32, pages 3109-3118, 2018.

Zhendong Chu, Jing Ma, and Hongning Wang. Learning from crowds by modeling common confusions. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 5832-5840, 2021.

Hansong Zhang, Shikun Li, Dan Zeng, Chenggang Yan, and Shiming Ge. Coupled confusion correction: Learning from crowds with sparse annotations. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 16732-16740, 2024.

David G Luenberger and Yinyu Ye. Linear and Nonlinear Programming , volume 2. Springer, 1984.

Stephen P Bradley, Arnoldo C Hax, and Thomas L Magnanti. Applied Mathematical Programming . Addison-Wesley Publishing Company, 1977.

Martin J Wainwright. High-Dimensional Statistics: A Non-Asymptotic Viewpoint . Cambridge University Press, 2019.

Jaeho Lee and Maxim Raginsky. Minimax statistical learning with wasserstein distances. In Advances in Neural Information Processing Systems , volume 31, pages 2687-2696, 2018.

John Hunter. The supremum and infimum. https://www.math.ucdavis.edu/~hunter/m125b/ch2. pdf .

Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning . MIT Press, 2018.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Hongwei Li and Bin Yu. Error rate bounds and iterative weighted majority voting for crowdsourcing. arXiv preprint arXiv:1411.4086 , 2014.

Maria Sofia Bucarelli, Lucas Cassano, Federico Siciliano, Amin Mantrach, and Fabrizio Silvestri. Leveraging inter-rater agreement for classification in the presence of noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3439-3448, 2023.

Songzhu Zheng, Pengxiang Wu, Aman Goswami, Mayank Goswami, Dimitris Metaxas, and Chao Chen. Error-bounded correction of noisy labels. In Proceedings of the 37th International Conference on Machine Learning , volume 119, pages 11447-11457, 2020.

Shahana Ibrahim, Tri Nguyen, and Xiao Fu. Deep learning from crowdsourced labels: Coupled crossentropy minimization, identifiability, and regularization. In The Eleventh International Conference on Learning Representations , 2023.

A Technical Details

A.1 Preliminaries about Linear Programming and Concentration Bounds

A linear program (LP) is an optimization problem of the form

where c ∈ R n and b ∈ R m are given, and A is a specified m × n matrix. Here, ' ≤ ' represents elementwise inequality for vectors. The expression c ⊤ x is called the objective function , and the set { x ∈ R n : Ax ≤ b , x ≥ 0 } defines the feasible region of the linear program. By introducing slack variables, any linear program can be converted to the following standard form :

We begin by introducing the concept of extreme point of related properties.

Definition A.1 ([37, Chapter 2]) . Apoint z in a convex set Θ is called an extreme point of Θ if there do not exist two distinct points z ′ , z ′′ ∈ Θ and a scalar ν with 0 < ν < 1 such that z = ν z ′ +(1 -ν ) z ′′ .

The following lemma shows that optimal solutions of a linear program are located among the extreme points.

Lemma 1 ([37, Chapter 2]) . If a linear programming problem has a finite optimal solution (i.e., a feasible solution that optimizes the objective function), then there is a finite optimal solution that is an extreme point of the constraint set.

The following lemma on strong duality in linear programming will be used in the proof of Proposition 2.1 and Remark 2.3.

Lemma 2 (Duality Theorem of Linear Programming; [38, Chapter 4]) . Let c = ( c 1 , . . . , c n ) ⊤ and b = ( b 1 , . . . , b m ) ⊤ be given vectors, and let A = [ a ij ] be a given m × n matrix with a ij being its ( i, j ) element. Define the primal problem as:

or equivalently written in compact form:

The corresponding dual linear problem is:

SUPPLEMENTARY MATERIAL

or equivalently,

If the primal (or dual) problem has a finite optimal solution, then the dual (or primal) problem also has a finite solution, and the optimal values of the primal and dual problems are equal.

Next we introduce a concentration bound along with associated concepts, which will be used in the proof of Theorem 2.2.

Let Ω denote a subset of R and f : Ω n → R . We say that a function f satisfies the bounded difference inequality [39] with parameters { L 1 , . . . , L n } if for any k ∈ [ n ] ,

̸

That is, for any k ∈ [ n ] , if we substitute s k with s ′ k while keeping other s j fixed for all j = k , the function f changes by at most L k .

Lemma 3 (Bounded Differences Inequality; [39, Corollary 2.21]) . Let S = ( S 1 , . . . , S n ) ⊤ represent a random vector with independent components defined on a sample space Ω n , and f ( S ) ≜ f ( S 1 , . . . , S n ) for a function f : Ω n → R . Suppose that f satisfies the bounded difference property with parameters { L 1 , . . . , L n } . Then, for any t ≥ 0 ,

We also introduce the following definitions and lemmas, which will be used to characterize the complexity of the function class.

Definition A.2 (Covering number; [39, Definition 5.1]) . Let Θ denote a set and ρ a metric on Θ . For t > 0 , a t -cover of Θ with respect to ρ is a set { θ i ∈ Θ : i = 1 , . . . , N } such that for each θ ∈ Θ , there exists some i ∈ [ N ] such that ρ ( θ, θ i ) ≤ t . The t -covering number N ( t ; Θ , ρ ) is the cardinality of the smallest t -cover.

Lemma 4. Let Θ j be a set equipped with a metric ρ j for j = 1 , 2 , and define Θ = Θ 1 × Θ 2 . Given α 1 , α 2 > 0 , define the metric ρ on Θ as ρ ( θ, θ ′ ) ≜ α 1 ρ 1 ( θ 1 , θ 1 ′ ) + α 2 ρ 2 ( θ 2 , θ 2 ′ ) for any θ ≜ ( θ 1 , θ 2 ) ∈ Θ and θ ′ ≜ ( θ 1 ′ , θ 2 ′ ) ∈ Θ . Then for t > 0 ,

Proof. For j = 1 , 2 , let ¯ Θ j ≜ { θ j 1 , . . . , θ j N j } denote the smallest t/ (2 α j ) -cover of Θ j with respect to ρ j . Then, by Definition A.2, N ( t/ (2 α j ); Θ j , ρ j ) = N j for j = 1 , 2 . For any θ = ( θ 1 , θ 2 ) ∈ Θ , by definition A.2, there exists i 1 ∈ [ N 1 ] and i 2 ∈ [ N 2 ] such that ρ 1 ( θ 1 , θ 1 i 1 ) ≤ t/ (2 α 1 ) and ρ 2 ( θ 2 , θ 2 i 2 ) ≤ t/ (2 α 2 ) . Then, θ i ≜ ( θ 1 i 1 , θ 2 i 2 ) ∈ ¯ Θ 1 × ¯ Θ 2 ⊂ Θ , and

Hence, by Definition A.2, ¯ Θ 1 × ¯ Θ 2 is a t -cover of Θ with respect to ρ and N ( t ; Θ , ρ ) ≤ | ¯ Θ 1 × ¯ Θ 2 | = N 1 × N 2 = N ( t/ (2 α 1 ); Θ 1 , ρ 1 ) × N ( t/ (2 α 2 ); Θ 2 , ρ 2 ) , where | · | represents the cardinality of a set.

Lemma 5. Let I ≜ [ a, b ] ⊂ R denote a closed interval on R with a < b . Define the metric ρ on I as ρ ( x, x ′ ) = | x -x ′ | for any x, x ′ ∈ I . Then, for any t > 0 , N ( t ; I , ρ ) ≤ b -a 2 t +1 if t < b -a 2 , and N ( t ; I , ρ ) = 1 if t ≥ b -a 2 .

Proof. Let n t = ⌊ b -a 2 ⌋ , where ⌊·⌋ represents the floor function. To prove the first result for b -a ≥ 2 t , we construct the following subset I of I :

Clearly, I ⊂ ∪ k ∈ [ n t ] [ a + 2 t ( k -1) , a + 2 tk ] ∪ [ a + 2 tn t , b ] . Next, we verify that I is a t -cover of I with respect to metric ρ . Specifically, for any x ∈ I , there exists k ∈ [ n t ] such that x ∈ [ a +2 t · ( k -1) , a +2 t · k ] , or x ∈ [ a +2 tn t , b ] . For the former case,

Hence, by Definition A.2, I is a t -cover of I , therefore N ( t ; I , ρ ) ≤ |I| = n t +1 ≤ b -a 2 t +1 . The first result is then established.

The second result for b -a ≤ 2 t follows from the fact that { a + b -a 2 } is a t -cover of I since ρ ( x, a + b -a 2 ) ≤ b -a 2 ≤ t for any x ∈ I .

Definition A.3 ([39, Definition 5.16]) . A collection of zero-mean random variables { S θ : θ ∈ Θ } is a sub-Gaussian process with respect to a metric ρ on Θ if, for all θ 1 , θ 2 ∈ Θ and t ∈ R ,

Lemma6 (Dudley's Entropy Integral Bound; modified from Theorem 5.22 of [39]) . Let { S θ : θ ∈ Θ } be a zero-mean sub-Gaussian process with respect to a metric ρ on Θ . Then,

where N ( t ; Θ , ρ ) represents the t -covering number of Θ with respect to ρ .

A.2 Proof of Proposition 2.1

Strong duality can be established using Theorem 1 in [18], which applies to general cases. However, by capitalizing the discrete nature of the sample space Y , we can present a more concise result. To see this, here we provide an alternative proof of strong duality using the duality principle in finite-dimensional linear programming, as detailed below.

For every fixed x ∈ X , ˜ y ∈ Y R , ψ ∈ Ψ , by re-writing the the constraint Q y | x , ˜ y ∈ Γ ϵ ( P y | x , ˜ y ) in (1) using Definition 2.1, we re-express the primal problem (1) as:

Given the discrete nature of the sample space Y , P ϵ can be reformulated as the following finitedimensional linear program:

and for the latter case,

where π jk = Π(Y = j, Y ′ = k ) for j, k ∈ [ K ] . By Lemma 2, each primal constraint corresponds to a dual variable. Introducing the dual variables γ and τ k for k ∈ [ K ] , the dual linear program for P ϵ is then expressed as

where first constraint can be written as τ k ≥ max j ∈ [ K ] { ℓ ( ψ ( x ) , j ) -γc p ( j, k ) } . Therefore, to minimize the objective function in (A4), the value of the dual variable τ k should be taken as max j ∈ [ K ] { ℓ ( ψ ( x ) , j ) -γc p ( j, k ) } for each γ ≥ 0 and k ∈ [ K ] . Hence, the proof is established.

A.3 Proof of Remark 2.3

As in the proof of Proposition 2.1, the optimization problem in (6) can be written as

where the first step is due to the definition of the empirical distribution P ( n ) x , ˜ y , and the second step holds by re-writing the constraint 1 n ∑ n i =1 /u1D4B9 ( Q y | x i , ˜ y i , P y | x i , ˜ y i ) ≤ ϵ using the definition of the the Wasserstein distance /u1D4B9 ( · , · ) given in Definition 2.1.

P ϵ can be further expressed as the finite- dimensional linear program:

where π ( i ) jk = Π ( i ) (Y = j, Y ′ = k ) for i ∈ [ n ] and j, k ∈ [ K ] .

By Lemma 2 and introducing dual variables γ and τ ( i ) k with i ∈ [ n ] and k ∈ [ K ] , the dual linear program for P ϵ is expressed as

where we let ˜ τ ( i ) k = nτ ( i ) k for i ∈ [ n ] and k ∈ [ K ] in the second step. Thus, the proof is completed.

A.4 Proof of Theorem 2.2

We begin by demonstrating that, for various choices of the reference distribution, the optimal value for γ in the relaxed dual problem, as stated in (4), is constrained to a compact set. The proof techniques in [40] are used.

Specifically, for a given ϵ > 0 and ψ ∈ Ψ , let

where the second inequality is due to the definition of γ ∗ ; the third inequality comes from the Lipschitz property in the assumption; and the last inequality holds because c ( y ′ , Y) = κ 1 ( y ′ = Y) takes values in { 0 , κ } , leading to sup y ′ ∈Y { L · c ( y ′ , Y) -γ · c p ( y ′ , Y) } = sup t ∈{ 0 ,κ } { Lt -γt p } ≤ sup t ≥ 0 { Lt -γt p } , which is a constant.

Now, we show that

̸

Indeed, when p = 1 , then taking γ = L in (A5) shows (A6). When p > 1 , then Lt -γt p in (A5) takes its maximum value at t ∗ = { L/ ( pγ ) } 1 / ( p -1) , and hence, (A5) yields that for any γ ≥ 0 ,

Therefore, taking γ = L/ ( pϵ p -1 ) leads to γ ∗ ϵ p ≤ Lϵ , i.e., (A6) holds.

Next, let ℓ γ,ψ ( x , ˜ y ) ≜ E P y | x , ˜ y [ sup y ′ ∈Y { ℓ ( ψ ( x ) , y ′ ) -γc p ( y ′ , Y) } ] for any ( x , ˜ y ) ∈ X × Y R . For every ψ ∈ Ψ , we have that

where the first equality comes from (4) and (5), the second equality follows from (A6) and the definition of ℓ γ,ψ , and the third step is due to the fact that | inf v ∈ A f ( v ) -inf v ∈ A g ( v ) | ≤ sup v ∈ A | f ( v ) -g ( v ) | for bounded functions f, g : A → R [41, Proposition 2.18].

In (A7), we use D to stress the dependence of E P ( n ) x , ˜ y { ℓ γ,ψ ( X , ˜ Y ) } on the observed data of size n , as defined in Section 2.1, and let Φ represent the resulting function mapping from ( X × Y R ) n to R , with Φ( D ) being the value for data D , where ( X × Y R ) n is the Cartesian product of multiplying X ×Y R n times. The function Φ : ( X × Y R ) n → R defined in (A7) satisfies the bounded difference property (A3) with parameters { M n , . . . , M n } , where M represents the upper bound of the loss function ℓ in the assumption of Theorem 2.2. Indeed, for any k ∈ [ n ] ,

where the second step holds since | sup v ∈ A f ( v ) -sup v ∈ A g ( v ) | ≤ sup v ∈ A | f ( v ) -g ( v ) | for bounded functions f, g : A → R [41, Proposition 2.18], the third step is due to the triangle inequality for absolute values, and the last step holds since ℓ γ,ψ ∈ [0 , M ] by definition.

Thus, by letting t = M √ log(1 /η ) 2 n in lemma 3, we have that, with probability at least 1 -η ,

Similar to the derivations for (3.8)-(3.13) in the proof of Theorem 3.3 in [42], we obtain that

where { σ i } n i =1 are independent random variables chosen from {-1 , +1 } with equal probability, and the expectation is taken with respect to all involved random variables.

Applying (A8) and (A9) to (A7) gives that with probability at least 1 -η ,

Now we identify an entropy based upper bound for the right-hand side of (A9) using the proof techniques in [40] and Example 5.24 of [39]. Specifically, for a given ψ , we define the random process { S γ ≜ 1 √ n ∑ n i =1 σ i ℓ γ,ψ ( X i , ˜ Y i ) : γ ∈ [0 , M ∗ ] } . By using E ( σ i ) = 0 and the independence between σ i and ( X i , ˜ Y i ) , we obtain that E ( S γ ) = 1 √ n ∑ n i =1 E ( σ i ) E { ℓ γ,ψ ( X i , ˜ Y i ) } = 0 . For any

γ 1 , γ 2 ∈ [0 , M ∗ ] ,

where the second inequality is due to the fact that | sup A f -sup A g | ≤ sup A | f -g | for bounded functions f, g : A → R [41, Proposition 2.18], and the last step is due to the fact that c ( y ′ , Y) can only take values in { 0 , κ } .

Hence, for t ∈ R , we have that

where the inequality is due to Hoeffding's lemma and (A11). Thus, { S γ : γ ∈ [0 , M ∗ ] } is a zeromean sub-Gaussian process with respect to metric ρ γ , defined as ρ γ ( γ 1 , γ 2 ) = κ p | γ 1 -γ 2 | for any γ 1 , γ 2 ∈ [0 , M ∗ ] .

Therefore, by Lemma 6, we obtain

where the third step comes from the fact that, by the definition of ρ γ , ρ γ ( γ 1 , γ 2 ) ≤ t if and only if | γ 1 -γ 2 | ≤ t/κ p ; the fourth step holds by letting s = t/κ p ; the fifth step follows from Lemma 5; the

sixth step comes from the fact that 1 < M ∗ 2 s +1 = M ∗ +2 s 2 s ≤ M ∗ s for s ∈ [0 , M ∗ / 2] ; the penultimate step holds by letting u = s/M ∗ ; and the last step arises from the fact that

where erfc ( x ) = 2 √ π ∫ ∞ x e -w 2 dw .

Applying (A13) and (A6) to (A10) gives that with probability at least 1 -η ,

where the last step is due to the fact that 8 √ 2( √ log 2 + √ π erfc ( √ log 2)) ≈ 14 . 2 < 15 . Hence, the proof is completed.

A.5 Proof of Corollary 2.3

By the definition of inf ψ ∈ Ψ R ϵ ( ψ ; P y | x , ˜ y ) , for any ζ > 0 , there exists ψ ζ ∈ Ψ such that R ϵ ( ψ ζ ; P y | x , ˜ y ) ≤ inf ψ ∈ Ψ R ϵ ( ψ ; P y | x , ˜ y ) + ζ . Therefore,

Since the inequality above is true for all ζ > 0 , we have that

where the second inequality arises from (A7).

Similar to the proof of Theorem 2.2 in Appendix A.4, we can derive that

For any γ 1 , γ 2 ∈ [0 , M ∗ ] and ψ 1 , ψ 2 ∈ Ψ ,

where the first step is due to the definition of ℓ γ,ψ defined after (A6), the second step is due to Jensen's inequality, and the last step is due to the Lipschitz property with respect to the cost function c ( · , · ) defined in Theorem 2.2 in the assumption, and ∥ ψ 1 -ψ 2 ∥ ∞ ≜ sup x ∈X ∥ ψ 1 ( x ) -ψ 2 ( x ) ∥ for some norm ∥ · ∥ .

Allowing ψ to vary, we modify the discussion for the random process { S γ : γ ∈ [0 , M ∗ ] } in Appendix A.4, and consider the collection of random variables { S γ,ψ ≜ 1 √ n ∑ n i =1 σ i ℓ γ,ψ ( X i , ˜ Y i ) : γ ∈ [0 , M ∗ ] , ψ ∈ Ψ } . Clearly, E ( S γ,ψ ) = 0 . Modifying the metric ρ γ ( γ 1 , γ 2 ) in Appendix A.4, we define the metric ρ γ,ψ (( γ 1 , ψ 1 ) , ( γ 2 , ψ 2 )) ≜ κ p | γ 1 -γ 2 | + L ′ ∥ ψ 1 -ψ 2 ∥ ∞ for any γ 1 , γ 2 ∈ [0 , M ∗ ] and ψ 1 , ψ 2 ∈ Ψ . Similar to deriving (A12), we obtain that for t ∈ R ,

Thus, { S γ,ψ : γ ∈ [0 , M ∗ ] , ψ ∈ Ψ } is a zero-mean sub-Gaussian process with respect to metric ρ γ,ψ .

Let the Cartesian product [0 , M ∗ ] × Ψ denote the 'parameter' space of ( γ, ψ ) . Then, by Lemma 6, we obtain

Here, for any set Ω and metric ρ on Ω , N ( t ; Ω , ρ ) denotes the t -covering number for t > 0 as defined in Definition A.2. In the derivation of (A16), the third step is due to Lemma 4; the fourth step is due to √ log( ab ) ≤ √ log a + √ log b for a ≥ 1 and b ≥ 1 ; the fifth step results from a change of variable; and the last line can be similarly proved as (A13).

By (A14)-(A16), we have that, with probability at least 1 -η ,

Therefore, the proof is completed.

A.6 Proof of Theorem 3.1

For ease of presentation, in this proof we omit the dependence on x and ˜ y in the notation. In particular, we let P 0 ≜ P 0 ( x , ˜ y ) , P 1 ≜ P 1 ( x , ˜ y ) , and ψ ≜ ψ ( x ) . Let the objective function in (8) be denoted as

To complete the proof, we begin by investigating the inner optimization problem in (8) by finding the optimal value of γ , γ ∗ ψ , as defined in Section 3.2, that minimizes g ( γ ; ψ ) for each given ψ , and then address the outer optimization problem in (8) by finding the optimal value ψ ⋆ that minimizes g ( γ ∗ ψ ; ψ ) . To this end, we eliminate the max operators in g ( γ ; ψ ) based on the values of ψ , and use the assumption that T is a decreasing function in (7). As ψ takes its value in [0 , 1] , we re-write the optimization problem (8) as

where ( γ ∗ ψ ∗ 1 , ψ ∗ 1 ) and ( γ ∗ ψ ∗ 2 , ψ ∗ 2 ) are the arguments of min ψ 1 ∈ [0 , 1 2 ] inf γ ≥ 0 g ( γ ; ψ 1 ) and min ψ 2 ∈ [ 1 2 , 1] inf γ ≥ 0 g ( γ ; ψ 2 ) , respectively. We complete the proof by considering the following two cases.

Case 1: ψ 1 ∈ [0 , 1 2 ] .

Consequently, we have that

For given ψ 1 ,

showing that g ( γ ; ψ 1 ) is continuous at γ 0 . Therefore, for any given ψ 1 , g ( γ ; ψ 1 ) is continuous in γ over R + .

Then, for any given ψ 1 ∈ [0 , 1 2 ] , corresponding to the first term in (A18), we obtain that

where we use the continuity of g ( γ ; ψ 1 ) in γ , the fact that g ( γ ; ψ 1 ) is increasing in γ when γ > γ 0 , and the fact that a continuous function attains its infimum within any closed and bounded set in R . Here,

for any ψ 1 ∈ [0 , 1 2 ] .

We complete the proof by the following two steps to examine the range of P 0 .

Step 1: If P 0 < ϱ ( ϵ ) , then, by (A19), for any given ψ 1 ∈ [0 , 1 2 ] , g ( γ ; ψ 1 ) is increasing in γ over [0 , γ 0 ] , showing that the optimal value in (A21) is γ ∗ ψ 1 = 0 . Furthermore, because g ( γ ∗ ψ 1 ; ψ 1 ) = T ( ψ 1 ) for any ψ 1 ∈ [0 , 1 2 ] , we obtain that

Consequently, (0 , 1 2 ) minimizes g ( γ ; ψ ) over [0 , γ 0 ] × [0 , 1 2 ] , i.e., ψ ∗ 1 = 1 2 and γ ∗ ψ ∗ 1 = 0 .

Step 2: If P 0 ≥ ϱ ( ϵ ) , then, by (A19), g ( γ ; ψ 1 ) is decreasing in γ when γ ≤ γ 0 , showing that the optimal value in (A21) is

and thus,

Consequently, the derivative of g ( γ ∗ ψ 1 ; ψ 1 ) with respect to ψ 1 is

leading to g ′ ψ 1 ( γ ∗ ψ 1 ; ψ 1 ) ∣ ∣ ψ 1 =0 = (1 -P 0 + ϱ ( ϵ )) T ′ (0) -( P 0 -ϱ ( ϵ )) T ′ (1) and g ′ ψ 1 ( γ ∗ ψ 1 ; ψ 1 ) ∣ ∣ ψ 1 =1 / 2 = (1 + 2 ϱ ( ϵ ) -2 P 0 ) T ′ ( 1 2 ) . Solving

for P 0 leads to solutions

respectively.

Next, we identify ψ ∗ 1 and γ ∗ ψ ∗ 1 by examining P 0 relative to P (1) 0 and P (2) 0 , in combination with the convexity or concavity of function T by the following two steps.

Step 2.1: Assume T is concave. Then by twice differentiability of T , T ′′ ( ψ 1 ) ≤ 0 for ψ 1 ∈ [0 , 1] , leading to T ′ (1) ≤ T ′ (0) < 0 , and hence P (1) 0 ≤ P (2) 0 . Additionally, g ′′ ψ 1 ( γ ∗ ψ 1 ; ψ 1 ) = ( P 1 + ϱ ( ϵ )) T ′′ ( ψ 1 ) + ( P 0 -ϱ ( ϵ )) T ′′ (1 -ψ 1 ) ≤ 0 , and thus, g ′ ψ 1 ( γ ∗ ψ 1 ; ψ 1 ) is non-increasing in ψ 1 for ψ 1 ∈ [0 , 1 2 ] .

If ϱ ( ϵ ) ≤ P 0 ≤ P (1) 0 , then g ′ ψ 1 ( γ ∗ ψ 1 ; ψ 1 ) ≤ 0 for ψ 1 ∈ [0 , 1 2 ] , and thus, g ( γ ∗ ψ 1 ; ψ 1 ) is nonincreasing in ψ 1 for ψ 1 ∈ [0 , 1 2 ] . Therefore, inf 0 ≤ ψ 1 ≤ 1 / 2 g ( γ ∗ ψ 1 ; ψ 1 ) = g ( γ ∗ 1 / 2 ; 1 2 ) = T ( 1 2 ) . Thus, ψ ∗ 1 = 1 2 and γ ∗ ψ ∗ 1 = 0 by (A22).

If P 0 ≥ P (2) 0 , then g ′ ψ 1 ( γ ∗ ψ 1 ; ψ 1 ) ≥ 0 , showing that g ( γ ∗ ψ 1 ; ψ 1 ) is non-decreasing in ψ 1 for ψ 1 ∈ [0 , 1 2 ] . Therefore, inf 0 ≤ ψ 1 ≤ 1 / 2 g ( γ ∗ ψ 1 ; ψ 1 ) = g ( γ ∗ 0 ; 0) = ( P 1 + ϱ ( ϵ )) T (0) + ( P 0 -ϱ ( ϵ )) T (1) . Thus, ψ ∗ 1 = 0 and γ ∗ ψ ∗ 1 = T (0) -T (1) κ p by (A22).

If P (1) 0 < P 0 < P (2) 0 , then g ′ ψ 1 ( γ ∗ ψ 1 ; ψ 1 ) is non-increasing in ψ 1 for ψ 1 ∈ [0 , 1 2 ] with g ′ ψ 1 ( γ ∗ ψ 1 ; ψ 1 ) ∣ ∣ ψ 1 =0 > 0 and g ′ ψ 1 ( γ ∗ ψ 1 ; ψ 1 ) ∣ ∣ ψ 1 =1 / 2 < 0 . Therefore, g ′ ψ 1 ( γ ∗ ψ 1 ; ψ 1 ) = 0 has a unique solution on [0 , 1 2 ] , denoted ψ ⋄ 1 , and furthermore, g ( γ ∗ ψ 1 ; ψ 1 ) is increasing in ψ 1 for ψ 1 ∈ [0 , ψ ⋄ 1 ] and decreasing on [ ψ ⋄ 1 , 1 2 ] . Therefore, the infimum of g ψ 1 ( γ ∗ ψ 1 ; ψ 1 ) on ψ 1 ∈ [0 , 1 2 ] is taken at ψ 1 = 0 or ψ 1 = 1 2 . If g ( γ ∗ 1 / 2 ; 1 2 ) ≤ g ( γ ∗ 0 ; 0) , i.e., P 0 ≤ ϱ ( ϵ )+ T (0) -T (1 / 2) T (0) -T (1) , the optimal value for ψ 1 in Case 1 is ψ ∗ 1 = 1 2 with γ ∗ ψ ∗ 1 = 0 ; otherwise, the optimal value is ψ ∗ 1 = 0 with γ ∗ ψ ∗ 1 = T (0) -T (1) κ p by (A22).

Summarizing the discussion in Step 2.1, we obtain that when ψ 1 ∈ [0 , 1 2 ] and T is concave,

(i) if P 0 > ϱ ( ϵ ) + T (0) -T (1 / 2) T (0) -T (1) , ( ψ ∗ 1 , γ ∗ ψ ∗ 1 ) in (A18) is given by ψ ∗ 1 = 0 and γ ∗ ψ ∗ 1 = T (0) -T (1) κ p , yielding g ( γ ∗ ψ ∗ 1 ; ψ ∗ 1 ) = ( P 1 + ϱ ( ϵ )) T (0) + ( P 0 -ϱ ( ϵ )) T (1) ;

(ii) otherwise, ψ ∗ 1 = 1 2 and γ ∗ ψ ∗ 1 = 0 , yielding g ( γ ∗ ψ ∗ 1 ; ψ ∗ 1 ) = T ( 1 2 ) .

Step 2.2: Assume T is convex. Then by twice differentiability of T , T ′′ ( ψ 1 ) ≥ 0 for ψ 1 ∈ [0 , 1] , leading to T ′ (0) ≤ T ′ (1) < 0 , and hence P (2) 0 ≤ P (1) 0 . Additionally, g ′′ ψ 1 ( γ ∗ ψ 1 ; ψ 1 ) = ( P 1 + ϱ ( ϵ )) T ′′ ( ψ 1 ) + ( P 0 -ϱ ( ϵ )) T ′′ (1 -ψ 1 ) ≥ 0 , and thus, g ′ ψ 1 ( γ ∗ ψ 1 ; ψ 1 ) is non-decreasing in ψ 1 for ψ 1 ∈ [0 , 1 2 ] .

If P 0 ≥ P (1) 0 , then g ′ ψ 1 ( γ ∗ ψ 1 ; ψ 1 ) ≥ 0 for ψ 1 ∈ [0 , 1 2 ] , and thus, g ( γ ∗ ψ 1 ; ψ 1 ) is non-decreasing in ψ 1 for ψ 1 ∈ [0 , 1 2 ] . Therefore, inf 0 ≤ ψ 1 ≤ 1 / 2 g ( γ ∗ ψ 1 ; ψ 1 ) = g ( γ ∗ 0 ; 0) = ( P 1 + ϱ ( ϵ )) T (0) + ( P 0 -ϱ ( ϵ )) T (1) . Thus, ψ ∗ 1 = 0 and γ ∗ ψ ∗ = T (0) -T (1) κ p by (A22).

1

If ϱ ( ϵ ) ≤ P 0 ≤ P (2) 0 , then g ′ ψ 1 ( γ ∗ ψ 1 ; ψ 1 ) ≤ 0 for ψ 1 ∈ [0 , 1 2 ] , and thus, g ( γ ∗ ψ 1 ; ψ 1 ) is nonincreasing in ψ 1 for ψ 1 ∈ [0 , 1 2 ] . Therefore, inf 0 ≤ ψ 1 ≤ 1 2 g ( γ ∗ ψ 1 ; ψ 1 ) = g ( γ ∗ 1 / 2 ; 1 2 ) = T ( 1 2 ) . Thus, ψ ∗ 1 = 1 and γ ∗ ∗ = 0 by (A22).

2 ψ 1

If P (2) 0 < P 0 < P (1) 0 , then g ′ ψ 1 ( γ ∗ ψ 1 ; ψ 1 ) is non-decreasing in ψ 1 for ψ 1 ∈ [0 , 1 2 ] with g ′ ψ 1 ( γ ∗ ψ 1 ; ψ 1 ) ∣ ∣ ψ 1 =0 < 0 and g ′ ψ 1 ( γ ∗ ψ 1 ; ψ 1 ) ∣ ∣ ψ 1 =1 / 2 > 0 . Therefore, g ′ ψ 1 ( γ ∗ ψ 1 ; ψ 1 ) = 0 has a unique solution on [0 , 1 2 ] , denoted ψ ⋄ 1 , and furthermore, g ( γ ∗ ψ 1 ; ψ 1 ) is decreasing in ψ 1 for ψ 1 ∈ [0 , ψ ⋄ 1 ] and increasing on [ ψ ⋄ 1 , 1 / 2] . Then, the infimum of g ψ 1 ( γ ∗ ψ 1 ; ψ 1 ) on ψ 1 ∈ [0 , 1 2 ] is taken at ψ 1 = ψ ⋄ 1 , that is, inf 0 ≤ ψ 1 ≤ 1 / 2 g ( γ ∗ ψ 1 ; ψ 1 ) = g ( γ ∗ ψ ⋄ 1 ; ψ ⋄ 1 ) = ( P 1 + ϱ ( ϵ )) T ( ψ ⋄ 1 ) + ( P 0 -ϱ ( ϵ )) T (1 -ψ ⋄ 1 ) . Thus, ψ ∗ 1 = ψ ⋄ 1 with γ ∗ ψ ∗ 1 = T ( ψ ⋄ 1 ) -T (1 -ψ ⋄ 1 ) κ p by (A22).

Case 2: ψ 2 ∈ [ 1 2 , 1] .

In this case, we set ψ 2 ≜ 1 -ψ 2 , yielding ψ 2 ∈ [0 , 1 2 ] , and the objective function g ( γ ; ψ 2 ) defined in (A17) can be written as

Hence, the derivation in Case 1 for any ψ 1 in [0 , 1 2 ] can be applied to ψ 2 by modifying the derivations based on the range of P 0 to be that for P 1 , as outlined below.

Step 1: If P 1 < ϱ ( ϵ ) , then following the results for ψ ∗ 1 and γ ∗ ψ ∗ 1 in Case 1, with only ψ ∗ 1 , P 1 , and P 0 there replaced by ψ ∗ 2 , P 0 , and P 1 , respectively, we obtain that ψ ∗ 2 = 1 2 and γ ∗ ψ ∗ 2 = 0 . Hence, ψ ∗ 2 is taken as 1 -ψ ∗ 2 = 1 2 and γ ∗ ψ ∗ 2 = 0 , yielding g ( γ ∗ ψ ∗ 2 ; ψ ∗ 2 ) = g (0; 1 2 ) = T ( 1 2 ) .

Step 2: If P 1 ≥ ϱ ( ϵ ) , then, by (A22), γ ∗ ψ ∗ 2 is set as γ ∗ ψ ∗ 2 = T ( ψ ∗ 2 ) -T (1 -ψ ∗ 2 ) κ p = T (1 -ψ ∗ 2 ) -T ( ψ ∗ 2 ) κ p .

Step 2.1: Assume T is concave. We can directly derive the following result from the summary in Step 2.1 of Case 1.

-If P 1 ≤ ϱ ( ϵ ) + T (0) -T (1 / 2) T (0) -T (1) , then ψ ∗ 2 = 1 2 and γ ∗ ψ ∗ 2 = 0 . Hence, ψ ∗ 2 = 1 -ψ ∗ 2 = 1 2 , and g ( γ ∗ ψ ∗ 2 ; ψ ∗ 2 ) = g (0; 1 2 ) = T ( 1 2 ) .

-If P 1 > ϱ ( ϵ ) + T (0) -T (1 / 2) T (0) -T (1) , then ψ ∗ 2 = 0 and γ ∗ ψ ∗ 2 = T (0) -T (1) κ p . Hence, ψ ∗ 2 = 1 -ψ ∗ 2 = 1 , and g ( γ ∗ ψ ∗ 2 ; ψ ∗ 2 ) = g ( T (0) -T (1) κ p ; 1) = ( P 0 + ϱ ( ϵ )) T (0)+( P 1 -ϱ ( ϵ )) T (1) .

Step 2.2: Assume T is convex. From the results on ψ ∗ 1 and γ ∗ ψ ∗ 1 in Step 2.2 of Case 1, we obtain the following conclusion.

-If P 1 ≥ ϱ ( ϵ )+ T ′ (0) T ′ (1)+ T ′ (0) , then ψ ∗ 2 = 0 and γ ∗ ψ ∗ 2 = T (0) -T (1) κ p . Hence, ψ ∗ 2 = 1 -ψ ∗ 2 = 1 , and g ( γ ∗ ψ ∗ 2 ; ψ ∗ 2 ) = g ( T (0) -T (1) κ p ; 1) = ( P 0 + ϱ ( ϵ )) T (0) + ( P 1 -ϱ ( ϵ )) T (1) .

-If ϱ ( ϵ ) ≤ P 1 ≤ ϱ ( ϵ ) + 1 2 , then ψ ∗ 2 = 1 2 and γ ∗ ψ ∗ 2 = 0 . Hence, ψ ∗ 2 = 1 -ψ ∗ 2 = 1 2 , and g ( γ ∗ ψ ∗ 2 ; ψ ∗ 2 ) = g (0; 1 2 ) = T ( 1 2 ) .

-If ϱ ( ϵ ) + 1 2 < P 1 < ϱ ( ϵ ) + T ′ (0) T ′ (1)+ T ′ (0) , then ψ ∗ 2 = ψ ⋄ 2 and γ ∗ ψ ∗ 2 = T ( ψ ⋄ 2 ) -T (1 -ψ ⋄ 2 ) κ p , where ψ ⋄ 2 is the unique solution to ( P 0 + ϱ ( ϵ )) T ′ ( ψ 2 ) -( P 1 -ϱ ( ϵ )) T ′ (1 -ψ 2 ) = 0 on [0 , 1 2 ] . Hence, ψ ∗ 2 = ψ ⋄ 2 and γ ∗ ψ ∗ 2 = T (1 -ψ ⋄ 2 ) -T ( ψ ⋄ 2 ) κ p , where ψ ⋄ 2 = 1 -ψ ⋄ 2 is the unique solution to -( P 0 + ϱ ( ϵ )) T ′ (1 -ψ 2 ) + ( P 1 -ϱ ( ϵ )) T ′ ( ψ 2 ) = 0 on [ 1 2 , 1] . Then g ( γ ∗ ψ ∗ 2 ; ψ ∗ 2 ) = ( P 0 + ϱ ( ϵ )) T (1 -ψ ⋄ 2 ) + ( P 1 -ϱ ( ϵ )) T ( ψ ⋄ 2 ) .

In summary, we present the derived results in Tables 3 and 4 for the scenarios where T is concave and convex, respectively.

Finally, for any given input x , applying the preceding results to the optimal solution ( γ ∗ ψ ∗ , ψ ∗ ) = arg inf ψ ∈ [0 , 1] inf γ ≥ 0 g ( γ, ψ ) in (A18), we obtain that the optimal action on the given instance x is given as below: for concave T ,

and for convex T ,

where t ∗ 0 is the unique solution of ( P 0 -ϱ ( ϵ )) T ′ (1 -t ) = ( P 1 + ϱ ( ϵ )) T ′ ( t ) on t ∈ (0 , 1 2 ) , and t ∗ 1 is the unique solution of ( P 0 + ϱ ( ϵ )) T ′ (1 -t ) = ( P 1 -ϱ ( ϵ )) T ′ ( t ) on t ∈ ( 1 2 , 1) . Hence, the proof is established.

A.7 Proof of Theorem 3.2

For ease of presentation, we omit the dependence on x and ˜ y in the notation for now. Specifically, for j ∈ [ K ] , we let P j ≜ P j ( x , ˜ y ) ≜ P (Y = j | x , ˜ y ) and ψ j ≜ ψ ( x ) j . Let the objective function in (9) be denoted as

We complete the proof in four steps. In Step 1, for each given ψ , we investigate the inner optimization problem in (9) by finding the optimal value of γ , defined as γ ⋆ ψ ≜ arg min γ ≥ 0 g ( γ ; ψ ) . Then, in Step 2, by substituting γ ⋆ ψ into g ( γ ; ψ ) , we find that the outer optimization problem in (9) can be written in a linear programming format under certain transformations. Next, in Step 3, we find the extreme points of the associated linear programming, and finally, in Step 4, we obtain the solution format of the optimal action ψ ⋆ .

Step 1: For any ψ ∈ Ψ , finding the optimal value of γ , defined as γ ⋆ ψ ≜ arg min γ ≥ 0 g ( γ ; ψ ) . Given ψ and x , we sort { ψ 1 , . . . , ψ K } in an decreasing order, denoted ψ (1) ≥ . . . ≥ ψ ( K ) , and hence, 1 -ψ (1) ≤ . . . ≤ 1 -ψ ( K ) . Assume that { ψ (1) , . . . , ψ ( K ) } corresponds to { ψ 1 , . . . , ψ K } via a permutation χ , that is, ψ ( j ) = ψ χ ( j ) for j ∈ [ K ] . Correspondingly, the P j 's with the associated indexes are denoted P ( j ) ≜ P χ ( j ) for j ∈ [ K ] . Then, for the χ ( j ) -th element in the summation of (A23), the maximum is taken between 1 -ψ ( K ) -γκ p and 1 -ψ ( j ) .

First, for given ψ , we examine the continuity of g ( γ ; ψ ) in γ by eliminating the max operators in (A23), which is conducted by comparing 1 -ψ ( K ) -γκ p and 1 -ψ ( j ) for j ∈ [ K ] as follows.

which is continuous in γ for γ ≥ ψ (1) -ψ ( K ) κ p .

On the other hand, if 1 -ψ (1) < 1 -ψ ( K ) -γκ p , i.e., 0 ≤ γ < ψ (1) -ψ ( K ) κ p , then we express the range of γ as:

Then we consider γ in each interval [ ψ ( s +1) -ψ ( K ) κ p , ψ ( s ) -ψ ( K ) κ p ) for s ∈ [ K -1] . In this case, 1 -ψ ( s ) < 1 -ψ ( K ) -γκ p ≤ 1 -ψ ( s +1) , and (A23) becomes

where the last step holds by re-arranging the arguments and using the definition of ϱ ( ϵ ) given after (8). Consequently,

where the last step comes from the expression (A25) for g ( γ ; ψ ) when ψ ( s ) -ψ ( K ) κ p ≤ γ < ψ ( s -1) -ψ ( K ) κ p . Thus, g ( γ ; ψ ) is continuous in γ for γ ∈ [ ψ ( s +1) -ψ ( K ) κ p , ψ ( s ) -ψ ( K ) κ p ] with s ∈ [ K -1] . Consequently, g ( γ ; ψ ) is continuous in γ for 0 ≤ γ ≤ ψ (1) -ψ ( K ) κ p .

Therefore, combining the discussion regrading (A24) and(A25), we obtain that given ψ , g ( γ ; ψ ) is continuous in γ for γ ≥ 0 .

Next, for each given ψ , we examine the monotonicity of g ( γ ; ψ ) in γ to find the γ that minimizes g ( γ ; ψ ) . To this end, we consider the following three cases by the values of ϱ ( ϵ ) .

then there exists an s ∗ ∈ { 2 , . . . , K } such that ∑ s ∗ -1 j =1 P ( j ) ≤ ϱ ( ϵ ) ≤ ∑ s ∗ j =1 P ( j ) . Then, by (A25), g ( γ ; ψ ) is decreasing in γ for γ ∈ [0 , ψ ( s ∗ ) -ψ ( K ) κ p ] and increasing for γ ∈ [ ψ ( s ∗ ) -ψ ( K ) κ p , ψ (1) -ψ ( K ) κ p ] ; and by (A24), g ( γ ; ψ ) is increasing in γ for γ ≥ ψ (1) -ψ ( K ) κ p . Therefore, γ ⋆ ψ = ψ ( s ∗ ) -ψ ( K ) κ p .

Case 2: If ϱ ( ϵ ) ≤ P (1) , then by (A25), g ( γ ; ψ ) is decreasing in γ for γ ∈ [0 , ψ (1) -ψ ( K ) κ p ] ; and by (A24), increasing in γ for γ ≥ ψ (1) -ψ ( K ) κ p . Therefore, γ ⋆ ψ = ψ (1) -ψ ( K ) κ p .

then by (A25), g ( γ ; ψ ) is increasing in γ for γ ∈ [0 , ψ (1) -ψ ( K ) κ p ] ; and by (A24), increasing in γ for γ ≥ ψ (1) -ψ ( K ) κ p . Therefore, γ ⋆ ψ = 0 .

Therefore, we conclude that

Step 2: Linear programming format.

For each fixed permutation χ , we now find the optimal ψ that minimizes g ( γ ⋆ ψ ; ψ ) by examining the three cases in Step 1.

In Case 3 of Step 1, g ( γ ⋆ ψ ; ψ ) = g (0; ψ ) = 1 -ψ ( K ) ≥ 1 -1 /K . Then the corresponding optimal action is ψ (1) = . . . = ψ ( K ) = 1 /K .

In Case 1 of Step 1, for a single data point ( x , ˜ y ) , by substituting γ ⋆ ψ = ψ ( s ∗ ) -ψ ( K ) κ p with s ∗ ∈ { 2 , . . . , K } into (A25), we obtain that

To find the optimal value that minimizes g ( γ ⋆ ψ ; ψ ) in (A26), we link it with a linear programming problem. Specifically, for j ∈ [ K ] , let z j ≜ 1 -ψ ( j ) and z ≜ ( z 1 , . . . , z K ) ⊤ . Define

̸

When s ∗ = K -1 , only the entries for j = s ∗ = K -1 and j = K need to be considered. Let V ( z ) = ∑ K j = s ∗ a j z j . Then, the optimal ψ that minimizes g ( γ ∗ ψ ; ψ ) in (A26) can be derived by solving the linear programming problem:

where the constraint ∑ K j =1 (1 -z j ) = 1 is due to ∑ K j =1 (1 -z j ) = ∑ K j =1 ψ ( j ) = 1 by the definitions of z and ψ , and the constraint 0 ≤ z 1 ≤ . . . ≤ z K ≤ 1 reflects the definition of ψ ( j ) for j ∈ [ K ] .

Similarly, in Case 2 of Step 1, by substituting γ ⋆ ψ = ψ (1) -ψ ( K ) κ p into (A24), we obtain that

which is a form similar to (A26) if letting s ∗ in (A26) equal 1. Hence, its optimal minimizer can be found through a linear programming problem similar to (A27). Consequently, in the next step, our discussion focuses on (A26) only.

Step 3: Extreme points.

The feasible region of (A27), denoted Ξ , can be expressed as follows:

where the second step holds since, for j ∈ [ K ] , j (1 -z j ) = ∑ j t =1 (1 -z j ) ≤ ∑ j t =1 (1 -z t ) ≤ 1 as z t ≤ z j for t ∈ [ j ] , and the last step holds by rearranging the equality ∑ K j =1 (1 -z j ) = 1 .

We next prove that the following K feasible solutions are the only extreme points of (A27):

We denote Ξ 0 ≜ { z 1 , . . . , z K } .

Firstly, we prove that each data point in Ξ 0 is an extreme point of (A27). To this end, consider any z j ∈ Ξ 0 . If there exist ν ∈ (0 , 1) , z ′ = ( z ′ 1 , . . . , z ′ K ) ⊤ ∈ Ξ , and z ′′ = ( z ′′ 1 , . . . , z ′′ K ) ⊤ ∈ Ξ , such that z j = ν z ′ +(1 -ν ) z ′′ , then z ′ = z ′′ = z j , as shown below. Let z j,t , z ′ t , and z ′′ t represent the t th element of z j , z ′ , and z ′′ , respectively.

If t = j +1 , . . . , K : then by νz ′ t +(1 -ν ) z ′′ t = z j,t , z j,t = 1 , and z ′ t , z ′′ t ≤ 1 , we have that z ′ t = z ′′ t = z j,t = 1 ;

If t = j : then νz ′ j +(1 -ν ) z ′′ j = z j,j = 1 -1 j , and z ′ j , z ′′ j ≥ 1 -1 j by (A28). Thus, we obtain that z ′ j = z ′′ j = z j,j .

If t = 1 , . . . , j -1 : then z ′ t ≤ z ′ j = 1 -1 j , z ′′ t ≤ z ′′ j = 1 -1 j , and νz ′ t +(1 -ν ) z ′′ t = z j,t = 1 -1 j . Thus, we can also obtain that z ′ t = z ′′ t = z j,t .

Therefore, z ′ = z ′′ = z j , and hence, z j is an extreme point of (A27) by Definition A.1.

Next, for any point ˜ z ≜ ( ˜ z 1 , . . . , ˜ z K ) ⊤ ∈ Ξ \ Ξ 0 , we prove that ˜ z is not an extreme point of (A27) by construction. Specifically, we have the following claims for ˜ z .

Claim 1: z > 1 -1 for t ∈ [ K ]

˜ t t : This claim can be proved by contradiction. Assume there exists t 0 ∈ [ K ] such that ˜ z t 0 ≤ 1 -1 t 0 . As ˜ z ∈ Ξ , by (A28) and the assumption, we have ˜ z t 0 = 1 -1 t 0 . Since ˜ z / ∈ Ξ 0 , one of the following statements must hold: (1) there exists j < t 0 such that ˜ z j < 1 -1 t 0 ; or (2) there exists j > t 0 such that ˜ z j < 1 for some j > t 0 . Therefore, ∑ K j =1 ˜ z j = ∑ t 0 j =1 ˜ z j + ∑ K j = t 0 +1 ˜ z j < ∑ t 0 j =1 ˜ z t 0 + ∑ K j = t 0 +1 1 = t 0 · ˜ z t 0 +( K -t 0 ) · 1 = K -1 since ˜ z 1 ≤ . . . ≤ ˜ z K and ˜ z t ≤ 1 for t ∈ [ K ] by (A28), where the strict inequality arises from the fact that either statement (1) or (2) holds. This conclusion contradicts the condition that ˜ z ∈ Ξ by (A28).

Claim 2: There exists t 1 ∈ [ K ] such that ˜ z t 1 -1 < ˜ z t 1 < 1 : This claim can be proved by contradiction:

-On one hand, if there exists t ′ ∈ [ K ] such that ˜ z t ′ -1 < ˜ z t ′ , then we must have ˜ z t = ˜ z t ′ -1 for t ≤ t ′ -1 ; otherwise, by letting t ′′ = arg max { t : ˜ z t < ˜ z t ′ -1 , t < t ′ -1 } , we obtain that ˜ z t ′′ < ˜ z t ′′ +1 = ˜ z t ′ < 1 and hence, t 1 can be set as t ′′ +1 , which contradicts the assumption. Additionally, we have ˜ z t = 1 for t ≥ t ′ ; otherwise, ˜ z t ′ -1 < ˜ z t ′ < 1 and t 1 can be set as t ′ , which contradicts the assumption. Summarizing the discussion for t ≤ t ′ -1 and t ≥ t ′ , we have ˜ z ∈ Ξ 0 .

-On the other hand, if ˜ z t -1 = ˜ z t for all t ∈ [ K ] , then ˜ z = z K ∈ Ξ 0 . In both cases, ˜ z ∈ Ξ 0 , contradicting the fact that ˜ z / ∈ Ξ 0 . Hence, Claim 2 holds.

Let t 2 ≜ max { t ∈ [ K ] : ˜ z t < 1 } . Then, t 2 ≥ t 1 . Let

By Claims 1 and 2, we have that /u1D4B8 1 > 0 and /u1D4B8 2 > 0 . Let /u1D4B8 ≜ min { ( t 1 -1) /u1D4B8 1 , ( t 2 -t 1 +1) /u1D4B8 2 } , /u1D4B8 1 ≜ /u1D4B8 / ( t 1 -1) , and /u1D4B8 2 ≜ /u1D4B8 / ( t 2 -t 1 +1) . Then we construct two points in Ξ :

Therefore, ˜ z = 1 2 z ′ + 1 2 z ′′ , and hence, ˜ z is not an extreme point of (A27).

Step 4: Solution format and optimal action.

By Steps 2 and 3, we obtain that for each fixed χ and s ∗ , the extreme points of the linear programming problem are given in Ξ 0 . By Lemma 1, every linear program has an extreme point that is an optimal solution. Hence, by the format of the K extreme points in Ξ 0 , we obtain that at least one optimal action of ψ can be found in the format:

for some k ∗ ∈ [ K ] .

If k ∗ = K , by (A23), we have that g ( γ ; ψ ) = γϵ p + ∑ K j =1 P j · (1 -1 K ) , and hence, the robust risk is g ( γ ⋆ ψ ; ψ ) = 1 -1 K by taking γ ⋆ ψ = 0 .

If k ∗ < K , we obtain that

Hence, for k ∗ < K , the robust risk is the minimum of g ( γ ⋆ ψ ; ψ ) = 1 by taking γ ⋆ ψ = 0 and g ( γ ⋆ ψ ; ψ ) = 1 + 1 k ∗ { ϱ ( ϵ ) -∑ k ∗ j =1 P ( j ) } by taking γ ⋆ ψ = 1 k ∗ κ p . Additionally, we observe that we should take the highest k ∗ values of { P 1 , . . . , P K } as P (1) , . . . , P ( k ∗ ) to minimize g ( γ ⋆ ψ ; ψ ) . Hence, we take the permutation χ such that P (1) ≥ . . . ≥ P ( K ) .

In summary, the optimal action ψ ⋆ that minimizes g ( γ ⋆ ψ ; ψ ) is given as below.

If 1 K ≥ 1 k ∗ ∑ k ∗ j =1 P ( j ) -1 k ∗ ϱ ( ϵ ) for all k ∗ ∈ [ K -1] , then ψ ⋆ j = 1 K for j ∈ [ K ] .

If there exists some k 0 ∈ [ K -1] , 1 k 0 ∑ k 0 j =1 P ( j ) -1 k 0 ϱ ( ϵ ) > 1 K , and 1 k 0 ∑ k 0 j =1 P ( j ) -1 k 0 ϱ ( ϵ ) ≥ 1 k ∗ ∑ k ∗ j =1 P ( j ) -1 k ∗ ϱ ( ϵ ) for all k ∗ ∈ [ K -1] , then ψ ⋆ ( j ) = 1 k 0 for j ∈ [ k 0 ] and ψ ⋆ ( j ) = 0 for j = k 0 +1 , . . . , K .

In particular, if P (1) ≥ max { 1 K + ϱ ( ϵ ) , P (2) + ϱ ( ϵ ) } , then the optimal action is given as: ψ ⋆ (1) = 1 and ψ ⋆ ( j ) = 0 for j = 2 , . . . , K . Thus, the proof is complete.

A.8 Proof of Theorem 3.3

For ease of presentation, we omit the dependence on x i and ˜ y i in the notation for now. Specifically, for i ∈ [ K ] and j ∈ [ K ] , let P i,j ≜ P j ( x i , ˜ y i ) ≜ P (Y = j | x i , ˜ y i ) and ψ i,j ≜ ψ ( x i ) j . For given x i , we sort the K elements of ψ ( x i ) , { ψ i, 1 , . . . , ψ i,K } , in a decreasing order, denoted ψ (1) i ≥ . . . ≥ ψ ( K ) i .

We first consider the difference between the Wasserstein robust loss (10) and the nominal loss (11):

where the second equality is due to the fact that ψ (1) i ≥ . . . ≥ ψ ( K ) i and that T is decreasing.

By definition, { α i,j ≜ T ( ψ ( K ) i ) -T ( ψ i,j ) : i ∈ [ n ] , j ∈ [ K ] } are ordered as α (1) ≥ . . . ≥ α ( nK ) , and correspondingly, the P i,j 's with the associated indexes are denoted P (1) , . . . , P ( nK ) . Consequently, we obtain that

Define α ( nK +1) = 0 . Then, h ( γ ) in (A30) can be expressed as

Since

and h ( γ ) is right-continuous at γ = | α ( s ) | /κ p by definition (A31), so we conclude that h ( γ ) is continuous at γ = | α ( s ) | /κ p for s ∈ [ nK ] . Hence, by (A31), h ( γ ) is continuous for γ ≥ 0 .

By (A31), h ( γ ) is increasing in γ on γ ∈ [ α ( s +1) /κ p , α ( s ) /κ p ) if ϱ ( ϵ ) -1 n ∑ s t =1 P ( t ) > 0 , and decreasing if ϱ ( ϵ ) -1 n ∑ s t =1 P ( t ) < 0 . Hence, to examine the monotonicity of h ( γ ) and find the γ that minimizes h ( γ ) , we consider the following three cases by the values of ϱ ( ϵ ) .

Case 1. If 1 n P (1) < ϱ ( ϵ ) < 1 n ∑ nK t =1 P ( t ) : then there exists s ∗ ∈ { 2 , . . . , nK } such that ϱ ( ϵ ) > 1 n ∑ s t =1 P ( t ) for s < s ∗ , and ϱ ( ϵ ) ≤ 1 n ∑ s t =1 P ( t ) for s ≥ s ∗ . Hence, by (A31), h ( γ ) is decreasing in γ for γ ∈ [0 , α ( s ∗ ) /κ p ] and increasing for γ ≥ α ( s ∗ ) /κ p . Consequently, γ ⋆ ψ = α ( s ∗ ) /κ p , and

Here the last step holds because by the definition of s ∗ , 0 < ϱ ( ϵ ) -1 n ∑ s ∗ -1 t =1 P ( t ) ≤ 1 n ∑ s ∗ t =1 P ( t ) -1 n ∑ s ∗ -1 t =1 P ( t ) = 1 n P ( s ∗ ) ≤ 1 n , where the last inequality holds since P ( s ∗ ) ∈ [0 , 1] .

Case 2. If ϱ ( ϵ ) ≤ 1 n P (1) : then, by (A31), h ( γ ) is decreasing in γ for γ ∈ [0 , α (1) /κ p ] and increasing for γ ≥ α (1) /κ p . Therefore, γ ⋆ ψ = α (1) /κ p , and

Case 3. If ϱ ( ϵ ) ≥ 1 n ∑ nK t =1 P ( t ) : then, by (A31), h ( γ ) is increasing in γ for γ ≥ 0 . Therefore, γ ⋆ ψ = 0 = α ( nK +1) /κ p , and

Hence, summarizing the discussion in the three cases above, we have that γ ⋆ ψ = α ( s ∗ ) /κ p , and the robust risk (10) is expressed as

where, to provide a unified expression, we define s ∗ ≜ 1 and s ∗ ≜ nK +1 in Case 2 and Case 3, respectively. Thus, the proof is completed.

B Experimental Details

B.1 Implementation Details

Datasets. We evaluate the effectiveness of the proposed AdaptCDRP on CIFAR-10 and CIFAR-100 [21] with synthetic annotations, and on four real-world datasets with human annotations: CIFAR10N, CIFAR-100N [22], LabelMe [23, 24], and Animal-10N [25]. CIFAR-10 has 10 classes of 32 × 32 × 3 color images, with 50,000 training images and 10,000 test images; CIFAR-10N provides three independent human annotated noisy labels per instance, with a majority vote yielding a 9.03% noise rate. CIFAR-100, with the same number and size of training and test images as CIFAR-10, features 100 fine-grained classes; for each instance in CIFAR-100, CIFAR-100N provides one human annotated noisy label, with a noise rate of 40.20%. LabelMe is an image classification dataset comprising 10,000 training images, 500 validation images, and 1,188 test images. The training set includes noisy and incomplete labels provided by 59 annotators, with each image being labeled an average of 2.547 times. The Animal-10N dataset contains 10 classes of 64 × 64 × 3 color animal images; it includes 5 pairs of similar-looking animals, where each pair consists of two animals that are visually alike. The training dataset contains 50,000 images and the test dataset contains 5,000 images. The noise rate (mislabeling ratio) of the dataset is about 8%. For all the datasets except LabelMe, we allocate 10% of the training data as validation data used for model selection, where we choose the model with the lowest validation accuracy during training. The test data is reserved for final evaluation of the model's performance on unseen data.

Noise generation. We generate synthetic instance-dependent label noise on the CIFAR-10 and CIFAR-100 datasets using Algorihtm 2 in [29]. Each annotator is classified as an IDNτ annotator if their mislabeling ratio is upper bounded by τ . We simulate R annotators independently, with R taking values from the set { 5 , 10 , 30 , 50 , 100 } . For each instance, one annotation is randomly selected from those provided by the R annotators' contributions, which evaluates methods under incomplete annotator labeling conditions. Additionally, for each R , we consider three groups of annotators with varying expertise levels, characterized by average mislabeling ratios of approximately 20%, 35%, and 50%. These groups are referred to as IDN-LOW, IDN-MID, and IDN-HIGH, indicating low, medium, and high error rates, respectively. We manually corrupt the datasets according to the following annotator groups:

R=5:

IDN-LOW. 2 IDN-10% annotators, 2 IDN-20% annotators, 1 IDN-30% annotator;

IDN-MID. 2 IDN-30% annotators, 2 IDN-40% annotators, 1 IDN-50% annotator;

IDN-HIGH. 2 IDN-50% annotators, 2 IDN-60% annotators, 1 IDN-70% annotator;

R=10:

IDN-LOW. 4 IDN-10% annotators, 4 IDN-20% annotators, 2 IDN-30% annotators;

IDN-MID. 4 IDN-30% annotators, 4 IDN-40% annotators, 2 IDN-50% annotators;

IDN-HIGH. R=30:

4 IDN-50% annotators, 4 IDN-60% annotators, 2 IDN-70% annotators;

IDN-LOW. 11 IDN-10% annotators, 11 IDN-20% annotators, 8 IDN-30% annotators;

IDN-MID. 11 IDN-30% annotators, 11 IDN-40% annotators, 8 IDN-50% annotators;

IDN-HIGH. 11 IDN-50% annotators, 11 IDN-60% annotators, 8 IDN-70% annotators;

R=50:

IDN-LOW. 18 IDN-10% annotators, 18 IDN-20% annotators, 14 IDN-30% annotators;

IDN-MID. 18 IDN-30% annotators, 18 IDN-40% annotators, 14 IDN-50% annotators;

IDN-HIGH. 18 IDN-50% annotators, 18 IDN-60% annotators, 14 IDN-70% annotators;

R=100:

IDN-LOW. 35 IDN-10% annotators, 35 IDN-20% annotators, 30 IDN-30% annotators;

IDN-MID. 35 IDN-30% annotators, 35 IDN-40% annotators, 30 IDN-50% annotators;

IDN-HIGH. 35 IDN-50% annotators, 35 IDN-60% annotators, 30 IDN-70% annotators.

Experiment setup. We employ the ResNet-18 architecture for CIFAR-10 and CIFAR-10N, and the ResNet-34 architecture for CIFAR-100 and CIFAR-100N datasets. Following [27], we use a pretrained VGG-16 model with a 50% dropout rate as the backbone for the LabelMe dataset. For the Animal-10N dataset, in line with [25], we use the VGG19-BN architecture [28] as the backbone. A batch size of 128 is maintained across all datasets. We use the Adam optimizer [43] with a weight decay of 5 × 10 -4 for CIFAR-10, CIFAR-100, CIFAR-10N, CIFAR-100N, and LabelMe datasets. The initial learning rate for CIFAR-10, CIFAR-100, CIFAR-10N, and CIFAR-100N is set to 10 -3 , with the networks trained for 120, 150, 120, and 150 epochs respectively. The first 30 epochs serve as a warm-up. For the LabelMe dataset, the model is trained for 100 epochs with an initial learning rate of 10 -2 and a 20-epoch warm-up. For the Animal-10N dataset, the network is trained for 100 epochs with an initial learning rate of 10 -1 and a weight decay of 10 -3 . The learning rate is reduced by a factor of 0.1 at the 50th and 75th epochs, with the first 40 epochs designated as the warm-up stage. Training times are approximately 3 hours on CIFAR-10 and 5.5 hours on CIFAR-100 using an NVIDIA V100 GPU.

Baselines. Our method addresses learning from noisy annotations, particularly when estimated true label posteriors may be misspecified. Thus, we select baselines that either use estimated transition matrices or true label posteriors (MBEM [9], CrowdLayer [27], TraceReg [12], Max-MIG [11], CoNAL [35]). We also include baselines that aggregate labels differently (CE (MV), CE (EM) [7], DoctorNet [34], CCC [36]). Since our theoretical framework applies to both single-annotator and multiple-annotator scenarios, we also include baselines designed for single noisy labels (LogitClip [33]), particularly those employing two networks (Co-teaching [30], Co-teaching+ [31], CoDis [32]), as our method similarly uses two networks that act as priors for each other. Details of the baselines are given as follows.

CE (Clean): Trains the network using the standard cross-entropy loss on clean datasets;

CE (MV): Trains the network using majority voting labels;

CE (EM) [7]: Aggregate labels using the EM algorithm;

Co-teaching [30]: Trains two networks and cross-trains on instances with small loss values;

Co-teaching+ [31]: Combines the "Update by Disagreement" with the Co-teaching method;

CoDis [32]: Selects possibly clean data that have high-discrepancy prediction probabilities between two networks;

LogitClip [33]: Clamps the norm of the logit vector to ensure it is upper bounded by a constant;

DoctorNet [34]: Models individual annotators and learns averaging weights by combining them;

MBEM[9]: Alternates between estimating annotator quality from disagreements with the current model and updating the model by optimizing a loss function that accounts for the current estimate of worker quality;

CrowdLayer [27]: Concatenates the classifier with multiple annotator-specific layers and learns the parameters simultaneously;

TraceReg [12]: Uses a loss function similar to CrowdLayer but adds regularization to establish identifiability of the confusion matrices and the classifier;

Max-MIG [11]: Jointly aggregates noisy crowdsourced labels and trains the classifier;

CoNAL [35]: Decomposes the annotation noise into common and individual confusions;

CCC [36]: Simultaneously trains two models to correct the confusion matrices learned by each other via bi-level optimization.

Among these methods, Co-teaching, Co-teaching+, CoDis, and LogitClip are strong baselines for handling single noisy labels. We adapt them to the multiple annotations setting by using majority vote labels for loss computation. The results demonstrate the effectiveness of the proposed pseudo-label generation method across various scenarios. Results for CE (Clean), CE (MV), CE (EM), DoctorNet, MBEM, CrowdLayer, Max-MIG, and CoNAL in Table 1 are sourced from [10]. Baselines (1)-(3)

Algorithm 1: Learning from Noisy Labels via C onditional D istributionally R obust True Label P osterior with an Adapt ive Lagrange multiplier (AdaptCDRP)

̸

are implemented according to their respective algorithms, while for the remaining baseline methods, we adapted the code from the GitHub repositories provided in their original papers, with further modifications to fit our setup.

Pseudo code for the algorithm. The training process described in Section 3.3 is presented in Algorithm 1.

B.2 Additional Experimental Results

Performance on the CIFAR-100 dataset with varying numbers of annotators. We conduct additional experiments on the CIFAR-100 dataset, varying the number of annotators from 5 to 100, with each instance labeled only once. Figure 3 presents the average accuracy across different annotator counts, highlighting the advantages of the proposed method across various settings. As the total number of annotators increases, labeling sparsity becomes more pronounced, which may lead to a performance collapse in methods that do not account for this sparsity, especially in datasets with a large number of classes, such as CIFAR-100.

5

10

30

50

100

Number of Annotators

40

45

50

55

60

Test Accuracy

CE(Clean)

CE(MV)

CE(EM)

Co-teaching

Co-teaching+

CoDis

LogitClip

DoctorNet

MBEM

CrowdLayer

TraceReg

CoNAL

CCC

AdaptCDRP (Ours)

(a) IDN-LOW

5

10

30

50

100

Number of Annotators

30

35

40

45

50

55

60

Test Accuracy

CE(Clean)

CE(MV)

CE(EM)

Co-teaching

Co-teaching+

CoDis

LogitClip

DoctorNet

MBEM

CrowdLayer

TraceReg

CoNAL

CCC

AdaptCDRP (Ours)

(b) IDN-MID

5

10

30

50

100

Number of Annotators

25

30

35

40

45

50

55

60

Test Accuracy

CE(Clean)

CE(MV)

CE(EM)

Co-teaching

Co-teaching+

CoDis

LogitClip

DoctorNet

MBEM

CrowdLayer

TraceReg

CoNAL

CCC

AdaptCDRP (Ours)

(c) IDN-HIGH

Performance with varying numbers of annotations per instance. To further evaluate model performance with varying numbers of annotations per instance, we use R = 30 annotators and

randomly select l = 1 , 3 , 5 , 7 , 9 labels from these R annotators for each instance. The test accuracies of the proposed method and other annotation aggregation methods are shown in Table 5.

Accuracy of robust pseudo-labels. To enhance the assessment of the effectiveness of the proposed robust pseudo-label generation method, we present the average accuracy of the robust pseudo-labels on the CIFAR-10 and CIFAR-100 datasets during the training process over 5 random trials, as shown in Figure 4. Additionally, the average accuracy of the robust pseudo-labels with varying numbers of annotators on the CIFAR-10 dataset is shown in Figure 5.

0

100

200

300

400

500

Iterations

90

92

94

96

98

100

Robust Pseudo-Label Accuracy

CIFAR-10 (IDN-LOW; R=5)

CIFAR-10 (IDN-MID; R=5)

CIFAR-10 (IDN-HIGH; R=5)

(a) CIFAR-10

0

100

200

300

400

500

Iterations

80

85

90

95

100

Robust Pseudo-Label Accuracy

CIFAR-100 (IDN-LOW; R=5)

CIFAR-100 (IDN-MID; R=5)

CIFAR-100 (IDN-HIGH; R=5)

(b) CIFAR-100

0

100

200

300

400

500

Iterations

90

92

94

96

98

100

Robust Pseudo-Label Accuracy

CIFAR-10 (IDN-LOW; R=5)

CIFAR-10 (IDN-MID; R=5)

CIFAR-10 (IDN-HIGH; R=5)

(a) CIFAR-10 (

R

= 5

)

0

100

200

300

400

500

Iterations

90

92

94

96

98

100

Robust Pseudo-Label Accuracy

CIFAR-10 (IDN-LOW; R=10)

CIFAR-10 (IDN-MID; R=10)

CIFAR-10 (IDN-HIGH; R=10)

(b) CIFAR-10 (

R

= 10

)

0

100

200

300

400

500

Iterations

90

92

94

96

98

100

Robust Pseudo-Label Accuracy

CIFAR-10 (IDN-LOW; R=30)

CIFAR-10 (IDN-MID; R=30)

CIFAR-10 (IDN-HIGH; R=30)

(c) CIFAR-10 (

R

= 30

)

0

100

200

300

400

500

Iterations

90

92

94

96

98

100

Robust Pseudo-Label Accuracy

CIFAR-10 (IDN-LOW; R=50)

CIFAR-10 (IDN-MID; R=50)

CIFAR-10 (IDN-HIGH; R=50)

(d) CIFAR-10 (

R

= 50

)

0

100

200

300

400

500

Iterations

90

92

94

96

98

100

Robust Pseudo-Label Accuracy

CIFAR-10 (IDN-LOW; R=100)

CIFAR-10 (IDN-MID; R=100)

CIFAR-10 (IDN-HIGH; R=100)

(e) CIFAR-10 (

R

= 100

)

Test accuracy during the training process. To further assess the effectiveness of the proposed method, we present the average test accuracy for the CIFAR-10 and CIFAR-100 datasets during the training process, as shown in Figures 6 and 7, respectively. The results indicate that the model tends to overfit during the warm-up stage, particularly under higher noise rates. This suggests that the results in Table 1 are not obtained with the optimal number of warm-up epochs. However, following the warm-up phase, the test accuracy of our method steadily improves, outperforming baseline methods across various scenarios.

0

20

40

60

80

100

120

Epochs

50

55

60

65

70

75

80

85

Test Accuracy

Warm-up stage

CE

EM

Co-teaching

Co-teaching+

CoDis

LogitClip

DoctorNet

MBEM

CrowdLayer

TraceReg

CoNAL

CCC

AdaptCDRP (Ours)

(a) IDN-LOW

0

20

40

60

80

100

120

Epochs

40

50

60

70

80

Test Accuracy

Warm-up stage

CE

EM

Co-teaching

Co-teaching+

CoDis

LogitClip

DoctorNet

MBEM

CrowdLayer

TraceReg

CoNAL

CCC

AdaptCDRP (Ours)

(b) IDN-MID

0

20

40

60

80

100

120

Epochs

20

30

40

50

60

70

80

Test Accuracy

Warm-up stage

CE

EM

Co-teaching

Co-teaching+

CoDis

LogitClip

DoctorNet

MBEM

CrowdLayer

TraceReg

CoNAL

CCC

AdaptCDRP (Ours)

(c) IDN-HIGH

0

20

40

60

80

100

120

140

Epochs

25

30

35

40

45

50

55

60

Test Accuracy

Warm-up stage

CE

EM

Co-teaching

Co-teaching+

CoDis

LogitClip

DoctorNet

MBEM

CrowdLayer

TraceReg

CoNAL

CCC

AdaptCDRP (Ours)

(a) IDN-LOW

0

20

40

60

80

100

120

140

Epochs

20

30

40

50

Test Accuracy

Warm-up stage

CE

EM

Co-teaching

Co-teaching+

CoDis

LogitClip

DoctorNet

MBEM

CrowdLayer

TraceReg

CoNAL

CCC

AdaptCDRP (Ours)

(b) IDN-MID

0

20

40

60

80

100

120

140

Epochs

10

20

30

40

50

Test Accuracy

Warm-up stage

CE

EM

Co-teaching

Co-teaching+

CoDis

LogitClip

DoctorNet

MBEM

CrowdLayer

TraceReg

CoNAL

CCC

AdaptCDRP (Ours)

(c) IDN-HIGH

Impact of the number of warm-up epochs. Following [10, 46], we use 30 warm-up epochs for the CIFAR-10 and CIFAR-100 datasets in our experiments. To rigorously assess the impact of the warm-up stage, we conduct additional experiments with varying numbers of warm-up epochs (10 , 20 , 30 , 40) on both our method and baseline approaches that also incorporate warm-up. The results, presented in Figure 8, illustrate how different warm-up durations affect performance.

10

20

30

40

60

65

70

75

80

85

90

IDN-LOW: Ours

IDN-LOW: MBEM

IDN-LOW: CrowdLayer

IDN-LOW: TraceReg

10

20

30

40

60

65

70

75

80

85

90

IDN-MID: Ours

IDN-MID: MBEM

IDN-MID: CrowdLayer

IDN-MID: TraceReg

10

20

30

40

50

55

60

65

70

75

80

85

90

IDN-HIGH: Ours

IDN-HIGH: MBEM

IDN-HIGH: CrowdLayer

IDN-HIGH: TraceReg

Number of warm-up epochs

(a) CIFAR-10

10

20

30

40

30

35

40

45

50

55

60

65

IDN-LOW: Ours

IDN-LOW: MBEM

IDN-LOW: CrowdLayer

IDN-LOW: TraceReg

10

20

30

40

25

30

35

40

45

50

55

60

IDN-MID: Ours

IDN-MID: MBEM

IDN-MID: CrowdLayer

IDN-MID: TraceReg

10

20

30

40

20

25

30

35

40

45

50

55

60

IDN-HIGH: Ours

IDN-HIGH: MBEM

IDN-HIGH: CrowdLayer

IDN-HIGH: TraceReg

Number of warm-up epochs

(b) CIFAR-100

Different transition matrix estimation methods. Our work does not focus on precise estimation of the noise transition matrix; instead, we use a simple frequency-counting method for noise transition estimation in our experiments. Nevertheless, our approach is versatile and can be integrated with various methods for estimating the noise transition matrix or the true label posterior. Additional experiments using advanced transition matrix estimation methods are presented in Table 6. As demonstrated, integrating these methods with AdaptCDRP significantly improves test accuracies compared to directly using the estimated noise transition matrices. Furthermore, applying advanced noise transition estimation methods enhances the performance of our method on real datasets. These results highlight the robustness and adaptability of our method.

Impact of sparse annotation. To further address the issue of annotation sparsity, we increase the total number of annotators, R , to 200, and manually corrupt the datasets according to the following annotator groups:

R=200:

IDN-LOW. 70 IDN-10% annotators, 70 IDN-20% annotators, 60 IDN-30% annotators;

IDN-MID. 70 IDN-30% annotators, 70 IDN-40% annotators, 60 IDN-50% annotators;

IDN-HIGH. 70 IDN-50% annotators, 70 IDN-60% annotators, 60 IDN-70% annotators.

The three groups of annotators, labeled as IDN-LOW, IDN-MID, and IDN-HIGH, have average labeling error rates of approximately 26%, 34%, and 42%, respectively. In this setup, we incorporate regularization techniques - specifically, GeoCrowdNet (F) and GeoCrowdNet (W) penalties [47] into our method. We then compare the results against those obtained using the traditional frequencycounting approach for estimating the noise transition matrices. Table 7 presents the performance of our proposed method on the CIFAR10 ( R = 200 ) dataset, where different approaches are used to estimate the noise transition matrices. In addition, Figure 9 displays the average accuracies of the robust pseudo-labels generated by our method during the training process. These pseudo-labels play a crucial role in constructing the pseudo-empirical distribution.

0

100

200

300

400

500

Iterations

90

92

94

96

98

100

Robust Pseudo-Label Accuracy

CIFAR-10 (IDN-LOW; R=200)

CIFAR-10 (IDN-MID; R=200)

CIFAR-10 (IDN-HIGH; R=200)

(a) Ours + Frequency Counting

0

100

200

300

400

500

Iterations

90

92

94

96

98

100

Robust Pseudo-Label Accuracy

CIFAR-10 (IDN-LOW; R=200)

CIFAR-10 (IDN-MID; R=200)

CIFAR-10 (IDN-HIGH; R=200)

0

100

200

300

400

500

Iterations

90

92

94

96

98

100

Robust Pseudo-Label Accuracy

CIFAR-10 (IDN-LOW; R=200)

CIFAR-10 (IDN-MID; R=200)

CIFAR-10 (IDN-HIGH; R=200)

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately reflect our contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of the work performed by the authors in the conclusion.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide the assumptions in the main text, and the proofs can be found in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide all the experimental details.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We upload our code and use public datasets.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide all the details and the code.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provide the error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the training time on different datasets.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: It's not appropriate for the scope and focus of our paper, and we don't see any direct negative social impacts of our paper.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our model doesn't have a high risk for misuse or dual-use.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the original papers.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide the data generation details.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not involve human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We do not involve human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper studies the problem of learning from noisy labels by using conditional distributionally robust optimization (CDRO) to estimate true label posterior. The authors formulate the problem as minimizing the worst-case risk within a distance-based ambiguity set centered around a reference distribution and derive upper bounds for the worst-case risk.

STRENGTHS:
1. Learning from noisy labels is an important and practical topic.
2. This paper provides rigorous theoretical analyses of the generalization bounds for the worst-case risk.
3. The authors offer a guideline for balancing robustness and model fitting by deriving the optimal value for the Lagrangian multiplier.

WEAKNESSES:
1. In section 3.1, the theoretical analyses are only conducted in the case of binary classification while multi-class classification is more common in practice.
2. In lines 192-193 and 198, the authors make assumptions about the function $\mathcal{T}$, however, the convexity or the concavity of $\mathcal{T}$ may be too strict.
3. According to Table 2, the performance improvement of the proposed method is not significant across four real datasets.

QUESTIONS:
1. Can the theoretical analyses be extended to the multi-classification case?
2. In Algorithm 1 line 1, does the procedure of "Warm up classifiers $\psi^{(1)}$ and $\psi^{(2)}$" affect the generation of pseudo-empirical distribution? What if the warm-up is not good enough for pseudo-empirical distribution?
3. The intuition and rationale of using the CDRO framework to improve the performance of learning from noisy labels are not clear, please explain them.

LIMITATIONS:
No, the authors do not mention the limitations of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studied the issue of potential misspecification of estimated true label posterior in learning from noisy labels. To alleviate the impact of this issue,  it formulated learning from crowds as a conditional distributionally robust optimization problem, where a robust pseudo-empirical distribution is used as a reference probability distribution. Experiments on multiple crowdsourcing datasets verified the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. The proposed methodology has a solid theoretical foundation.\n2. The writing is very clear and easy to understand.\n3. The performance of the proposed method is very promising in the experiments.\n\nWEAKNESSES:\n1. The focused problem, i.e., the potential misspecification of estimated true label posterior, has not been defined and measured. And it is not very clear to what extent the proposed method has solved the impact of this problem.\n2. The contribution of this work is about \"learning from noisy labels\", while the context of this work seems limited to \"learning from crowds\". They are not consistent, since \"learning from noisy labels\" also includes the case with one annotator. However, this work didn't discuss and test on the case with one annotator.\n3. Although the way to construct a robust pseudo-empirical distribution has a theoretical motivation, it seems similar to the pseudo-labeling methods [1,2] in learning with noisy labels. What are the differences and advantages of the proposed way and the pseudo-labeling methods?\n4. As we know, in learning with noisy labels or learning from crowds, the estimation of noise transition probabilities is very important, while the way to approximate noise transition probabilities in this work is heuristics. Why not use those thoroughly studied methods, e.g.,  [3,4,5], or more advanced instance-dependent transition matrix estimation methods [6,7]？\n\n[1] Joint optimization framework for learning with noisy labels. CVPR 2018\n\n[2] Deep Self-Learning From Noisy Labels. ICCV 2019\n\n[3] Deep learning from crowdsourced labels: Coupled cross-entropy minimization, identifiability, and regularization. ICLR 2023\n\n[4] Learning from noisy labels by regularized estimation of annotator confusion. CVPR 2019\n\n[5] Deep learning from crowds. AAAI 2018\n\n[6] Label correction of crowdsourced noisy annotations with an instance-dependent noise transition model. NeurIPS 2023\n\n[7] Transferring annotator- and instance-dependent transition matrix for learning from crowds. TPAMI 2024\n\nQUESTIONS:\n1. What {(·, ·)}^p means in Line 133?\n2. There is a little abuse of notations.  ψ means the classifier in Line 88, while it represents the predicted probabilities in Line 190.\n3. How to address the annotation sparse problem when approximating noise transition probabilities?\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work addresses learning from noisy annotations by using conditional distributionally robust optimization (CDRO). \nTo account for variability in estimating the true label posteriors, the authors propose an approach that minimizes the maximum expected risk  with respect to a probability distribution within a distance-based ambiguity set centered around a reference distribution (the posterior distribution of the true labels).\nDeriving the dual problem, the authors are able to provide upper bounds for risk. Moreover they derive generalization bounds for the upper bound of the risk.\nAdditionally, a closed-form expression for empirical robust risk and the optimal Lagrange multiplier is provided.\nAn analytical solution for dual robust risk is found for loss functions of a particular form. Starting from the analytical solution and introducing a robust pseudo-label collection algorithm. \nExperiments are performed on CIFAR-10 and CIFAR-100 introducing on synthetic and on the real-world datasets CIFAR-10N, CIFAR-100N, LabelME, animal-10N.\n\nSTRENGTHS:\n1) The proposed approach is really interesting, novel and principled, with a sound theoretical foundation.\n2) The paper is clearly written, making the concepts and methodology understandable.\n3) The authors provide a generalization bound for the upper bound of the risk function they define.\n\nWEAKNESSES:\n1) Some results in the experiments are not fully convincing me, I would like to hear the authors feedback:\n- Regarding Figure 1. How can the accuracy change so little with an increasing number of annotators? According to Figure 1 in [7], the noise rate of aggregated labels decreases significantly as the number of annotators increases, even for high initial noise rates. How is it possible that the accuracy in Figure 1 remains almost constant despite the increase in annotators?\n\n- Why is the performance of the ResNet34 model on the clean CIFAR-100 dataset so low? The overall performances  for CIFAR 100 and CIFA10-N seem lower than those reported in ProMix [1] (Tables 1 and 2) or SOP [3] (Tables 1 and 3). Although the noise settings differ, the model trained on clean data should exhibit higher performance on CIFAR 100.\n Similarly, the results for Co-teaching method appear inconsistent with those in other studies. \n\n2) I believe a better justification for the chosen baselines is needed. For example, since Co-teaching, using two networks and which is designed for scenarios with a single label per sample is included, why not include more SOTA methods such as ProMix [1], Divide-Mix [2], and SOP [3]? \nI am also curious why these methods were not included. Additionally, for methods that aggregate labels, considering other approaches like IWMV [4], or those that train models on soft labels, such as IAA [5] and soft-labels average [6], would be beneficial.\nWhile not asking the authors to include all these methods, I would appreciate a clearer justification for the choice of baselines.\n\n3) In my opinion, the limitations are not fully discussed. Please see the Limitations section for further details.\n\n[1] Wang, Haobo, et al. \"Promix: Combating label noise via maximizing clean sample utility.\" arXiv preprint arXiv:2207.10276 (2022).\n[2] Li, Junnan, Richard Socher, and Steven CH Hoi. \"Dividemix: Learning with noisy labels as semi-supervised learning.\" arXiv preprint arXiv:2002.07394 (2020).\n[3] Liu, Sheng, et al. \"Robust training under label noise by over-parameterization.\" International Conference on Machine Learning. PMLR, 2022.\n[4] Li, Hongwei, and Bin Yu. \"Error rate bounds and iterative weighted majority voting for crowdsourcing.\" arXiv preprint arXiv:1411.4086 (2014).\n[5] Bucarelli, M. S., Cassano, L., Siciliano, F., Mantrach, A., & Silvestri, F. (2023). Leveraging inter-rater agreement for classification in the presence of noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3439-3448).\n[6] Collins, Katherine M., Umang Bhatt, and Adrian Weller. \"Eliciting and learning with soft labels from every annotator.\" Proceedings of the AAAI conference on human computation and crowdsourcing. Vol. 10. 2022.\n[7] Wei, Jiaheng, et al. \"To aggregate or not? learning with separate noisy labels.\" Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2023.\n\nQUESTIONS:\n- What loss function is used in the experiments, namely what function $ \\mathcal{Tau}$ is used in the loss? \n\nsee also questions in Weaknesses.\n\nSuggestions: \nWrite in the main paper the type of network used in the experiments would facilitate the reader (ResNet-18 architecture for CIFAR-10 and CIFAR-10N, and ResNet-34 architecture for CIFAR-100 and CIFAR-100N datasets. )\n\nLIMITATIONS:\nPartially. some possible improvements are shortly discussed in the conclusions.\n\nSome limitations:\n- An overlooked limitation is the need for $P_j(x,\\tilde{y}y)$, that is the posterior  $ P(Y=j| \\mathbf{X}= \\mathbf{x}, \\mathbf{\\tilde{Y}} = \\mathbf{\\tilde{y}})$ to define the optimal action. This requires the posterior distribution, which can only be obtained after training the model for some epochs. The number of epochs necessary depends on the dataset's characteristics and how quickly the model overfits, but the sensitivity to this factor is not discussed. Also the number of samples for the $\\mathcal{D}^{*}_0$ dataset are not specified. \n\n- Another limitation that is not mentioned is the fact that the method needs to classifiers.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}