Title: Weak Supervision Performance Evaluation via Partial Identification
Venue: NeurIPS 2024
Accepted: yes
Keywords: weak supervision

evaluation

Frechet bounds

partial identification
Abstract: Programmatic Weak Supervision (PWS) enables supervised model training without direct access to ground truth labels, utilizing weak labels from heuristics, crowdsourcing, or pre-trained models. However, the absence of ground truth complicates model evaluation, as traditional metrics such as accuracy, precision, and recall cannot be directly calculated. In this work, we present a novel method to address this challenge by framing model evaluation as a partial identification problem and estimating performance bounds using Fréchet bounds. Our approach derives reliable bounds on key metrics without requiring labeled data, overcoming core limitations in current weak supervision evaluation techniques. Through scalable convex optimization, we obtain accurate and computationally efficient bounds for metrics including accuracy, precision, recall, and F1-score, even in high-dimensional settings. This framework offers a robust approach to assessing model quality without ground truth labels, enhancing the practicality of weakly supervised learning for real-world applications.

Weak Supervision Performance Evaluation via Partial Identification

Abstract

Programmatic Weak Supervision (PWS) enables supervised model training without direct access to ground truth labels, utilizing weak labels from heuristics, crowdsourcing, or pre-trained models. However, the absence of ground truth complicates model evaluation, as traditional metrics such as accuracy, precision, and recall cannot be directly calculated. In this work, we present a novel method to address this challenge by framing model evaluation as a partial identification problem and estimating performance bounds using Fréchet bounds. Our approach derives reliable bounds on key metrics without requiring labeled data, overcoming core limitations in current weak supervision evaluation techniques. Through scalable convex optimization, we obtain accurate and computationally efficient bounds for metrics including accuracy, precision, recall, and F1-score, even in high-dimensional settings. This framework offers a robust approach to assessing model quality without ground truth labels, enhancing the practicality of weakly supervised learning for real-world applications. 2

1 Introduction

Programmatic weak supervision (PWS) is a modern learning paradigm that allows practitioners to train their supervised models without the immediate need for ground truth labels Y [50, 48, 47, 49, 58, 64]. In PWS, practitioners first acquire cheap and abundant weak labels Z through heuristics, crowdsourcing, external APIs, and pretrained models, which serve as proxies for Y . Then, they fit a label model , i.e. , a graphical model for P Y,Z [50, 49, 22, 17], which, under appropriate modeling assumptions, can be fitted without requiring Y 's. Finally, a predictor h : X → Y is trained using samples ( X i , Z i ) 's and a noise-aware loss constructed using this fitted label model [50].

One major unsolved issue with the weak supervision approach is that even if we knew P Y,Z , evaluation metrics such as accuracy, recall, precision, or F 1 cannot be estimated for model validation without any ground truth labels. In fact, these quantities are not identifiable (not uniquely determined) since we only have partial information about the joint distribution P X,Y through the marginals P X,Z and P Y,Z . As a consequence, any performance metric based on h cannot be estimated without making extra strong assumptions, e.g. , X ⊥ ⊥ Y | Z . Unfortunately, these conditions are unlikely to arise in many situations. A recent work [66] investigated the role and importance of ground truth labels on model evaluation in the weak supervision literature. They determined that, under the current situation,

the good performance and applicability of weakly supervised classifiers heavily rely on the presence of at least some high-quality labels, which undermines the purpose of using weak supervision since models can be directly fine-tuned on those labels and achieve similar performance. Therefore, in this work, we develop new evaluation methods that can be used without any ground truth labels and show that the performance of weakly supervised models can be accurately estimated in many cases, even permitting successful model selection. Our solution relies on partial identification by estimating Fréchet bounds for bounding performance metrics such as accuracy, precision, recall, and F 1 score of classifiers trained with weak supervision.

Fréchet bounds: Consider a random vector ( X,Y,Z ) ∈ X × Y × Z is drawn from an unknown distribution P . We assume X ⊂ R d is arbitrary while Y and Z are finite. In this work, we develop and analyze the statistical properties of a method for estimating Fréchet bounds [53, 54] of the form

when g is a fixed bounded function, with Π being the set of distributions π for ( X,Y,Z ) such that the marginal π X,Z (resp. π Y,Z ) is identical to the prescribed marginal P X,Z (resp. P Y,Z ). Our proposed method can efficiently obtain estimates for the bounds by solving convex programs, with the significant advantage that the computational complexity of our algorithm does not scale with the dimensionality of X , making it well-suited for applications dealing with high-dimensional data. In previous work, for example, Fréchet bounds were studied in the financial context ( e.g. , see Rüschendorf [55], Bartl et al. [7]). However, our focus is on applying our methods of estimating Fréchet bounds to the problem of assessing predictors trained using programmatic weak supervision (PWS). For example, the upper and lower bounds for the accuracy of a classifier h can be estimated using our method simply by letting g ( x, y, z ) = 1 [ h ( x ) = y ] in (1.1). At a high level, our method replaces P Y,Z with the fitted label model, and P X,Z with its empirical version in the Fréchet bounds in (1.1), and reformulates the problem in terms of a convex optimization problem.

Contributions: Our contributions are

Developing a practical algorithm for estimating the Fréchet bounds in (1.1). Our algorithm can be summarized as solving convex programs and is scalable to high-dimensional distributions.

Quantifying the uncertainty in the computed bounds due to uncertainty in the prescribed marginals by deriving the asymptotic distribution for our estimators.

Applying our method to bounding the accuracy, precision, recall, and F 1 score of classifiers trained with weak supervision. This enables practitioners to evaluate classifiers in weak supervision settings without access to ground truth labels .

1.1 Related work

Weak supervision: With the emergence of data-hungry models, the lack of properly labeled datasets has become a major bottleneck in the development of supervised models. One approach to overcome this problem is using programmatic weak supervision (PWS) to train predictors in the absence of high-quality labels Y [50, 48, 47, 49, 58, 64]. PWS has shown the potential to solve a variety of tasks in different fields with satisfactory performance. For example, some works have applied weak supervision to named-entity recognition [32, 21, 57], video frame classification [22], bone and breast tumor classification [61]. More recently, Smith et al. [59] proposed a new approach to integrating weak supervision and pre-trained large language models (LLMs). Rather than applying LLMs in the usual zero/few-shot fashion, they treat those large models as weak labelers that can be used through prompting to obtain weak signals instead of using hand-crafted heuristics. Recently, Zhu et al. [66] showed that in many situations, the success of weakly supervised classifiers depends on the availability of ground truth validation samples, undermining the purpose of weak supervision. Then, we develop a new method for model evaluation that does not depend on the availability of any ground truth labels.

A relevant line of research within the realm of weak supervision that is closely related to this work is adversarial learning [4, 5, 42, 41]. Often, adversarial learning aims to learn predictors that perform well in worst-case scenarios. For example, Mazzetto et al. [41] develops a method to learn weakly supervised classifiers in the absence of a good label model. In their work, the authors use a small set of labeled data points to constrain the space of possible data distributions and then find a predictor that performs well in the worst-case scenario. Our work relates to this literature in the sense that we

are interested in the worst and best-case scenarios over a set of distributions. However, we focus on developing an evaluation method instead of another adversarial learning strategy.

Partial identification: It is often the case that the distributions of interest cannot be fully observed, which is generally due to missing or noisy data [43, 23]. In cases where practitioners can only observe some aspects of those distributions, e.g. , marginal distributions or moments, parameters of interest may not be identifiable without strong assumptions due to ambiguity in the observable data. Partial identification deals with the problem without imposing extra assumptions. This framework allows estimating a set of potential values for the parameters of interest (usually given by non-trivial bounds) and has been frequently considered in many areas such as microeconometrics [38-40, 43], causal inference [25, 23], algorithmic fairness [20, 46]. Our work is most related to Rüschendorf [53, 54, 55], Bartl et al. [7], which study bounds for the uncertainty of a quantity of interest for a joint distribution that is only partially identified through its marginals, i.e. , Fréchet bounds. Compared to the aforementioned works, the novelty of our contribution is proposing a convex optimization algorithm that accurately estimates the Fréchet bounds with proven performance guarantees in a setup that is realized in numerous weak-supervision applications.

1.2 Notation

We write E Q and Var Q for the expectation and variance of statistics computed using i.i.d. copies of a random vector W ∼ Q . Consequently, P Q ( A ) = E Q 1 A , where 1 A is the indicator of an event A . If the distribution is clear by the context, we omit the subscript. If ( a m ) m ∈ N and ( b m ) m ∈ N are sequences of scalars, then a m = o ( b m ) is equivalent to a m /b m → 0 as m →∞ and a m = b m + o (1) means a m -b m = o (1) . If ( V ( m ) ) m ∈ N is a sequence of random variables, then (i) V ( m ) = o P (1) means that for every ε > 0 we have P ( | V ( m ) | > ε ) → 0 as m →∞ , (ii) V ( m ) = O P (1) means that for every ε > 0 there exists a M > 0 such that sup m ∈ N P ( | V ( m ) | > M ) < ε , (iii) V ( m ) = a m + o P (1) means V ( m ) -a m = o P (1) , (iv) V ( m ) = o P ( a m ) means V ( m ) /a m = o P (1) , and (v) V ( m ) = O P ( a m ) means V ( m ) /a m = O P (1) .

2 Estimating Fréchet bounds

A roadmap to our approach follows. We first reformulate the Fréchet bounds in (1.1) into their dual problems, which we discuss in (2.1). Then, we replace the non-smooth dual problems with their appropriate smooth approximations, as discussed in (2.2). Finally, we propose estimators for the smooth approximations (2.3) and derive their asymptotic distributions in Theorem 2.5.

2.1 Dual formulations of the bounds and their approximations

This section presents a result that allows us to efficiently solve the optimization problems in (1.1) by deriving their dual formulations as finite-dimensional convex programs. Before we dive into the result, let us define a family of matrices denoted by

With this definition in place, we introduce the dual formulation in Theorem 2.1.

Theorem 2.1. Let g : X × Y × Z → R be a bounded measurable function. Then, where

Moreover, L and U are attained by some optimizers in A .

Theorem 2.1 remains valid if we maximize/minimize over R |Y|×|Z| instead of A . However, this is not necessary because the values of f l and f u remain identical for the following shifts in a : a · z ← a · z + b z where b z ∈ R . By constraining the set of optimizers to A , we eliminate the possibility of having multiple optimal points. The proof of Theorem 2.1 is placed in Appendix B and is inspired by ideas from Optimal Transport; see Appendix A.

The computation of these bounds entails finding a minimum or maximum over a discrete set, meaning that straightforward application of their empirical versions could result in optimizing non-smooth functions, which is often challenging. To mitigate this, we consider a smooth approximation of the problem that is found to be useful in handling non-smooth optimization problems [3, 6]. We approximate the max and min operators with their 'soft' counterparts:

where ε > 0 is a small constant that dictates the level of smoothness. As ε nears zero, these soft versions of max and min converge to their original non-smooth forms. Using these approximations, we reformulate our dual optimization in (2.1) into smooth optimization problems:

where

and ε > 0 is kept fixed at an appropriate value. As a consequence of Lemma 5 of An et al. [3], we know that L ε and U ε are no more than ε log |Y| units from L and U . Thus, that distance can be regulated by adjusting ε . For example, if we are comfortable with an approximation error of 10 -2 units when |Y| = 2 , we will set ε = 10 -2 / log(2) ≈ . 014 .

2.2 Estimating the bounds

In practice, it is not usually possible to solve the optimization problems in (2.2), because we may not have direct access to the distributions P X,Z and P Y | Z . We overcome this problem by assuming that we can estimate the distributions using an available dataset.

To this end, let us assume that we have a sample { ( X i , Z i ) } n i =1 iid ∼ P X,Z , and thus we replace the relevant expectations with P X,Z by its empirical version. Additionally, we have a sequence { ˆ P ( m ) Y | Z , m ∈ N } that estimates P Y | Z with greater precision as m increases. Here, m can be viewed as the size of a sample to estimate P Y | Z . Although the exact procedure for estimating the conditional distribution is not relevant to this section, we have discussed in our introductory section that this can be estimated using a label model [49, 22] in applications with weak supervision or in a variety of other ways for applications beyond weak supervision. Later in this section, we will formalize the precision required for the estimates. To simplify our notation, we omit the superscript m in ˆ P ( m ) Y | Z , whenever it is convenient to do so.

Thus, the Fréchet bounds are estimated as

where

In our practical implementations we eliminate the constraint that ∑ y a yz = 0 for all z ∈ Z by adding a penalty term ∑ z ∈Z ( ∑ y ∈Y a yz ) 2 to ˆ U ε (and its negative to ˆ L ε ) and then solve unconstrained convex programs using the L-BFGS algorithm [33]. Since the penalty term vanishes only when ∑ y a yz = 0 for all z ∈ Z , we guarantee that the optimal solution is in A .

2.3 Asymptotic properties of the estimated bounds

In the following, we state the assumptions required for our asymptotic analysis of ˆ L ε and ˆ U ε . We start with some regularity assumptions.

Assumption 2.2. L ε and U ε are attained by some optimizers in A (2.2) .

Assumption 2.3. Let ˆ a represent the optimizer for any problem in (2.3) , which is assumed to exist. Suppose ∥ ˆ a ∥ ∞ = O P (1) as m →∞ .

We show in Lemmas C.4, C.5, and C.6 that Assumptions 2.2 and 2.3 can be derived in the binary classification case ( |Y| = 2 ) if P ( Y = y | Z = z ) is bounded away from both zero and one, i.e. κ < P ( Y = y | Z = z ) < 1 -κ for some κ > 0 for every y ∈ Y and z ∈ Z .

In our next assumption, we formalize the degree of precision for the sequence { ˆ P ( m ) Y | Z , m ∈ N } of estimators that we require for desired performances of the bound estimates.

Assumption 2.4. Denote the total variation distance (TV) between probability measures as d TV . For every z ∈ Z , for some λ > 0 , we have that d TV ( ˆ P ( m ) Y | Z = z , P Y | Z = z ) = O P ( m -λ ) .

From Ratner et al. [49]'s Theorem 2 and a Lipschitz property of the label model 3 , we can conclude λ = 1 / 2 for a popular label model used in the PWS literature. The asymptotic distributions for the estimated bounds follow.

Theorem 2.5. Assume 2.2, 2.3, and 2.4, and let n be a function of m such that n → ∞ and n = o ( m 2 λ ) when m →∞ . Then, as m →∞

where σ 2 l,ε ≜ Var f l,ε ( X,Z,a ∗ l,ε ) , σ 2 u,ε ≜ Var f u,ε ( X,Z,a ∗ u,ε ) , and a ∗ l,ε and a ∗ u,ε are the unique optimizers to attain L ε and U ε (2.2) .

Theorem 2.5 tells us that, if the label model is consistent (Assumption 2.4), under some mild regularity conditions (Assumption 2.2 and 2.3), our estimators and will be asymptotically Gaussian with means L ε and U ε and variances σ 2 l,ε /n and σ 2 u,ε /n . The above theorem requires m 2 λ to grow faster than n implying that, through assumption 2.4, P Y | Z is estimated with a precision greater than the approximation error when we replace P X,Z with 1 n ∑ i δ X i ,Z i . In the case which λ = 1 / 2 , this condition translates to n/m → 0 as n →∞ . This allows us to derive the asymptotic distribution when combined with classical results from M-estimation (see proof in Appendix C).

Construction of confidence bounds: One interesting use of Theorem 2.5 is that we can construct an approximate confidence interval for the estimates of the bounds. For example, an approximate 1 -γ confidence interval for L ε can is constructed as

where τ γ = Φ -1 (1 -γ/ 2) and ˆ σ l,ε is the empirical standard deviation of f l,ε ( X,Z, · ) , substituting the estimate ˆ a (solution for the problem in 2.3). For such interval, it holds P ( L ε ∈ ˆ I ) ≈ 1 -γ , i.e. , with approximately 1 -γ confidence we can say that the true L ε is in the interval above. An interval for U ε can be constructed similarly.

3 Evaluation of model performance in weak supervision

In this section, we describe how to use the ideas presented in Section 2 to estimate non-trivial bounds for the evaluation metrics of a weakly supervised classifier h when no high-quality labels are available. In the standard weak supervision setup, only unlabeled data ( X ) is available, but the practitioner can extract weak labels ( Z ) from the available data. More specifically, we assume access to the dataset { ( X i , Z i ) } i m =1 , i.i.d. with distribution P X,Z , used in its entirety to estimate a label model ˆ P Y | Z [49, 22] and where part of it, e.g. , a random subset of size n , is used to estimate bounds 4 . To simplify the exposition, we assume the classifier h is fixed 5 .

3.1 Risk and accuracy

Let ℓ be a generic classification loss function. The risk of a classifier h is defined as R ( h ) = E [ ℓ ( h ( X ) , Y )] , which cannot be promptly estimated in a weak supervision problem, where we do not observe any Y . In this situation, we can make use of our bound estimators in Section 2.2, where we set g ( x, y, z ) = ℓ ( h ( x ) , y ) to obtain bounds for R ( h ) . Furthermore, we can estimate an uncertainty set for the accuracy of the classification simply by letting g ( x, y, z ) = 1 [ h ( x ) = y ] .

3.2 Precision, recall, and F 1 score

For a binary classification problem, where Y = { 0 , 1 } , the precision, recall, and F 1 score of a classifier h are defined as

The quantities P ( h ( X ) = 1) and P ( Y = 1) in the above definitions are identified, since the marginals P X,Z and P Y,Z are specified in the Fréchet problem in (1.1). The P ( h ( X ) = 1) can be estimated from the full dataset { ( X i , Z i ) } i m =1 simply using ˆ P ( h ( X ) = 1) ≜ 1 m ∑ m i =1 1 [ h ( X i ) = 1] . On the other hand, in most weak supervision applications, P ( Y = 1) is assumed to be known from some prior knowledge or can be estimated from an auxiliary dataset, e.g. , using the method described in the appendix of Ratner et al. [49]. Estimating or knowing P ( Y = 1) is required to fit the label model [49, 22] in the first place, so it is beyond our scope of discussion. Then, we assume we have an accurate estimate ˆ P ( Y = 1) .

The probability P ( h ( X ) = 1 , Y = 1) , which is the final ingredient in the definition of precision, recall, and F1 score is not identifiable as P X,Y is unknown. The uncertainty bounds for this quantity can be estimated using our method simply by letting g ( x, y, z ) = 1 [ h ( x ) = 1 and y = 1] . Let ˆ L ε and ˆ U ε denote the estimated lower and upper bounds for P ( h ( X ) = 1 , Y = 1) obtained using (2.3). Naturally, the lower bound estimators for precision, recall, and F 1 score are

while the upper bound estimators ˆ p u,ε , ˆ r u,ε , and ˆ F u,ε are given by substituting ˆ L ε by ˆ U ε above. In the following corollary, we show that the bounds converge asymptotically to normal distributions, which we use for calculating their coverage bounds presented in our applications.

Corollary 3.1. Let n be a function of m such that n → ∞ and n = o ( m (2 λ ) ∧ 1 ) when m → ∞ . Assume the conditions of Theorem 2.5 hold. Then as m →∞

where L ε , σ 2 l,ε are defined in Theorem 2.5. Asymptotic distributions for √ n ( ˆ p u,ε -p u,ε ) , √ n ( ˆ r u,ε -r u,ε ) , and √ n ( ˆ F u,ε -F u,ε ) are obtained in a similar way by changing L ε to U ε and σ 2 l,ε to σ 2 u,ε .

Reiterating our discussion in the final paragraph in Section 2.2, asymptotic distributions are important for constructing confidence intervals for the bounds, which can be done in a similar manner.

4 Experiments

All experiments are structured to emulate conditions where high-quality labels are inaccessible during training, validation, and testing phases, and all weakly-supervised classifiers are trained using the noise-aware loss [50]. To fit the label models, we assume P Y is known (computed using the training set). Unless stated, we use l 2 -regularized logistic regressors as classifiers, where the regularization strength is determined according to the validation noise-aware loss.

Wrench datasets: To carry out realistic experiments within the weak supervision setup and study accuracy/F1 score estimation, we utilize datasets incorporated in Wrench ( W eak Supe r vision B ench mark) [63]. This standardized benchmark platform features real-world datasets and pregenerated weak labels for evaluating weak supervision methodologies. Most of Wrench's datasets are designed for classification tasks, encompassing diverse data types such as tabular, text, and image; all contain their pre-computed weak labels. Specifically, we utilize Census [27], YouTube [1], SMS [2], IMDB [37], Yelp [65], AGNews [65], TREC [31], Spouse [12], SemEval [24], CDR [14], ChemProt [29], Commercial [22], Tennis Rally [22], Basketball [22]. For text datasets, we

youtube

imdb

cdr

commercial

tennis

1.0

1.0

1.0

1.0

1.0

Oracle

accuracy

accuracy

0.5

0.5

0.5

0.5

0.5

0.0-

0.0

0.0+

0.0+

0.0+

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

threshold

threshold

threshold

threshold

threshold

1.0

1.0

1.0

1.0

1.0

Snorkel

accuracy

accuracy

0.5

0.5

0.5

0.5

0.5

0.0+

0.0+

0.0+

0.0+

0.0+

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

threshold

threshold

threshold

threshold

threshold

lowerbound

upper bound

test accuracy/F1

employ the paraphrase-MiniLM-L6-v2 model from the sentence-transformers 6 library for feature extraction [51]. Features were extracted for the image datasets before their inclusion in Wrench.

Hate Speech Dataset [15]: This dataset contains sentence-level annotations for hate speech in English, sourced from posts from white supremacy forums. It encompasses thousands of sentences classified into either Hate ( 1 ) or noHate ( 0 ) categories. This dataset provides an ideal ground for examining recall and precision estimation. Social media moderators aim to maximize the filtering of hate posts, i.e. , increasing recall, while ensuring that non-hate content is rarely misclassified as offensive, maintaining high precision. Analogously to the Wrench text datasets, we utilize paraphrase-MiniLM-L6-v2 for feature extraction.

4.1 Bounding the performance of weakly supervised classifiers

In this section, we conduct an empirical study using some of the Wrench and Hate Speech datasets to verify the validity and usefulness of our methodology. We compare results for which P Y | Z is estimated using the true labels Y ('Oracle') and those derived using Snorkel's [48, 47] default label model with no hyperparameter tuning and a thousand epochs. Such a comparison facilitates an evaluation of our method's efficacy, especially in cases where the label model could be incorrectly specified. Results for other Wrench datasets and one extra label model (FlyingSquid, [22]) are presented in Appendix F.

In Figure 1, we demonstrate our approaches for bounding test metrics, such as accuracy and F1 score (shown in green), when no true labels are available to estimate performance at various classification thresholds for binary classification tasks on Wrench datasets. In the first row ('Oracle'), true labels are used to estimate the conditional distribution P Y | Z , representing a (close to) ideal scenario with a

well-specified label model. In the second row ('Snorkel'), however, we use a label model to estimate P Y | Z without relying on any true labels. Despite potential inaccuracies in Snorkel's label model, it achieves results close to those obtained using true labels to estimate P Y | Z , yielding approximate but useful bounds. This indicates that even if Snorkel's label model is imperfectly specified, its effectiveness in estimating bounds remains similar to that of the 'Oracle' approach, underscoring the value of bounding metrics regardless of label model accuracy. Delving deeper into Figure 1, results for 'youtube', 'commercial', and 'tennis' highlight that our uncertainty about out-of-sample performance is small, even without labeled samples. However, there is a noticeable increase in uncertainty for 'imdb' and 'cdr', making weakly supervised models deployment riskier without

additional validation. Yet, the bounds retain their informative nature. For instance, for those willing to accept the risk, the 'imdb' classifier's ideal threshold stands at . 5 . This is deduced from the flat worst-case and peaking best-case accuracy at this threshold. Table 1 presents some results for 'agnews' (4 classes) and 'semeval' (9 classes). From Table 1, we can see that both 'Oracle' and 'Snorkel' approaches produce valid bounds.

Now, we present bounds on the classifiers' precision and recall across different classification thresholds for the hate speech dataset. This dataset did not provide weak labels, so we needed to generate them. We employed four distinct weak labelers. The initial weak labeler functions are based on keywords and terms. Should words or phrases match those identified as hate speech in the lexicon created by Davidson et al. [13], we categorize the sentence as 1 ; if not, it's designated 0 . The second weak labeler is based on TextBlob's sentiment analyzer [36]: a negative text polarity results in a 1 classification, while other cases are labeled 0 . Our final pair of weak labelers are language

1.0

1.0

Oracle

precision

recall

0.5

0.5

0.0+

0.0

0.0

0.5

1.0

0.0

0.5

1.0

threshold

threshold

1.0

1.0

Snorkel

precision

recall

0.5

0.5

0.0

0.0

0.0

0.5

1.0

0.0

0.5

1.0

threshold

threshold

lower bound

upper bound

testrecall/precision

models, specifically BERT [16] and RoBERTa [34], that have undergone fine-tuning for detecting toxic language or hate speech [35, 28]. Figure 2 presents both recall and precision bounds and test estimates for the weakly-supervised hate speech classifier. Mirroring observations from Figure 1, Snorkel's standard label model gives valuable bounds analogous to scenarios where we employ labels to estimate P Y | Z . If used by practitioners, Figure 2 could help trade-off recall and precision by choosing an appropriate classification threshold in the absence of high-quality labels.

4.2 Choosing a set of weak labels

In this experiment, we examine how our approach performs under the influence of highly informative weak labels as opposed to scenarios with less informative weak labels. Using the YouTube dataset provided by Wrench, we attempt to classify YouTube comments into categories of SPAM or HAM , leveraging Snorkel to estimate P Y | Z . Inspired by Smith et al. [59], we craft three few-shot weak labelers by prompting 7 the large language model (LLM) Llama-2-13b-chat-hf [60]. For each dataset entry, we pose three distinct queries to the LLM. Initially, we inquire if the comment is SPAM or HAM . Next, we provide clear definitions of SPAM and HAM , then seek the classification from LLM. In the third prompt, leveraging in-context learning ideas [17], we provide five representative comments labeled as SPAM / HAM prior to requesting the LLM's verdict on the comment in question. In cases where LLM's response diverges from SPAM or HAM , we interpret it as LLM's abstention.

After obtaining this triad of weak labels, we analyze two situations. Initially, we integrate the top five 8 weak labels ('high-quality' labels) from Wrench. In the subsequent scenario, we synthetically generate weak labels ('low-quality' labels) that do not correlate with Y . The first plot in Figure 3 depicts the bounds of our classifier based solely on weak few-shot labels, which unfortunately do not provide substantial insights. Enhancing the bounds requires the inclusion of additional weak labels. Yet, as indicated by the subsequent pair of plots, it becomes evident that only the incorporation of 'high-quality' weak labels results in significant shrinkage and upward shift of the bounds. As confirmed by the test accuracy, if a practitioner had used our method to select the set of weak labels, that would have led to a significant boost in performance.

4.3 Model selection strategies using the Fréchet bounds

In Sections 4.1 and 4.2, we implicitly touched on the topic of model selection when discussing the classification threshold and weak label selection. Here, we explicitly discuss the use of our Fréchet bounds for model selection purposes. Consider a set of possible models H ≜ { h 1 , · · · , h K } from which we wish to find the best model according to a specific metric, e.g. , accuracy, or F1 score. We consider three approaches for model selection using the Fréchet bounds: choosing the model with the best possible (i) lower bound, (ii) upper bound, and (iii) average of lower and upper bounds on the metric of interest. Strategy (i) works well for the worst-case scenario and can be seen as the

LLMfew-shotWLs+extra

LLM few-shotWLs+extra

LLM few-shot WLs(#WLs=3)

lowqualityWLs(#WLs=8)

high qualityWLs(#WLs=8)

1.00

1.00

1.00

youtube

0.75

0.75

0.75

lower

0.50

0.50

0.50

upper

testacc

0.25

0.25

0.25

0.00

0.00

0.00

0.00

0.25

0.50（

0.75

1.00

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.50

0.75

1.00

threshold

threshold

threshold

distributionally robust optimization (DRO) [9] solution when the uncertainty set is given by Π in (1.1), while (ii) is suitable for an optimistic scenario, and (iii) is suggested when one wants to balance between the worst- and best-case scenarios. Please check Appendix E for more details.

In this experiment, we select multilayer-perceptrons (MLPs). The considered MLPs have one hidden layer with a possible number of neurons in { 50 , 100 } . Training is carried out with Adam [26], with possible learning rates in { . 1 , . 001 } and weight decay ( l 2 regularization parameter) in { . 1 , . 001 } . For those datasets that use the F1 score as the evaluation metric, we also tune the classification threshold in { . 2 , . 4 , . 5 , . 6 , . 8 } (otherwise, they return the most probable class as a prediction). In total, H is composed of 8 trained models when evaluating accuracy and 40 models when evaluating the F1 score. We also consider directly using the label model (Snorkel [47]) to select models. For example, when the metric considered is accuracy, i.e. , we use select the model arg max h k ∈H 1 n ∑ n i =1 E ˆ P Y | Z 1 [ h k ( X ) = Y | Z = Z i ] , which is a natural choice when X ⊥ ⊥ Y | Z . As baselines, we consider having a few labeled samples.

In Table 2, we report a subset of our results (please check Appendix E for the full set of results). In this table, we report the average test scores of the chosen models over 10 repetitions for different random seeds (standard deviation re-

port as subscript). We can extract some lessons from the table. First, using metrics derived from the Fréchet bounds is most useful when our uncertainty about the model performance is low, e.g. , ''commercial' and 'tennis' in Figure 1. In those cases, using our metrics for model selection gives better results even when compared to a labeled validation set of size n = 100 . Moreover, once the practitioner knows that the uncertainty is low, using the label model approach also does well.

5 Discussion

5.1 Extensions

An extension we do not address in the main text is the evaluation of end-to-end weak supervision methods [62, 56, 52], where the separation between the label model and the final predictor is less clear than in our primary setting. Our approach remains compatible with these methods as long as we can fit a label model ( e.g. , Snorkel) separately and utilize it solely for the evaluation step. Another possible extension is the application of the ideas presented in this work in different fields of machine learning or statistics. One could consider applying our ideas to the problem of 'statistical matching' (SM) [18, 10, 19, 30, 11], for example. The classic formulation of SM involves observing two distinct datasets that contain replications of ( X,Z ) and ( Y, Z ) , but the triplet ( X,Y,Z ) is never observed. The primary goal is to make inferences about the relationship between X and Y . For instance, if our focus is on bounding P (( X,Y ) ∈ B ) = E [ 1 B ( X,Y )] for a certain event B , we could define g ( x, y, z ) = 1 B ( x, y ) and apply our method.

5.2 Limitations

We discuss some limitations of our methods. Firstly, our method and theoretical results are only applicable to cases where Y and Z are finite sets, such as in classification problems. Extending the dual formulation in Theorem 2.1 to general Y and Z is possible but would require optimizing over function spaces, which is computationally and theoretically challenging. Additionally, if |Z| is large, convergence may be slow, necessitating a large unlabeled dataset for accurate bounds. Using a smaller, curated set of weak labels, may be more effective for bounds estimation and performance. We end this subsection with two other limitations related to misspecification in the label model and informativeness of the bounds. The proofs of the results introduced in this section are placed in Appendix D.

̸

Label model misspecification: In our asymptotic results we assumed that the label models are well-specified, i.e. , the estimates { ˆ P ( m ) Y | Z , m ∈ N } converge to the true label model P Y | Z at m → ∞ . To understand the qualities of our bound when this assumption is violated, we introduce the misspecification: ˆ P ( m ) Y | Z → Q Y | Z and Q Y | Z = P Y | Z . In our investigation on the misspecification of the label model, we control the level of misspecification as d TV ( Q Y | Z = z , P Y | Z = z ) ≤ δ and then study the subsequent errors in our Fréchet bounds. The following theorem formalizes the result.

Theorem 5.1. Recall from equation (2.2) that L ϵ and U ϵ are the smoothened upper and lower Fréchet bounds with the true P Y | Z = z . Additionally, let us define similar ˇ L ϵ and ˇ U ϵ bounds, but with a misspecified Q Y | Z = z , i.e.

Assume Q Y | Z is in a set of conditional distributions such that the optimizers for (5.1) , which are assumed to exist, are uniformly bounded. If d TV ( Q Y | Z = z , P Y | Z = z ) ≤ δ , then for some C > 0 which is independent of δ > 0 , we have

The above theorem reveals how misspecification translates to the errors in subsequent Fréchet bounds. In an ideal scenario, with access to an { ( Y i , Z i ) } i m =1 sample we can consistently estimate P Y | Z , leading to δ = 0 . In situations when this { ( Y i , Z i ) } i m =1 sample is not accessible and we have to rely on a label model, we require the misspecification in this model to be small.

Informativeness of the bounds: The bounds L and U are especially useful when their difference U -L is small because in that case, we obtain a tight bound for E [ g ( X,Y,Z )] and narrow it down with high precision even if the joint random vector ( X,Y,Z ) is never observed. But when is this bound small? In the next theorem, we provide an upper bound on this difference.

Theorem 5.2. Let L and U be defined as in equation (1.1) . Then

where H ( X | Z ) (resp. H ( Y | Z ) ) denotes the conditional entropy of X (resp. Y ) given Z .

To understand the result better, recall our setting: we do not observe the joint distribution P X,Y,Z and only observe the marginals P Y,Z and P X,Z . The only way we can infer about the joint distribution is by connecting these two marginals through Z . So, readers can guess that it is more favorable when Z is informative for either X or Y . For example, take the extreme case when Y = h ( Z ) for a function h : Y → Z . In this case the P X,Y,Z = P X,g ( Z ) ,Z is precisely known from the P X,Z and we can exactly pinpoint the E [ g ( X,Y,Z )] as E [ g ( X,h ( Z ) , Z )] . In this case, H ( Y | Z ) = 0 , leading to U -L = 0 , i.e. , its Fréchet bounds can precisely pinpoint it as well; ideally at least one of the H ( X | Z ) and H ( Y | Z ) is small.

References

T.C. Alberto and J.V. Lochter. YouTube Spam Collection. UCI Machine Learning Repository, 2015. DOI: https://doi.org/10.24432/C58885.

Tiago Almeida and Jos Hidalgo. SMS Spam Collection. UCI Machine Learning Repository, 2011. DOI: https://doi.org/10.24432/C5CC84.

Dongsheng An, Na Lei, Xiaoyin Xu, and Xianfeng Gu. Efficient optimal transport algorithm by accelerated gradient descent. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 10119-10128, 2022.

Chidubem Arachie and Bert Huang. Adversarial label learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 3183-3190, 2019.

Chidubem Arachie and Bert Huang. A general framework for adversarial label learning. J. Mach. Learn. Res. , 22:118-1, 2021.

Azam Asl and Michael L Overton. Behavior of limited memory bfgs when applied to nonsmooth functions and their nesterov smoothings. In Numerical Analysis and Optimization: NAO-V, Muscat, Oman, January 2020 V , pages 25-55. Springer, 2021.

Daniel Bartl, Michael Kupper, Thibaut Lux, Antonis Papapantoleon, and Stephan Eckstein. Marginal and dependence uncertainty: bounds, optimal transport, and sharpness. SIAM Journal on Control and Optimization , 60(1):410-434, 2022.

Mathias Beiglböck and Walter Schachermayer. Duality for borel measurable cost functions. Transactions of the American Mathematical Society , 363(8):4203-4224, 2011.

Ruidi Chen, Ioannis Ch Paschalidis, et al. Distributionally robust learning. Foundations and Trends® in Optimization , 4(1-2):1-243, 2020.

Pier Luigi Conti, Daniela Marella, and Mauro Scanu. Statistical matching analysis for complex survey data with applications. Journal of the American Statistical Association , 111(516): 1715-1725, 2016.

Pier Luigi Conti, Daniela Marella, and Mauro Scanu. An overview on uncertainty and estimation in statistical matching. Analysis of Integrated Data , pages 73-100, 2019.

David PA Corney, Dyaa Albakour, Miguel Martinez-Alvarez, and Samir Moussa. What do a million news articles look like? In NewsIR@ ECIR , pages 42-47, 2016.

Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. Automated hate speech detection and the problem of offensive language. In Proceedings of the international AAAI conference on web and social media , volume 11, pages 512-515, 2017.

Allan Peter Davis, Cynthia J Grondin, Robin J Johnson, Daniela Sciaky, Benjamin L King, Roy McMorran, Jolene Wiegers, Thomas C Wiegers, and Carolyn J Mattingly. The comparative toxicogenomics database: update 2017. Nucleic acids research , 45(D1):D972-D978, 2017.

Ona de Gibert, Naiara Perez, Aitor García-Pablos, and Montse Cuadros. Hate Speech Dataset from a White Supremacy Forum. In Proceedings of the 2nd Workshop on Abusive Language Online (ALW2) , pages 11-20, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 10.18653/v1/W18-5102. URL https://www.aclweb.org/anthology/ W18-5102 .

Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey for in-context learning. arXiv preprint arXiv:2301.00234 , 2022.

Marcello D'Orazio, Marco Di Zio, and Mauro Scanu. Statistical matching: Theory and practice . John Wiley & Sons, 2006.

Marcello D'Orazio. Statistical learning in official statistics: the case of statistical matching. Statistical Journal of the IAOS , 35(3):435-441, 2019.

Riccardo Fogliato, Alexandra Chouldechova, and Max G'Sell. Fairness evaluation in presence of biased noisy labels. In International conference on artificial intelligence and statistics , pages 2325-2336. PMLR, 2020.

Jason A Fries, Ethan Steinberg, Saelig Khattar, Scott L Fleming, Jose Posada, Alison Callahan, and Nigam H Shah. Ontology-driven weak supervision for clinical entity classification in electronic health records. Nature communications , 12(1):2017, 2021.

Daniel Fu, Mayee Chen, Frederic Sala, Sarah Hooper, Kayvon Fatahalian, and Christopher Ré. Fast and three-rious: Speeding up weak supervision with triplet methods. In International conference on machine learning , pages 3280-3291. PMLR, 2020.

Wenshuo Guo, Mingzhang Yin, Yixin Wang, and Michael Jordan. Partial identification with noisy covariates: A robust optimization approach. In Conference on Causal Learning and Reasoning , pages 318-335. PMLR, 2022.

Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. SemEval2010 task 8: Multi-way classification of semantic relations between pairs of nominals. In Katrin Erk and Carlo Strapparava, editors, Proceedings of the 5th International Workshop on Semantic Evaluation , pages 33-38, Uppsala, Sweden, July 2010. Association for Computational Linguistics. URL https://aclanthology.org/S10-1006 .

Nathan Kallus. What's the harm? sharp bounds on the fraction negatively affected by treatment. Advances in Neural Information Processing Systems , 35:15996-16009, 2022.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Ron Kohavi. Census Income. UCI Machine Learning Repository, 1996. DOI: https://doi.org/10.24432/C5GP7S.

Petra Kralj Novak, Teresa Scantamburlo, Andraž Pelicon, Matteo Cinelli, Igor Mozetiˇ c, and Fabiana Zollo. Handling disagreement in hate speech modelling. In International Conference on Information Processing and Management of Uncertainty in Knowledge-Based Systems , pages 681-695. Springer, 2022.

Martin Krallinger, Obdulia Rabal, Saber A Akhondi, Martın Pérez Pérez, Jesús Santamaría, Gael Pérez Rodríguez, Georgios Tsatsaronis, Ander Intxaurrondo, José Antonio López, Umesh Nandal, et al. Overview of the biocreative vi chemical-protein interaction track. In Proceedings of the sixth BioCreative challenge evaluation workshop , volume 1, pages 141-146, 2017.

Israa Lewaa, Mai Sherif Hafez, and Mohamed Ali Ismail. Data integration using statistical matching techniques: A review. Statistical Journal of the IAOS , 37(4):1391-1410, 2021.

Xin Li and Dan Roth. Learning question classifiers. In COLING 2002: The 19th International Conference on Computational Linguistics , 2002.

Pierre Lison, Aliaksandr Hubin, Jeremy Barnes, and Samia Touileb. Named entity recognition without labelled data: A weak supervision approach. arXiv preprint arXiv:2004.14723 , 2020.

Dong C Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical programming , 45(1-3):503-528, 1989.

Yinhan Liu. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019.

Varvara Logacheva, Daryna Dementieva, Sergey Ustyantsev, Daniil Moskovskiy, David Dale, Irina Krotova, Nikita Semenov, and Alexander Panchenko. ParaDetox: Detoxification with parallel data. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 6804-6818, Dublin, Ireland, May 2022. Association for Computational Linguistics. URL https://aclanthology.org/2022.acl-long.469 .

Steven Loria et al. textblob documentation. Release 0.15 , 2(8):269, 2018.

Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1 , HLT '11, page 142-150, USA, 2011. Association for Computational Linguistics. ISBN 9781932432879.

Charles F Manski. Anatomy of the selection problem. Journal of Human resources , pages 343-360, 1989.

Charles F Manski. Nonparametric bounds on treatment effects. The American Economic Review , 80(2):319-323, 1990.

Charles F Manski. Partial identification of probability distributions , volume 5. Springer, 2003.

Alessio Mazzetto, Cyrus Cousins, Dylan Sam, Stephen H Bach, and Eli Upfal. Adversarial multi class learning under weak supervision with performance guarantees. In International Conference on Machine Learning , pages 7534-7543. PMLR, 2021.

Alessio Mazzetto, Dylan Sam, Andrew Park, Eli Upfal, and Stephen Bach. Semi-supervised aggregation of dependent weak supervision sources with performance guarantees. In International Conference on Artificial Intelligence and Statistics , pages 3196-3204. PMLR, 2021.

Francesca Molinari. Microeconometrics with partial identification. Handbook of econometrics , 7:355-486, 2020.

Wojciech Niemiro. Asymptotics for m-estimators defined by convex minimization. The Annals of Statistics , pages 1514-1533, 1992.

Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trends® in Machine Learning , 11(5-6):355-607, 2019.

Flavien Prost, Pranjal Awasthi, Nick Blumm, Aditee Kumthekar, Trevor Potter, Li Wei, Xuezhi Wang, Ed H Chi, Jilin Chen, and Alex Beutel. Measuring model fairness under noisy covariates: A theoretical perspective. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society , pages 873-883, 2021.

Alex Ratner, Braden Hancock, Jared Dunnmon, Roger Goldman, and Christopher Ré. Snorkel metal: Weak supervision for multi-task learning. In Proceedings of the Second Workshop on Data Management for End-To-End Machine Learning , pages 1-4, 2018.

Alexander Ratner, Stephen H Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. Snorkel: Rapid training data creation with weak supervision. In Proceedings of the VLDB Endowment. International Conference on Very Large Data Bases , volume 11, page 269. NIH Public Access, 2017.

Alexander Ratner, Braden Hancock, Jared Dunnmon, Frederic Sala, Shreyash Pandey, and Christopher Ré. Training complex models with multi-task weak supervision. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 4763-4771, 2019.

Alexander J Ratner, Christopher M De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. Data programming: Creating large training sets, quickly. Advances in neural information processing systems , 29, 2016.

Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bertnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, 2019.

Salva Rühling Cachay, Benedikt Boecking, and Artur Dubrawski. End-to-end weak supervision. Advances in Neural Information Processing Systems , 34:1845-1857, 2021.

Ludger Rüschendorf. Bounds for distributions with multivariate marginals. Lecture NotesMonograph Series , pages 285-310, 1991.

Ludger Rüschendorf. Fréchet-bounds and their applications. In Advances in Probability Distributions with Given Marginals: beyond the copulas , pages 151-187. Springer, 1991.

Ludger Rüschendorf. Risk bounds and partial dependence information. From Statistics to Mathematical Finance: Festschrift in Honour of Winfried Stute , pages 345-366, 2017.

Dylan Sam and J Zico Kolter. Losses over labels: Weakly supervised learning via direct loss construction. In Proceedings of the AAAI conference on artificial intelligence , volume 37, pages 9695-9703, 2023.

Agam Shah, Ruchit Vithani, Abhinav Gullapalli, and Sudheer Chava. Finer: Financial named entity recognition dataset and weak-supervision model. arXiv preprint arXiv:2302.11157 , 2023.

Changho Shin, Winfred Li, Harit Vishwakarma, Nicholas Roberts, and Frederic Sala. Universalizing weak supervision. arXiv preprint arXiv:2112.03865 , 2021.

Ryan Smith, Jason A Fries, Braden Hancock, and Stephen H Bach. Language models in the loop: Incorporating prompting into weak supervision. arXiv preprint arXiv:2205.02318 , 2022.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Paroma Varma, Bryan D He, Payal Bajaj, Nishith Khandwala, Imon Banerjee, Daniel Rubin, and Christopher Ré. Inferring generative model structure with static analysis. Advances in neural information processing systems , 30, 2017.

Yue Yu, Simiao Zuo, Haoming Jiang, Wendi Ren, Tuo Zhao, and Chao Zhang. Fine-tuning pre-trained language model with weak supervision: A contrastive-regularized self-training approach. arXiv preprint arXiv:2010.07835 , 2020.

Jieyu Zhang, Yue Yu, Yinghao Li, Yujing Wang, Yaming Yang, Mao Yang, and Alexander Ratner. Wrench: A comprehensive benchmark for weak supervision. arXiv preprint arXiv:2109.11377 , 2021.

Jieyu Zhang, Cheng-Yu Hsieh, Yue Yu, Chao Zhang, and Alexander Ratner. A survey on programmatic weak supervision. arXiv preprint arXiv:2202.05433 , 2022.

Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems , 28, 2015.

Dawei Zhu, Xiaoyu Shen, Marius Mosbach, Andreas Stephan, and Dietrich Klakow. Weaker than you think: A critical look atweakly supervised learning. arXiv preprint arXiv:2305.17442 , 2023.

A Connection to optimal transport

The optimizations in the Frechet bounds (1.1) can be connected to an optimization problem [45]. We only explain this connection for the lower bound, but the connection to the upper bound is quite similar. The optimization lower bound is

where we notice that the inner minimization is an optimal transport problem between the probability distributions P X | Z = z and P Y | Z = z with the cost function d z ( x, y ) = g ( x, y, z ) .

B A proof for the duality result

Proof of Theorem 2.1. We start proving the result for L . See that

with Π z , is defined as

That is, for each z ∈ Z , Π z represents the set of couplings such that marginals are given by P X | Z = z and P Y | Z = z . We can represent the problem in this way since the marginal distribution of Z is fixed and, given that distribution, { Π z } specifies the same set of distributions as Π .

Realize that we have broken down our initial maximization problem in |Z| smaller minimization problems. Each of those minimization problems can be treated as an optimal transportation problem. Consequently, by Beiglböck and Schachermayer [8, Theorem 1], for each z ∈ Z , we get the following duality result

with

Moreover, partially optimizing on β z ( x ) , we can set β ∗ z ( x ) = min y ∈Y [ g ( x, y, z ) -α z ( y )] and then

where α z is a simple function taking values in the real line.

Consequently,

Because each α z is a function assuming at most |Y| values and we have |Z| functions (one for each value of z ), we can equivalently solve an optimization problem on R |Y|×|Z| . Adjusting the notation,

From Beiglböck and Schachermayer [8, Theorem 2], we know that the maximum is attained by some a ∗ ∈ R |Y|×|Z| . To show that there is a maximizer in A , we need to update the solution

for every z ∈ Z . The objective function is not affected by such translations.

To prove the result for U , first realize that because g is bounded, with no loss of generality, we can assume its range is a subset of [0 , 1] . Define c ( x, y, z ) = 1 -g ( x, y, z ) and see that

Proceeding as before, we can obtain the final result by finding the dual formulation for inf π ∈ Π E π [ c ( X,Y,Z )] .

C Proofs for the estimation results

We will analyze the estimator for U (results for the estimator of L can be obtained analogously). For the next results, we define

Proof of Theorem 2.5. By Assumption 2.2 and Lemmas C.1 and C.2, we can guarantee that E [ ˜ f u,ε ( X,Z,a )] is minimized by a unique a ∗ u,ε (equalling it to U ε ) and that ∇ 2 a E [ ˜ f u,ε ( X,Z,a ∗ u,ε )] is positive definite. Also, from the proof of Lemma C.2, we can see that ˜ f u,ε is convex in a (because its Hessian is positive semidefinite). It is also true that the second moment of ˜ f u,ε ( X,Z,a ) is well defined (exists and finite) for each a since g is bounded. Define

and let ˜ a ε denote a value that attains that minimum; from Niemiro [44, Theorem 4] and the conditions discussed above, we know that √ n (˜ a ε -a ∗ u,ε ) = O P (1) . The existence of ˜ a ε is discussed by Niemiro [44]. Then

where the first term is obtained by a second-order Taylor expansion of the summing functions around ˜ a ε ( ¯ a is some random vector). Also, from the standard central limit theorem, we know that

Given that √ n (˜ a ε -a ∗ u,ε ) = O P (1) and that the Hessian has bounded entries (then O P (1) as well), the first term in C.1 is o P (1) . Because a ∗ u,ε ∈ A , we have that f u,ε ( X,Z,a ∗ u,ε ) = ˜ f u,ε ( X,Z,a ∗ u,ε ) and then

by Slutsky's theorem. Since

if we can show that we are done.

Let ˆ a be solution for the problem in 2.3 and see that

where the last equality is obtained using Assumptions 2.3 and 2.4, and the fact that ∥ ˜ a ε ∥ ∞ is tight (derived from √ n (˜ a ε -a ∗ u,ε ) = O P (1) ). Consequently,

Finally, using Slutsky's theorem,

Proof of Corollary 3.1. We prove the asymptotic distribution of √ n ( ˆ p u,ε -p u,ε ) . The result for the lower bound can be obtained analogously.

First, note that

by the standard central limit theorem.

Next, see that

where the (i) fifth and sixth lines equality is obtained using Taylor's theorem, (ii) sixth and seventh lines equality is obtained using observation C.2 and the fact that n = o ( m (2 λ ) ∧ 1 ) , and (iii) seventh to eighth lines equality is obtained using observation C.2, Theorem 2.5, and Slutsky's theorem.

We prove the asymptotic distribution of √ n ( ˆ r u,ε -r u,ε ) . The result for the lower bound can be obtained analogously. From Lemma C.3, we know that there is an estimator ˆ P ( Y = 1) such that ˆ P ( Y = 1) -P ( Y = 1) = O P ( m -( λ ∧ 1 / 2) ) , i.e. , it has enough precision. We use that estimator.

Next, see that

Finally, we prove the asymptotic distribution of √ n ( ˆ F u,ε -F u,ε ) . The result for the lower bound can be obtained analogously. From the facts stated above, we know that

Then,

where all the steps are justified as before.

C.1 Auxiliary lemmas

From Assumption 2.2, we know that there exists some a ∗ u,ε ∈ R |Y|×|Z| such that

For that specific a ∗ u,ε , we have that

Then

Proof. First, see that

Consequently,

Lemma C.2. The function of a given by E [ ˜ f u,ε ( X,Z,a )] has positive definite Hessian, i.e.,

and, consequently, it is strictly convex.

Proof. See that

We start computing the first term in the sum.

First, for an arbitrary pair ( k, l ) ∈ Y × Z , define

Now, see that and

̸

See that ∂ 2 ∂a pb ∂a kl f u,ε ( x, z, a ) = 0 if b = l . Consequently, the Hessian ∇ 2 a f u,ε ( x, z, a ) is block diagonal.

Consequently, because the second derivatives are bounded, we can push them inside the expectations and get

Because ∇ 2 a f u,ε ( x, z, a ) is block diagonal, we know that the Hessian

is block diagonal (one block for each segment a · z of the vector a ). Now, realize that

is also block diagonal, with each block being matrices of ones. In this case, we also have one block for each segment a · z of the vector a . Consequently, H ε ( a ) is block diagonal, and it is positive definite if and only if all of its blocks are positive definite. Let us analyse an arbitrary block of H ε ( a ) , e.g. , ∇ 2 a · l E [ ˜ f u,ε ( X,Z,a ) ] . Let s · l ( x ) be the vector composed of s kl ( x ) for all k . If 1 ∈ R |Y| denotes a vector of ones, then,

where ˜ 1 = √ ε P ( Z = l ) 1 . See that diag ( s · l ( x ) ) has rank |Y| (full rank) while s · l ( x ) s · l ( x ) ⊤ is rank one for every x ∈ R d X . Consequently, the rank of the difference

is greater or equal |Y| 1 . It is the case that rank ( D ( x )) = |Y| 1 because ˜ 1 is in the null space of D :

Moreover, the range of D ( x ) and ˜ 1 ˜ 1 ⊤ are orthogonal. For any two vectors v , u ∈ R |Y| , we have that

That implies D ( x ) + ˜ 1 ˜ 1 ⊤ is full rank. To see that, let v ∈ R |Y| be arbitrary and see

̸

Because D ( x ) v = 0 , it means that v = θ ˜ 1 for some constant θ . If θ = 0 , then ˜ 1 ˜ 1 ⊤ v = θ |Y| ˜ 1 = 0 . Therefore, θ = 0 and v = 0 .

Now, let u ∈ R |Y| be arbitrary non-null vector and see

by the Cauchy-Schwarz inequality. Then, D ( x ) is positive semidefinite, and because ˜ 1 ˜ 1 ⊤ is also positive semidefinite, their sum needs to be positive definite for all x (that matrix is full rank). Each block of H ε ( a ) is positive definite; consequently, H ε ( a ) is positive definite.

Lemma C.3. Assume Assumption 2.4 holds. Let

Proof. To derive this result, see that

where the standard central limit theorem obtains the third step, the sixth step is obtained by the formula of the total variation distance for discrete measures, and the seventh step is obtained by Assumption 2.4.

Then

̸

Lemma C.4. For some κ > 0 and any y ∈ Y assume that κ ≤ p y ≤ 1 -κ . Define A = { a y ∈ R : ∑ y a y = 0 } . Then the optima of the following problems

are attained in a compact set K ( κ, L ) ⊂ A where ∥ g ∥ ∞ ≤ L .

Proof of lemma C.4. We shall only prove this for the minimization problem. The conclusion for the maximization problem follows in a similar way.

Strict convexity: The second derivation of f u is

where p ( x, y, a ) ≜ exp ( g ( X,y )+ ay ϵ ) ∑ i ∈Y exp ( g ( X,i )+ a i ϵ ) . For any u ∈ R Y we have

where σ 2 ( x, u, a ) is the variance of a categorical random variable taking the value u i with probability p ( x, i, a ) . This leads to the conclusion that the function is convex.

̸

To establish strict convexity, we fix a ∈ A and notice that 0 < p ( x, y, a ) < 1 (because ∥ g ∥ ∞ ≤ L ). Therefore, the σ 2 ( x, u, a ) can be zero only when the u i 's are all equal. Since A = { a ∈ R Y : ∑ y a y = 0 } , such u belongs to the space A in the unique case u i = 0 . This leads to the conclusion that for any u ∈ A and u = 0

Therefore, f u is strictly convex and has a unique minimizer in A . In the remaining part of the proof, we focus on the first-order condition.

First order condition: The first-order condition is

To show that (C.6) has a solution in a compact ball K ( ϵ, κ, L ) ≜ { a ∈ A : ∥ a ∥ 2 ≤ M ( ϵ, κ, L ) } we construct an M ( ϵ, κ, L ) > 0 such that min y h ( a, y ) < κ whenever ∥ a ∥ 2 > M ( ϵ, κ, L ) . Since κ ≤ p y ≤ 1 -κ for all y ∈ Y this concludes that the solution to (C.6) must be inside the K ( ϵ, κ, L ) .

Construction of the compact set: Let us define y min ≜ arg min y a y and y max ≜ arg max y a y . To construct M ( κ, L, ϵ ) (we shall write it simply as M whenever it is convenient to do so) we notice that

where

We rewrite the constraints of the optimization as:

where we use the Popoviciu's inequality on variance to obtain

Thus R ≥ 2 M √ |Y| whenever ∥ a ∥ 2 > M . We use this inequality in (C.7) and obtain

Finally, we choose M = M ( ϵ, κ, L ) > 0 large enough such that

For such an M we concludes that h ( a, y min ) < κ whenever ∥ a ∥ 2 > M .

Lemma C.5. L ε and U ε are attained by some optimizers in a compact set K ( ϵ, κ z , z ∈ Z ; L ) ⊃ A ( (2.2) ), where κ z = min { p y | z , 1 -p y | z : y ∈ Y} .

Proof of lemma C.5. We shall only prove the case of the minimization problem. The proof for the maximization problem uses a similar argument.

A decomposition of the minimization problem according to the values of Z follows.

where p z = P ( Z = z ) and p y | z = P ( Y = y | Z = z ) . We fix a z and consider the corresponding optimization problem in the decomposition. Then according to lemma C.4 the optimal point is in a compact set K ( ϵ, κ z , L ) . Thus the optimal point of the full problem is in the Cartesian product ∏ z K ( ϵ, κ z , L ) ⊂ A , which a compact set. Thus we let K ( ϵ, κ z , z ∈ Z ; L ) = ∏ z K ( ϵ, κ z , L ) .

Lemma C.6. The optimizers for the problems in (2.3) are tight with respect to m,n →∞ .

Proof of the lemma C.6. Let p y | z = P ( Y = y | Z = z ) and ˆ p ( m ) y | z = ˆ P ( m ) Y | Z = z ( y ) . Since

according to the assumption 2.4, for sufficiently large m with high probability p ( m ) ( p ( m ) → 1 as m →∞ ) κ z 2 ≤ ˆ p ( m ) y | z ≤ 1 -κ z 2 for all y . Fix such an m and ˆ p ( m ) y | z . In lemma C.5 we replace P X,Z with ∑ n i =1 δ X i ,Z i and p y | z with ˆ p ( m ) y | z to reach the conclusion that the optimizer is in the compact set K ( ϵ, κ z / 2 , z ∈ Z ; L ) . Thus, for sufficiently large m and any n

This establishes that the optimizers are tight.

D Proof of extra results

Proof of Theorem 5.1. We only prove the theorem for upper Fréchet bound. The proof of lower bound is similar.

First, notice that and thus

Defining

we now establish that

This is easily established from the following arguments:

and similarly

By assumption, a ⋆ and ˇ a are bounded. Thus, we can find a C > 0 , which is independent of δ > 0 , such that

and the theorem holds for this C .

Proof of Theorem 5.2. For clarity, we shall assume X is a continuous random variable in this proof, even though this is not strictly needed.

Let Q (1) X,Y,Z and Q (2) X,Y,Z be two distributions in Π , i.e. , with marginals P X,Z and P Y,Z . Let the densities of Q (1) X,Y,Z and Q (2) X,Y,Z as q (1) X,Y,Z and q (2) X,Y,Z . Then, see that

where step D.9 is justified by triangle inequality, step D.10 is justified by Pinsker's inequality, step D.11 is justified by Jensen's inequality, and step D.12 is justified by the fact that both expected KL terms are conditional mutual information terms, which can be bounded by the conditional entropy.

Following the same idea, we can show that ∣ ∣ ∣ ∫ g d Q (1) X,Y,Z -∫ g d Q (2) X,Y,Z ∣ ∣ ∣ ≤ √ 8 ∥ g ∥ 2 ∞ H ( Y | Z ) . Consequently,

Because this statement is valid for any distributions Q (1) X,Y,Z and Q (2) X,Y,Z in Π , we have that

E Model selection using performance bounds

In this section, we propose and empirically evaluate three strategies for model selection using our estimated bounds in Equation 2.3.

E.0.1 Introducing model selection strategies

Assume, for example, g ( x, y, z ) = 1 [ h ( x ) = y ] for a given classifier h , i.e. , we conduct model selection based on accuracy, even though we can easily extend the same idea to different choices of metrics, such as F1 score. The model selection problem consists of choosing the best model from a set H ≜ { h 1 , · · · , h K } in order to maximize out-of-sample accuracy. Define ˆ L ε ( h ) and ˆ U ε ( h ) as the estimated accuracy lower and upper bounds for a certain model h . The first strategy is to choose the model with highest accuracy lower bound, i.e. ,

Maximizing the accuracy lower bound approximates the distributionally robust optimization (DRO) [9] solution when the uncertainty set is given by Π in 1.1. That is, we optimize for the worst-case distribution in the uncertainty set. Analogously, we can choose the model that optimizes the best-case scenario

Moreover, if we want to guarantee that both worst and best-case scenarios are not bad, we can optimize the average of upper and lower bounds, i.e. ,

E.0.2 Experiment setup

In this experiment, we select multilayer-perceptrons (MLPs). The considered MLPs have one hidden layer with a possible number of neurons in { 50 , 100 } . Training is carried out with Adam [26], with possible learning rates in { . 1 , . 001 } and weight decay ( l 2 regularization parameter) in { . 1 , . 001 } . For those datasets that use the F1 score as the evaluation metric, we also tune the classification threshold in { . 2 , . 4 , . 5 , . 6 , . 8 } (otherwise, they return the most probable class as a prediction). In total, H is composed of 8 trained models when evaluating accuracy and 40 models when evaluating the F1 score. We also consider directly using the label model (Snorkel [47]) to select models. For example, when the metric considered is accuracy, i.e. , we use

which is a natural choice when X ⊥ ⊥ Y | Z . As benchmarks, we consider having a few labeled samples.

In Table 3, we report the average test scores of the chosen models over 10 repetitions for different random seeds (standard deviation report as subscript). The main message here is that, for those datasets in which our uncertainty about the score (given by the different upper and lower bounds) is small, e.g. , ''commercial' and 'tennis', using our approaches leads to much better results when compared to using small sample sizes.

Now, we explore a different way of comparing models. Instead of making an explicit model selection, this experiment uses the proposed metrics, i.e. , accuracy lower/upper bounds, bounds average, to rank MLP classifiers. We rank the models using both the test set accuracy/F1 score (depending on Zhang et al. [63]) and alternative metrics, i.e. , accuracy/F1 score lower/upper bounds, bounds average, label model, and small labeled sample sizes. Then, we calculate a Pearson correlation between rankings and display numbers in Table 4. If the numbers are higher, it means that the proposed selection method is capable of distinguishing good from bad models. Table 4 shows that the bounds average and label model methods usually return the best results when no labels are used. Moreover, in some cases using a small labeled sample for model selection can relatively hurt performance (when the validation set is small, there is a chance all models will have the same or similar performances, leading to smaller or null rank correlations).

metric

Lower bound

Upper bound

Bounds avg

Label model

Labeled (

n

= 10

)

Labeled (

n

= 25

)

Labeled (

n

= 50

)

Labeled (

n

= 100

)

acc

0

.

77

±

0

.

00

0

.

78

±

0

.

00

0

.

78

±

0

.

00

0

.

77

±

0

.

00

0

.

75

±

0

.

02

0

.

75

±

0

.

02

0

.

75

±

0

.

03

0

.

77

±

0

.

00

trec

acc

0

.

27

±

0

.

01

0

.

29

±

0

.

02

0

.

29

±

0

.

02

0

.

29

±

0

.

02

0

.

28

±

0

.

02

0

.

28

±

0

.

02

0

.

28

±

0

.

02

0

.

28

±

0

.

02

semeval

acc

0

.

32

±

0

.

01

0

.

26

±

0

.

01

0

.

32

±

0

.

01

0

.

32

±

0

.

01

0

.

30

±

0

.

02

0

.

29

±

0

.

03

0

.

31

±

0

.

02

0

.

31

±

0

.

02

chemprot

acc

0

.

40

±

0

.

00

0

.

38

±

0

.

00

0

.

40

±

0

.

00

0

.

40

±

0

.

00

0

.

40

±

0

.

01

0

.

40

±

0

.

01

0

.

40

±

0

.

00

0

.

39

±

0

.

01

youtube

acc

0

.

89

±

0

.

02

0

.

84

±

0

.

05

0

.

87

±

0

.

01

0

.

89

±

0

.

02

0

.

90

±

0

.

02

0

.

90

±

0

.

02

0

.

90

±

0

.

02

0

.

91

±

0

.

02

imdb

acc

0

.

72

±

0

.

00

0

.

74

±

0

.

00

0

.

73

±

0

.

00

0

.

73

±

0

.

00

0

.

69

±

0

.

05

0

.

71

±

0

.

02

0

.

73

±

0

.

01

0

.

72

±

0

.

01

yelp

acc

0

.

81

±

0

.

00

0

.

81

±

0

.

00

0

.

81

±

0

.

00

0

.

81

±

0

.

00

0

.

81

±

0

.

04

0

.

81

±

0

.

03

0

.

81

±

0

.

01

0

.

82

±

0

.

01

census

F1

0

.

49

±

0

.

00

0

.

18

±

0

.

00

0

.

49

±

0

.

00

0

.

49

±

0

.

00

0

.

49

±

0

.

00

0

.

49

±

0

.

00

0

.

49

±

0

.

00

0

.

50

±

0

.

00

tennis

F1

0

.

76

±

0

.

01

0

.

76

±

0

.

01

0

.

76

±

0

.

01

0

.

75

±

0

.

01

0

.

71

±

0

.

01

0

.

71

±

0

.

01

0

.

71

±

0

.

01

0

.

71

±

0

.

02

sms

F1

0

.

00

±

0

.

00

0

.

14

±

0

.

02

0

.

14

±

0

.

02

0

.

14

±

0

.

02

0

.

00

±

0

.

00

0

.

00

±

0

.

00

0

.

00

±

0

.

00

0

.

00

±

0

.

00

cdr

F1

0

.

48

±

0

.

00

0

.

29

±

0

.

00

0

.

48

±

0

.

00

0

.

48

±

0

.

00

0

.

47

±

0

.

00

0

.

47

±

0

.

00

0

.

47

±

0

.

00

0

.

47

±

0

.

00

basketball

F1

0

.

16

±

0

.

01

0

.

27

±

0

.

01

0

.

26

±

0

.

01

0

.

17

±

0

.

01

0

.

20

±

0

.

05

0

.

17

±

0

.

03

0

.

16

±

0

.

01

0

.

16

±

0

.

01

spouse

F1

0

.

27

±

0

.

01

0

.

27

±

0

.

01

0

.

27

±

0

.

01

0

.

29

±

0

.

01

0

.

00

±

0

.

00

0

.

00

±

0

.

00

0

.

00

±

0

.

00

0

.

10

±

0

.

12

commercial

F1

0

.

96

±

0

.

00

0

.

96

±

0

.

00

0

.

96

±

0

.

00

0

.

96

±

0

.

00

0

.

90

±

0

.

01

0

.

90

±

0

.

02

0

.

91

±

0

.

01

0

.

91

±

0

.

01

metric

Lower bound

Upper bound

Bounds avg

Label model

Labeled (

n

= 10

)

Labeled (

n

= 25

)

Labeled (

n

= 50

)

Labeled (

n

= 100

)

acc

0

.

91

±

0

.

01

0

.

86

±

0

.

00

0

.

86

±

0

.

00

0

.

92

±

0

.

01

0

.

39

±

0

.

31

0

.

35

±

0

.

54

0

.

50

±

0

.

53

0

.

76

±

0

.

19

trec

acc

-

0

.

32

±

0

.

36

0

.

37

±

0

.

32

0

.

19

±

0

.

34

0

.

44

±

0

.

39

-

0

.

03

±

0

.

55

-

0

.

06

±

0

.

39

-

0

.

10

±

0

.

34

0

.

17

±

0

.

52

semeval

acc

0

.

87

±

0

.

04

-

0

.

84

±

0

.

04

0

.

88

±

0

.

04

0

.

88

±

0

.

04

0

.

13

±

0

.

64

0

.

17

±

0

.

69

0

.

64

±

0

.

32

0

.

62

±

0

.

41

chemprot

acc

0

.

75

±

0

.

08

-

0

.

66

±

0

.

11

0

.

38

±

0

.

11

0

.

49

±

0

.

15

-

0

.

01

±

0

.

47

-

0

.

12

±

0

.

45

-

0

.

02

±

0

.

43

0

.

06

±

0

.

41

youtube

acc

0

.

38

±

0

.

21

-

0

.

42

±

0

.

19

-

0

.

01

±

0

.

21

0

.

17

±

0

.

29

0

.

41

±

0

.

35

0

.

42

±

0

.

27

0

.

38

±

0

.

31

0

.

58

±

0

.

19

imdb

acc

0

.

84

±

0

.

01

0

.

76

±

0

.

00

0

.

71

±

0

.

02

0

.

92

±

0

.

01

0

.

17

±

0

.

49

0

.

40

±

0

.

46

0

.

61

±

0

.

28

0

.

65

±

0

.

30

yelp

acc

0

.

01

±

0

.

08

-

0

.

50

±

0

.

05

-

0

.

49

±

0

.

05

-

0

.

31

±

0

.

10

0

.

08

±

0

.

65

0

.

09

±

0

.

53

0

.

14

±

0

.

45

0

.

45

±

0

.

30

census

F1

0

.

85

±

0

.

00

0

.

94

±

0

.

00

0

.

96

±

0

.

00

0

.

98

±

0

.

00

0

.

00

±

0

.

00

0

.

00

±

0

.

00

0

.

03

±

0

.

09

0

.

18

±

0

.

17

tennis

F1

0

.

83

±

0

.

02

0

.

74

±

0

.

03

0

.

80

±

0

.

02

0

.

90

±

0

.

03

0

.

33

±

0

.

24

0

.

41

±

0

.

22

0

.

51

±

0

.

12

0

.

52

±

0

.

12

sms

F1

0

.

00

±

0

.

00

0

.

99

±

0

.

02

0

.

99

±

0

.

02

1

.

00

±

0

.

00

0

.

00

±

0

.

00

0

.

00

±

0

.

00

0

.

00

±

0

.

00

0

.

00

±

0

.

00

cdr

F1

0

.

80

±

0

.

00

0

.

89

±

0

.

00

0

.

92

±

0

.

00

0

.

98

±

0

.

00

0

.

03

±

0

.

08

0

.

07

±

0

.

11

0

.

07

±

0

.

11

0

.

09

±

0

.

11

basketball

F1

0

.

85

±

0

.

01

0

.

69

±

0

.

00

0

.

73

±

0

.

01

0

.

81

±

0

.

00

0

.

35

±

0

.

29

0

.

53

±

0

.

18

0

.

59

±

0

.

00

0

.

59

±

0

.

00

spouse

F1

0

.

53

±

0

.

00

0

.

86

±

0

.

00

0

.

86

±

0

.

00

1

.

00

±

0

.

00

0

.

00

±

0

.

00

0

.

00

±

0

.

00

0

.

00

±

0

.

00

0

.

10

±

0

.

12

commercial

F1

0

.

95

±

0

.

00

0

.

96

±

0

.

00

0

.

99

±

0

.

00

1

.

00

±

0

.

00

0

.

14

±

0

.

13

0

.

26

±

0

.

14

0

.

28

±

0

.

13

0

.

31

±

0

.

06

F More on experiments

F.1 Extra results for the Wrench experiment

Extra results for binary classification datasets can be found in Figures 4 and 5. In Table 5, we can see the results for multinomial classification datasets.

youtube

imdb

cdr

commercial

tennis

1.0

1.0

1.0

1.0

1.0

Oracle

accuracy

accuracy

0.5

0.5

0.5

F0.5-

0.5 -

0.0+

0.0+

0.0+

0.0+

0.0+

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

threshold

threshold

threshold

threshold

threshold

1.0

1.0

1.0

1.0

1.0

Snorkel

accuracy

accuracy

0.5

0.5

0.5

0.5

0.5

0.0+

0.0+

0.0+

0.0+

0.0+

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

threshold

threshold

threshold

threshold

threshold

FlyingSquid

1.0

1.0

1.0

1.0

1.0

accuracy

accuracy

0.5

0.5

F0.5

0.5

0.5

0.0+

0.0+

0.0+

0.0+

0.0+

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

threshold

threshold

threshold

threshold

threshold

lowerbound

upperbound

test accuracy/F1

yelp

sms

basketball

spouse

census

1.0

1.0

1.0

1.0

1.0

Oracle

accuracy

0.5

0.5

0.5

0.5

0.5

0.0+

0.0+

0.0+

0.0+

0.0+

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

threshold

threshold

threshold

threshold

threshold

1.0

1.0

1.0

1.0

1.0

Snorkel

accuracy

0.5

0.5

0.5 -

0.5

0.5

0.0 +

0.0 +

0.0+

0.0 +

0.0+

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

threshold

threshold

threshold

threshold

threshold

FlyingSquid

1.0

1.0

1.0

1.0

1.0

accuracy

0.5

0.5

0.5

0.5

0.5 -

0.0

0.0

0.0

0.0 +

0.0+

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

threshold

threshold

threshold

threshold

threshold

lower bound

upper bound

test accuracy/F score

F.2 Extra plots for the hate speech detection experiment

In Table 6, we can see the same results already in the main text plus the results for FlyingSquid.

G Computing resources

All experiments were conducted using a virtual machine with 32 cores. The experiments are not computationally intensive and everything can be run within a few hours.

1.0

1.0

Oracle

precision

recall

0.5

0.5

0.0+

0.0

0.0

0.5

1.0

0.0

0.5

1.0

threshold

threshold

1.0

1.0

Snorkel

precision

recall

0.5

0.5

0.0+

0.0+

0.0

0.5

1.0

0.0

0.5

1.0

threshold

threshold

FlyingSquid

1.0

1.0

precision

recall

0.5

0.5

0.0+

0.0+

0.0

0.5

1.0

0.0

0.5

1.0

threshold

threshold

lowerbound

upperbound

test recall/precision

G.1 Examples of prompts used in Section 4.2

Prompt 1

You s h o u l d c l a s s i f y t h e t a r g e t s e n t e n c e as " spam" or "ham " . I f d e f i n i t i o n s or examples are i n t r o d u c e d , you s h o u l d c o n s i d e r them when c l a s s i f y i n g s e n t e n c e s . Respond with " spam" or "ham " .

Target s e n t e n c e : i f your l i k e drones , plz s u b s c r i b e t o Kamal Tayara . He t a k e s videos with hi s drone t h a t are a b s o l u t e l y b e a u t i f u l . --Response :

Prompt 2

You s h o u l d c l a s s i f y t h e t a r g e t s e n t e n c e as " spam" or "ham " . I f d e f i n i t i o n s or examples are i n t r o d u c e d , you s h o u l d c o n s i d e r them when c l a s s i f y i n g s e n t e n c e s . Respond with " spam" or "ham " .

Definition of spam : spam i s a t e r m r e f e r e n c i n g a broad c a t e g o r y of postings which abuse web-based f o r m s t o post u n s o l i c i t e d a d v e r t i s e m e n t s as comments on forums , blogs , wikis and online guestbook .

Definition of ham: t e x t s t h a t are not spam .

Target s e n t e n c e : i f your l i k e drones , plz s u b s c r i b e t o Kamal Tayara . He t a k e s videos with hi s drone t h a t are a b s o l u t e l y b e a u t i f u l . --Response :

Prompt 3

You s h o u l d c l a s s i f y t h e t a r g e t s e n t e n c e as " spam" or "ham " . I f d e f i n i t i o n s or examples are i n t r o d u c e d , you s h o u l d c o n s i d e r them when c l a s s i f y i n g s e n t e n c e s . Respond with " spam" or "ham " .

Example 0: 860 ,000 ,000 l e t s make i t f i r s t f e m a l e t o r e a c h one b i l l i o n ! ! Share i t and r e p l a y i t ! --Response : ham

Example 1: Waka waka eh eh --Response : ham

Example 2: You guys s h o u l d check out t h i s EXTRAORDINARY website c a l l e d ZONEPA.COM . You can make money online and s t a r t working from home t o d a y as I am! I am making over $3 ,000+ per month a t ZONEPA.COM ! Visit Zonepa . com and check i t out ! How does t h e mother approve t h e axiomatic i n s u r a n c e ? The f e a r a p p o i n t s t h e r o l l . When does t h e s p a c e prepare t h e h i s t o r i c a l shame? --Response : spam

Example 3: Check out t h e s e I r i s h guys cover of Avicii &#39; s Wake Me Up! J u s t s e a r c h . . . &quot ; wake me up Fiddle Me S i l l y&q u o t ; Worth a l i s t e n f o r t h e gorgeous f i d d l e pl a y e r ! --Response : spam

Example 4: i f you want t o win money a t hopme c l i c k here <a href =" h t t p s : / / www. p a i d v e r t s . com / r e f / sihaam01 "> h t t p s : / / www. p a i d v e r t s . com / r e f / sihaam01 </ a> i t &#39;s work 100/100 --Response : spam

Target s e n t e n c e : i f your l i k e drones , plz s u b s c r i b e t o Kamal Tayara . He t a k e s videos with hi s drone t h a t are a b s o l u t e l y b e a u t i f u l . --Response :

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In the abstract and Introduction section, we summarize the contributions and the scope of the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have included a section to discuss limitations in the conclusion.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The assumptions are discussed in the statements of the theorems and the proofs are provided in the Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide a general setting of the experiments in the paper and details are provided in the code itself submitted as supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide simulation settings that are accessible and reproducible through the submitted zip file.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We do specify those parameters in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provide errorbars for the plots.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We include a section in the appendix about this.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Yes, we followed the NeurIPS Code of Ethics

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work may have potential societal consequences, none of which we feel must be specifically highlighted here.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We make citations when needed.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes solutions via convex programs to estimate Frechet bounds for Programmatic Weak Supervision (PWS). This approach uses estimates of the true labels via labelmodels (i.e., different aggregation schemes that exist in the literature). With these estimates of the labels, they provide an approach to estimate bounds on the accuracy (and other quantities) of the weak labelers. They provide experiments to check the validity of their bounds and also provide experiments with weak labelers generated via prompting to examine how their bounds perform under instances of weak labelers with different qualities/accuracies.\n\nSTRENGTHS:\nThe strengths of this paper are that it analysis the PWS setup from a theoretical perspective, tackling the problem of analysis the performance of individual weak labelers. \n\n1. The authors provide a nice analysis of their estimation scheme.\n2. They derive the asymptotic distributions of their estimators and show they are consistent.\n3. They also provide CI for their estimates given finite sample data.\n4. A novel application of Frechet bounds (and the corresponding literature) to the field of programmatic weak supervision.\n\nWEAKNESSES:\nMy only substantiative concern with this paper is that its applicability to the task of label set selection is weirdly motivated. Given a correctly specified label model and a better than random weak labeler (which is exactly the setting that where this method achieves valid bounds), then there is no need to prune away worse-performing weak labelers as they are helpful for the labelmodel. I agree that removing these low-accuracy weak labelers can be helpful given misspecification or a violation of assumptions, but in the setting where this paper achieves valid bounds, then I believe there is no need for this pruning.\n\nQUESTIONS:\n1. Do you have any intuitions about settings where the usage of your method for label set selection is well-motivated? \n2. There's also a line of work in PWS that looks to train end models directly [1, 2, 3]. Can the bounds of this work be applied directly to these models to analyze their error rates? This added discussion would be interesting to broaden the applicability of this theoretical framework.\n\n[1] Sam, et. al. Losses over Labels: Weakly Supervised Learning via Direct Loss Construction.\n\n[2] Cachay, et. al. End-to-End Weak Supervision.\n\n[3] Yu, et. al. Fine-Tuning Pre-trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approach.\n\nLIMITATIONS:\nYes, limitations have been adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method for bounding the performance of models obtained using weak-supervision, without the need for a gold-standard label set. The proposed approach starts with Frechet bounds, and casts them as dual optimization problems. From there, the dual optimization problems are relaxed to a smooth functions which can be solved using L-BFGS or similar methods. Finally, the authors explore the performance and utility of the proposed method on several empirical datasets. The evaluation demonstrates a non-trivial ability to bound performance without gold-standard labels and to perform model selection from a small set of models.\n\nSTRENGTHS:\nThe authors attack a thorny weakness in the weak-supervision literature; the need for a gold-standard dataset. This need has been present going back to the seminal work by Ratner et al. and undermines the approach. It also addresses model evaluation, which is arguably a more important and difficult problem than model training. After all, it's not a real surprise that heuristics could be cobbled together to produce a model. But, quantifying exactly how good that model might be without direct access to labels is another story.\n\nWEAKNESSES:\nThe paper is somewhat technical and difficult to follow in places. For example, I stared at Thm 2.1 for some time before realizing that Appendix C existed. The writing could be improved by noting connections to the appendices in the main body of the text, where appropriate. In addition, noting the primary tool(s) used to get each result would help the reader intuit the type of machinery being leveraged. \n\nFinally, the experiments show some utility of the method. Could the authors give an opinion on the applications or circumstances in which this method could be used for real-world problems? Do we need confidence that the label model is correct? How many weak labels do we need? etc. Appendix A.2 goes some way to helping here. Again, it would be great to mention that this content exists in the main text.\n\nQUESTIONS:\nNo additional questions.\n\nLIMITATIONS:\nThe authors have addresses the limitations of the work.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nProgrammatic weak supervision is a machine learning approach where labeled training data is generated using heuristic rules, domain-specific functions, or other programmatic methods, rather than manual annotation. This technique allows for the creation of large training datasets quickly and cost-effectively by leveraging expert knowledge and automated processes. Uncertainty in programmatic weak supervision arises from the noisiness and variability of labels generated by heuristic rules, domain-specific functions, or automated methods. Unlike manually annotated datasets, programmatically generated labels often contain errors and inconsistencies due to imperfect heuristics that may not accurately capture the true patterns. The paper prososes techniques that estimate Fréchet bounds to validate the performance of programmatic weak supervision models. The proposed algorithms estimate upper and lower bounds for multiple metrics.\n\nSTRENGTHS:\nThe algorithm developed in this work has solid theoretical justification, which reinforces its validity and applicability. Additionally, the presented theory is very interesting and makes a significant contribution to the field.\n\nProposing upper and lower bounds for this framework is very useful and interesting, as well as novel.\n\nWEAKNESSES:\nNot explaining the problem along with the state-of-the-art issues in the introduction makes it difficult to understand the contributions and the objective of the paper.\n\nI believe the paper could be written more clearly; currently, it is difficult to follow.\n\nAs the authors mention, the labels could be obtained using a label model. If from Section 2.2 onwards a label model is not proposed, but instead the work deals with a dataset where some labels may be incorrect, why not directly state that this is a supervised classification framework with noisy labels instead of PWS?\n\nTheorem 2.4 is not easy to interpret.\n\nThe quality of Figure 1 can be improved, and this figure needs more explanation.\n\nQUESTIONS:\nSee Weaknesses Section.\n\nLIMITATIONS:\nThe authors describe limitations in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the problem of Frechet bound which is the task of determining the infimum and supremum of a function g(X,Y,Z) over the set of joint distribution with fixed marginals. Asymptotic behavior is derived when estimation of condition distribution P_{Y|z} is available with vanishing total variation error. This bound is based on a nice dual formalization of the original optimization problem.\n\nThis approach is then applied in a weak supervised learning setup to derive confidence interval for the precision and recall. Weak supervised means that cheap label is available which is the random variable Z, and the expensive target label is Y. So if one can estimate the condition probability of Y given Z, the performance guaranties can be derived for label Y which is the target label.\n\nSTRENGTHS:\nI think the dual formalization of the optimization problem is nice part of the paper. I am not sure how novel it is, since the proof heavily relies on some results on previous work. The problem which is considered is relevant.\n\nWEAKNESSES:\nIt is not clear what is the merit of the approach, see my third question. I am a little bit confused about the contribution of the paper. Estimating the precision/recall seems cheap, but estimating the conditional distribution of the true label conditioned on the proxy label might be expensive? Especially in multi-class setting.\n\nQUESTIONS:\n1) why it is not possible to get high probability results? \n2) Regarding Theorem 2.4, if P_{Y|z} can be estimated with an error O(m^{-1/2}) in terms total variation distance (which is the optimal rate for discrete distributions), then the variance of the optimal solution of the problem based on estimates is shrinking of order with the same order? Is that right?\n3) Why it is cheaper to estimate the conditional distribution of Y given z, then to estimate the precision and recall of the model based on Y directly?\n\nLIMITATIONS:\nI think the this approach can be applied only for small label spaces. In the conclusion, the authors points out that this approach can be applied for finite set, but I believe that this approach has only merit if the label space is very small.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-18"}