Title: Statistical Analysis of an Adversarial Bayesian Weak Supervision Method
Venue: NeurIPS 2025
Accepted: yes
Keywords: weak supervision

bayesian

adversarial

convergence

consistency
Abstract: Programmatic Weak Supervision (PWS) aims to reduce the cost of constructing large high quality labeled datasets often used in training modern machine learning models.  A major component of the PWS pipeline is the label model, which amalgamates predictions from multiple noisy weak supervision sources, i.e. labeling functions (LFs), to label datapoints.  While most label models are either probabilistic or adversarial, a recently proposed label model achieves strong empirical performance without falling into either camp.  That label model constructs a polytope of plausible labelings using the LF predictions and outputs the "center" of that polytope as its proposed labeling.  In this paper, we attempt to theoretically study that strategy by proposing Bayesian Balsubramani-Freund (BBF), a label model that implicitly constructs a polytope of plausible labelings and selects a labeling in its interior.  We show an assortment of statistical results for BBF: log-concavity of its posterior, its form of solution, consistency, and rates of convergence.  Extensive experiments compare our proposed method against twelve baseline label models over eleven datasets.  BBF compares favorably to other Bayesian label models and label models that don't use datapoint features -- matching or exceeding their performance on eight out of eleven datasets.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper suggests BBF, a label model that aggregates labeling functions predictions for weak supervision.\nThe paper investigates the model theoretically and empirically.\nIn empirical evaluation the model outperforms other Bayesian label models.\n\nSTRENGTHS AND WEAKNESSES:\nThe authors do a mathematical analysis of their model.\nOn the other hand they also do intensive empirical evaluation.\n\nThe benchmark uses many models and datasets, and the method shows improvement over Bayesian models.\n\nAssumptions, setting,  and therefore the proof are not presented clearly.\nI tried to point out a few unclear points.\n\nAlso, there might be certain assumptions and constraints of the models used in the benchmark, hence their straight forward usage might be under performing.\nFor example, “On Tennis, the three LFs that each predicted on 99% of the datapoints closely matched each other” hints that the labeling functions are not conditionally independent given the concept. Hence, performance might be better on a subset of the functions.\n\nA general comment, not part of the judgment:\n“then regardless of what h(w) predicts, its accuracy can only be 1/K.” - Isn’t it since some labels can be positive for a sample yet the LF can hit only a single label?\nIf so, adjusting them might help improve performance and bound.\n\nI like the paper and I think that its weaknesses can be taken care of during the rebuttal.\nIf so, I’ll be happy to accept.\n\nQUESTIONS:\nThe assumptions on the labeling functions themselves are not clearly given.\nShould they all be weak classifiers?\nSchapire, Robert E. \"The strength of weak learnability.\" Machine learning 5 (1990): 197-227.\nShould they be independent given the concept? Any other constraint preventing using the same labeling function ten times?\n\n\nAssumptions never exactly match reality.\nIt will be interesting to add lower bounds of the models on the datasets and their comparison to actual performance in Table 2.\n\nThe EM algorithm influenced the entire area, via Dawid & Skene and more. \nIt will be nice to add a comparison to it.\nEspecially, is the process a single step or iterative?\nDempster, Arthur P., Nan M. Laird, and Donald B. Rubin. \"Maximum likelihood from incomplete data via the EM algorithm.\" Journal of the royal statistical society: series B (methodological) 39.1 (1977): 1-22.\nDawid, Alexander Philip, and Allan M. Skene. \"Maximum likelihood estimation of observer error‐rates using the EM algorithm.\" Journal of the Royal Statistical Society: Series C (Applied Statistics) 28.1 (1979): 20-28.\n\nThe learning framework is not clear.\nYou report F1 which requires labels, yet possibly after the modeling, which is OK.\nHowever, you mention using accuracy in the modeling yet write “No labeled data is provided to the label models.”\nIf you need labels in the modeling, this is a semi-supervised framework so please specify how many labels are needed.\nIf you use a different method (e.g., disagreement minimization Amit, Idan, Eyal Firstenberg, and Yinnon Meshi. \"Framework for semi-supervised learning when no labeled data is given.\" U.S. Patent No. 11,468,358. 11 Oct. 2022.) then the term accuracy is confusing and method should be explained more.\nIf it is simply accuracy with respect to a proxy, it should be written when accuracy is used.\n\nHyperLM and Denoise share the top results.\nIt will be beneficial to try and explain the intuition of these results.\nIt is especially interesting since it seems that when one of the models is not the top performer, its performance tends to be very bad.\nIt hints that the datasets differ in some properties with a large impact on performance.\n\nLIMITATIONS:\nSee weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-17"}

comments: SUMMARY:
The paper theoretically studies a new label model that is neither probabilistic nor adversarial but performant in practice by proposing BBF. A series of statistical results are presented for BBF, including log-concavity of its posterior, consistency and rates of convergence. Empirically, BBF compares favorably over other label models. Overall, the paper offers valuable theoretical insights of a new category of performant label model, and proposed a practical method, it makes a solid contribution both theoretically and empirically.

STRENGTHS AND WEAKNESSES:
Strong points:
1. The paper offers novel theoretical insights into HyperLM’s strong performance in real-world by proposing Bayesian Balsubramani-Freund (BBF). This make a solid contribution to the programmatic weak supervision field as this is the first theoretical analysis of a new category of label model that is performant in practice.  
2. The paper shows BBF has some nice properties including log-concavity of posterior, form of exponential family solution, consistency, and rates of convergence.
3. Writing is clear and also strong empirical results are also shown.

Weak points:
1. I did not find any major weakness for this paper and one minor point is that, in experiments, since some methods involve randomness, it would also be nice to present the variance as in practical applications variance is also important.

QUESTIONS:
see W1

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a Bayesian label model for programmatic weak supervision. This approach (named BBF) combines the polytope‐center strategy of HyperLM with the minimax guarantees of Balsubramani–Freund in a fully unsupervised Bayesian approach. \n\nThe authors first show that BBF has a log-concave posterior distribution over labels. They also show that the learned solution of their approach is contained within an exponential family of weighted combinations of weak labeler outputs. The authors provide consistency guarantees on their approach.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n* The provided Bayesian approach is well motivated and the authors provide a theoretical analysis on (a) what the posterior form is and (b) the consistency of the resulting method.\n\nWeaknesses\n* The authors only show consistency when the priors of labeler accuracies concentrates around the empirical accuracy. While the authors claim this approach to be fully unsupervised (contrasting with prior minimax approaches that require labeled data to compute labeler accuracies), this consistency only arises when priors are properly specificed – which would indeed likely require labeled data without other domain knowledge.\n* Lack of discussion with [1, 2], which both are relevant constrained optimization or minimax approaches (albeit not Bayesian in nature) in PWS and both provide guarantees\n* No comparison with any of the minimax-based approaches above or in the related work – while their formulation does require labeled data, they can be run assuming a fixed accuracy as a hyperparameter, which is not too dissimilar from specifying the uniformative prior distributions in BBF; I believe such an experiment would greatly strengthen the support for using such a proposed Bayesian approach for PWS\n* As highlighted by the authors, scalability is a concern – the number of parameters of the approach scales with the input data, which is certainly undesirable\n* Empirically, the performance of the approach isn’t that strong – it’s almost always outperformed by HyperLM and does not noticeably improve upon the MV baseline\n\n[1] Arachie, et. al. Constrained labeling for weakly supervised learning.\n\n[2] Mazetto, et. al. Adversarial multi class learning under weak supervision with performance guarantees.\n\nQUESTIONS:\nQ1: Can you provide guidelines for choosing the hyperparameters for each of the priors in practice? \n\nQ2: Could the authors compare against some subset of the minimax based approaches, assuming a fixed constant accuracy for each labeler (e.g., say 0.75 for binary classification tasks)?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the BBF method (for Bayesian Balsubramani-Freund), an unsupervised Bayesian label model for weak supervision, where inputs $x_i$ with unknown labels $y_i$ are annotated by multiple noisy label functions (LFs). It weights the various LF to create a posterior on the $y_i$, by first drawing class priors, sampling latents $(y_i)$, and compute accuracy posterior for each LF on these latents before combining them in some adversarial fashion.\n\nThe method comes with theoretical guarantees and offers decent empirical performance.\nIf I understand correctly, Theorem 5.2 states that, if the class prior is a Dirac on  the right class proportion and if the accuracy posterior for each LF concentrates to the right one, then this method find the optimal blend of the LFs (based on the KL divergence to the true labeling). Theorem 5.3 seems to offer some speed for this convergence, it seems to corresponds to how much the estimated accuracies of the LFs implied by the model (which is captured by $Ag^{bbf}$) matches the true ones ($b^*$).\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**:\n- A method with theoretical guarantee and decent empirical performance.\n\n**Weaknesses**:\n- It was hard for me to follow the paper, understand the motivations of both the setting, and the various steps in the algorithm.\n- The empirical results are decent but not groundbreaking\n- I do not really know how \"nice\" the consistency and convergence results are, they seems to depend on quite restrictive assumptions. Basically, it is assumed that the priors having already done the heavy lifting. And the convergence result is more of a \"calibration result\" as per Bartlett et al. JASA 2006 (with this paper in mind, the consistency result can be viewed as a \"Fisher consistency\" result)\n\nQUESTIONS:\nHow would this paper potentially lead to a groundbreaking finding?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 2, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}