Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a framework for weak supervision which works for both discrete and continuous labels. The proposed framework assumes the sources are coming from an exponential family distribution and generalizes the accuracy and correlation terms in previous approaches. They use a discriminative learning setting and use triplet methods in order to compute accuracies. They evaluated the performance for regression, ranking and hyperbolic and graph learning tasks. 

The paper's main contribution is putting an exponential family distribution assumption which allows to use any metric space. In their experiments, they choose distances based on the problem and label type, mostly weighted L2 distance except hyperbolic dataset. 





main_review: Strengths:

- The paper is clear and well written with proper notations.
- The authors emphasize an important aspect of weak supervision (allowing different types of labels and unifying.)

Weaknesses:

- I find the theoretical contributions on the more trivial side. Finding the roots of the triplets methods is just plain algebra and error term easily follows from bernoulli assumption.
- The dataset is constructed by the authors, so it is hard to evaluate its performance. The criteria metrics might not be fair since they are directly related to this paper's distance choices.
- Since the paper claims to be a universal framework, I would expect using different types of labels in the same task (like if we have binary labels for some data and probabilistic outputs for others), or coexisting more than one task type is doable. However, in their formulation it is not, since the distances would need to be mapped on the same space or at least a similar range.
- I believe Related Work section is significiant and should be in the main paper, not in the supplementary.


Questions:
- How do you compare your dataset with [1]?
- How do you compare method for varying  label density(low, middle, high) vs majority voting? Why snorkel works worse than majority voting in your experiments? 
- How would you adapt your method for the problems that use different label types at the same time? (rather than choosing one type of distance based on the problem)



[1] Zhang, Jieyu, et al. "WRENCH: A Comprehensive Benchmark for Weak Supervision." arXiv preprint arXiv:2109.11377 (2021).
summary_of_the_review: In general, the paper is written clearly, connects previous work in a good way, so it is valuable in that sense. I am willing to chance my vote towards accept if the authors could clarify my concerns and convince me their work has sufficient contributions to be published.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work studies a weakly supervised learning setup of aggregating multiple weak sources of labels into high-quality pseudo labels for learning. 

The proposed approach is based on building graphical models with the true labels as latent variables following existing techniques, but generalizes existing works in 1) supporting more types of task labels such as ranking, regression and in non-Euclidean spaces through approximating the graphical model terms using dot product between embeddings, and 2) algorithms for efficient learning under the embedding approximation and theoretical bounds on the estimation error using embedding-based approximation. 

Experiments focus on comparing weakly supervised learning performance of learning a graphical model versus using majority vote on the newly supported tasks and fully supervised learning with varying amount of labels. Results show that learning the graphical model achieved better performance than majority vote and can achieve parity reach a certain level of fully supervision.
main_review: Strength
* The proposed embedding-based approximate estimation approach is novel.
* Theoretical bounds are derived to monitor the closeness of this approximate estimation.
* Experiments are performed in multiple domains with different types of labels

Weakness
1. Writing can be substantially improved. 

 A lot of context is missing in the main paper, which makes the paper very hard to follow. Specifically

* Needs a bit more background introduction. "LF" (label function) in section 1 is defined in section 2. 
* How θa and θab are inferred from accuracies and correlations is not clear. Seems to be based on results from prior works. As this is the key innovation, would of great help to include a illustration of the construction of algorithms 1 and 3.
* The meaning of correlation set E in equation (1) is undefined.
* I find the advantage of a learned model over majority vote straightforward and well-studied, and can be compressed to include more background discussions.

 Experiments are lousily organized. Readers have to go to the appendix for a more comprehensive picture of the proposed approach.

* Experiments in Figure 1,2 and 3 are repetitive and can be summarized.
* A selection of experiments on estimation quality (figure 5, 6 and 7) would strengthen the main paper.

2. Discussions and experiments on the universality aspect is lacking. This work claims to be a universal approach for WS, which boils down to an embedding-based approximation of distance function d(,). It is a question whether such an approximation could be made close enough to a usable level for truly universal application. It would be nice to remark at Theorem 4.2 on how accurate the embedding approximations have to be for θ to be accurate using a practical example. Is being able to tractably estimate θ but not being loyal to d(,) a superior outcome than using a not-so-close estimate of θ?

-----------------------------------------

Update: My concerns on writing and universality have been sufficiently addressed with the latest revision. 
summary_of_the_review: My recommendation of this paper is based on the novelty with respect to previous works along this line of work, also considering the writing and experiment quality. I am not a domain expert and I didn't verify the correctness of equations and proofs.

My main concerns are 1) the proposed approach is not explained clearly (or am I missing the context?) and 2) the experiments included in the main paper seem shallow and repetitive.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors present a framework that intends to extend the label types considered traditionally in learning from weak labels. 
main_review: I commend the authors on their thoroughness and the aims of their approach - which are much needed in the field of weak supervision. I find some of the potential scenarios really interesting and appealing, particularly learning in hyperbolic space.

My concerns are threefold. The first has to do with novelty. There are many approaches in the past that also move past classification to consider ranking and regression. In general in loss factorizations of the form of 

Loss factorization, weakly supervised learning and label noise robustness
Giorgio Patrini, Frank Nielsen, Richard Nock, Marcello Carioni Proceedings of The 33rd International Conference on Machine Learning, PMLR 48:708-717, 2016.

that apply to most of those settings. In the following, the authors can also find the explicit generalization including classification, ranking or regression, with a similar notion of pseudolabels:

Cid-Sueiro J., García-García D., Santos-Rodríguez R. (2014) Consistency of Losses for Learning from Weak Labels. In: Calders T., Esposito F., Hüllermeier E., Meo R. (eds) Machine Learning and Knowledge Discovery in Databases. ECML PKDD 2014. Lecture Notes in Computer Science, vol 8724. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-662-44848-9_13

And even further generalizations from weak to superset learning, which arguably could be seen as including the former. 

Hüllermeier E., Cheng W. (2015) Superset Learning Based on Generalized Loss Minimization. In: Appice A., Rodrigues P., Santos Costa V., Gama J., Jorge A., Soares C. (eds) Machine Learning and Knowledge Discovery in Databases. ECML PKDD 2015. Lecture Notes in Computer Science, vol 9285. Springer, Cham. https://doi.org/10.1007/978-3-319-23525-7_16

It is not clear to me how these overlap with this work. Also, the structure of the paper does not help in understanding this, with the related work being in the supplementary material. Finally, the title seems a bit to general for what it is shown in the paper. 
summary_of_the_review: In its current form, the structure of the paper makes it difficult to assess the novelty. Also, there are previous approaches in the literature that seem to overlap with the aim of the paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: A novel general approach for weakly supervised learning.
+ A new model for incorporating weak sources based on an exponential family parameterization
+ Theoretical guarantees on the estimators for learning the parameterization
+ Empirical results show that the proposed approach can apply weak supervision to novel problem types
main_review: A new general framework for weak supervision is proposed. The main idea is to learn embeddings over the labels tractably into two embedding spaces (Boolean/Euclidean). Then, estimators for the parameters of the labeling model is learned using MLE. It is proved that if the embeddings are isometric the estimators are consistent and convergence rates are proven for these estimators. Experiments are performed on varied tasks and the results compare the proposed approach with Snorkel to show improved performance in weakly supervised learning for these tasks.

The paper is well written and has a very general framework for weak supervision. The experiments show the benefits of this framework in varied tasks. Also, there are new problems that the proposed framework is applied to (not previously possible by other weakly supervised approaches). The paper seems well written, has both good experiments and theoretical guarantees. One minor point is perhaps there is not a lot in discussion in terms of prior work for someone who is not familiar with the specific area. Is snorkel the only possible comparison baseline (particularly since a lot of the tasks are novel and cannot be done by snorkel).

summary_of_the_review: Overall, this seems like a strong paper with good theoretical justification and experimental results. The one possible weakness is perhaps lack of comparison baselines (though not sure if it is possible or not)
review_title: null

Meta-review: The paper propose a universal technique that enables weak supervision over any label type while still offering desirable properties, including practical flexibility, computational efficiency, and theoretical guarantees.

Over the course of the rebuttal, the authors have made a substantial overhaul on writing and experimentation. The universality claims are now better supported by bounds, and experiments cover comparison to snorkel, majority vote and supervised learning, on multiple applications. The authors are encouraged to move the related work section to the main body of the paper. The authors should also clarify to what extent the contributions they make pertain to Snorkel as opposed to weak supervision more generally. This may require revisiting both the introduction as well as perhaps the title.
Decision: Accept (Poster)