Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors proposed for the first time a theoretically motivated label model for programatic weak supervision (PWS), called a hyper label model based on GNN. The model can be used to aggregate any list of labeling functions (LFs) for any task and infer ground-truth labels in a single pass, without need any dataset-specific training of the label model. Empirical results are quite strong compared to SOTA.
strength_and_weaknesses: Strengths:

1. The paper takes on a novel perspective for PWS, that trains a hyper label model that can be used off the shelf universally for any task, any list of LFs and any training data. The training is entirely unsupervised, theoretically founded (Sec 4) and tractably approximated by a data generation process (Sec 5).
2. The model can either be used on its own (as a hyper label model), or further fine tuned if some ground truth labels are used. To me the idea is an interesting counterpart in PWS to the well studied concept of "unsupervised pretraining" in Vision and NLP. IMHO this is a very refreshing first step that can inspire a whole new branch of ideas and solutions in PWS. (Perhaps the authors can also comment their view on this, since my impression is that the paper wasn't positioned this way, e.g. no mention of related works in transfer learning?)

Weaknesses: I think the paper is in a very good shape and only have some minor questions/comments/clarification.

1. There seems to be a gap between the loss functions in the theoretical analysis (L2 loss in Eq. 4) and the one used in training GNN (CE in Eq. 5). Can the authors please comment on this? In particular, I wonder what happens if L2 loss is used for training (ablation?)
2. Following the last question, have the authors considered/tried, in Step 2 (Sec 5.1), generating one X and several valid y's, and add (X, \bar{y}) as a training point (using L2 loss as now \bar{y} would not be binary anymore), and why not do this?
3. The data generation process uses a uniform distribution throughout. Have the authors tried other distributions? In particular, in Step 2 (Sec 5.1), the method simply samples each cell in X i.i.d. uniformly from {-1, 1, 0} (if I wasn't mistaken?) Real world LFs have varying degree of accuracy and correlation (Snorkel's assumption), the data generation model seems oversimplified and can be very different from real data. For instance, would it make sense to pull a large set of real world X's (e.g. combining all X's from WRENCH datasets) and only simulate y?
4. Given the difference between the synthetic vs real world distribution of X, I'm quite surprised that fine tuning didn't really improve much over the pretrained model (Fig 2). Do the authors have some intuitions?
5. Why does one have to convert to one-vs-rest binary tasks in order to train multiclass hyper label model? Perhaps I miss the point, I didn't see any blocker from simply simulating multiclass y and using multiclass CE in Eq 5?
6. The proposed model is referred to as hyper label model before Sec 6 and as LELA after, it would be good to unify them?
clarity,_quality,_novelty_and_reproducibility: The paper proposes a brand novel model for PWS that is theoretically founded, and empirically validated in benchmark. The contribution is significant (See strengths above). The paper is well written and very pleasant to read.

Not sure if the authors will release their pretrained models, but it would definitely be very impactful to the community if they do.
summary_of_the_review: To the best of my knowledge, the paper is a significant contribution in PWS with a brand new idea of (pre)training a one-fits-all label model. Many readers should find the paper interesting.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a general method (hyper label model) to infer pseudo-labels in weak supervision that doesn't require learning parameters for each dataset separately. They characterize an optimal analytical model that is computationally intractable but gives true labels for any dataset under mild assumptions on labeling function -- better than random for each class. They give a novel hyper label model based on Graph neural networks to approximate this optimal analytical model. The choice of GNN is motivated by the setting -- it requires the model to be invariant to the permutation of LFs (or data points). They empirically evaluated the proposed solution on 14 real-world benchmark datasets for this problem and found that the proposed hyper label model works better than the baselines on most of the  datasets.

strength_and_weaknesses: Strengths:
1. I like the idea to have a single label model that can work across various datasets and the use of GNNs as label model so that the model is invariant to the permutations of LFs. 
2. The empirical evaluation on benchmark datasets is comprehensive and shows the benefit of their method.
3. They give a method to generate training data to train the hyper label model so that it can work across various datasets. 

Weaknesses/Questions:
1. The assumptions in Section 3 are reasonable. However I am not quite sure if the assumption per class is a weaker form or stronger form ? It seems stronger to me? Moreover won't majority vote alone give accurate inference under such assumption? Could you please share some analysis on this aspect and justify why would one need a hyper label model when this assumption is satisfied?

2. What happens to the noise rates of different labeling functions? How are they getting accounted in the inference procedure? 

3. A single solution for all datasets might be too much to ask for and it is very much possible that by optimizing for generality one looses the specific advantages while optimizing for a specific setting. Are there ways to optimize the process for the dataset at hand?
 
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and the contributions are novel. The authors have shared the code and experiments are on public benchmark datasets so should be reproducible.
summary_of_the_review: The paper brings in a novel and valuable perspective on learning label models for weak supervision. The evaluation is satisfactory. I have a few questions and concerns listed above, except those I liked reading the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a new approach that uses programmatic weak supervision. First, labeling functions are used to generate weak labels for each example. Then, a hyper label model is used to infer the ground-truth labels from the weak labels. The hyper label model is Graph Neural Network (GNN) that is trained on synthetic data, and it can be used on any new dataset without retraining. The paper also introduces a weaker assumption about the labeling functions:  the majority of labeling functions have to be better than random. The proposed approach is evaluated on 14 classification datasets and shows better results on average.
strength_and_weaknesses: **Strengths**
-The general problem tackled in this paper is interesting. Reducing annotation cost is an interesting research direction because it is cheaper to create new datasets or extend existing datasets.
- I like the proposed assumption “the majority of labeling functions are better than random” because it seems to be more realistic as claimed in the paper.
- I like the assumptions about the model architecture: (1) ability to accept arbitrary input size; (2) invariance to permutation of labeling functions. (3) equivariance to permutation of data points
The paper presented well the motivations behind these assumptions and how the proposed GNN model respects these assumptions. 
- I like the idea of training the hyper label model on synthetic data to not increase the annotation cost. It is also possible to improve the hyper label model performances by generating more data. I also like the fact that the hyper label model is trained only one time and can be used on multiple datasets without retraining.  
- The hyper label model can work with missing weak labels. 
- The proposed approach shows better performances on average on multiple datasets (Table 2). The evaluation is performed in both unsupervised and semi-supervised settings. The proposed method seems faster than most of the baselines. An ablation study shows the impact of each key contributions  


**Weaknesses**
- I think some parts of Section 4 are a bit misleading because p(y|X) is approximated by a uniform distribution, so theorem 1 is an optimal solution of an approximate problem. There is no strong evidence the uniform distribution is the best approximation for p(y|X), and should maybe be presented as a good approximation instead of the best. I agree that the uniform distribution has optimalities in both the worst case and the average case, but it does not mean it is always optimal. *[post rebuttal] The paper has been updated to take into account this comment and remove the misleading part.*
- I am not sure I understand the motivation of using a GNN instead of a simpler set-based model like DeepSet [A]. I agree that a GNN respects the proposed assumptions but I am not sure to understand why modeling the edges is important. I think adding an experiment with a DeepSet could help to justify this design choice. *[post rebuttal] The rebuttal gives a motivation to this design choice.*
- The paper claims the data generation process is efficient but the probability of generating a valid pair is about 0.2, which does not seem very efficient. Maybe it could be interesting to propose a better generation process or to not claim the data generation process is efficient. *[post rebuttal] The paper has been updated to remove the claim.*
- The results about the training of the hyper label model are in the supplementary and not in the main paper. I think the main paper should show at least some of them because it seems to be a key contribution of the paper. I would suggest reducing Section 3 and 4, by moving some parts to the supplementary, to have space to present some results about the training of the hyper label model.
- The related section does not mention some other approaches to reduce the annotation cost like self-supervised learning methods. I think the related work section should discuss some of the other methods.  *[post rebuttal] The related work section has been updated to cover other approaches.*

[A] Zaheer M., Kottur S., Ravanbakhsh S., Poczos B., Salakhutdinov R., Smola A. Deep Sets. In NeurIPS, 2017.

clarity,_quality,_novelty_and_reproducibility: The paper is easy to read and understand.The general problem tackled in this paper (reducing annotation cost) is interesting. The proposed approach seems novel and shows better results than baselines. The code is provided as supplementary material so it should not be too difficult to reproduce the results.
summary_of_the_review: Overall, I think the paper is good and interesting but I have some concerns about one of the contributions and some design choices. Please check the Strength And Weaknesses section for more information.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a "hyper label model" for weak supervision that can infer ground-truth labels for any weak supervision dataset without retraining. The authors evaluate their hyper label model in accuracy and efficiency on 14 benchmark datasets. The hyper label model is more efficient and on average slightly more accurate.
strength_and_weaknesses: + Clear idea, clear evaluation.
+ The method seems to be empirically well-suppported in terms of accuracy of the final model
+ Theoretical support for existence of an optimal label model and conditions for finding one

- Theoretical analysis is unclear in the case of class-imbalanced data (which is the common case in practice)
- Experiments on efficiency a little incomplete (see Q's below)
clarity,_quality,_novelty_and_reproducibility: Clarity:
The paper is well-written and clear to follow.

Quality:
I have a few questions about the method and results.

1. How do the theoretical results adjust for class-imbalanced data? This is the common case in practice, so understanding the role of class balance in the theoretical results is important.
2. Is the "weaker form" of the better-than-random assumption actually weaker for imbalanced labeling functions?
3. Questions about the efficiency results: Can you report training time of all the methods? Can you report runtime on CPU as well? Some results, such as METAL's low running time do not match practical experience -- since METAL requires training a new label model for each dataset.

Novelty:
This problem statement is novel to the best of my knowledge. Parts of the approach have analogues in meta-learning/pretraining.

Reproducibility:
Paper seems reproducible from the writing.
summary_of_the_review: Overall I think this paper is strong, with a clear contribution and results. I have a few questions about the results and experiments, but they can be addressed fairly easily. I plan to give this paper a score of "accept" - contingent on having my questions answered in rebuttal.

Update after the rebuttal: I am decreasing my score to a 5 because of concerns about the characterization of previous better-than-random assumptions.

Second update after the rebuttal: Raising my score to a 6 - I think the paper is would be a positive contribution to ICLR.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The papers discusses an approach to programmatic weak supervision using graph neural nets. The paper justifies the method theoretically and the empirical results are supportive. The paper is reasonably well written. In general the reviewers were in agreement that the paper makes a useful and justifiable contribution to the literature. 

justification_for_why_not_higher_score: During the discussion there remained some doubts about the depth of the theoretical contributions and their overall validity in practice. 
justification_for_why_not_lower_score: The paper has above threshold scores from all reviewers.
Decision: Accept: poster