Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper focuses on weakly-supervised classification tasks where (i) multiple weak sources (i.e., labeling rules) are used in a label model (e.g., weighted majority voting) to generate soft labels for unlabeled data; (ii) the soft labels are used to train an end model (e.g., logistic regression, neural classifier) with a smooth cross-entropy loss. 

The paper proposes a method to evaluate the influence of each weak source on the end model's performance by considering the choice of the label model, a critical component in this weak supervision pipeline. The main idea behind the proposed method is to decompose the training loss of the end model into multiple components corresponding to the individual weak sources. Then, two techniques are proposed to compute a source-aware influence function, namely reweighting and weight-moving. In cases of label models involving an exponential function in their generation process, the paper applies the proposed source-aware influence function techniques by training an approximate label model based on the identify function.

The paper evaluates and compares the proposed method with various alternatives for multiple use cases and datasets across tabular, text, and image modalities. 
strengths_and_weaknesses: Strengths:
* Overall, the paper is clearly written and provides clear and substantiated arguments.
* The paper addresses an interesting and challenging problem. Evaluating the influence of weak sources on the end model's performance is an increasingly important research direction given the increasing adoption of programmatic labeling in both research and industrial settings. 
* The proposed method shows promising experimental results on an extensive experimental evaluation of multiple techniques on several scenarios and datasets. It is demonstrated that including the label model into the computation of IF helps across multiple use cases. 


Weaknesses: 
* It is hard to understand why (in theory) the proposed method should be effective across label model choices.
  *  The proposed method unifies three types of label models into a single equation (Eq. (6) in Section 3.1) without having discussed the inner workings of each label model (e.g., weighted aggregation in Snorkel). Thus, it is hard for a reader not familiar with these label models to understand Eq. (6) and possibly the rest of the method. 
  * For cases where $\sigma(\cdot)$ is not the identity function, it is not clear whether the proposed method explains the real influence of each component. The approximated label model (that uses the identity function) is simpler and is not guaranteed to have the same behavior as the original label model. (Indeed, the two label models lead to different rankings as shown in Figure 1. Thus, it is not clear whether the estimated influence scores explain the real influence of each component. In addition to comparing the rankings, it would also help to compare the predicted labels of the original and approximated label models to give an idea of how well the latter approximates the former. 
  * It is not clear why the proposed method (resorting into approximations of label models) would in theory be more effective than a simpler method that first computes (source-agnostic) influence scores for each instance and then aggregates instance-level scores into source-level scores. 
* The problem addressed in this paper is a simplified (and possibly unrealistic?) case compared to problems addressed in practice in weak supervision
  * According to the problem definition no labeled data are assumed for training, however, labeled data are considered in a validation set. In practice, (at least few) labeled data are considered during training (either in the label model or in the end model) and have been shown to improve the end model's performance. For, example, clean labeled data could be combined with weakly labeled data with weights. It is not clear whether the proposed method can be applied in this setting. 
questions: * What is the size of the validation set used in this paper? Does it make sense to use at least a few labeled data per class while training either the label model or the end model? 

limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces a method called source-aware Influence Function (IF) to study the “influence” of individual data, source and class tuples on the performance of different label functions in the programmatic weak supervision paradigm. This differs from previous methods that calculate ordinary IF which can only identify influential training data and not the labeling function or the training data source responsible for mislabeling. The paper introduces a framework for estimating the influence of each training data on the test loss and prediction. The authors introduce two methods to calculate this source-aware IF. They support their method with theorems and proofs. They also show a variety of use-cases on different datasets as to how source-aware IF can improve the performance and understanding of PWS pipelines. 
strengths_and_weaknesses: Strengths:
1. The paper tackles a very interesting and important aspect of understanding programmatic weak supervision. The problem is well motivated. 

2. The paper includes theoretical proofs of the claims and methods used for developing source-aware IF.

3. The generalization of the generation process of probabilistic labels using eq 6 is nice.

4. The paper includes experiments on a wide range of benchmark datasets and classification tasks.

5. The contributions of the paper are clear, original and significant. 

Weaknesses:
1. The motivation for certain aspects of the method and experiments are unclear. For e.g:
         a. The intuition behind why there are two methods for calculating source-aware IF is not clear - why reweighting vs weight-moving. There isn’t a clear study of how one is better or worse than the other and in what cases and why. 
         b. The significance of RelatIF in the context of the contributions of this paper is not clear. 

2. In the experiments, the training data most responsible for mislabeling is removed and the resulting test loss is reported. However, it is unconvincing if this is the best strategy. What happens if this process is repeated multiple times after removing 1 data point each time or the top n most responsible data points are removed?

3. The test loss is shown to reduce after removing the most negatively influencing tuples, however, there is no mention of how this may translate to classification accuracy or other metrics. 

questions: 1. The two methods of calculating source-aware IF correspond to two different ways of perturbing the training loss. How do these 2 differ (maybe through examples) in identifying the most responsible (i,j,c) tuples? From tables 3 & 4, on average, RW seems to be doing better than WM. It would be nice to have a discussion on why that may be. 

2. What is the significance of RelatIF here? How is it better/different from ordinary IF and source-aware IF? R-RW seems to be outperforming RW (table 3) for Majority Vote but not for others. 

3. It would also be interesting to look at some examples from other datasets (tabular/textual) and see which LFs are responsible. In many real world non-image applications, the LFs may not be created similar to the experimental setup in the paper for the DN datasets. 

4. Having identified which LFs have mislabelings, it would be nice to see what can be done about it. For eg, how much does correcting these mislabelings affect the end performance? Is there a minimum number of corrections to be made to improve the classification results? I.e, how many mislabelings need to be identified to positively make an impact on the results?

5. It would also be interesting to look at the difference in classification accuracy/F1 score on an unseen test set with the usual PWS setup and after removing the tuples most responsible for mislabeling (based on a validation set). 

6. In table 2, it might be more helpful to have the ‘Misclassified Test Data’ as the first column since the explanation goes from misclassification to the responsible components. 

7. There is a small grammatical error in the table 3 caption - “The larger the AP is, the better the method identify mislabeling of LFs.” 

limitations: The authors have mentioned some limitations of their work. The proposed framework may be difficult to implement in the case of more complicated label models. It has not been defined for one-stage PWS pipelines which are becoming more popular recently. The paper does not include experiments on using complicated deep learning architectures as the end model. Additionally, this framework is also limited in its ability to decide/recommend how many samples (responsible for mislabeling) need to be discarded or down weighted for optimal classification performance.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: ## Summary
In this work, the authors propose a general framework for quantifying the influence of individual PWS components on the end model. The proposed framework can be used to look into the influence of individual PWS components' effects on the end model. A source-aware influence function is proposed, which leverages the knowledge of the probabilistic label generation process and uses that knowledge to decompose the training loss into multiple terms and eventually individual influence scores. These influence scores are used to quantify the effect of PWS components (source vote, supervision source, and training data). 

strengths_and_weaknesses: ## Strengths
1. The effective use of the source-aware influence function along with the knowledge of the probabilistic label generation process provides a fine-grained analysis tool for the behavior of the end-model.
2. The influence of individual PWS components can be analyzed with the proposed framework.
3. The proposed framework has the capability to work with diverse data domains (tabular, image, textual). Secondly, an ample number of datasets are used in the study. 
4. The most responsible LFs as well as their capacity to mislabel can be identified, which is different from identifying the influential training data. 
5. The proposed model steers the end-model towards generalization, which is always desirable.
6. The paper is well written.


## Weaknesses
1. The proposed framework hasn’t been shown to work with one-stage methods.
2. Experimental results are reported for two-layer neural networks. It is still to be seen how the proposed method performs with intricate and deep neural architectures.

questions: n/a
limitations: n/a
review_title: null

Meta-review: metareview: This paper proposes source-aware Influence Function (IF) to study the “influence” of individual data, source, and class tuples on the performance of different label functions in the programmatic weak supervision paradigm. The proposed method has the capability to work with diverse data domains (tabular, image, textual). An ample number of datasets are used in the experiments.

The reviewers agree that the proposed method is interesting and sound, the experiments are thorough, and the results provide valuable insights for future work. Reviewers' raised concerns and questions are properly addressed by the author's response. 
confidence: Certain
Decision: Accept