Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the effect of label error on the model’s disparity metrics (e.g., calibration, FPR, FNR) on both the training and test set. Empirically, the authors have found that label errors have a larger influence on minority groups than on majority groups. To mitigate the impact of label errors, The authors have proposed a method to estimate the influence of changing a single training input’s label on a model’s group disparity metric.
strength_and_weaknesses: Strength:
+ The research problems are important and may have many practical applications. The real-world machine learning dataset can easily contain label errors. Improving the robustness of learning models trained on noisy data is important. Existing methods mainly focus on downstream accuracy, but group-based disparity metrics have been ignored which are also important for designing a robust algorithm. 
+ The proposed method is well-motivated. Estimating the influence of a single training input on a model’s group disparity metric is important for confident example selection and dataset purification.



Weakness:
+ The technical insight may not be enough. The authors have empirically illustrated that minority groups are more sensitive to label errors than majority groups. To make the conclusion more meaningful and practical, I think it would be great to add some theoretical analysis on the influence of label errors with different minority and majority group sizes.

+ The proposed method for estimating the ‘influence’ of perturbing a training point’s label on a disparity metric may not practical. The computational cost of the method seems very expensive and needs a lot of retraining processes to detect the effect of all training inputs, which can be hard to apply to a dataset with high-dimensional features. In addition, to demonstrate the performance of the proposed methods, some SOTA methods should be compared (e.g., JoCoR, CVPR’20; DivideMix, CVPR’20; MEIDTM, CVPR’22). The benchmark datasets such as CIFAR10 and CIFAR100 with different types of synthetic noise should also be compared.


+ The experiment setting is not clear to me. For example, it is not clear how the minority group and majority group in Fig. 1 and Fig.2 are obtained. I think the authors may also need to discuss that how to apply the convolutional network Resnet-18 to tabular and text datasets. 

clarity,_quality,_novelty_and_reproducibility: This paper generally is well-written and easy to follow, but most discussions are based on experimental results obtained from a few datasets. The experimental settings and comparison should be more detailed and comprehensive.
summary_of_the_review: For me, the motivation and research problems of this paper are strong and important. My major concerns are that the technical contribution may not that strong, and the proposed method may not practical and hard to be applied to real-world machine learning datasets.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers an important problem of label noise in the training data. Specifically, it studies the effect of label error on a model's group-based disparity metrics, with more focus on smaller groups in the data. Then, the authors of the paper take a step further by considering a method based on influence function to identify training samples that significantly impact the test disparity metric of interest. The authors of the paper conduct a series of experiments to answer these questions and offer valuable insight into this important problem. 
strength_and_weaknesses: Strength:
- This paper is very well organized and written in general. Most of the claims are supported by ample experimental analysis. 
- The problem of concern has a unique fairness perspective, which has great practical significance. 

Weaknesses: 
- Learning with noisy labels is a widely studied topic, especially in the context of neural networks. As an empirical paper, it would be nice if the authors of the paper can conduct some additional analysis to show the effect of some of the recently proposed noise-robust algorithms on such group-based disparity metrics. 
- Similar to the above point, it would be nice if the authors of the paper benchmarked the proposed "IF-Calib-Label" against some other recently proposed noise-robust algorithms that can potentially identify label errors.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written in general, and ample details are provided to help reproduce the results shown in the paper. 
summary_of_the_review: Despite the interesting perspective and a well series of well-conducted experiments, I feel like the authors of the paper can provide some additional experimental insight to the paper, as suggested above. As such, I recommend a weak accept for now.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: 1. This paper studies the effect of label error on a model’s group-based disparity metrics. Differences in terms of calibration error can be observed for the minority (smallest) group and the majority (largest) group.
2. They also propose an approach (influence function) to estimate how changing a single training input’s label affects a model’s group disparity metric on a test set. Based on the influence function, they can identify training points that have a high effect on a model’s test disparity metric.

strength_and_weaknesses: Strength

1. They try to answer two significant questions regarding label errors and fairness:
    What is the sensitivity of a model’s disparity metric to label errors in training and test data? Does the effect of label error vary based on group size?
    How can a practitioner identify training points whose labels have the most influence on a model’s group disparity metric?
2. Various disparity metrics, models, and datasets are considered
    model disparity metrics: expected calibration error (ECE), the Brier Score (BS), False Positive Rate (FPR), False Negative Rate (FNR), and Error Rate (ER).
    datasets across different modalities: 4 tabular, and a text dataset.
    models: a logistic regression model, a Gradient-boosted Tree (GBT), and ResNet-18.
3. The closed form influence function is potentially useful for regularizing or mitigating the label error while training.

Weaknesses
1. Although multiple datasets are used, how to make sure that the conclusions and analysis drawn is not dataset-specific is still an open question.
2. The label flipping is indeed useful for empirical results but not interesting as a technical contribution.

clarity,_quality,_novelty_and_reproducibility: 1. Their theoretical analysis seems to be more interesting. But is the chain rule analysis covered by other work? This is one part which I am not sure about regarding novelty.
2. I feel like the key approach for the influence function regarding label errors is not brand new, but modified for the new target applications (regarding fairness).
3. Although label noise and label errors are not exactly the same,  but I would appreciate it if the authors can briefly explain the difference (in terms of the key approach) between this work and the paper [Fair Classification with Group-Dependent Label Noise]. 

summary_of_the_review: 1. The overall quality of the paper is good, and the key ideas are clear enough to make reviewers easy to follow.
2. If as they claimed, label errors’ influence on group disparity is not yet covered by literature, this is a strong submission.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper investigates the effect of label error on the model’s disparity metrics (e.g., calibration, FPR, FNR) on both the training and test set. The authors found that empirically, label errors have a larger influence on minority groups than on majority groups. The authors proposed a method to estimate the influence of changing a single training input’s label on a model’s group disparity metric. Reviewers agree that the studied problem is important and may have many practical implications and that the proposed method is well-motivated. At the same time, reviewers also have several sensible concerns; e.g., the technical contribution may not be strong enough, and the proposed method may not practical to deal with real-world machine learning datasets. However, overall, I believe the value overweights the issues in the paper.
justification_for_why_not_higher_score: Reviewers agree that the studied problem is important and may have many practical implications and that the proposed method is well-motivated. 
justification_for_why_not_lower_score: Reviewers also have several sensible concerns; e.g., the technical contribution may not be strong enough, and the proposed method may not practical to deal with real-world machine learning datasets.
Decision: Accept: poster