Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to combine a measure of the difficulty of an example in a supervised classification task (AUM) with the performance of annotators, calculated from their confusion matrix on a crowd-sourcing task. This combined estimator, called WAUM (weighted AUM), allows to take into account both the intrinsect difficulty of an example and the confidence in its label, when the latter comes from a crowd-sourcing campaign. The method is described in detail and compared to four other standard methods in the literature. The experiments are conducted on 2 synthetic datasets and 1 real dataset (CIFAR-10H). The WAUM method achieves slightly better classification results than the other methods on the synthetic data set and similar to the other methods on the real data set. 


strength_and_weaknesses: * The proposed method is motivated, very clearly described and new as far as I can tell.
* As far as the performance of the proposed method is concerned, the experiments do not really allow to highlight a major contribution. As indicated by the authors, the CIFAR-10H dataset may already be too clean to show the contribution of the proposed method.

It seems to me that the results should be confirmed on other real datasets, as indicated by the authors in their conclusion to give more strength to their proposal.


clarity,_quality,_novelty_and_reproducibility: The article is particularly well written, very clear, with references to the main articles in the literature on the subject. The appendices detail the state of the art papers used in the comparisons and give further analysis of the results.
summary_of_the_review: The proposed metric developed in the paper is well motivated and clearly described. The experimental evaluation is still limited and has not been able to show performances clearly superior to the state of the art. The results described in the article still seem too preliminary.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a way to combine crowdsourced multiple labels per task by using a weighting areas under the margin method (WAUM), which uses area under the margin method (AUM) used for detecting mislabeled data in the training dataset with a single hard label per task. It is a weighted version of AUM based on the trust factor. The trust factor can be derived as the inner product between the diagonal of the confusion matrix and the softmax vector. Experiments show how the proposed method is helpful in achieving higher test accuracy and lower ECE scores.
strength_and_weaknesses: 
Strengths

1. Taking into account task ambiguity is an important research topic.
2. The proposed method can achieve a higher test accuracy for the simulated synthetic datasets and achieve the same performance as the naive soft method in the CIFAR-10H dataset.

Questions (and Weaknesses)

1. One of the motivation of the paper is to remove intrinsically ambiguous tasks (which might even fool expert workers) because these tasks may be harmful for the learning step. My understanding is that intrinsically ambiguous tasks are helpful for learning, and I was not sure why these tasks may be harmful. For example, the most intrinsically ambiguous task may have p(y=+1|x) = p(y=-1) = 0.5 in binary classification (assuming y \in {+1, -1}), where p(y|x) is the underlying true class probability. If we collect many hard labels (n labels) for this same task, close to half of them will be positive hard labels and the rest of the half will be negative hard labels. When we naively regard this as n samples of (x,y) pairs (as well as other tasks), and train a classifier, our (probabilistic) classifier will try to learn hatp(y|x) = 0.5 for this task, and perhaps this will lead to improved ECE. It will do no harm for generalization performance, because test samples close to this x is intrinsically hard. On the other hand, I understand that noisy mislabelled samples (which is not related to the intrinsic difficulty of the task) may be harmful for learning. In Section 3 of the AUM paper by Pleiss et al. (2020), the following sentence "In general, we assume both easy and hard correctly-labeled samples in D_train improve model generalization, whereas mislabeled examples hurt generalization" is similar to my understanding. It would be helpful if the paper can provide some discussions about why removing intrinsically ambiguous tasks will be beneficial.
2. The paper explains that the margin in Eq. 2 is modified from the original version, following Yang & Koyejo (2020). I did not have time to fully go through Yang & Koyejo (2020), but I am predicting that this corresponds to using softmax_y - softmax_2 in Eq. 2 instead of softmax_y - max_{j\neq y} softmax_j which is used in Pleiss et al. (2020), assuming we ignoring the softmax-logit difference. I wasn't sure if the nice theoretical properties discussed in Yang & Koyejo (2020) directly transfer to AUM as a mislabel identification metric. Intuitively, the original margin definition seems to be easier to understand, e.g., if we consider the right example in Fig.2 of Pleiss et al. (2020), the modified margin will be near zero because Dog logit is the 2nd largest value for most of the epochs. It would be helpful if the paper can further discuss the benefits of the modified margin.
3. Is it also possible to compare the unmodified and modified margins in the experiments?
4. The paper explains that CIFAR-10 has very few (intrinsically) ambiguous tasks, because of the careful creation of the dataset. Since the goal of the proposed method is to remove ambiguous tasks, is this a good benchmark dataset to use to test the perfomance of the proposed method?

Minor comments, suggestions
1. Page 4, typo: "Appendix A" should be "Appendix B".
2. What is bold in Table 2, 3, and 4? Are they results of a t-test?
3. Is it possible to provide a practical suggestion for the number of hard labels per task?
clarity,_quality,_novelty_and_reproducibility: Clarity is high. The paper is well organized, but I some important contents are included in the supplementary sections, such as Section B. I did not understand the motivation of the paper (see the previous section on "Strength And Weaknesses").

Quality is high. The experiments are based on a single dataset that does not have many ambiguous tasks which does not seem to be a good fit (however, I also understand that there are not many datasets with multiple labels per tasks, which is also discussed in the final section of the paper.) The analysis and visualization with simulated datasets is helpful to gain a better understanding of the proposed method.

Novelty seems to be high. It is based on a very recent paper that was published in Dec. 2020 and I have not heard of a weighted version of the AUM with crowdsourced labels.

The optional reproducibility statement was not provided in the paper. There are some details of the experiments written throughout the paper. The code was provided as a supplementary file, but I did not read the code.
summary_of_the_review: The paper works on an interesting problem and provides a novel algorithm that aims to achieve better generalization performance. However I had a hard time understanding the main motivation of the paper along with some other questions/weaknesses. More discussions around those questions would be helpful for the reader of the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work considers the crowdsourcing problems where there exist ambiguous tasks. To address this problem, a new framework that used a similar technique in AUM method to identify ambiguous tasks has been proposed. To show the effectiveness of the proposed method, evaluations on both the synthetic datasets and the CIFAR-10H crowdsourced dataset have been implemented.
strength_and_weaknesses: This paper is well-organized, the structure is clean and clear, and some nice graphs are provided to help the readers to understand the proposed framework. Also, the authors provided links for almost all the involved datasets and baseline methods, which brings convenience to the readers and shows respect to the original authors at the same time. However, this work can be further improved in the following aspects:

- The baseline methods used are classic methods but not very new. It would be better to use some more recent methods as baseline methods. 
- Since the proposed algorithm used the idea of AUM method, can you compare these two methods in depth?
- In this paper, the comparison with some vote aggregation algorithms is provided, however, these methods are not new, can you compare the proposed method with more recent aggregation methods [1][2][3]
[1] https://jmlr.org/beta/papers/v21/19-359.html
[2] https://arxiv.org/pdf/1909.12325.pdf
[3]https://proceedings.neurips.cc/paper/2020/file/f86890095c957e9b949d11d15f0d0cd5-Paper.pdf
- The format of the tables used in this paper should meet the requirement of the conference.
- It would be better to give some theoretical insights or analyses why the proposed algorithm works well.

clarity,_quality,_novelty_and_reproducibility: Please see the detailed comments above.
summary_of_the_review: In my opinion, this paper is a borderline paper. The motivation is valid, and the organization is clear. However, the baseline methods used are not very new which is my concern.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel supervised learning model in the scenarios of crowdsourcing based on AUM, which is a confidence indicator for each task. The proposed method generalizes the existing AUM indicator to the crowdsourcing setting as WAUM (the Weighted Area Under the Margin), and the WAUM identifies harmful data and prunes the ambiguous tasks to improve the results of label aggregation. Experimental results show the effectiveness of the proposed method on several simulated and real-world datasets.
strength_and_weaknesses: This study is intuitive and technically sound. The proposed method is simple but interesting. In addition, the paper is presented clearly and is easy to follow and reproduce.

However, there are several weaknesses existing in this paper:

- First, the main idea of this paper is straightforward and not very impressive, and the novelty of the method is limited. The proposed WAUM seems like a simple combination of existing works, i.e. AUM and confusion matrix (which is mentioned in many EM-based label aggregation methods). I suggest the authors discuss the contribution of this paper in more detail and highlight the innovations.
- Second, the design of the experiments is somewhat weak. 1) The proposed WAUM-based method measures the task difficulty by incorporating feature information and trains a neural network classifier simultaneously, but the comparison methods such as MV, DS, and GLAD are all feature-blind methods, also the comparison methods are classical algorithms but not the latest research works, so the experimental results are less convincing. I suggest that more algorithms (better to use the feature of tasks) should be compared with the proposed methods, e.g., (Raykar et al, 2010), (Rodrigues et al, 2018), and so on.
- Furthermore, since the main idea of the proposed method is to identify the harmful tasks and prune them to reconstruct a new sub-dataset, could the sub-dataset be used on other existing methods to show the effectiveness of the pruned dataset?

Raykar VC, Yu S, Zhao LH, Valadez GH, Florin C, Bogoni L, Moy L (2010), Learning from crowds. The Journal of Machine Learning Research 11:1297–1322.
Filipe Rodrigues and Francisco C. Pereira. Deep learning from crowds. AAAI’18, 2018.

clarity,_quality,_novelty_and_reproducibility: This study is intuitive and technically sound. In addition, the paper is presented clearly and easy to follow and reproduce. The method is simple but interesting and seems effective, but the novelty is limited.
summary_of_the_review: Overall, the paper has its merits to show a new method introducing a crowdsourcing-specific confidence indicator to prune harmful tasks and improve the performance of the model trained on the pruned dataset. However, it also has  some  weaknesses. Thus, this paper cannot meet the high-quality requirements of ICLR.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: 
The reviewers agreed that the paper presents an interesting approach, in a paper that is well written. However, the main weaknesses of the paper -- fairly limited technical novelty (combining AUM with weighting techniques), lack of theoretical support or strong intuitive motivation for the approach, as well as limited experimental results (small improvements and limited baselines) remained despite the additional experiments provided in the rebuttal.
justification_for_why_not_higher_score: A higher score could have been given if the experimental results had shown larger differences with the baselines on the real dataset, or if the authors had articulated a rationale for the approach, for instance supported by a theoretical analysis under reasonable assumptions.
Decision: Reject