Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In crowd-computing tasks there are often two causes for wrong answers: 1) low reliability of worker (i.e. possibly a spammer or someone making a random guess), 2) confusion due to task (or question) difficulty (i.e. the question is simply harder than others or the potential answers could be confusing). The David-Skene model only captures the worker reliability portion of this. The authors have built a model for multi-choice crowdsourcing, that infers the top-two answers and the confusion probability. This would benefit downstream uses of these crowdsource tasks by providing the most plausible answer other than the ground truth and how plausible that second answer is. The authors show how using this top-two information can add neural network training versus using a single hard label.
strength_and_weaknesses: - The authors evaluate on both synthetic and real-world datasets. They also demonstrate how their model can improve neural network performance.
- As I was reading through the paper I wondered why only top-two, the authors provided good explanation in the appendix using analysis of public crowdsourcing datasets.

Weakness
- 5.3, It isn't clear to me that top-two is much better than full distribution. 
clarity,_quality,_novelty_and_reproducibility: The paper is well written. I have seen other approaches to using the confusion of raters versus hard labels. 
summary_of_the_review: Overall it was a well written paper. The evaluations on the synthetic and real-world datasets were well done. I am unsure of the results of the neural network training.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper considers the multiple-class crowdsourcing problem, where the confusing answers and confusion probability are considered. To address this problem, a new model was proposed to identify the top-two plausible answers for every task. Based on the proposed model, a new algorithm was also developed. To show the effectiveness of the proposed model, both theoretical guarantees and empirical evaluations are provided.
strength_and_weaknesses: This work is technically sound, to show the effectiveness of the proposed method, some theoretical guarantees are provided, also some experiments on both synthetic and real datasets were implemented. However, this work can be further improved in the following aspects:

- The baseline methods used are not state-of-art methods. The most recent crowdsourcing method is from 2018. It would be better to compare with more recent works like [1] and 
[1]https://proceedings.neurips.cc/paper/2020/file/f86890095c957e9b949d11d15f0d0cd5-Paper.pdf
[2] https://dl.acm.org/doi/10.5555/3454287.3454992

- All the figures are not vector graphs. It should be revised.

- It is not quite clear how the proposed model and algorithm can be applied to real applications.

- The organization of this work is not clear enough, especially the theoretical analyses part. It would be better to have some discussions after the proposition and theorem.

- I also have concerns about the novelty of the proposed model and algorithm. There already exists some work to divide the prediction in crowdsourcing into two stages, where the first stage is focusing on filtering the top answers, and the second stage is doing the prediction. Also, there exist some works to predict the reliability of the workers to do prediction. The proposed model gives me the impression that it is a combination of these methods. I am not sure how much insight this work can bring to the study of crowdsourcing or the ML community.

clarity,_quality,_novelty_and_reproducibility: Please see detailed comments above.
summary_of_the_review: In my opinion, this work is not ready to be published yet. The baseline methods used are not start-of-art methods. The novelty of this work is kind of limited. Also, the organization of this paper can be further improved.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a "top-two" model in which crowdsourcing tasks admit two plausible answers (the ground truth and a "confusing" answer). The authors argue and show that this model can be applied to many of the crowdsourcing tasks used for existing public datasets. In this framework, they describe a two-stage algorithm to infer the two types of answers and the workers' abilities. The first stage uses SVD on a transformation of a split of the workers x task answers matrix to estimate the difficulty of the task (defined in their eq 1). The second stage obtains the final estimates of the workers' abilities and task difficulty via maximum likelihood. The authors conduct a performance analysis of the algorithm and show that it recovers the two most plausible answers. Lastly, they conduct experiments on both synthetic and real-world datasets, including training a neural network with the top two answers rather than soft labels or the majority vote. 
strength_and_weaknesses: I found that the paper provided a thorough description of the existing methods (although I am not an expert on this), clearly characterized the proposed model, and contained an extensive description of the the theoretical and empirical properties of the algorithm. The experiments seemed to be reasonable. I only have the following minor comments: 

* 10 repetitions of the experiments to estimate the standard deviation is not enough to get accurate estimates (neither of the mean nor of its sampling variation). This should be increased. In addition, please report the 95% confidence intervals. Based on the reported errors, the intervals might reveal that the performances of the algorithms are not significantly different. The shaded regions in Figure 2a should be described. 
* What happens in the "easiest" case where the q=1? Does any of the other algorithms outperform Toptwo2 or Toptwo1?
* Can the model be easily extended to more than two plausible tasks? 
* How does performance vary in settings where crowdworkers are all experts with similar ability (p=1)?
* Details about IRB approval for the experiment that has been run should be provided. 

Typos: * "ssume" in footnote 2. "achieveed" in page 8. 
* "lower bound for the performance" -> upper bound for the performance and lower bound for the error probably?
* \eta is described in footnote 2 and then appears in algorithm 1. It might be worth reintroducing it.  
clarity,_quality,_novelty_and_reproducibility: The writing was clear and the experiments are reproducible. The proposed model and related methods seem to be novel. 
summary_of_the_review: The authors propose to model the difficulty of crowdsourcing tasks with two plausible answers and carefully details the properties of the proposed algorithm in this framework. I only have minor concerns and questions about the existing work that I hope the authors can address in their answer.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In order to simulate the effect of confusion in multi-choice crowdsourcing problems, this paper proposes a new multi-choice crowdsourcing task model and provides a two-stage inference algorithm to recover the first two answers and the confusion probability of each task. Finally, it shows the potential application of the proposed algorithm.
strength_and_weaknesses: Please refer to Summary Of The Review.
clarity,_quality,_novelty_and_reproducibility: Clarity, 
In this paper, the introduction of the model and algorithm is clear, but the logic is not strong in introducing the significance and background of the work in this paper. There are some problems as fellow：
1. For example, the abstract part lacks a clear description of the background and significance of the research.
2. I have a question, when the confusion probability is not 0.5, how to understand the confusion probability and how its value affects the result?

Quality, 
The theoretical and experimental parts of this paper are relatively complete, but there are some detailed problems as follows:
1. The organization of related work in the introduction is strange.
2. The latest works are barely presented in the relevant work section
3. In the conclusion section, the reference is very strange

Novelty 
This paper proposes a new multi-choice crowdsourcing task model and proposes a two-stage algorithm to recover the top two answers. The method recovers the top-two answers and the confusion probability of each task at the minimax optimal convergence rate. The method in this paper is quite innovative.

Reproducibility
According to the content of this paper, the method in this paper is reproducible
summary_of_the_review: Most previous multiple-choice crowdsourcing models cannot simulate this kind of confusion of plausibility. This paper provides an effective two-stage inference algorithm to reply to the first two plausible answers and confusion probability and can achieve the optimal convergence speed. This method is compared with other recent crowdsourcing algorithms and achieves superior results. This paper is relatively innovative.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Training on the top and a second label is certainly a useful idea, and its simple. Several more complex models have been developed for this task.  Experiments are done on predicting difficulty and on considering the training of top 2 labels versus other scenarios, both worthy applications of the idea.  The authors did a good job of addressing some of the reviewers concerns.

Comparisons misses some crowd-sourcing work published since 2020.   Comparisons also missed prior work and survey papers on related tasks of estimating difficulty, estimating subjectivity, estimating skill levels, which are inter-related and make the seemingly black-and-white task of estimating difficulty more nuanced.  However, some additional work such as Ma etal 2020 was compared against.

The experimental results are good demonstrating the system is working, but due to lack of other comparisons they may not be state of the art.  It is a good simple idea, however, so I expect with this comparison the paper- will be much stronger.

Author gives further results which would be better placed in the main paper since experiments in main paper can be improved.

justification_for_why_not_higher_score: The paper had substantial disagreement from reviewers, although the AC and the Reject reviewer had more confidence in their reviews.  The Accept reviewer was not confident.
justification_for_why_not_lower_score: N/A
Decision: Reject