Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the author proposed two different regularised versions of CCEM to enhance the identifiability of the method by introducing geometric conditions. The comprehensive experiment is conducted to demonstrate the effectiveness of the proposed approach where GeoCrowdNet(F) performs better when more annotated items available, where GeoCrowdNet(W) shows advantages when class specialists are available.
strength_and_weaknesses: Strength
Overall, the paper is logically organised. The research problem is clear and formulated. Theoretically seems sound. Comprehensive experiments were conducted to show the effeteness of the proposed approaches.

Weaknesses
In session 1, too much spend on the background of the course source while the importance of identification is not mentioned.
The author claims that the existing identifiability result on CCEM was derived under restricted conditions, while the proposed method needs to hold under five different assumptions, and some of the assumptions are difficult to hold under real-world conditions.
The performance improvement seems marginal by comparing it to the existing methods such as Max-MIG.
The discussion on the experiment's result is not sufficient. Especially the performance difference between Max-MIG needs to be discussed.
clarity,_quality,_novelty_and_reproducibility: Quality
=====================
The paper is logically organised; however, it is not easy to follow. Comprehensive experiments were conducted.

Novelty
========================
Marginal the improvement by comparing to the existing method seems marginal.

Reproducibility
========================
The reproducibility is high, and the necessary setting is given. However, it will be greatly appreciated if the source code can be published.
summary_of_the_review: The weakness outweighed the strengths in the current version.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper designed new regularization terms of coupled cross entropy minimization criterion in crowdsourcing tasks. It relaxed the assumption of conditional independence among the annotators.
strength_and_weaknesses: Strength: This paper relaxed the assumption conditional independence among the annotators in crowdsourcing tasks with coupled cross entropy minimization criterion. The newly designed regularization terms based on the relaxed assumption consider 1) when the annotation data is large (voting matrix is not sparse), no expert/dominate is also ok, can set the classifier-related regularization terms; 2) when there are more annotators, can set the voting matrix as regularzation term. This paper transfers an intuitive impression "the more annotators/annotations and the stronger classifiers, the better the performance of the crowdsourcing task" in a mathematical way.

Weakness/problems: 
- The author should provide a more detailed analysis of how and when to select the regularization term, with varying N and M. Table 1 has only 6 combinations, it's really hard to make an accurate conclusion of how to choose the regularization term.  
 - When N and M are large, are the performances of GeoCrowdNet(M) and GeoCrowdNet(F) come close to majority voting? Does it still works when the voting marix is sparse?
 - In Table 1, case 2, N = 5000, why TraceReg perform much worse than GeoCrowdNet(λ = 0)? TraceReg has trace constraints compared with GeoCrowdNet(λ = 0).

clarity,_quality,_novelty_and_reproducibility: Clarity: this paper is clearly written.

Quality: this paper provide strong theoretical proofs of their proposed method.

Novelty: this paper is designed on top of (Rodrigues & Pereira, 2018) and provides better regularization terms. This is novel.

Reproducibility: the author has provided their code for re-implementing their experiments.
summary_of_the_review: This paper relaxed the assumptions of current coupled cross entropy minimization criterion based crowdsourcing tasks. More experiments needed to be provided to illustrate how to choose the regularization terms.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper provides theoretical analysis to guarantee the performance of the CCEM criterion for the first time. Furthermore, to improve the identifiability of the classifiers and the workers, the authors also proposed two variants of the CCEM by introducing two different regularization terms. Experimental results on several synthetic and real-world datasets show corroborate the theoretical claims and the effectiveness of the proposed methods.
strength_and_weaknesses: The presentation of this paper is easy to follow and the study is technically sound. The design and technique details of the software structure are disclosed and reproducible. In addition, the theoretical analysis presented in this paper is conducive to research on the CCEM criterion.

There are several issues:
- The experimental results in Table 1 are somewhat weak, and the authors should discuss the results in more detail, e.g. the performance of Max-MIG is competitive in most cases, and in case 1, GeoCrowdNet(W) does not show the advantages well compared with GeoCrowdNet(F) though the N is small and M is relatively large.
- Whether there is an appropriate strategy or method to make decisions about which variant (GeoCrowdNet(W) or GeoCrowdNet(F)) should be used on a specific dataset?


clarity,_quality,_novelty_and_reproducibility: The presentation of this paper is easy to follow and the study is technically sound. The design and technique details of the software structure are disclosed and reproducible.
summary_of_the_review: Overall, this is a valuable piece of work. The proposed methods and the theoretical analysis are conducive and attractive for researchers in the areas of CCEM in crowdsourcing. However, there are also some weaknesses in the paper. The quality of the paper could be further improved.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper deals with learning with crowdsourcing.
It introduces a new metric (CCEM) with two different types of regularization (depending on the number of annotators) with competitive performance on test accuracy in simulated datasets and good performance in real datasets.
It builds on Rodrigues et. al (2018) to improve results with new regularization terms.

justification_for_why_not_higher_score: Assumptions are hard to verify and understand in practice
Over the two proposed E2E strategies, only one seem to perform best in practice (the F-regularized).
At the end of the paper, we do not know if the main goal was presenting the CCEM (highlited in eq 6a) theory or the importance of a new regularization term in crowdsourced losses (or both, but most of the paper seem to be on the second part).
Background methods could have been presented more deeply in appendix and less in the main paper to leave room for more on the 1st part.
Annotators are simulated with a confusion matrix, and the model estimates such a matrix. Even though it is classical to do this to obtain a comparison of estimated confusion matrices, this puts aside methods not based on such a tool.
Writting could be polished.

justification_for_why_not_lower_score: The work Builds on Rodrigues et. al (2018) to improve results with new regularization terms.
Theoretical garantees over the model proposed are of interest, and based on confusion matrices (easy to interpret parameters) and it leads to competitive methods.
Code is provided
Decision: Accept: poster