Venue: ICLR 2023 TinyPapers

Reviews:

final_score: rating: Needs Clarification (NC): a submission which does not meet the reviewing criteria and needs clarification for its described problem or solution
review_content: summary_of_contributions: The paper proposes to use statistical tools to identify disagreements introduced during auding with human reviews.
strengths_and_weaknesses: Strengths:
1. The paper proposes multiple scenarios and factors that could potentially impact the reviewer's decision choice and might introduce a source of bias in evaluation, for ex: question criteria, sampling bias.

Weaknesses:
1. While the paper presents the usefulness of the tools to determine the source of agreement, there are no results that grounds their hypotheses.
suggested_changes: It is unclear what has been studied and what is to be studied/proposed. It is also unclear what the source of the dataset for the analysis is.
review_title: null

final_score: rating: Clear, Correct, and Reproducible (CCR): a submission which meets the reviewing criteria
review_content: summary_of_contributions: This paper presents several statistical analysis methods to help researchers and auditors identify problematic human reviews/annotators. These methods include prospective analysis (chi-square test, t-test, and ANOVA test) as well as retrospective analysis (experiment design, difference-in-difference)
strengths_and_weaknesses: ## Strengths and weaknesses

#### Clarity

1. Overall the paper is clear and easy to read.
2. What is "inter-rater reliability"?

#### Correctness

1. The proposed statistical methods look sound and useful, but there is no experiment to evaluate the proposed methods.

#### Reproducibility

1. This paper proposes several statistical methods to evaluate human reviews. It discusses these methods in detail in sections 2 and 3. The appendix also includes the formula for the proposed methods.

#### Follows basic requirements

1. This paper follows the basic formatting requirements.

suggested_changes: ## Suggested changes

1. It would be great if the paper's main text can include a formula to compute Fleiss Kappa score, as it might be unfamiliar to many readers.
2. Table 1 needs to explain what "sample groups" and "rubric questions" are.
review_title: null

Meta-review: metareview: 1. Clarity: author has communicated clearly with relevant literature
2. Correctness: justification missed in the conclusion
3. Reproducibility: code & data not provided
4. Pros:
     * novel application to reduce human bias & judgement
5. Cons:
    * spelling mistakes
    * gist or explanation of few unknown terms:  inter-rater reliability
    * fails to provide explanation of how well their novel application outperforms in the conclusion
confidence: 3: The area chair is somewhat confident
Decision: Invite to archive