Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: First of all I want to thank the author(s) to notice and study the quality of reviews in ICLR.

Quality:

The authors put a lot of effort in collecting and cleaning the data, which is not easy. The authors did thorough statistical analysis over the data and considered many possible pitfalls. I still have some questions but I think it's of good value for the community to see and discuss the paper in the conference.

Clarity:

The paper is well written and easy to follow. 

Originality:

Good original thinking. 

Significance:

The analysis and conclusion is of good value for the ICLR community. Great significance.

I gave the rating of marginally below acceptance threshold. Why not higher? I think there are some fundamental questions that need more clarification from the authors (4,5, 6,7,10 and 12 in the detailed comments). But to be clear I still think there's great value for it to be accepted given the topic, and if the authors can answer the questions in the comments.

Detailed comments:

1) Page 1, intro: What do you mean by "censorship" here?
2) Page 1 last paragraph. I think the reviewers have no knowledge of author's gender, given the double blind review process? I know you might be able to dig it through by searching arxiv ,but doubt anyone would do that to see the gender.
3) Page 2, data set. It's better to explain the size of data clearly, is it 2560 papers or 5569 papers?  
4) Page 3. the simulation. Given the whole simulation set up, I think the key factor is the variation of reviewer score and AC's decision noise. Why don't we just look at those directly? Simulation gives a more intuitive number (re-acceptance rate) but with more noise injected in.
5) Page 3, I'm not a statistician, but short research on Hosmer-Lemeshow test suggests that it's not a good way to exam the quality of LR model. Could you also show the x-validation precision/recall of the prediction? AC might take other factors into consideration, such as the content of review comments, reviewer's confidence and background, or the order of papers (esp the ones on the border of a/r).
6) Page 3, If the same 2020 logistic regress model was used, then the result only shows that the variance of review score increased over time. Have you considered the change in number of papers and number of reviewers over the years?
7) Page 4, Sec 3.3. Number of citation increases exponentially over time. simply dividing by time will punish papers published sooner.
8) Page 5, figure 4, could you add the papers that never before seen online too?
9) Page 6, paragraph 4, is it 0.15 out of 10? seems a trivial amount.
10) Page 7, reputation, Is this a correlation or causality? It could be that papers from authors with better reputation did have an edge in quality. Remember ACs have access to the review notes and the paper themselves too.
11) Page 7, second last paragraph.  We can't mix woman % in the US with ICLR which is a global venue. Global % could be lower.
12) Page 8 Conclusion. This is a weak point, Conclusion is not supported by the content of the paper, but more like a speculation.

review_title: Interesting point of view with room to improve.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper focuses on understanding and analyzing the reviewing process for a large conference such as ICLR and understand the reproducibility of the review process through Monte Carlo simulations.  Further, the authors also aim to study the impact of factors such as institutional bias, gender and study if higher review scores ensure more number of citations.

Comments:
1. How were the gender labels produced? Was it done through a manual process? With regard to gender bias analysis, one missing type of analysis in terms of reviewers' scores is to take into consideration the arxiv/ resubmission effect. This would provide better insights.
2. The timing of this study is important and some of the findings in this paper raise concerns about the overall review process. Also, a big thanks to the authors of this work for providing the entire codebase to reproduce their results
3. "As more reviewers are added, the high level of disagreement among area chairs remains constant, while the standard error in mean scores falls slowly" --> How is the high level of disagreement quantified? or is that an assumption? And if there is high-level of disagreement, how does the logistic regression model take that into consideration?
4.  Figure 4 is really interesting in terms of showing the impact of making submissions available earlier. This raises concerns about how making papers available online is biasing the reviewers in terms of the scores provided.
5. I have concerns over the way interrater reliability is calculated in section 4. The reviewers are randomized and this may not lead to the right number of samples per review and may affect the interrater reliability. Also, the assumptions mentioned by the authors seems to be wrong for this process.

 Suggestions:
1. "then" should be corrected to "the" in the last line of the first paragraph of section 2.
2. Can the mention of NIPS conference be changed to Neurips conference even though the change only happened in 2018?


review_title: This paper focuses on understanding and analyzing the reviewing process for a large conference such as ICLR and understand the reproducibility of the review process.

final_score: rating: 3: Clear rejection
review_content: The paper analyzes ICLR submissions and trends in the reviewing and selection process.

The paper is very interesting, and would be quite valuable for the senior members of the community to read and be familiar with. However, it does not seem to have anything to do with the ICLR topics. Hence, in this reviewer's view, it should either be privately disseminated to area chairs, or published as some form of appendix, but should not be part of the conference.

Getting into the details, it is not clear why the metric for the regression is just the mean reviewer score, not including all grades (or at least a variance). There is information in a paper that a reviewer gives a particular high/low grade to. Constructing then simulated data that uses the variation data and being surprised they don't match sounds as a potential pitfall for this analysis. 

In addition, there are several biases identified in the paper. But since the AC are not identified as biased, and the papers are anonymous, it is not clear what is the mechanism suggested by the authors of how these biases manifest themselves. Is there a suggestion that CMU/Cornell/MIT have a specific way of writing papers? Do women? Or is there a suggestion that anonymity does not genuinely exist, and most reviewers have good knowledge on who are the authors of the papers they review?
review_title: Interesting, but out of scope

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
In this paper, the authors conduct an empirical study on major machine learning reviewing process. The goal of the study is to answer questions like how is the reproducibility or what is the bias in the review process.

In general the paper makes some timely attempts for the conference review process. Here are some of my concerns:

- In Sec 3.1, the authors basically assume that papers with similar mean review scores have similar variability in scores. I think more justifications are needed for this assumption.

- In Sec 5.1, the authors only consider the average reviewer score, and an indicator variable for whether a paper came from one of the top 10 most highly ranked institutions as two factors for ACs to make the decision. I think the ACs decision process is too simplified. The same problem exists for other sections when the authors try to fit a logistic regression model between the acceptance decisions and a number of factors that the authors pre-select. The authors may want to clearly explain if only these factors contribute to the final decision (other confounding issues matter?), or if the relationship could be captured by a LR model. 

- In Sec 5.4, the authors assigned labels based on gendered pronouns appearing on personal webpages when possible, and on the use of canonically gendered names otherwise. The authors should report some statistics for how accurate this process is.

review_title: An empirical study on conference reviewing process

Meta-review: Reviewers appreciated the care and substantial effort that went into the paper, for instance:
AR3) I think it's of good value for the community to see and discuss the paper in the conference.
AR4) would be quite valuable for the senior members of the community to read and be familiar with.

The main argument for rejection is the the analysis done in the paper is not typical of ICLR research.  Arguably, the paper could fall under the topic "societal considerations of representation learning including fairness, safety, privacy", but this does not apply because the subject of analysis is the conference ICLR, not representation learning.   I support this argument.

The reviewers posed a good number of questions and issues with the paper, and largely these were addressed well by the authors.  In some cases they addressed the issues properly, and others they argued their case.  For instance AR2 says  "think the ACs decision process is too simplified" and the response summed up as "our ability to do multi-factor studies is limited by the size of our dataset".

An important one of these discussions is as follows:
AR4)  But since the AC are not identified as biased, and the papers are anonymous, it is not clear what is the mechanism suggested by the authors of how these biases manifest themselves.
Authors)  <extensive points>  .... we find the idea that anonymity does not genuinely exist to be entirely plausible.
I would argue that neither party can claim to have won this argument, and I am not really sure how it can be resolved.  Fortunately, though, no evidence for gender bias in ACs was found.

In conclusion, the paper is not topical to ICLR material, and the reviewer consensus is Reject.  However, the paper is both valuable and interesting to the community, and it has seen substantial improvement through the review process and a lot of the issues defended well.  

The paper should be brought to the attention of the various committees and made available somehow at the conference and acknowledged as a useful publication. 
Decision: Reject