An Open Review of OpenReview: A Critical Analysis of the Machine Learning Conference Review Process
Authors: David Tran
Alexander V Valtchanov
Keshav R Ganapathy
Raymond Feng
Eric Victor Slud
Micah Goldblum
Tom Goldstein
Venue:  (2021)
Keywords: Conference Review
OpenReview
Gender
Bias
Reproducibility
Fairness

Abstract: Mainstream machine learning conferences have seen a dramatic increase in the number of participants, along with a growing range of perspectives, in recent years.  Members of the machine learning community are likely to overhear allegations ranging from randomness of acceptance decisions to institutional bias.  In this work, we critically analyze the review process through a comprehensive study of papers submitted to ICLR between 2017 and 2020.  We quantify reproducibility/randomness in review scores and acceptance decisions, and examine whether scores correlate with paper impact.  Our findings suggest strong institutional bias in accept/reject decisions, even after controlling for paper quality. Furthermore, we find evidence for a gender gap, with female authors receiving lower scores, lower acceptance rates, and fewer citations per paper than their male counterparts.  We conclude our work with recommendations for future conference organizers. 

Under review as a conference paper at ICLR 2021
AN OPEN REVIEW OF OPENREVIEW: A CRITICAL ANALYSIS OF THE MACHINE LEARNING CONFERENCE REVIEW PROCESS
Anonymous authors Paper under double-blind review
ABSTRACT
Mainstream machine learning conferences have seen dramatic increase the number of participants, along with growing range of perspectives, in recent years. Members of the machine learning community are likely to overhear alle- gations ranging from randomness of acceptance decisions to institutional bias. In this work, we critically analyze the review process through a comprehensive study of papers submitted to ICLR between 2017 and 2020. We quantify reproducibil- ity/randomness in review scores and acceptance decisions, and examine whether scores correlate with paper impact. Our findings suggest strong institutional bias in accept/reject decisions, even after controlling for paper quality. Furthermore, we find evidence for a gender gap, with female authors receiving lower scores, lower acceptance rates, and fewer citations per paper than their male counterparts. We conclude our work with recommendations for future conference organizers.
INTRODUCTION
Over the last decade, mainstream machine learning conferences have been strained by deluge of conference paper submissions. At ICLR, for example, the number of submissions has grown by an order of magnitude within the last years alone. Furthermore, the influx of researchers from disparate fields has led to diverse range of perspectives and opinions that often conflict when it comes to reviewing and accepting papers. This has created an environment where the legitimacy and randomness of the review process is common topic of discussion. Do conference reviews consistently identify high quality work? Or has review degenerated into a process orthogonal to meritocracy?
In this paper, we put the review process under a microscope using publicly available data from across the web, in addition to hand-curated datasets. Our goals are
Quantify reproducibility in the review process We employ statistical methods disen- tangle sources of randomness in the review process. Using Monte-Cate simulations, we quantify the level of outcome reproducibility. Simulations indicate that randomness is effectively mitigated by recruiting more reviewers.
Measure whether high-impact papers score better We see that review scores are only
weakly correlated with citation impact.
Determine whether the process has gotten worse over time We present empirical ev- idence that the level of reproducibility of decisions, correlation between reviewer scores and impact, and consensus among reviewers has decreased over time.
Identify institutional bias We find strong evidence that area chair decisions are impacted by institutional name-brands. ACs are more likely to accept papers from prestigious in- stitutions (even when controlling for reviewer scores), and papers from more recognizable authors more likely to be accepted as well.
Present evidence for gender gap in the review process We find that women tend to receive lower scores than men, and have a lower acceptance rate overall (even after controlling for differences in the topic distribution for men and women).

---

Under review as a conference paper at ICLR 2021
2 DATASET CONSTRUCTION
We scraped data from multiple sources to enable analysis of many factors in the review process. OpenReview was a primary source of data, and we collected titles, abstracts, authors lists, emails, scores, and reviews for ICLR papers from 2017-2020. We also communicated with OpenReview maintainers to obtain information on withdrawn papers. There were a total of 5569 ICLR submis- sions from these years: ICLR 2020 had 2560 submissions, ICLR 2019 had 1565, ICLR 2018 had 960, and ICLR 2017 had 490. Authors were associated with institutions using both author profiles from OpenReview and the open-source World University and Domains dataset. CS Rankings was chosen to rank academic institutions because it includes institutions from outside the US.
The arXiv repository was scraped find papers that first appeared in non-anonymous form before review. We were able to find 3196 papers on arXiv, 1415 of them from 2020.
Citation and impact measures individual papers and individual authors, in addition to the publication counts of each author. Se- manticScholar search results were screened for duplicate publications using an edit distance metric. The dataset was hand-curated to resolve a number of difficulties, including finding missing authors whose names appear differently in different venues, checking for duplicate author pages that might corrupt citation counts, and hand-checking for duplicate papers when titles/authors are similar.
obtained from SemanticScholar. This includes citations for
To study gender disparities in the review process, we produced gender labels for first and last au- thors papers in 2020. We assigned labels based on gendered pronouns appearing on personal webpages when possible, and on the use of canonically gendered names otherwise. This produced labels for 2527 out of 2560 papers. We acknowledge the inaccuracies and complexities inherent in labeling complex attributes like gender, and its imbrication with race. However we do not think these complexities should prevent the impact of gender reviewer scores from being studied.
2.1 TOPIC BREAKDOWN
To study how review statistics vary by subject, and control for topic distribution the analysis below, define rough categorization of papers common ML topics. To keep things simple and interpretable, we define a short list hand curated keywords for each topic, and identify paper with that topic if it contains least of the relevant keywords. The topies used were theory, computer vision, natural language processing ng, adversarial fairness, generalizate on,
MI, generative modelling, meta-learnin opt imiioniion, graphs, Bayesian methods, and Other In total, 1605 papers from 2020 fell into the above categories, and 772 of 1605 papers fell into multiple categories.
Figure 1: Breakdown of papers and review outcomes by topic, ICLR 2020
(a) Acceptance rate
(b) Topic distribution
Acceptance Rate
05
Distribution
0.125
0.13
311
0.2
0.24
0.29
0.25
3:22
1:27
0.
0.100
3.075
5.13
0.10
0,10
0.050
9.025
0.030
Theory
Vision
NLP
Adversarial
Generative
Meta- learning
Fairnsss
Generalization
Optimization
Graph
Bayesian
All
Theory
Vision
NLP
Adversarial
Gonerative
leariingn
Fairess
Gonoralization
Optimizatioon
Graph
Buyesian
Meta- Meta-learning
See Appendix A.1 for details the keywords that chose.
2

---

Under review as a conference paper at ICLR 2021
3 BENCHMARKING THE REVIEW PROCESS
We measure the quality of the review process by several metrics, including reproducibility and correlation between scores and paper impact. We also study how these key metrics have evolved over time, and provide suggestions on how to improve the review process.
3.1 REPRODUCIBILITY OF REVIEWS
The reproducibility of review outcomes has been a hot-button issue, and many researchers are con- cerned that reviews are highly random. This issue was brought to the forefront by the well-known "NIPS experiment" (Lawrence & Cortes 2014), in which papers were reviewed by two different groups This experiment yielded the observation that 43% of papers accepted to the conference would be accepted again if the conference review process were repeated. In this section, we conduct an analysis quantify the randomness in the review process for papers submitted ICLR from 2017 to 2020. Our analysis is post hoc, unlike the controlled experiment from NeurlPS 2014.
Monte-Carde simulation To produce an interpretable metric of reproducibilty that accounts for variation in both reviewers and area chairs, we use Monte-Carlo simulations to estimate the "NIPS experiment" metrie: if accepted paper were reviewed anew, would it be accepted second time?
Our simulation samples from review scores from all 2560 papers submitted to ICLR in 2020. We simulate area chair randomness using a logistic regression model that predicts the AC's accept/reject decision as a function of a paper's s mean reviewer score. This model is fit using 2020 paper data, and the goodness of fit of this model is explored in Appendix A.8
To estimate reproducibility, we treat the empirical distribution of mean paper scores as if it were the true distribution of "endogenous" paper quality. We simulate paper by (i) drawing a mean/endogenous score from this distribution. We then (ii) examine all 2020 papers with similar (within unit) mean score, and compute the differences between the review scores and mean score for each such paper. We (iii) sample 3 of these differences at random, and add them to the endoge- nous score for the simulated paper to produce 3 simulated reviewer scores. We then (iv) use the simulated reviewer scores as inputs to the 2020 logistic regression model to predict whether the pa- per is accepted. Finally, we (v) generate a second set of reviewer scores using the same endogenous score, and use the logistic regression to see whether the paper is accepted second time.
We run separate simulations for years 2017 through 2020 with their own respective logistic regres- sion models. We observe downward trend in reproducibility, with scores decreasing from 75% in 2017, to 70% in 2018 and 2019, to 66% in 2020.
Reproducibility scores by topic appear in Figure 10 of the Appendix. We observe that Computer Vision papers have the lowest reproducibility score with 59%, while General aiatt zat ion papers have the highest score with 70%, followed by Theory papers with 68%.
Analysis of Variance A classical method for quantifying randomness in the review process is anal- ysis of variance (ANOVA). In this very simple generative model, (i) each paper has an endogenous quality score that is sampled from a Gaussian distribution with learned mean/vare variance, and (ii) re- viewer scores are sampled from a Gaussian distribution centered around the endogenous score, but with a learned variance. The ANOVA method generates unbiased estimates for the mean/variance parameters the above model. An ANOVA using all of scores in 2020 finds that endogenous paper quality has a standard deviation of 2.95, while review scores have deviation 1.72. This can be inter- preted to mean that the variation between reviewers is roughly 60% as large as the variation among papers. This does not account for the randomness of the area chair. Note the Gaussian model is a fairly rough approximation of the actual score distribution (ANOVA assumptions as discussed in Appendix A.2). ily interpretable. Simulations in which papers and scores are drawn from an ANOVA model agree closely with simulations using empirical distributions (Appendix A.2
(..2). Nonetheless, we report ANOVA results because this model is widely used and cas-
TThe NeuriPS conference was officially called NIPS prior 2018.
3

---

Under review as a conference paper at ICLR 2021
3.2 SHOULD WE HAVE MORE REVIEWERS PER PAPER?
A number of recent conferences have attempted mitigate the perceived randomness of ac- cept/reject decisions by increasing the number of reviewers per paper. Most notably, efforts were made at NeuRIPS 2020 to ensure that most papers received 5 reviews. Unfortunately, simulations suggest that increasing the number of reviewers is not particularly effective at mit- igating randomness.
Figure 2: Reproducibility by number of review- ers, ICLR 2020.
Reproducibility Level
0.8
0.62
0.04
0.57
2,66
0.70
0.
We apply the above reproducibility model to re- view scores in ICLR 2020 while increasing the
number of simulated reviews per paper, and we
plot results in Figure 2 While reproducibility scores increase with more reviewers, gains are
Number of Reviewers
marginal; increasing the number of reviewers from 2 to 5 boosts reproducibility by just 3%. Note that as the number of reviewers goes to infinity, the reproducibility score never exceeds 75%. This limiting behavior occurs because area chairs, who make final accept/reject decisions, vary in their standards for accepting papers. This uncertainty is captured by a logistic regression model in our simulation. As more reviewers are added, the level of disagreement among area chairs remains constant, while the standard error in mean scores falls slowly. The result is paltry gains in repro- ducibility. The reproducibility on rejected papers and on all papers can be found in Appendices A.5 and A.6 respectively.
Our study suggests that program chairs should avoid assigning large numbers of reviewers per paper as tactic to improve the review process. Rather, it is likely most beneficial to use a small number of reviews per paper (which gives reviewers more time for each review), and add ad-hoc reviewers to selected papers in cases where first-round reviews are uninformative.
Figure 3: Citation rank score rank for papers submitted to ICLR 2020
3.3 CORRELATION WITH IMPACT
Much of the conversation around the review process focuses on reproducibility and random- ness. However, perfectly reproducible pro- cess can still be quite bad if reprodueibly re-
Accepe (Drn))
Accepe (Spanigan)
Rejact
Rate Rank (1 (ighoot
jects high-impact and transformative papers. It
is thought by some that the ML review process favors non-controversial papers with incremen- tal theoretical results over papers with big new ideas.
1583
3390
In this section, we analyze whether reviewer scores correlate with paper impact. We measure impact using citation rate, calculated by divid- ing citation count by the number of days since
Score Ronk bhighent
the paper was first published online. After examining the data, citation rates are approximately linear in time although rare papers with extremely high citation count may exhibit exponential behavior. A similar study under the exponential model for citation count can be found in Appendix IT Though average scores for papers are roughly normally distributed, the distribution of citation rates is highly right-skewed and heavy tailed, with some papers receiving thousands of citations and oth- ers receiving none. To mitigate this issue, we use Spearman's 's rank correlation coefficient as a metric to measure the strength of the nonlinear relationship between average review score and citation rate.
We assign each paper a "citation rank" (the number papers with lower citation rate) and "score rank" (the number of papers with lower mean reviewer score), then calculate Spearman's 's rank cor- relation coefficient between citation rank and score rank for all submitted papers. At first glance, the Spearman correlation (0.46. p 0.001) seems to indicate moderate relationship between scores and citation impact. However, we find that most of this trend is easily explained by the fact that higher scoring papers are accepted to the conference, and gain citations from the public exposure
4

---

Under review as a conference paper at ICLR 2021
ICLR. This trend is clearly seen in Figure 3 in which there is sharp divide between the citation rates of accepted and rejected papers.
To remove the effect of conference exposure, calculate Spearman coefficients separately cepted posters and rejected/retrrateed papers, obtaining Spearman correlations of 0.17 and 0.22, re- spectively. While these small correlations still statistically significant (p 0.001) due to the large number of papers being analyzed, the effects are quite small; an inspection of Figure [ does reveal any visible relationship between scores and citation rates within each paper category.
Spotlight papers had a Spearman correlation of -0.043 (p = 0.66, = 107), and oral papers had Spearman correlation -0.0064 (p- 0.97, 48).
3.4 SHOULD WE DISCOURAGE RE-SUBMISSION PAPERS?
It is likely that papers that appeared on the web long before the ICLR deadline were previously rejected from another conference. We find that papers that have been online for more than months at the time of the ICLR submission date, in fact, have higher acceptance rates at ICLR as well as higher average scores. This fact observed each of past years (see Figure 4aa As these papers likely have been previously rejected, our findings suggest that resubmission should not be discouraged, as these papers are more likely to be accepted than fresh submissions. Interestingly, papers which did not appear online prior to the submission deadline, denoted by "not visible online", have the lowest acceptance rate.
Figure 4: Papers statistics by time online before submission deadline
(a) Acceptance
(b) Average reviewer score
visible online
visible
> <= months
> & <= months
0.8
months
8
> months
Average Score
> months
Acceptance
0.6
2
0.0
2017
2018
2019
2020
2017
2018 2019
2,200
Year
Year
HAS THE REVIEW PROCESS GOTTEN "WORSE" OVER TIME?
Our dataset shows that reproducibility scores, correlations with impact, and reviewer agreement have all gone down over the years. Downward temporal trends were already seen in reproducibility scores. Spearman correlations between scores and citations decreased from 0.582 in 2017 to 0.471 in 2020. We also measure the "inter-rater reliability," which is well-studied statistical measure of the agreement between reviewers (Hallgren) 2012), using the Krippendorf alpha statistic (Krippendorfi 2011 De Swert 2012). We observe that alpha decreases from 56% in 2017 to 39% in 2020. More details can be found in Appendix A.3
It should be noted that ICLR changed the rating scale used by reviewers in 2020 and also decreased the paper acceptance rate. These factors may be responsible in part for the sharp fall-off in repro- ducibility in 2020. We address this issue by bumping 2020 reviewer scores up so that the acceptance rate matches 2019, and then re-running our simulations. After controlling for acceptance rate in this way, we still observe an almost identical falloff 2020, indicating that the change in reprodueibility is not explained by the decreasing acceptance rate alone.
5 BIASES
We explore sources of bias in the review process, including institutional bias and bias toward rep- utable authors. We also discuss how gender correlates with outcomes.
5

---

Under review as a conference paper at ICLR 2021
5.1 INSTITUTIONAL BIAS
Does ICLR give certain institutions preferential treatment? In this section, we see whether the rank of institution (as measured by CS Rankings) influences decisions. We focus on academic institutions because they fall within widely known quantitative ranking system. We found that 85% of papers across all years (87% in 2020) had at least one academic author. In the event that a paper has authors from multiple institutions, we replicate the paper and assign copy of it to each institution. Figure 13 in Appendix A.7 plots the average of all paper scores received by each institution, ordered by institution rank. We find meaningful trend between institution rank and paper scores.
Area Chair Bias It is difficult to detect institutional bias reviews because higher scores among more prestigious institutions could be explained by either bias differences in paper quality. It is easier to study institutional bias at the AC level, as we can analyze the accept/reject decisions an AC makes while controlling for paper scores (which serves proxy for paper quality).
To study institutional bias at the AC level, we use a logistic regression model to predict paper ac- ceptance as a function of (i) the average reviewer score, and (ii) an indicator variable for whether a paper came from one of the top 10 most highly ranked institutions.
We found that, even after controlling for reviewer scores, being top ten institution leads to a boost in the likelihood of getting accepted. This boost is equivalent to a point increase in the average reviewer score (p = 0.028) Regression details can found in Table 4 located in Appendix A.9
Prestigious institutions are more likely than others to have famous researchers, and the acceptance bias could be attributed to the reputation of the researcher rather than the university. Section 5.3 we control for last-author reputation and still find significant effect for institution rank (Table D
We find that most of the institutional bias effect is explained by universities: Carnegie Mellon, MIT, and Cornell. We repeat the same study, but instead of a top 10 indicator, we use a separate indicator for each top 10 school. Most institutions have large p-values, but CMU, MIT, and Cor- nell each have statistically significant boosts equating to 0.31, 0.31, and 0.58 points in the average reviewer score, respectively. See
in Appendix
While 2020 area chairs came from a diverse range of universities, 26/1 15 of them had Google or Deepmind affiliation, which lead us to investigate bias effects for large companies (Table 7 A.9. The 303 papers from Google/Deepmind have insignificant boost (0.007 points, p 0.932) as did the 110 papers from Facebook (0.07 points, p = 0.630). However, the 92 papers from Microsoft had strong penalty (-0.50 points, p- 0.003).
5.2 THE ARXIV EFFECT
it better to post a paper on arXiv before the review process begins, or stay anonymous? To explore the effects of arXiv, we introduce a new indicator variable that detects if a paper was posted online by a date one week after the ICLR submission deadline. We fit logistic regression models to predict paper acceptance as a function of (i) the average score given by the reviewers of paper, and (ii) an indicator variable for papers that were visible on arXiv during the review process.
We fit models separately on 3 subsets of the 2020 papers: schools not in the top 10 (Table 9 A.9 top 10 schools excluding CMU/MI//Cornell (Table I A.9). and papers from CMU, MIT, and Cornell (Table 10 A.9). We find that by having papers visible on arXiv during the review process, CMU, MIT, and Cornell as a group receive a boost of 0.67 points (p = 0.001). Conversely, institutions not in the top 10 received statistically insignificant boost (0.08 points, p 0.235), did top 10 schools excluding CMU, MIT, and Cornell (0.05 points, p = 0.737).
Overall, found that papers appearing arXiv tended to do better (after controlling for reviewer scores) across the board, including schools that did not make the top 50, in addition to Google, Facebook, and Microsoft. However, these latter effects are not statistically significant.
We compute this effect by taking the ratio of the indicator coefficient and the average reviewer score coefficient, which represents the relative impact on the likelihood of paper being accepted.

---

Under review as a conference paper at ICLR 2021
5.3 REPUTATION BIAS
Are papers by famous authors more likely to be accepted even after controlling for reviewer scores? We quantify the reputation of an author by computing (citaatinnn///aa wwwtiion/pppprrat.tee/e + for cach author, where citations is the total citations for the author, and papers is their total number of papers (as indexed by Semantic Scholar). The raw ratio citations, haap/ppperss has a highly skewed and heavy tailed distribution, and we find that the log transform of the ratio has symmetric and nearly Gaussian distribution for last authors (see Figure 15
Appendix A.8).
We run logistic regression predicting paper acceptance as a function of (i) the average reviewer score, (ii) the reputation index for the last author, and (iii) an indicator variable for whether a paper came from a top-ten institution. We include the third variable to validate the source of bias, because institution and author prestige may be correlated. Table shows that last author reputation has a sta- tistically significant positive relationship with AC decisions when controlling mean reviewer score and institution. Taking into account the distribution of the last author reputation index (Figure [5] Appendix [A these coefficients indicate that last author near the mean of the author reputation index will gain boost equivalent to a 0.16 point increase in average reviewer score, while last au- thors at the top (2 standard deviations above the mean) receive a boost equivalent to a 0.29 point increase.
Table 1: Relationship between author reputation/innstiuuiinn and acceptance. This logistic gression model has 93% accuracy on a hold-out set containing 30% of 2020 papers.
Variable
Coefficient
Std. Error
Z-score
p-value
mean reviewer score
2,4354
0.098
24.807
0.000
log (citati ons / papers+1)
0.1302
0.057
2.270
0.023
top ten school?
0.2969
0.165
1.802
0,072
constant
-13.6620
0.562
-24.313
0.000
5.4 GENDER BIAS
There is well-known achievement gap between women and men in the sciences. Women in engi- neering disciplines tend to have their papers accepted to high impact venues, and yet receive fewer citations (Ghiasi et al.] 2015: Larivière et al., 2013 Lariviere 2014 Rossiter 1993). These cita- tion disparities exist among senior researchers at ICLR. Over their whole career, male last authors from 2020 have on average 44 citations per publication while women have 33. This gap in total citations can be attributed in part to differences in author seniority; the 2019 Taulbee survey reports that women are more severely under-represented among senior faculty (15.7%) than among junior faculty (23.9%) (Zweben & Bizot, 2020), and more senior faculty have had more time to accumu- late citations. To remove the effect of time, we focus in on only 2020 ICLR papers, where male last authors received on average 4.73 citations to date compared to 4.16 citations among women. This disparity flips for first authors: male first authors received just 4.4 citations vs 6.2 among women.
We find that women are under-represented at ICLR relative to their representation in computer sci- ence as a whole. In 2019, women made up 23.2% of all computer science PhD students (Zweben) & Bizot 2020) in the US. However, only 12.1% of publications at ICLR 2020 from US universities have a female first author. While women make up 22.6% of CS faculty in the US, they make up only 12.8% of last authors from US institutions ICLR 2020. Similar trends occur in the experimental sciences (Gaule & Piacentini] (2018) Pezzoni et al. 2016 Kato & Song 2018). Among papers from t0p-10 schools in the US, women make up 12.3% of first authors and 13.4% of last authors.
We observe a gender gap in the review process, with women first authors achieving a lower accep- tance rate than men (23.5% vs 27.1%) )).. This performance gap is reflected in individual reviewer scores, as depicted by histograms in Figure 5 Women on average score 0.16 points lower than their male peers. A Whintey U test generated a (two sided) p-value of 0.085 No significant differences were observed when scores were split by the gender of last author.
* 'A large gap in acceptance rates is not seen for last authors. See Table 13 in Appendix A.10 We use non-parametric test due to the non- Gan-Gassiian and sometimes multi-modal distribution of paper scores.
7

---

Under review as a conference paper at ICLR 2021
Figure 5: Histogram of average scores by gender, first and last author, ICLR 2020.
(a) First Author
(b) Last Author
Males Females
0.35
Males Females
0.30
0.30
Distribution
0.25
0.25
Distribution
0.20
0.20
0.15
0.15
0.10
0.10
0.05
0.05
0.00
0.00
Average Reviewer Score
Average Reviewer Score
As noted in Section [ there is a wide range of acceptance rates among the different research topics at ICLR. Among papers that fall into labeled topic (excluding other), male and female first author papers are accepted with rates 27.6% and 22. 1%, respectively. One may suspect that this disparity is explained by differences in the research topic distribution of men and women, shown in Figure However, we find that differences in topic distribution are actually more favorable to women than men. If women kept their same topic distribution, but had the same per-topic acceptance rates their male colleagues, we should expect women to achieve rate of 27.8% among topic papers (slightly more than their male colleagues) number far greater than the 22.1 they actually achieve.
Figure 6: Acceptance rate and Distribution by topic for male and female first authors (a) Acceptance
(b) Distribution
0.5
Acceptance Rate
Weenen
0.172
Distribution
0,300
0.375
911
80
Theory
Vision
NLP
Advorsanial
Genorative
Mote: leaming
Fairmas
Coneralization
Optimization
Graph
Bopesion
0,000
Theary
Vision
NLP
Advarsarial
Groerative
Mote-leaman Pairnoss
Meta-leaming
Coneralization
Optiicaation
Graph
Beyesien
We test for area chair bias with a regression predicting paper acceptance as a function of review scores and gender. We did not find evidence for AC bias (Figure 12] Appendix A.9
CONCLUSION
We find the level of reproducibility ICLR (66% in 2020) be higher than expect when considering the much lower acceptance rate (26.5%), which seemingly contradicts the notion that reviews "random." " Nonetheless, many authors find large swings reviews as they resubmit papers to different conferences and find it difficult to identify a home venue where their ideas feel respected. We speculate that the perceived randomness of conference reviews the result of several factors. First, differences in paper matching and bidding systems used by different conferences can sway the population of reviewers that are recommended source of coter-conference randomness that is not represented in the above intra-conference study. Second, the influx researchers from disparate backgrounds means that the value system paper's reviewers is often mismatched with that of its authors.
bid on an article, resulting in a major
Finally, we interpret the empirical studies in this paper to suggest that acceptance rates at mainstream conferences just too low. This is indicated by the prevalence of paper re-submission within the ML community, combined with the relatively high marks re-submitted papers we observed here. When the number of papers with merit is greater than the number of papers that will be accepted,
8

---

Under review as a conference paper at ICLR 2021
is inevitable that decisions become highly subjective and that biases towards certain kinds of papers, topics, institutions, and authors become endemic.
In work, study number of hand-selected factors for which sufficient data exists to draw statistically significant conclusions. Additional interesting factors and confounding variables likely exist. However, nulti-factor studies limited by the number of historic ICLR submissions. For example, it is not possible to simultaneously study the effect gender and institution rank because there are nearly enough women authors at ICLR get statistically meaningful results regarding gender after the dataset is sliced into bins based on institution. Despite the limitations stemming from the high number of interesting variables along with limited data, this study yields number of findings which are informative for the machine learning community and conference organizers.
REFERENCES
Knut De Swert.
Calculating inter-coder reliability in media content analysis using Krippendorff's alpha. Center for Politics and Communication, pp. 1-15, 2012.
Patrick Gaule and Mario Piacentini. An advisor like me? Advisor gender and post-graduate careers in science. Research Policy, 47(4))805-8))) 813, 2018.
Gita Ghiasi, Vincent Larivière, and Cassidy Sugimoto. On the compliance of women engineers with a gendered scientific system. PloS one, 10(12)xe0149911 2015.
Kevin A Hallgren. Computing inter-rater reliability for observational data: an overview and tutorial. Tutorials quantitative methods for psychology, 8(1):23, 2012.
Takao Kato and Yang Song. An advisor like me: Does gender matter? IZA Institute of Labor Economics Discussion Paper Series, Jun 2018.
Klaus Krippendorff. Computing Krippendorff's alpha-reliability. Computing, 1:25-2011, 2011.
Vincent Lariviere. Women and science: The first global data validate inequality. Acfas Magazine, 2014.
Vincent Larivière, Chaoqun Ni, Yves Gingras, Blaise Cronin, and Cassidy R Sugimoto. Bibliomet- rics: Global gender disparities science. Nature News, 504(7479):211, 2013.
Neil Lawrence and Corinna Cortes. The NIPS Experiment. inverseprobabiiit.comm 2014.
Michele Pezzoni, Jacques Mairesse, Paula Stephan, and Julia Lane. Gender and the publication output of graduate students: case study. PLoS One, 11()):0044144, 2016.
Margaret W Rossiter. The Matthew Matilda effect in science. Social studies of science, 23(2): 325-341, 1993.
Stuart Zweben and Betsy Bizot. The 2019 Taulbee Survey, volume I. Computing Research Associ- ation, 2020.

---

Under review as a conference paper at ICLR 2021
A
APPENDIX
Additional experimental details and results are included below.
A.1 BREAKDOWN OF PAPERS BY TOPIC
We identify a paper with a topie contains least one of the relevant keywords. The topics [and keywords] used were theory [theorem, prove, proof, theory, bound], computer vision [com- puter vision, object detection, segmentation, pose estimation, optical character recognition, structure from motion, facial recognition, face recognition), natural language processing [nat- ural language processing, nlp, named-entity, machine translation, language model, word embed- dings, part-of-speech, natural language], adversarial ML [adversarial attack, poison, back- door, adversarial example, adversarially roburst, adversarial training, certified roburst, certifiably roburst], generative ve modell 1 iiiinn ng [generative adversarial network, gan, vae, variational toencoder]. meta- -aaaaaleeninnng [few-shot, meta learning, transfer learning, zero-shot), fa rness [gender, racial, racist, biased, unfair, demographic, ethnic]. genera veneralization [generalization), opt ietimizaati [optimization theory, convergence rate, convex optimization, rate of convergence, global convergence, local convergence, stationary point], graphs [graph], Bayesiar ian methods [bayesian], and Other which includes the papers that do not fall into any of the above categories.
10

---

10 Under review as a conference paper at ICLR 2021
A.2 ANOVA METHOD
Here we show histograms of mean scores over all papers, and we plot the variance of individual reviewer scores as a function of paper quality. We observe that the distribution paper scores has overall Gaussian structure, however this structure is impacted by the fact that certain mean scores can be achieved using a large number of individual score combinations, while others cannot, resulting in
spiked behavior. We also find at the homogeneity assumptions of ANOVA (i.e., the reviewer variance remains the same for all papers) is not well satisfied for all conference years. For this reason, we do not want to rely heavily on ANOVA for our reliability analysis, and were motivated to construct the Monte Carlo simulation model. Interestingly, repeatability simulations using our ANOVA model yield results in close agreement with the Monte Carlo model.
Figure 7: Histogram for average scores of all papers. This plot demonstrates that the average scores in each year follows a roughly Gaussian distribution. In some years there are particular scores that appear to chosen relatively infrequently, causing multi-modal behavior. This is what inspired to use Monte-Carlo simulation in addition to the ANOVA results. Interestingly, our Monte- Carlo simulations produce similar results regardless of whether we sample the empirical score distribution, or the Gaussian approximation from ANOVA.
0.35
0.35
0.30
0.25
Distribution
Distribution
0.25
0.20
0.20
0.15
0.10
0.05
0.00
0.00
0
4
10
0
2
8
Average Score
Average Score
(a) 2017
(b) 2018
0.35
Distribution
Distribution
0.20
0.15
0.10
0.10
0.05
0.00
Average
Score
(c) 2019
(d) 2020
11

---

Std. Within Under review as a conference paper at ICLR 2021
Figure 8: Bar plot for average standard deviation within each score range. This plot demon- strates that the standard deviations are roughly equal across different ranges in 2017, 2018 and 2019. However, we observe that the standard deviations from score ranges between 3 and are higher than the rest in 2020. Note that in 2018 and 2019, there are no papers that have average score greater than 9.
Average Standard Deviation
3.0
Average Standard Deviation
3.0
2.5
2.5
2.0
2.0
1.5
1.5
1:15
1:30
111
1.0
1.20
3.42
0:92
1.0
3.20
188
1.00
3.03
6:57
0.50
0.5
0.5
0.0
0.0
1-2
2-3
3-1
4-5
5-6
6-7
7-8
8-9
9-10
1-2
2-3
3-1
4-5
5-6
6-7
7-8
8-9
9-10
Score Range
Score Range
(a) 2017
(b) 2018
Average Standard Deviation
3.0
Average Standard Deviation
3.0
2.5
2.5
2.0
2.0
1.71
1.5
1.5
1:25
1.4
3.12
2.55
0.55
Score Range (c) 2019
Score Range
(d) 2020
Table 2: Standard deviations from ANOVA model.. Summary for standard deviation between all average scores and standard deviation for review scores within paper.
Year
Std. Between
2017
2.3334
1.0307
2018
2.1080
1.1947
2019
2.1038
1.1036
2020
2.9595
1.7209
12

---

Under review as a conference paper at ICLR 2021
Figure 9: Reproducibility score over time. This plot compares the reproducibility score over time using the ANOVA model.
0.9
Reproducibility Level
0.6
0.5
0.4
0.3
2017
2018
2019
2020
Year
A.3 KRIPPENDORF ALPHA STATISTIC
Krippendorff's alpha is general statistical measure of agreement among raters, observers, or coders of data, designed to indicate their reliability. The advantage of this statistic is that it supports ordinal ratings, handles missing data, and is applicable to any number of raters, each assigning ratings to any other unit of analysis. As result, the Krippendorff's alpha is appropriate choice for our dataset given that there are many reviewers in the conference, each reviewer the conference usually rates several papers, and no reviewer rates all papers. Since the computation of the Krippendorf's is alpha depends only on the frequency of scores given to each paper, we can calculate the statistics despite the anonymous nature of the review process. We observe downward trend in agreement between reviewers, with alpha of 56% in 2017, 42% 2018, 47% in 2019, and 39% in 2020 where 100% indicates perfect agreement and 0% indicates absence of agreement.
13

---

Under review as a conference paper at ICLR 2021
A.4
REPRODUCIBILITY LEVEL BY TOPIC
Figure 10: Reproducibility level by topic, ICLR 2020.
Reproducibility Level
0.8
0.6
0.68
0.59
0.64
0.62
0,70
0.61
0.66
0.4
0.2
0.0
Theory
Vision
NLP
Adversarial
Generative
Meta-learning
Fairness
Generalization
Optimization
Graph
Bayesian
All
A.5 REPRODUCIBILITY LEVEL BY NUMBER OF REVIEWERS FOR REJECTED PAPERS
Figure 11: Reproducibility level by number of reviewers, ICLR 2020. The reproducibility level denotes the probability that rejected paper rejected second time.
1.U
100,000
0.84
0.86
0.87
0.89
Reproducibiity Level
0.80
0.8
0.4
0.2
0.0
2
4
6
8
Number of Reviewers
14

---

Under review as a conference paper at ICLR 2021
A.6 REPRODUCIBILITY LEVEL BY NUMBER REVIEWERS FOR ALL PAPERS
Figure 12: Reproducibility level by number of reviewers, ICLR 2020. The reproducibility level denotes the probability that a paper achieves the same outcome twice.
1.0
0.87 at 100,000 reviewers
Reproducibility Level
0.78
0.80
0.82
0.83
0.83
0.84
0.6
0.4
0.2
Number of Reviewers
A.7 CORRELATIONS BETWEEN INSTITUTION RANK AND PAPER SCORES
Figure 13: Institution rank vs average score for the top 100 institutions. For each institution and each paper decision type, all reviewer scores were averaged to generate a data point. Shaded areas represent 95% confidence intervals.
Average Reviewer Score
00
Accept (Oral)
Accept (Spollight)
Accept (Poster)
Reject
80
Institution Rank
15

---

Under review as a conference paper at ICLR 2021
A.8 VERIFYING ASSUMPTIONS OF STATISTICAL TESTS
Figure 14: Logistic Regression Assumptions Checking. We Hosmer-Lemesshow plot check the assumption for our logistic regression model. The null hypothesis is that there is no difference between observed deciles and those predicted by the logistic model. If the values are the same, we have model that fits perfectly. This graph compares the observed number of accepted papers and expected number of accepted papers across multiple groups.
Number of Accepted Papers
Observed
Number of Accepted Pupers
120
Observed
50
Expected
100
Expected
80
40
10
0.125
12.522
25-375
37.5-5
50-62
62 R2.5-75
75-87.5 7.
87:5-190
0.12.
12.5-25
26-37.5
37.5-50
50 5
62. 5-.5-5
75-87.5
87.5 87.5-100
Decile Rank by Acceptance Rate
Decile Rank by Acceptance Rate
2017
(b) 2018
Number Accepted Papers
175
Observed
Number Acceptod Papers
300
Observed
Expected
Expected
150
125
200
100
042.5
12.5-25
25-37.5
37,5-50
50-62.5
02.5778
79-87.5
07.3-100
032.5
12,5-25
25-37.5
37,5-50
50462.5
(2.3-75
75-87.5
87.5-100
62
62
Decile Rank by Acceptance Rate
Decile Rank by Acceptance Rate
(c) 2019
(d) 2020
Table Stukel's Test. We also use Stukel's test, another goodness-of-fit test for logistic regression. A statistically significant p-value indicates a poor fit. This test validates the logistic regression models on years 2017, 2018, and 2020, while indicates that the 2019 model provides poorer fit.
Year
p-value
2017
0.46
2018
0.37
2019
0.00
2020
0.75
16

---

Under review as a conference paper at ICLR 2021
Figure 15: Histogram for last author reputation index. This plot demonstrates that the transfor- mation log citaaii iatiiiin ons/ htpppplcattiino 1) achieves roughly Gaussian distribution with mean of 2.95 and standard deviation of 1 1.28. Last authors were taken from the 2020 conference only.
0.35
0.30
Density
0:25
logfciiations/puulcatiiosss + 1)
A.9 ADDITIONAL LOGISTIC REGRESSION SUMMARIES
Table 4: Top 10 institutions' impact acceptance, together, ICLR 2020. Summary of the
Logistic Regression predicting paper acceptance as function of mean reviewer score and top 10 indicator. The accuracy of the classifier was 92% on hold-out set containing 30% of 2020 papers
Variable
Coefficient
Std. Error
Z-score
p-value
mean reviewer score
2.4350
0.097
25.118
0.000
top ten school? constant
0.3536
0.161
2.194
0.028
-13.2707
0.526
-25.212
0.000
Table 5: Top 10 institutions' impact on acceptance, separate, ICLR 2020. Logistic regression summary for predicting paper acceptance. The top 10 institution ranks each have their indicator variable. Statistically significant effects are bold. The accuracy of the classifier was 92% on a hold-out set containing 30% of 2020 papers
Variable
Coefficient
Std. Error Z-score
p-value
mean reviewer score Carnegie Mellon
2,4600
0.099
24.949
0.000
0.7510
0.363
2.071
0.038
MIT
0.7498
0.357
2.100
0.036
U. Illinois, trbana-Champaign
0.6698
0.535
1.252
0.211
Stanford
-0.0312
0.371
-0.084
0.933
U.C. Berkeley
0.2299
0.337
0.681
0,496
U. Washington
-0.8189
0.684
-1.198
0.231
Cornell
1.4267
0.607
2.350
0.019
Tsinghua U. U. Michigan (tied)
-0.0188
0.369
-0.032
0.974
ETH Zurich
0.0608
0.640
0.095
0.924
constant
-13.4048
0.535
-25.038
0.000
17

---

Under review as a conference paper at ICLR 2021
Table 6: Institution & last author reputation impact on acceptance, CMU, MIT, and Cornell, ICLR 2020. Logistic regression predicting paper acceptance as function of mean reviewer score, last author reputation index, and 3 institution indicators. The accuracy of the classifier was 90% on a hold-out set containing 30% of 2020 papers.
Variable
Coefficient
Std. Error
Z-score
p-value
mean reviewer score
2,4511
0.099
24.718
0.000
log (citations/ paperr + 1)
0.1306
0,057
2.285
0.022
Carnegie Mellon (rank 1)
0.6730
0.368
1.829
0.067
MIT (rank 2)
0.6783
0.358
1.895
0.058
Cornell (rank 7)
1,3482
0.611
2.207
0.027
constant
-13.7425
0.567
-24.238
0.000
Table 7: Institution impact acceptance, Google, Facebook, and Microsoft, ICLR 2020. Lo- gistic regression predicting paper acceptance as a function of mean reviewer score and institution indicators. The accuracy of the classifier was 91% on hold-out set containing 30% of 2020 papers.
Variable
Coefficient Std. Error
Z-score
p-value
mean reviewer score
2.3515
0.075
31.557
0.000
google
-0.0177
0.209
-0.085
0.932
facebook
0.1745
0.362
0.482
0.630
microsoft
-1.1837
0.402
-2.948
0.003
constant
-12.7907
0.402
-31.843
0.000
Table 8: Visibility arXiv during the review period, top 10 institutions ICLR 2020. Logistic regression predicting paper acceptance as function mean reviewer score and an indicator de- marcating if the paper classifier was 93% on a hold-out set containing 30% of 2020 papers.
arXiv up to week after the submission date. The accuracy of the
Variable
Coefficient
Std. Error
Z-score
p-value
mean reviewer score visible on arXiv? 0.6599 constant
2.3904
0.190 0.288
12.551
0.000
2,294
0.022
-12.9311
1.014
-12.754
0.000
Table 9: Visibility on arXiv during the review period, institutions not in the top 10 ICLR 2020. Logistic regression predicting paper acceptance as a function of mean reviewer score and an indicator demarcating if the paper was on arXiv up to one week after the submission date. The accuracy of the classifier was 92% on hold-out set containing 30% of 2020 papers.
Variable
Coefficient Std. Error
Z-score
p-value
mean reviewer score
2,4309
0.112
21.650
0.000
visible on arXiv? constant
0.2033
0.171
1.187
0.235
-13.3262
0.610
-21.859
0.000
18

---

Under review as a conference paper at ICLR 2021
Table 10: Visibility arXiv during the review period, CMU, MIT, and Cornell at ICLR 2020. Logistic regression predicting paper acceptance as function of mean reviewer score and an indica- tor demarcating if the paper was on arXiv up to one week after the submission date. The accuracy of the classifier was 94% on a hold-out set containing 30% of 2020 papers.
Variable
Coefficient
Std. Error
Z-score
p-value
mean reviewer score
2.7144
0.372 0.551
7.295
0.000
visible arXiv?
1.8285
3.317
0.001
constant
-14.4988
1.967
-7.372
0.000
Table 11: Visibility on Xixiv during the review period, top 10 institutions excluding CMU, MIT, and Cornell at ICLR 2020. Logistic regression predicting paper acceptance as function of mean reviewer score and an indicator demarcating if the paper the submission date. The accuracy the classifier was 91% on a hold-out set containing 30% of 2020 papers.
arXiv up to one week after
Variable
Coefficient
Std. Error
Z-score
p-value
mean reviewer score
2.3874
0.239
9.987
0.000
visible on arXiv?
0.1203
0.358
0.336
0.737
constant
-12.9840
1,276
-10.175
0,000
Table 12: Gender impact acceptance, ICLR 2020. Logistic regression predicting paper accep-
tance as function of mean paper reviewer score and a gender indicator variable for both the first and last author. The accuracy of the classifier 93% hold-out containing 30% of 2020 papers. The results were inconclusive. Papers with last authors that were unlabelled were excluded.
Variable
Coefficient
Std. Error
Z-score
p-value
mean reviewer score gender indicator, first author constant
2.3956
0.109
21.877
0,000
-0.1045
0.266
-0.392
0.695
-13.0241
0.633
-20.590
0.000
mean reviewer score gender indicator, last author constant
2.3813
0.107
22.349
0.000
-0.0520
0.272
0.191
0.849
-13.0441
0.627
-20.800
0,000
A.10 ADDITIONAL STATISTICS ON GENDER
Table 13: Summary of gender statistics. Some percents do not add up to 100%; we were unable
label 5.6% of first authors and 2.5% of last authors.
First Author
Last Author
Statistic
Male
Female
Male
Female
Percent of Papers
83.75%
10.63%
87.81 %
9.65%
Percent of Accepted Papers
84.43%
9.32%
88.50%
9.46%
Acceptance Rate
27.05%
23.53%
27.05%
26.32%
Average Reviewer Score
4.21
4.06
4.20
4.18
19

---

Under review as a conference paper at ICLR 2021
A.11 CORRELATION WITH IMPACT UNDER AN EXPONENTIAL MODEL FOR THE GROWTH RATE OF CITATION COUNT
Here, calculate citation rate as logícitations + (time))) As with our experiments under the linear model in the main body, Figure [6 shows no visible relationship between scores and citation rates within each paper category. We obtain a Spearman correlation 0.46 (p < 0.001) for all papers. Accepted posters and rejected papers have Spearman correlations of 0.13 (p = 0.002) and 0.21 (p < 0.001) respectively. Spotlight papers have Spearman correlation 0.045 (p 0.65, 107) and oral papers have Spearman correlation 0.13 (p 0.39, = 48). Thus, we observe the same behavior under this model; specifically, score only has a small relationship with impact within decision categories.
Figure 16: Log-transformed citation rate rank score rank for papers submitted to ICLR 2020.
(Oral)
Accept (Spotlight)
(Pooter)
Reject
2500
Log-transformee Citatin Rate Rank -highest)
2000
1500
1000
500
1000
1500
2000
2500
Rank (1 highest)
20


Reviews:
[{"review_id": "soTZcQJo7bR", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nIn this paper, the authors conduct an empirical study on major machine learning reviewing process. The goal of the study is to answer questions like how is the reproducibility or what is the bias in the review process.\n\nIn general the paper makes some timely attempts for the conference review process. Here are some of my concerns:\n\n- In Sec 3.1, the authors basically assume that papers with similar mean review scores have similar variability in scores. I think more justifications are needed for this assumption.\n\n- In Sec 5.1, the authors only consider the average reviewer score, and an indicator variable for whether a paper came from one of the top 10 most highly ranked institutions as two factors for ACs to make the decision. I think the ACs decision process is too simplified. The same problem exists for other sections when the authors try to fit a logistic regression model between the acceptance decisions and a number of factors that the authors pre-select. The authors may want to clearly explain if only these factors contribute to the final decision (other confounding issues matter?), or if the relationship could be captured by a LR model. \n\n- In Sec 5.4, the authors assigned labels based on gendered pronouns appearing on personal webpages when possible, and on the use of canonically gendered names otherwise. The authors should report some statistics for how accurate this process is.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "U8Xb4cYv88G", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper analyzes ICLR submissions and trends in the reviewing and selection process.\n\nThe paper is very interesting, and would be quite valuable for the senior members of the community to read and be familiar with. However, it does not seem to have anything to do with the ICLR topics. Hence, in this reviewer's view, it should either be privately disseminated to area chairs, or published as some form of appendix, but should not be part of the conference.\n\nGetting into the details, it is not clear why the metric for the regression is just the mean reviewer score, not including all grades (or at least a variance). There is information in a paper that a reviewer gives a particular high/low grade to. Constructing then simulated data that uses the variation data and being surprised they don't match sounds as a potential pitfall for this analysis. \n\nIn addition, there are several biases identified in the paper. But since the AC are not identified as biased, and the papers are anonymous, it is not clear what is the mechanism suggested by the authors of how these biases manifest themselves. Is there a suggestion that CMU/Cornell/MIT have a specific way of writing papers? Do women? Or is there a suggestion that anonymity does not genuinely exist, and most reviewers have good knowledge on who are the authors of the papers they review?", "extra_scores": {}, "extra_text": {}}, {"review_id": "P3360aJi8gQ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper focuses on understanding and analyzing the reviewing process for a large conference such as ICLR and understand the reproducibility of the review process through Monte Carlo simulations.  Further, the authors also aim to study the impact of factors such as institutional bias, gender and study if higher review scores ensure more number of citations.\n\nComments:\n1. How were the gender labels produced? Was it done through a manual process? With regard to gender bias analysis, one missing type of analysis in terms of reviewers' scores is to take into consideration the arxiv/ resubmission effect. This would provide better insights.\n2. The timing of this study is important and some of the findings in this paper raise concerns about the overall review process. Also, a big thanks to the authors of this work for providing the entire codebase to reproduce their results\n3. \"As more reviewers are added, the high level of disagreement among area chairs remains constant, while the standard error in mean scores falls slowly\" --> How is the high level of disagreement quantified? or is that an assumption? And if there is high-level of disagreement, how does the logistic regression model take that into consideration?\n4.  Figure 4 is really interesting in terms of showing the impact of making submissions available earlier. This raises concerns about how making papers available online is biasing the reviewers in terms of the scores provided.\n5. I have concerns over the way interrater reliability is calculated in section 4. The reviewers are randomized and this may not lead to the right number of samples per review and may affect the interrater reliability. Also, the assumptions mentioned by the authors seems to be wrong for this process.\n\n Suggestions:\n1. \"then\" should be corrected to \"the\" in the last line of the first paragraph of section 2.\n2. Can the mention of NIPS conference be changed to Neurips conference even though the change only happened in 2018?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "CCzgPzqxJJ4", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "First of all I want to thank the author(s) to notice and study the quality of reviews in ICLR.\n\nQuality:\n\nThe authors put a lot of effort in collecting and cleaning the data, which is not easy. The authors did thorough statistical analysis over the data and considered many possible pitfalls. I still have some questions but I think it's of good value for the community to see and discuss the paper in the conference.\n\nClarity:\n\nThe paper is well written and easy to follow. \n\nOriginality:\n\nGood original thinking. \n\nSignificance:\n\nThe analysis and conclusion is of good value for the ICLR community. Great significance.\n\nI gave the rating of marginally below acceptance threshold. Why not higher? I think there are some fundamental questions that need more clarification from the authors (4,5, 6,7,10 and 12 in the detailed comments). But to be clear I still think there's great value for it to be accepted given the topic, and if the authors can answer the questions in the comments.\n\nDetailed comments:\n\n1) Page 1, intro: What do you mean by \"censorship\" here?\n2) Page 1 last paragraph. I think the reviewers have no knowledge of author's gender, given the double blind review process? I know you might be able to dig it through by searching arxiv ,but doubt anyone would do that to see the gender.\n3) Page 2, data set. It's better to explain the size of data clearly, is it 2560 papers or 5569 papers?  \n4) Page 3. the simulation. Given the whole simulation set up, I think the key factor is the variation of reviewer score and AC's decision noise. Why don't we just look at those directly? Simulation gives a more intuitive number (re-acceptance rate) but with more noise injected in.\n5) Page 3, I'm not a statistician, but short research on Hosmer-Lemeshow test suggests that it's not a good way to exam the quality of LR model. Could you also show the x-validation precision/recall of the prediction? AC might take other factors into consideration, such as the content of review comments, reviewer's confidence and background, or the order of papers (esp the ones on the border of a/r).\n6) Page 3, If the same 2020 logistic regress model was used, then the result only shows that the variance of review score increased over time. Have you considered the change in number of papers and number of reviewers over the years?\n7) Page 4, Sec 3.3. Number of citation increases exponentially over time. simply dividing by time will punish papers published sooner.\n8) Page 5, figure 4, could you add the papers that never before seen online too?\n9) Page 6, paragraph 4, is it 0.15 out of 10? seems a trivial amount.\n10) Page 7, reputation, Is this a correlation or causality? It could be that papers from authors with better reputation did have an edge in quality. Remember ACs have access to the review notes and the paper themselves too.\n11) Page 7, second last paragraph.  We can't mix woman % in the US with ICLR which is a global venue. Global % could be lower.\n12) Page 8 Conclusion. This is a weak point, Conclusion is not supported by the content of the paper, but more like a speculation.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers appreciated the care and substantial effort that went into the paper, for instance:
AR3) I think it's of good value for the community to see and discuss the paper in the conference.
AR4) would be quite valuable for the senior members of the community to read and be familiar with.

The main argument for rejection is the the analysis done in the paper is not typical of ICLR research.  Arguably, the paper could fall under the topic "societal considerations of representation learning including fairness, safety, privacy", but this does not apply because the subject of analysis is the conference ICLR, not representation learning.   I support this argument.

The reviewers posed a good number of questions and issues with the paper, and largely these were addressed well by the authors.  In some cases they addressed the issues properly, and others they argued their case.  For instance AR2 says  "think the ACs decision process is too simplified" and the response summed up as "our ability to do multi-factor studies is limited by the size of our dataset".

An important one of these discussions is as follows:
AR4)  But since the AC are not identified as biased, and the papers are anonymous, it is not clear what is the mechanism suggested by the authors of how these biases manifest themselves.
Authors)  <extensive points>  .... we find the idea that anonymity does not genuinely exist to be entirely plausible.
I would argue that neither party can claim to have won this argument, and I am not really sure how it can be resolved.  Fortunately, though, no evidence for gender bias in ACs was found.

In conclusion, the paper is not topical to ICLR material, and the reviewer consensus is Reject.  However, the paper is both valuable and interesting to the community, and it has seen substantial improvement through the review process and a lot of the issues defended well.  

The paper should be brought to the attention of the various committees and made available somehow at the conference and acknowledged as a useful publication. 
Decision: Reject