An Investigation into the Role of Author Demographics in ICLR Participation and Review
Authors: Keshav Ganapathy
Emily Liu
Zain Zarger
Gowthami Somepalli
Micah Goldblum
Tom Goldstein
Venue: ICLR 2022 Submitted (2022)
Keywords: Conference Review
OpenReview
Gender
Bias
Fairness

Abstract: As machine learning conferences grow rapidly, many are concerned that individuals will be left behind on the basis of traits such as gender and geography.  We leverage historic ICLR submissions from 2017 to 2021 to investigate the impact of gender and country of origin both on representation and paper review outcomes at ICLR.  We also study various hypotheses that could explain gender representation disparities at ICLR, with a focus on factors that impact the likelihood of an author returning to the conference in consecutive years. Finally, we probe the effects of paper topic on the review process and perform a study on how the inclusion of theorems and the number of co-authors impact the success of papers in the review process.

Under review as a conference paper at ICLR 2022
AN INVESTIGATION INTO THE ROLE OF AUTHOR DE- MOGRAPHICS IN ICLR PARTICIPATION AND REVIEW
Anonymous authors Paper under double-blind review
ABSTRACT
As machine learning conferences grow rapidly, many are concerned that individ- uals will be left behind on the basis of traits such as gender and geography. We leverage historic ICLR submissions from 2017 to 2021 to investigate the impact of gender and country of origin both representation and paper review outcomes at ICLR. We also study various hypotheses that could explain gender represen- tation disparities at ICLR, with focus factors that impact the likelihood of an author returning to the conference in consecutive years. Finally, we probe the interplay between paper topic and review outcomes, and perform study how the inclusion of theorems in a paper and the size of the author list correlates with paper scores.
INTRODUCTION
It is well known that the field of computer science has strong representation disparities across both racial/ethnic and gender categories (Zweben & Bizot 2021 Muro et al. 2018 Dillon Jr et al.) 2015). For example, women comprised just 23.6% of enrollees in computation graduate programs in 2020, and only 1.2% of awarded PhDs were to Black students (Zweben & Bizot 2021). While these statistics paint concerning picture of the field's ability to recruit and retain students from diverse backgrounds, representation at ICLR is considerably more distorted. Last year, Tran et al. (2020) reported that only 12.1% of ICLR papers from US universities were lead by woman first author. Using methods described in this paper, we observe that this number decreased to 10.62% in 2021.
Gender representation disparity is quite extreme at ICLR, even within the context of well-known disparities in computer science at large. This motivates us to study these representation disparities more deeply, and we seek to identify and analyze theories for why they might exist. In particular, we are interested in whether issues concerning authorship, community, and the review process can be identified as factors that influence the representation of different groups We consider these issues:
Retention We find that women who attend ICLR are less likely to return to the conference the following year than men. We study number of factors that may contribute this effect. We find that women are much more likely to return when they are mentored by woman and that both women and men are far less likely return after receiving negative scores during review. At the same time, women are advantaged by working with slightly larger authorship teams, which tends result more positive outcomes.
Country of Origin We find that papers from western nations tend score higher, while papers from East and South Asia score lower than the conference average, although no significant bias against Asian papers is detected when acceptance decisions made, trolling for reviewer scores.
Topic Breakdown & Theory Empirical papers We see that papers containing theo- rems are far more likely to be accepted than non-theory papers. Further, women tend to submit papers to topics with slightly lower acceptance rates than men.
Industry Affiliations Papers from Google, Facebook, and Microsoft are much more likely to be accepted on average. Interestingly, women first authors are slightly more prevalent among industry papers than among academic papers.

---

Under review as a conference paper at ICLR 2022
While the focus of our study is demographic differences in review outcomes, number of recent studies have been conducted on the review process at large, with a focus on reproducibility and quality of reviews (Rogers & Augenstein, 2020, [hhradhwaj et al. 2020; Stelmakh et al. 2020b a Fiez et al. 2020) Manzoor & Shah] 2020; Shah et al. 2018) Sttelmakh et al. 2019))
2 DATASET CONSTRUCTION
Our amalysis is based on a dataset we acquired using the APIs of OpenReview and SemanticScholar, followed by extensive hand-labeling procedures conducted by the authors. From OpenReview, obtained titles, paper text, author lists, emails, scores, and reviews for ICLR papers from 2017- 2021. In total, 8,553 submissions were obtained: 2,978 from ICLR 2021, 2,560 from ICLR 2020, 1,565 from ICLR 2019, 960 from ICLR 2018, and 490 from ICLR 2017. To identify an author's institution, we used the author's email listed on their OpenReview profile and then mapped them to institutions with the open source World University and Domains dataset.
We used SemanticScholar scrape author- and paper-specific data. Author-specific data includes the number of citations. publications, influential papers, and h-index of each author on the paper. These data was obtained by scraping every paper by each author on SemanticScholar, identifying pa- pers with titles that were either identical or similar as measured by edit distance, and hand-val sand-validating to see if identified papers were correct. Paper-specific data includes the number of citations.
To study the role that gender plays in the review process, we hand-labeled gender for first and last authors on ICLR 2021 submissions. These labels were produced without relying on an automated process and rather by searching for gendered pronouns that appeared on personal webpages, social media, and CV' if applicable, and choosing the canonical gender for the name otherwise. We chose this process because ICLR has broad international participation, and automated gender inference tools, while more reproducible, are known to have unusually strong biases and inaccuracies on non-western identities (Santamaría & Mihaljević, 2018). We consider only binary gender labels. Future work should consider the experiences of non-binary and transgender people, however these populations cannot be easily studied using the statistical methods used in this paper.
GENDER REPRESENTATION AT ICLR
Tran et al. (2020) identified that women are highly under-represented at ICLR; in 2019, women made up 23.2% all CS PhD students in the US (Zweben & Bizot 2020), while only 12.1% of first authors at ICLR from US universities were women. We find that this disparity widened in 2020, with women compromising just 10.62% of first authors and 10.77% of last authors. For comparison. women comprised 23.4% of enrollees in doctoral CS programs and 24.8% of computer science, computer engineering, and information programs combined (Zweben & Bizot 2021).
There are many complex social and economic factors that likely contribute to the under- representation of women in computer science, including confidence gaps between men and women in STEM classrooms (Lubienski et al. 2013), stereotypes of skill levels and gender roles in aca- demics (Lubienski et al. 2013), differences in workplace treatment between men and women (Funk) Parker 2018), and even differences in income after graduation (Xu) 2013). However, the repre- sentation disparity at ICLR is strikingly more lopsided than what is observed in the computer science community at large, which indicates community-spectio factors that influence representation.
In this section, we consider various hypotheses for why such a strong disparity in representation might exist at ICLR. While there may be a range of sociological factors at play that are outside the scope of our study, we focus on factors that can be analyzed through publicly available data. Throughout this section, we focus on the likelihood of women returning to ICLR after appearing once. We focus only on the return rate of first authors. This is because last authors are unlikely to be first time attendees and often have long professional careers in ML that make them likely to return to the conference regardless of the factors identified below.
2

---

Under review as a conference paper at ICLR 2022
3.1 RETENTION
We observe that ICLR has somewhat more difficulty retaining women participants than men. Among first-time conference attendees in 2020, of the women who appeared as first author on paper, 19.79 19.79% returned 2021, compared to 23.83% for men, Expanding to account for all 2020 attendees regardless of their number of appearances, this mild discrepancy remains; the return rate for women was 28.12%
12% compared to 31.64% for men.
As for why this discrepancy exists, the difference in return rate could be explained in part by out- comes in the review process. We hypothesize that authors receiving strongly negative feedback on their contributions may be less likely to return to the conference and that the relatively lower scores women receive make them less likely to return. Tran et al. (2020) reported that papers from women first authors average scored 0.16 points lower than men in 2020. Our 2021 data shows present but smaller gender gap with women first authors on average scoring 0.13 points lower than men first authors (p 0.03). This gender discrepancy is clearly visible in the score histograms in Figure T There is no discernible gender gap for last authors.
(a) First Author
(b) Last Author
0.20
Density
Density
0.5
Legend Pietate Rest
Legend
Fan Procaleaa Last
0.10
Mala Pock
0:10
Make Lest
0:35
0.55
10
Score
Score
Figure 1: Histogram of average scores by gender, first and last author, ICLR 2017 ICLR 2021. We observe pronounced gap in first author scores, but mostly equal distribution for last authors.
This hypothesis is indeed supported by the data review scores strongly correlate with retention rates. logistic model predicting author return probability as function of mean reviewer score and gender identifies reviewer scores strong factor in predicting author return 0.015. Table T This trend is a significant factor within both the men and women uub-populations (see Table in the Appendix). In separate test, women with rejected papers had return rate of 24.28% while men with rejected papers had return rate of 30.28% (p 0.073).
Research on small minority peer groups STEM fields (Etzkowitz et al. 1992 Rosser & Lane 2002) suggests that the effect of reviewer scores may be stronger for women than for men, however we did not detect this as a significant factor in Table T
Table I: Return rate for first authors as function of mean reviewer score and gender indicator.
The logistic regression model had 70.8% accuracy and hold-out set containing 30% of 2,122 first authors.
Variable
Coefficient
Std. Error
Z-score
p-value
mean reviewer score
0,236
0.097
2.430
0.015
gender indicator
0.374
0.460
0.811
0.417
gender x mean review
-0.045
0.102
-0,441
0.659
constant
-1.960
0.437
-4.88
0.000
3

---

Under review as a conference paper at ICLR 2022
3.2 AMOUNT OF PROFESSIONAL EXPERIENCE
Negative reviewer scores are commonplace in the machine learning community at large. One may expect experienced authors to be well acquainted with this fact, and therefore less strongly impacted by negative scores than those authors receiving their first ever round of reviews. In this section, we quantify how author experience correlates with the return rate of first authors. We consider the relation between professional experience and first author outcomes, in addition to last author's experience level.
We identify "first-time" authors people who did not submit paper to ICLR in 2017-2019. Note that we use "first author" to refer to author order, and "first time author" to refer to lack of prior submissions. We find that first time authors are less likely to return ICLR than other authors (p<c0.001). This effect is still highly significant even after controlling for reviewer scores. Women first-time authors return with rate of 20.36%, while men first-time authors return with rate of 24.54%. The observations in this study are applicable to many attendees, as 65% of women and 63% of men are first-time authors.
We also consider the number of publications an author has and how it correlates with their return rate. After controlling for reviewer scored we find that the number of publications of the first author was correlated with return rate, but not to statistically significant degree (p = 0.143, Table o). We find that the number of citations is not a good indicator of return rate, possibly because varies wildly for young researchers.
3.3 MENTORSHIP AND GENDER PAIRING
Gender researchers in STEM fields have often cited the gender pairing of mentors and mentees major factor in predicting author productivity (Gaule Piacentini 2018 Pezzoni et al. 2016 Kato & Song 2018). At ICLR, we find that first authors their mentor of the same gender as they are. Women first authors with women last authors had return rate of 42.22% while women first authors with male last authors had a return rate of only 25.40% (p- = 0.02). A similar trend existed for men, who returned 46.36% of the time with a male last author, and only 31.01% % of the time with a women last author (p<0.001).
more likely return to the conference if
Curiously, women teams had lower acceptance rate (23.5% vs 27.3%) than women-mas teams, and yet women- women-women teams still result in much higher retention. Furthermore, when we control for paper scores, a logistic model still identifies gender pairing as yielding a highly significant boost return rate (p=0.01).
Finally, we see that the experience of the last author, as measured by the log of their total citation count, correlates strongly with first author return rate. After controlling for reviewer scores, we find that the citation count of the last author is a significant factor in predicting first author return rate, with more experienced last authors making return more likely (p- 0.002; Table 7).
4 AUTHORSHIP TEAMS
In this section, we investigate whether larger authorship teams perform better. Our study is motivated by the prevalence of personal accounts of isolation among women in computer science (Sankar) 2013 Fisher& & Margolis 2002 Frenkel 1990) which posits that the relatively smaller peer group for women may result in having fewer collaborators, and as a result fewer co-authors. Our study does not find evidence of this and, in fact, indicates that women tend to have larger authorship teams than men.
For this study, we consider papers with author, authors, 3 authors, 4 authors, 5 authors, 6 authors, 7 authors, and 8 or more authors. These categories had 274, 1396, 2144, 1896, 1276, 767, 410, and 383 submissions, respectively.
We find that mean reviewer scores increase monotonically with the number of authors (Figure This is consistent with published observations that collaborative papers in the sciences generally
Less experienced authors likely receive lower scores, making them less likely to return. Our model includes reviewer score control variable remove this effect.
4

---

Under review as a conference paper at ICLR 2022
(a) First Author, Author List Size Distribution
(b) Last Author, Author List Size Distribution
925
0.0
Density
Density
0.35
Legund Fazale
0.30
046
0.00
300
Number of Authors
Number of Authors
Figure 2: Distributions of number of co-authors based on gender. With respect the gender of (left) first author (right) last author.
score better and accrue citations faster than single author papers (Shneiderman 2018 2016 Perk mann et al., 2013). A regression model controlling for reviewer scores does find significant correlation between author number and acceptance rate, i.c., ACs do not appear to preferentially accept papers with more authors.
0.09
0.09
Mean Reviewer Score
5.4
5.0
0.05
0.05
4.6
42
2
Number of Authors
Number of Authors
Figure 3: (left) Mean reviewer scores as a function of number of authors, ICLR 2017 to ICLR 2021. (right) Percent women-led papers in each authorship category, ICLR 2017 to ICLR 2021.
It might be possible that papers with more authors more likely to be accepted due to the higher probability of personal connections with an AC. We do not find enough support for this hypothesis; regression model controlling for reviewer scores does not find significant correlation between author number and acceptance rate, i.e., ACs do not appear to preferentially accept papers with more authors.
We find that women on average were part of larger authorship teams. Men first author papers had on average 4.06 authors, while women had 4.25. Men last author papers had 4.06 authors average, Women had 4.37. See Figure for the distributions.
5 GEOGRAPHIC REPRESENTATION
This section breaks down how paper's country of origin correlates with review outcomes. We focus on academic (rather than industrial) papers, as they are associated with university with a well-defined location. We attribute paper to the national affiliation of its last author. We separate the papers into ten different regions: Australia and New Zealand (96 papers), Canada (333 papers), East Asia (including China, Japan, Mongolia, North Korea, South Korea, and Taiwan, 498 papers),
5

---

Under review as a conference paper at ICLR 2022
Mainland Europe (498 papers), the Middle East (including Bahrain, Cyprus, Egypt, Iran, Iraq. Israel, Jordan, Kuwait, Lebanon, Oman, Palestine, Qatar, Saudi Arabia, the Syrian Arab Republic, Turkey, the United Arab Emirates, and Yemen, 113 papers), Russia (13 papers), South America (13 papers), South Asia (including Afghanistan, Bangladesh, Bhutan, India, Nepal, Pakistan, Sri Lanka and the Maldives, 355 papers), the UK and Ireland (438 papers), and the United States (2345 papers).
Charts showing the geographic distribution of submitted and accepted papers appear in Figure Papers from the US and Canada on average scored 0.1 and 0.22 points higher than the conference average respectively, while papers from East Asia and South Asia scored 0.36 and 0.55 points lower than the conference average, respectively. Other regions did not deviate from the average score in substantial way. The acceptance rate of the conference over the years studied was 30.19%, while the acceptance rate for the US was 34.58% (p < 0.001), Canada was 35.73% (p 0.012), East Asian was 19.27% (p < 0.001), and South Asia was 22.53% (p < 0.001). See Table T0 in the appendix.
logistic model that predicts acceptance as a function of mean reviewer score and an indicator vari- able for East and South Asia indicates that, controlling for scores, papers from East and South Asia were slightly less likely to be accepted by ACs. This discrepancy in acceptance rates is equivalent to change in mean reviewer score of 0.098 and had borderline p-value 0.11 (Table 2
(a) All Papers
(b) Accepted Papers
0,51
0.57
822
0.07
0.09
0.09
0.08
0.09
0.09
0,06
0.02
0.02
0.01
0.02
0.02
0.0L
Metbeel
Bon Adw
Sead Alc
Figure 4: Geographic distribution of papers submitted and accepted
It appears that the scoring difference between papers of Western and Asian origins is likely explained in part by Asian authors having less professional experience on average than authors from other regions. 84% of papers of East and South Asian origin were lead by first-time author, whereas first-time authors lead 66% of papers from the US. We also suspect that scores are affected by gaps in English language experience in addition cultural differences in how non- Western speakers write and communicate, although we are not able to quantify these latter impacts.
Table Impact of East Asian origin on acceptance, all submissions ICLR from 2017 to 2021. Logistic regression predicting paper acceptance as function of mean reviewer score and an indica- tor of whether the paper was from East Asia. The model achieves 90% accuracy on hold-out set containing 30% of 2021 papers.
Variable
Coefficient
Std. Error
Z-score
p-value
mean reviewer score east asia indicator
3.0912
0.074
41,506
0,000
-0.2880
0.180
-1.598
0.110
constant
-18.2774
0.437
-41.796
0.000
TOPIC DISTRIBUTION
To understand how the topic of a paper correlates with the likelihood of it being accepted,
we define a high-level categorization of papers based on hand-curated keywords. When
any of these keywords appear in the paper title and/or abstract, assign the topic the

---

Under review as a conference paper at ICLR 2022
(a) Topic vs Submission rate
(b) Topic vs Acceptance rate
55.91
20.2
17.3
20
33 BG
15.3
15.0
29.20
12.5
20
10.3
2.5
2.31
Adhrgaam
Model
Hesst earing
Gyomication
Planey
Trandiomaa
Advizanne
Parraee
Ded Mudel
Crogan
Mesd sande
anue
optionatie
Therry
Viste
Figure 5: Topic level submission and acceptance on papers submitted from ICLR 2017- -021.
paper (some papers appear in multiple topies). The high-level topics used in this study Adversarial, Bayesian, Fairness, Generative models, Graph ML, Meta-Learning, NLP, Optimization, Reinforcement Learning (RL), Theory, Transformers, and Vision. We refer the reader to Appendix A.1 for the keywords used this study.
are:
In Figure 5a we show the percentage of papers falling into each of the topic categories. The numbers are computed on ICLR papers from 2017 2021. Theory forms the biggest chunk followed by Cenerative models and Adversarial ML. In Figure 56 we show the percentage of accepted papers for given category. Reinforcement Learning was most successful topic followed by Opt imi zation on and Theory. We see in Fig. 6a that women were less likely than men to submit a paper in these three categories and more likely to submit Vision and Trans formers which were the least successful topics at ICLR. When it comes to topic level acceptance rates, women did better than men in Meta-Learning, Fai rness, and Theory and almost comparable in Vision (Fig. (bb.
We also examine how paper submission rates per topic changed based on the geography of the first author in Figure 8 (left). We examine regions USA, Canada, Europe, Asia. US submitted the highest proportion of papers in Theory, while Asia was highest NLP, Fransformerss Europe in Rayesian, Generative Mode 1s and Canada in Theory, Meta-Tearning, and F Fairness.
(a) Topic level submission rate Men Women
(b) Topic level acceptance rate vs Women
Rasi-outae ponder
Fist-asllur gendes
35
17.5
1x.2
12.5
15
7.5
5.3
2.3
Atromun
Papebo
Ga. MAKL
Grapl
M2
ThAo
Trandheseos
Pamae
On Model
Opuzantlee
Vollo
Figure 6: Topic breakdown by gender, submission vs accepted papers, ICLR 2017 ICLR 2021.
6.1 Do PAPERS WITH THEOREMS DO BETTER THAN EMPIRICAL PAPERS?
is common across sub-fields to categorize papers as theory papers and empirical papers, and is colloquially thought by many that papers containing theorem more likely be
7

---

Under review as a conference paper at ICLR 2022
cepted to conference than purely empirical studies. We classify paper theorem paper if the paper body contains any of the following patterns: "Theorem: "Theorem <number> pripossition: ", "Proposition <number>* ", "Lemma: "Lemma <number> ", Out of the 8553 papers in the data set, 2321 submissions had at least one of the aforementioned patterns.
Theorem papers average scored 0.34 points higher than papers classified non-theorem papers (p<c0.001). The score distribution for the two kinds of papers is depicted in Figure 7 In all paper topics, papers with theorems out-performed those without theorems, although always by a sta- tistically significant margin, with Fairness and NLP getting the biggest boosts (Table D2]. Among women first authors, 26.7% of papers contained a theorem compared 31.1% for men (p 0.028). Finally, although theorem papers may receive higher mean reviewer scores, they do not receive bonus at the AC level (Table We also examine how likely researchers from different demograph- ies added a theorem in papers from different topics in Figures 7 (right) arig(tiight). We see women were more likely to have theorem Theory and Fa rness topies, while men were more likely to have one in Opt iimiiiize imizat ion and RL topics. When it comes region-level trends, the US was more likely to have a theorem in most the topics while Canada was less likely.
0.25
Probality it fiwoenn in eoeettee poger
WIDLES
Density
8
Theeen
8
0.35
de Mood
Gnsb
Score
Figure 7: (left) The distribution of paper scores with and without a theorem. (right) Fraction of
theorem papers by topic Men Women
Rogoon Chaada
-o of offttame hh othninat piper
Regios Canaad
Uis
Salmishin
60
Alumadal
Sopebon
Modd
Westaamr
Woon
Ropeton
Dimoe
Ons M004
Figure 8: In this figure study topic distribution with respect geography. (left) Fraction of papers by topic across regions (right) Fraction of theorem papers by topic across regions.
7 HOW DO INDUSTRY PAPERS REVIEW?
In this section, we break down the review statistics for industry papers submitted and published by Google (including DeepMind), Facebook, and Microsoft. In 2021, we find that papers from each of these organizations scored better average than typical conference paper. Without controlling for scores, all three organizations were significantly more likely the have papers accepted in 2021. The conference acceptance rate for 2021 was 28.87%, while the rate for Google 43.29% (p<,001),
8

---

Under review as a conference paper at ICLR 2022
Table 3: Theorem impact on acceptance, all submissions to ICLR from 2017 2021. Logistic regression predicting paper acceptance as function of mean reviewer score and theorem indicator. The model achieves 89% accuracy on hold-out set containing 30% of 2021 papers.
Variable
Coefficient
Std. Error
Z-score
p-value
reviewer score
3.0926
0.075
41.470
0.000
theorem indicator
0.0518
0,085
0,609
0.542
constant
-18.3113
0.438
-41.795
0.000
Table Industry impact on acceptance, Google, Facebook, and Microsoft, ICLR 2021. Logistic regression predicting paper acceptance as function of 3 industry indicators. The model achieves 70% accuracy on a hold-out set containing 30% of 2021 papers.
Variable
Coefficient
Std. Error
Z-score
p-value
google
0.7050
0.110
6.428
0.000
facebook
0.4859
0.168
2.896
0,004
microsoft 0.3573
0.166
2.147
0.032
constant
-25.6433
0.636
-40.350
.000
Facebook was 38.80% (p 0.004), and Microsoft was 42.42% (p 0.032). Women made up slightly more first authors on industry papers (12.86%) than they did on all papers (10.62%).
Tran (2020) finds that, controlling for reviewer scores, ACs were relatively less likely to accept a paper from Microsoft (-550 points, = 0.003) while submissions from Google (.007 points, p 0.932) and Facebook (.07 points, p 0.630) received siggsignifi acceptance boosts at the AC level at ICLR 2020. When controlling for score using the same procedures as TTan et al. (020) for consistency, we find that at ICLR 2021, although negative bias towards Microsoft submissions disappeared (.08 points, 0.204), positíve bonus on Google submissions surfaced (.118 points, p = 0.008). See Table
CONCLUSION
Using publicly available historical data and hand-curated datasets, this article examines the interplay between demographies, representation, and review outcomes at ICLR. This study has several impor- tant limitations that must be acknowledged. First, this is observational study where able to do controlled experiments or identify causal relationships. Second, non-i.id. review scores caused by people reviewing multiple papers, and violations of parametric assumptions, can cause Type-I errors. These issues have been particularly problematic when detecting bias in reviews (Stel- makh al.) 2019). Finally, impact demographics. Furthermore, while we did find significant results for some factors, we also report trends that, in some situations. did not rise the level of statistical significance. Unfortu- nately, our sample size is limited by the number of submissions, and the relatively small number of women at ICLR limits the statistical power of our tests.
not
that ICLR was virtual conference in 2020 and 2021, which may
The study has two major goals: to raise awareness of representational issues at ICLR and to pro- duce actionable conclusions. One of the strongest actionable conclusions reached in this paper that mentorship is important; women were far more likely to return to the conference when work- ing with a women last author (25% vs 42%). Furthermore, we suspect much of the disparity in acceptance rate of papers from Asia is due to English language experience another factor that can be addressed by mentorship. Finally, we know that first authors paired with more experienced last authors scored better (which promotes retention) and have improved retention rates overall. Collec- tively, this evidence supports the use of mentorship and role model pairing programs to enhancing representation.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
This study was conducted using publicly available datasets and without surveying participants. For this reason, do not consider this study raise privacy or data security issues, constitute human subject research in the US federally defined sense.
Finally, there risk that observations of lower reviewer scores among some sub-populations may promote stereotypes or reinforce negative opinions. We emphasize here that correlation does not imply causation, and the negative effects observed in this study are the result of confounding variables, some of which we tried and succeeded to identify, and some of which we did not.
REFERENCES
Homanga Bharadhwaj, Dylan Turpin, Animesh Garg, and Ashton Anderson, De-anonymization of authors through arxiv submissions during double-blind review. arXiv preprint arXiv:2007.00177, 2020.
Edward C Dillon Jr, Juan E Gilbert, Jerlando FL Jackson, and LJ Charleston. The state of african americans in computer science-the need to increase representation. Computing Research News, 21(8):2-6, 2015.
Henry Etzkowitz, Carol Kemelgor, Michael Neuschatz, and Brian Uzzi. Athena unbound: Barriers to women in academic science and engineering. Science and Public Policy, 19(3):157-179, 1992.
Tanner Fiez, Nihar Shah, and Lillian Ratliff. super* algorithm optimize paper bidding in peer review. In Conference on Uncertainty in Artificial Intelligence, pp. 580-589. PMLR, 2020.
Allan Fisher and Jane Margolis. Unlocking the clubhouse: the carnegie mellon experience. ACM SIGCSE Bulletin, 34(2):79-83 83, 2002.
Karen A Frenkel. Women and computing. Communications of the ACM, 33(11):34 46, 1990.
Cary Funk and Kim Parker. Women and men in stem often odds over workplace equity. 2018.
Patrick Gaule and Mario Piacentini. An advisor like me? Advisor gender and post-graduate careers in science. Research Policy, 47(4))805-8)) 2018.
Takao Kato and Yang Song. An advisor like me: Does gender matter? IZA Institute of Labor Economics Discussion Paper Series, Jun 2018
Sarah Lubienski, Joseph Robinson, Corinna C Crane, and Colleen Ganley. Brief report: Girls' and boys' mathematics achievement, affect, and experiences: Findings from ecls-k. Journal for Research in Mathematics Education, 44(4):634-645. 2013.
Emaad Manzoor and Nihar Shah. Uncovering latent biases in text: Method and application to peer review. arXiv preprint arXiv:2010. /5300, 2020.
Mark Muro, Alan Berube, and Jacob Whiton. Black and hispanic underrepresentaation in tech: It's time change the equation. The Brookings Institution, 2018.
Markus Perkmann, Valentina Tartari, Maureen McKelvey, Erkko Autio, Anders Broström, Pablo D' eeste, Riccardo Fini, Aldo Geuna, Rosa Grimaldi, Alan Hughes, et al. Academic engagement and commercialisation: A review of the literature on university-industry relations. Research policy, 42(():423 442, 2013.
Michele Pezzoni, Jacques Mairesse, Paula Stephan, and Julia Lane. Gender and the publication output of graduate students: case study. PLoS One, 11()::0044514,, 2016.
Anna Rogers and Isabelle Augenstein. What can we do to improve peer review in nlp? arXiv preprint ArXiv:20 03866, 2020.
Sue V Rosser and Eliesh O'Neil Lane. Key barriers for academic institutions seeking to retain female scientists and engineers: -unillyunfriendly policies. low numbers, stereotypes, and harassment. Journal of Women and Minorities in Science and Engineering, 8(2), 2002.
10

---

Under review as a conference paper at ICLR 2022
Pooja Sankar. The pervasive bias against female computer science majors. Fortune Magazine, 2015.
Lucía Santamaría and Helena Mihaljevié. Comparison and benchmark of name-to-gender inference services. PeerJ Computer Science, 4:e156, 2018.
Nihar В Shah, Behzad Tabibian, Krikamol Muandet, Isabelle Guyon, and Ulrike Von Luxburg. Design and analysis of the nips 2016 review process. Journal of machine learning research, 2018.
Ben Shneiderman. The new ABCs of research: Achieving breakthrough collaborations. Oxford University Press, 2016.
Ben Shneiderman. Twin-win model: A human-centered approach to research success. Proceedings of the National Academy of Sciences, (55550):2590-12294) 2018.
Ivan Stelmakh, Nihar Shah, and Aarti Singh. On testing for biases in peer review. Advances in Neural Information Processing Systems, 32:5286-555 -2966,5226622 2019.
Ivan Stelmakh, Charvi Rastogi, Nihar B Shah, Aarti Singh, and Hal Daumé III. A large scale ran- domized controlled trial on herding in peer-review discussions. arXiv preprint arXiv:201..1503, 2020a.
Ivan Stelmakh, Nihar Shah, Aarti Singh, and Hal Daumé III. A novice-reviewer experiment to address scarcity of qualified reviewers in large conferences. arXiv preprint arXiv:2011.15050, 2020b.
David Tran, Alex Valtchanov, Keshav Ganapathy, Raymond Feng, Eric Slud, Micah Goldblum, and Tom Goldstein. An open review of openreview: A critical analysis of the machine learning conference review process. arXiv preprint arXiv:2010.05137, 2020.
Yonghong Xu. Focusing on women in stem: longitudinal examination of gender-based carning gap of college graduates. The Journal of Higher Education, (6(4))489-5233 2015.
Stuart Zweben and Betsy Bizot. 2019 taulbee survey. Computing Research News, 2020.
Stuart Zweben and Betsy Bizot. 2020 taulbee survey. Computing Research News, 2021.
A APPENDIX
A.1 BREAKDOWN OF PAPERS BY TOPIC
The topics [and keywords] used were Vision [computer vision, object detection, segmentation, pose estimation, optical character recognition, structure from motion, facial recognition, face recog- nition], NLP [nlp, named-entity, machine translation, language model, word embeddings, part-of- speech, natural language, BERT, GPT), Meta-learninng [few-shot, meta learning, transfer learn- ing, zero-shot], Adversaríal ML [adversarial attack, poisoning, backdoor, adversarial exam- ple, adversarially robust, adversarial training, certified robust, certifiably robust], Generati medelling Igenerative adversarial network, gan, vae, variational autoencoder, diffusion models], Fairness [gender, racial, racist, biased, unfair, demographic, ethnic, ethical], Opt iitiiiin zat on foptimization theory, convergence rate, convex optimization, rate of convergence, global conver- gence, local convergence, stationary point], Graph [graph, tabular), Transforme Learning (RL) [temporal difference
[attention,
transformer), Bayesian [bayesian], Reinforremee learning, value function, value network, policy gradient, actor critic, actor-critic, A2C, PPO, Q- learning, Q learning, Q value, Bellman, DQN, Deep Q-network, mujoco, OpenSim RL, PyGame Learning Environment, Unity ML, OpenAl Gym].
A.2 ADDITIONAL LOGISTIC REGRESSION SUMMARIES
11

---

Under review as a conference paper at ICLR 2022
Table 5: Return rate for first authors function of mean reviewer score for men (top) and
women (bottom). The logistic regression models have a 69.2% and 78. % accuracy and a hold-out containing 30% of 1,848 male and 30% of 243 female first authors. Mean reviewer score is stronger factor on the return rate of women than men.
Variable
Coefficient Std. Error
Z-score
p-value
mean reviewer score constant
0.1987 -1.5866
0.031 0.146
6.092
0.000
10..86
0.000
mean reviewer score constant
0.2356
0,097
2.43
0,015
-1.9602
0.437
-4.88
0.000
Table 6: Experience level of the first author on return rate The logistic regression models have 69.5% and 67. 84,, accuracy, respectively, on a hold-out set containing 30% of 2122 first authors.
Variable
Coefficient
Std. Error
Z-score
p-value
first time author indicator mean reviewer score constant
-1.1427
0.099
-1.586
0.000
0.1820
0,030
6.120
0.000
-0.8021
0.153
-5.241
0.000
log base 10 publications mean reviewer score constant
0.1388
0.095
1.464
0.143
0.1927
0.030
6.451
0,000
-1.8050
0.183
-9.859
0.000
Table 7: Experience of the last author the return rate of the first author The logistic regres- sion models have 64.78% accuracy on a hold-out set containing 30% of 2,122 first authors.
Variable
Coefficient
Std. Error
Z-score p-value
log base 10 citation 0.1668
0.053
3.155
0,002
mean reviewer score constant
0.1788
0,035
4.888
0.000
-1.9845
0.223
-8,891
0.000
Table Return rate for first authors as function of gender indicator. The logistic regression model had 70.8% accuracy and hold-out set containing 30% of 2122 first authors.
Variable
Coefficient
Std. Error
Z-score
p-value
gender indicator constant
0.2104 -0,9977
0.152 0.144
1.381
0.167
-6.928
0.000
12

---

Under review as a conference paper at ICLR 2022
Table 9: Gender impact on acceptance, ICLR 2021. Logistic regression predicting paper accep- tance function of mean paper reviewer score and a gender indicator variable for both the first and last author. The model achieves 91 % accuracy on hold-out set containing 30% of 2021 papers. The results were inconclusive. Papers with first and last authors that were unlabelled were excluded.
Variable
Coefticient
Std. Error
Z-score
p-value
mean reviewer score
4.2505
0.182
23.312
0.000
gender indicator, first author constant
-0.1363
0.215
-0.635
0.525
-25.4785
1.098
-23.210
0.000
mean reviewer score gender indicator, last author constant
4.2523
0.179
23.778
0.000
0.0283
0.235
0.120
0.904
-25.6643
1.091
-23.514
0.000
Table 10: Country of origin impact on acceptance, all submissions to ICLR from 2017 to 2021.
Logistic regression predicting paper acceptance function of an indicator of what country it was from. The model achieves 70% accuracy on hold-out set containing 30% of 2021 papers.
Variable
Coefficient
Std. Error
Z-score
p-value
US
0.2506
0.056
4.492
0,000
Canada
0.3011
0.120
2.518
0.012
South America
0,0770
0.602
0.128
0.898
Australia/NZ
0.0006
0.227
0.003
0.998
Middle East
-0.2050
0.230
-0.893
0.372
UK/Ireland
-0.0233
0.147
-0.159
0.874
Mainland Europe
0.0921
0.109
0.844
0.398
Russia
0.3259
0.579
0.562
0.574
South Asia East Asia
-0.3468
0.132
-2.632
0.008
-0.5442
0.119
-4.578
0.000
constant
-0.8879
0.035
-25.347
0.000
Table 11: Country of origin impact acceptance, all submissions to ICLR from 2017 to 2021. Logistic regression predicting paper acceptance as a function mean reviewer score and an indi- cator of what country it from. The model achieves 90% accuracy hold-out set containing 30% of 2021 papers.
Variable
Coefficient
Std. Error
Z-score
p-value
mean reviewer score
3,0962
0.075
41.426
0.000
US Canada
0.0770
0.093
0.827
0.408
0.0714
0.196
0.364
0.716
South America
1.4728
0.973
1.514
0.130
Australia/NZ
-0.0628
0.356
-0.176
0.860
Middle East
-0.1418
0.375
-0.378
0.705
UK/Ireland
0.3342
0.241
1.384
0.166
Mainland Europe
-0.1479
0.181
-0.818
0,414
Russia
-0.0601
0.833
-0.072
0.942
South Asia
0.0807
0.236
0.341
0.733
East Asia
-0.2593
0.185
-1.401
0.161
constant
-1.3355
0.440
-41.666
0.000
13

---

Under review as a conference paper at ICLR 2022
Table 12: Theorem impact on acceptance for each category, all submissions ICLR from 2017 to 2021. Logistic regression predicting paper acceptance for each category as function of theorem indicator. The models have 68. 72%, 74.83%, 66.91%, 68.63%, 70. 16, 72. 12%, 66. 66.66%, 69.68%, 63.63%, 70.19%, 70.96%, 61.4% accuracy on a hold-out set containing 30% of all papers.
Regressions are for topies in the following order: Theory, Vision, NLP, Adversarial, Generative Modelling, Meta-learming, Fairness, Transformers, Optimization, Graph, Bayesian, and RI..
Variable
Coefficient
Std. Error
Z-score
p-value
theorem indicator constant
0.1612
0.097
1.667
0.095
-0.7764
0.071
-10.904
0.000
theorem indicator constant
0.3625
0.273
1.327
0.185
-1,0557
0.110
-9.580
0.000
theoren indicator constant
0.6300
0.205
3,066
0.002
-0.7925
0.077
-10.271
0.000
theorem indicator constant
0.3160 -0.9762
0.129
2.454
0.014
0.075
-12.969
0.000
theoren indicator constant
0.3597 -0.9391
0.142
2.536
0.011
0.074
-12.616
0.000
theorem indicator constant
0.1627
0.257
0.633
0.526
-0,8372
0.100
-8.334
0.000
theorem indicator constant
1 1,2009
0.489
2,456
0.014
-1,4523
0.335
-4.336
0.000
theorem indicator constant
0.3199
0.197 0.084
1,624
0.104
-0,9705
-11.511
0.000
theorem indicator constant
0.3474
0.406
0.856
0.392
-0.8755
0.376
-2.326
0.020
theorem indicator constant
0.2986 -0.9793
0.164
1.821
0.069
0.091
-10.758
0.000
theorem indicator constant
0,4187
0.271
1.547
0.122
-1,0622
0.155
-6.852
0.000
theorem indicator constant
0.3792 -0.7056
0.221
1.715
0.086
0.137
-5.163
0.000
14

---

Under review as a conference paper at ICLR 2022
Table 13: Industry impact on acceptance, Google, Facebook, and Microsoft, ICLR 2021. Lo- gistic regression predicting paper acceptance function of mean reviewer score and 3 industry indicators. The model achieves 91 % accuracy on a hold-out set containing 30% of 2021 papers.
Variable
Coefficient
Std. Error
Z-score
p-value
mean reviewer score
4.2521
.106
40.006
0.000
google
.5036
.191
2.639
.008
facebook
-0.1219
.291
-0419
.675
microsoft
.3420
269
1.270
.004
constant
-25.6333
0.636
40.350
.000
15


Reviews:
[{"review_id": "vRDSOJbeEqq", "reviewer": "Reviewer_4yLt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "STRENGTHS:\n- This paper studies an important topic regarding how conference (here specifically ICLR) submission/score/acceptance is correlated with various aspects that are in and out of authors' control, including their choice of topic, gender, location, etc. This has implications for how conferences can be designed to be more inclusive, guide scholars toward more friendly/specific venues, and how as a community we may re-think the assignment of prestige/importance to conferences.\n- The authors examine several cross-variable impacts, including gender/topic, experience/geographic region, and gender/experience. It would be interesting to measure how impactful such variables are compared to each other, perhaps via some sort of ANOVA or recursive elimination feature importance assessment, or the proposal of a method for measuring confounding/combinatorial impacts of each of the studied variables.\n\nWEAKNESSES:\n- It would have been interesting to see how gender representation could have changed if non-first/last authors were also labeled with their gender identity, although I recognize this may have been too expensive to label.\n- It is very difficult in general to make strong comments about the impact of various demographic factors on conferences and the review process, and while the authors propose several important insights and hypotheses to explain differences in e.g. return rate of women, it is difficult to establish how likely these hypotheses are (or to establish causality)\n- It would be interesting to see how the authors account for the self-selection bias of the fields/organizations that submit to or have members who attend ICLR, and perhaps compare this to other conferences (including those that are more specialized to specific fields like ACL, as well as other more general conferences like NeurIPS).\n- There is not much discussion of possible reasons for the scoring difference among geographic regions; it would be nice to see more clarity around what causes could be hypothesized and what data the authors believe is missing/can be collected to test such hypotheses.", "summary_of_the_paper": "This paper aims to quantify to some degree how gender, country of origin, and paper topic impact the review process, decision to submit, and paper review outcomes at ICLR, using data from 2017 to 2021. The authors report several interesting outcomes, including a lower return/retention rate for women attendees of ICLR (as well as authors with low review scores), higher scores for papers of Western origin, the increased likelihood of theorem-containing papers to be accepted, and the higher rate of acceptance for industry papers.", "summary_of_the_review": "This paper touches on the very important topic of investigating explanatory effects of demographic and meta-factors in conference attendance, submission, and acceptance. This is an extraordinarily wide mandate, and the paper attempts to cover a fair amount of ground - this also results in several intuitive, unanswered questions when reading. Despite the opportunity for deeper analysis, I believe this paper is valuable in understanding representation at conferences (specifically ICLR) and provides some starting points for improving the same.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TcojsHF5Gv", "reviewer": "Reviewer_b44g", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The exploratory data analysis and regression results presented in the paper contain some interesting results. Although the authors do not provide explanations behind each of these results, it might be helpful to raise awareness of these patterns in the community. At the same time, most of the findings from this exploratory analysis overlap with those of (Tran et al., 2020), thus I am unsure of how novel the authors' contributions are, besides presenting an updated analysis.\n\nAs the authors correctly note in the Ethics Statement, this is an observational study. All regression models control only for the reviewers' scores, thus in many instances it's unclear what kind of effect, if any, these models are actually capturing. The paper contains statements such as \"probe the effects of paper topic on the review process\", \"review score impacts retention\", \"how it impacts their return rate\", \"we remove the effect of reviewer score by controlling for this variable, there is still a small residual effect of gender\" etc. These regressions simply uncover associational patterns and thus causal statements should be avoided.\n\nAbout Section 3.1: \n* In Section 3.1, the histogram on the left of Figure 1 would suggest a large difference in scores between men and women, so I found the fact that the difference was only 0.13 surprising. Is this difference of 0.13 based on the data whose distribution is displayed in the histogram?\n* Was any residual diagnostics analysis for the logistic regression models conducted? All models are simple (i.e., they include only a few variables) so it should be rather straightforward to check whether the assumptions are met (e.g., by using the diagnostic plots presented here https://arxiv.org/pdf/1612.03257.pdf)\n* In the regression of retention ~ reviewers scores+gender, how were observations of individuals that appeared in multiple papers as first authors handled? These observations are dependent and thus it might be a good idea to cluster the standard errors by the first author\n* Are these results robust to changes in the way that reviewers scores are taken into account? e.g., if the most negative score instead of the mean score is accounted for, do the results still hold? \n* Past work has shown that type I error may not be controlled for when running logistic regression on peer-reviewed data (https://arxiv.org/abs/1912.13188), so this issue should be acknowledged\n* Based on Table 1, we cannot really say much about whether the difference between the coefficients in the two regression models is statistically significant. The most straightforward way to make some inference on that would be to fit a single model of retention ~ intercept + I(is woman) + reviewer score + reviewer score * I(is woman) and do a Wald test on the last coefficient. That said, the difference in the  probabilities predicted by the current models are very small\n* some of the concerns and suggestions raised above apply also to the remaining sections of the paper\n\n\nAbout Section 3.2:\n* \"we find that the number of publications of the first author had a positive impact on return rate (p = 0.143, Table 6)\". This p-value is above any of the significance levels that are generally considered (0.001, 0.01, 0.05, 0.1), so I would interpret this coefficient (and not \"impact\") as not being statistical significant. I read that \"borderline p-value 0.11\", so I assume that the significance level the authors adopted is 0.1.\n\nMinor details:\n* some of the summary statistics that are reported contain two decimal digits, while others only contain one\n* the manuscript contains some typos (e.g., \"These data was\")\n* Section 1: In (Zweben & Bizot, 2021) I could not find the information about women comprising 23.6% of the enrollees in computation graduate programs\n* Section 2: Perhaps it might useful to mention that the papers included in the dataset represent almost (but not) all of the submissions to ICLR in the period considered\n* Captions of figures 3-7 need to be improved. Table 1 needs to be improved too, e.g., by adding an additional column on the left containing the labels \"men\" and \"women\"\n* It would be helpful if the authors could provide the code to replicate these results", "summary_of_the_paper": "The paper conducts an observational study of disparities in the ICLR 2017-2021 reviewing process. The work centers around the analysis of disparities in reviewing scores and acceptance rates across gender, number of authors, countries, papers topics, and industry/academia. The authors present several findings that are likely of interest to the community and provide some potential explanations behind some of such findings.  ", "summary_of_the_review": "The analysis presented in the paper contains some interesting results. However, many of the findings overlap with those of (Tran et al., 2020), which conducted a similar analysis. In addition, I raised several concerns and questions on the methodology used in the paper. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NFu3_i6iutq", "reviewer": "Reviewer_fQri", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong points:\n\n-The peer review process in machine learning, and diversity and/or bias issues in CS/AI/ML, are both crucially important issues that are in dire need of research.  This work addresses both of these issues simultaneously, and is therefore likely to obtain substantial attention.\n\n-This work studies not only the impact of gender and geographic location, but also industry affiliation, the use of theorems, topics, experience level, etc.\n\n-Substantial and labor-intensive manual annotation of ICLR authors' gender was performed in order to execute the study. The authors chose to perform this work to get more reliable results, instead of simply relying on automatic tools which are known to have a bias toward performing better on Western names.\n\n-Attempting to control for various factors, due to the observational nature of the dataset, makes the analysis much more rigorous.\n\n-There are several very interesting findings in the research, which could have real-world impact on the peer review process, and on the academy at large. For example: ICLR authors are more likely to return or have their paper accepted if their mentor is the same gender, larger ICLR teams have higher review scores on average, women tend to have larger co-author teams than men.\n\nWeak points:\n\n-The precise relationship to the work of Tran et al. (2020) is unclear, e.g. what are the methodological differences and the novel findings? (Note, if this paper turns out to be by the same authors as the Tran et al. arXiv paper, and is actually a conference version of that same paper, its significance would be higher. I cannot assess this due to the double blind.)\n\n-It is a complicated question as to whether this paper is a good \"fit\" for ICLR.  On one hand, this work does not have the methodological novelty we would expect of an accepted ICLR paper, and this type of research is not normally published at this venue.  On the other, since it studies the ICLR conference itself, its findings are clearly of interest to the ICLR community.  Furthermore, it would behoove the ICLR community to be open to submissions such as this which are \"outside the box\" but still insightful.\n\n-Arguably, a journal format would be more appropriate for this work, as it would provide more space for a thorough analysis. Then again, there is no journal which is directly affiliated with the ICLR conference, so this would have disconnected the paper from the ICLR community.\n\n-I would like to have seen a more sophisticated level of analysis which leverages techniques developed by the ICLR community, and from machine learning at-large.  For example, NLP methods including topic models, word embeddings, and/or BERT could have contributed to the analysis.  This work only scratches the surface of the analyses on this dataset which are possible when leveraging the authors' gender annotations.\n\n-The paper could be strengthened by a deeper investigation into whether the disparities identified are due to factors relating to systemic bias, sexism, racism, etc.  Similarly, the paper could have delved deeper into the extent to which English language capabilities are the reason for the disparities, e.g. by reporting results when controlling for the percentage of English language speakers per country.\n\n-The potential impact of COVID-19 on the population under study should have been discussed.\n\n-Since much of the analysis hinges on annotations, it would be appropriate to report inter-annotator agreement metrics.\n\n\n\nQuestions to the authors:\n\n-Why is it particularly important to perform this study on the ICLR conference (other than the fact that the data are available on OpenReview)?\n\n\n\nAdditional feedback / minor suggestions:\n\n-Pg 3, the phrase \"limited evidence\" is a bit ambiguous as it is unclear whether authors are saying there is evidence (the data suggest the statement is true, but the evidence is limited), or there isn't evidence (the data suggest the statement is false, but the evidence is limited)\n\n-Pg 3, use parenthetical citations (\\citep) for \"Etzkowitz et al. (1992); Rosser & Lane (2002)\"\n\n-Pg 5, \"increase monotonically with number of authors\" (missing \"the\")\n\n-Pg 6, \"papers from East and South Asian\" (Asia)\n\n-Pg 7, \"women did doing better than men\" (remove \"doing\")\n\n-Pg 9, \"as a function 3 industry indicators\" (of)\n\n-Pg 9, \"10.62%\" (missing parentheses around the number)\n\n-Pg 9, \"mentorship is import\" (important)\n\n-The authors should mention that this work assumes a gender binary, and that future work could and should study nonbinary and transgender populations.\n\n-Once this work is published, it would be extremely valuable if the authors could make their annotated dataset publicly available.", "summary_of_the_paper": "This paper presents a study on the relationship between demographics and the submission and review of papers to ICLR, leveraging publicly available data from OpenReview.  The authors perform a substantial manual annotation effort on this dataset, and then study how paper acceptance and \"retention\" (i.e., whether authors submit to the conference again the following year) are related to gender and geographical location, by attempting to control for various factors. Topic, industry affiliation, experience level, and the use of theorems are also examined.", "summary_of_the_review": "This paper does not fit the typical mold of an ICLR paper, in that it is mainly empirical work rather than presenting novel methods.  As always, more analysis could have been done.  However, its findings regarding participation and review in ICLR are valuable to the ICLR community, and it would likely obtain substantial attention in the literature and have an impact on the machine learning community at large.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BjSC5UgSor", "reviewer": "Reviewer_kcQ1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positive aspects:\n-\tThe observations are interesting, specially for the organizers and senior members of ICLR.\n-\tThe methodology and analysis is clear and easy to follow.\n\n\nNegative aspects:\n-\tThe paper is not well-motivated and does not seem to fit within the general scope of ICLR (see more below).\n-\tThe analyses are univariate and possibly misleading (see more below).\n-\tThe study is non-conclusive and lacks a coherent structure.\n-\tNo in-depth study of the observations is provided and the results raise more questions than answers.\n\n\n\nMain Comments:\n\n-\tThe main problem with the study is the focus on univariate analysis. The phenomena in social sciences often involve more than one variable and a univariate analysis gives a biased and often misleading answer. A notable example is the work by Blau and Kahn: \nBlau, Francine D., and Lawrence M. Kahn. 2017. \"The Gender Wage Gap: Extent, Trends, and Explanations.\" Journal of Economic Literature, 55 (3): 789-865. \nFor drawing meaningful conclusions, I encourage authors to perform a multivariate analysis on all the variables considered in this study and possibly more variables drawn from similar studies in social sciences.\n-\tThe study is not well-motivate and it is not clear (1) how it fits withing the scope of ICLR, and (2) what conclusion should be drawn from the multiple correlations found in the study. The analysis done in this paper is narrowed to ICLR data only and does not relate to representation learning and broader applications. The paper is not conclusive and is built as a set of seemingly independent correlation-based analysis without a coherent structure.\n-\tThere are no deeper analysis of the observations. For example, when a certain bias exists (assuming a correct analysis), what are the underlying dynamics that lead to these biases. Considering the anonymity of the reviewing process, in many cases where the authors found there is a gender bias in the accepted papers (per topic), the question arises that how can a gender “bias” appear in this kind of reviewing process. Isn’t this just a byproduct of the small dataset and even smaller subset of female authors within the dataset?\n-\tI find the justification behind considering only the first and last authors for the mentorship analysis rather poor. The papers submitted in CS venues are more often than not the product of more than one team’s effort in which the senior heads of each team do not necessarily have mentorship relationship with the first author of the paper (often times a student). I think authors should consider the affiliations, experience, and gender of all authors of a paper and draw conclusion on the mentorship. Also, there can be more than one mentor (e.g., two senior authors with the same affiliation as the first author). The same goes for attributing a location to a paper; last author is not a good representation necessarily. A better approach is to make a new class of “multi-national” papers. \n-\tIn Section 3.1, authors study data from 2020 and 2021 to infer the statistics for first authors who came back the next year. The conference in these two years was virtual and, quite possibly, an anomaly in terms of the behavioral patterns. Authors should consider previous years as well. \n", "summary_of_the_paper": "The is a data analysis study on ICLR submissions from 2017 to 2021. Several variables such as gender, location, prior experience, and topic of the papers is considered. The study is mainly univariate and tries to find meaningful patterns for inferring representation of minorities at ICLR.", "summary_of_the_review": "I evaluate the paper as a reject for the following reasons:\n-\tThe paper is not well-motivated and does not seem to fit within the general scope of ICLR (see more below).\n-\tThe analyses are univariate and possibly misleading (see more below).\n-\tThe study is non-conclusive and lacks a coherent structure.\n-\tNo in-depth study of the observations is provided and the results raise more questions than answers.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors present an investigation of the impact of demographics on the peer review outcomes of ICLR. This is an important topic, as the demographics of ICLR and similar conferences are seriously skewed and may cause some people to feel excluded. The authors look into this complex problem with extensive manual annotations and analyses. 

The main weakness of this paper is that it is observational, and while the results are interesting, it is difficult to take away a clear and convincing message for the future. Part of the reason is that the whole problem is quite complex, and the hypotheses that are presented and tested in this paper reveal relatively shallow findings. Compared to the NeurIPS experiments which are carefully designed, these are not causal (see one of the reviewers' comments), so it is difficult to draw conclusions beyond correlations.

In summary, the results are interesting, and despite some of the reviewers' concerns, I would not exclude this paper because of the topic being irrelevant to the cfp, but I think the paper needs a more clear and convincing message.
Decision: Reject