An Investigation into the Role of Author Demographics in ICLR Participation and Review

As machine learning conferences grow rapidly , many are concerned that individuals will be left behind on the basis of traits such as gender and geography . We leverage historic ICLR submissions from 2017 to 2021 to investigate the impact of gender and country of origin both on representation and paper review outcomes at ICLR . We also study various hypotheses that could explain gender representation disparities at ICLR , with a focus on factors that impact the likelihood of an author returning to the conference in consecutive years . Finally , we probe the interplay between paper topic and review outcomes , and perform a study on how the inclusion of theorems in a paper and the size of the author list correlates with paper scores . 1 INTRODUCTION . It is well known that the field of computer science has strong representation disparities across both racial/ethnic and gender categories ( Zweben & Bizot , 2021 ; Muro et al. , 2018 ; Dillon Jr et al. , 2015 ) . For example , women comprised just 23.6 % of enrollees in computation graduate programs in 2020 , and only 1.2 % of awarded PhDs were to Black students ( Zweben & Bizot , 2021 ) . While these statistics paint a concerning picture of the field ’ s ability to recruit and retain students from diverse backgrounds , representation at ICLR is considerably more distorted . Last year , Tran et al . ( 2020 ) reported that only 12.1 % of ICLR papers from US universities were lead by a woman first author . Using methods described in this paper , we observe that this number decreased to 10.62 % in 2021 . Gender representation disparity is quite extreme at ICLR , even within the context of well-known disparities in computer science at large . This motivates us to study these representation disparities more deeply , and we seek to identify and analyze theories for why they might exist . In particular , we are interested in whether issues concerning authorship , community , and the review process can be identified as factors that influence the representation of different groups . We consider these issues : • Retention We find that women who attend ICLR are less likely to return to the conference the following year than men . We study a number of factors that may contribute to this effect . We find that women are much more likely to return when they are mentored by a woman and that both women and men are far less likely to return after receiving negative scores during review . At the same time , women are advantaged by working with slightly larger authorship teams , which tends to result in more positive outcomes . • Country of Origin We find that papers from western nations tend to score higher , while papers from East and South Asia score lower than the conference average , although no significant bias against Asian papers is detected when acceptance decisions are made , controlling for reviewer scores . • Topic Breakdown & Theory vs Empirical papers We see that papers containing theorems are far more likely to be accepted than non-theory papers . Further , women tend to submit papers to topics with slightly lower acceptance rates than men . • Industry Affiliations Papers from Google , Facebook , and Microsoft are much more likely to be accepted on average . Interestingly , women first authors are slightly more prevalent among industry papers than among academic papers . While the focus of our study is on demographic differences in review outcomes , a number of recent studies have been conducted on the review process at large , with a focus on reproducibility and quality of reviews ( Rogers & Augenstein , 2020 ; Bharadhwaj et al. , 2020 ; Stelmakh et al. , 2020b ; a ; Fiez et al. , 2020 ; Manzoor & Shah , 2020 ; Shah et al. , 2018 ; Stelmakh et al. , 2019 ) . 2 DATASET CONSTRUCTION . Our analysis is based on a dataset we acquired using the APIs of OpenReview and SemanticScholar , followed by extensive hand-labeling procedures conducted by the authors . From OpenReview , we obtained titles , paper text , author lists , emails , scores , and reviews for ICLR papers from 2017- 2021 . In total , 8,553 submissions were obtained : 2,978 from ICLR 2021 , 2,560 from ICLR 2020 , 1,565 from ICLR 2019 , 960 from ICLR 2018 , and 490 from ICLR 2017 . To identify an author ’ s institution , we used the author ’ s email listed on their OpenReview profile and then mapped them to institutions with the open source World University and Domains dataset . We used SemanticScholar to scrape author- and paper-specific data . Author-specific data includes the number of citations , publications , influential papers , and h-index of each author on the paper . These data was obtained by scraping every paper by each author on SemanticScholar , identifying papers with titles that were either identical or similar as measured by edit distance , and hand-validating to see if identified papers were correct . Paper-specific data includes the number of citations . To study the role that gender plays in the review process , we hand-labeled gender for first and last authors on ICLR 2021 submissions . These labels were produced without relying on an automated process and rather by searching for gendered pronouns that appeared on personal webpages , social media , and CVs if applicable , and choosing the canonical gender for the name otherwise . We chose this process because ICLR has broad international participation , and automated gender inference tools , while more reproducible , are known to have unusually strong biases and inaccuracies on non-western identities ( Santamarı́a & Mihaljević , 2018 ) . We consider only binary gender labels . Future work should consider the experiences of non-binary and transgender people , however these populations can not be easily studied using the statistical methods used in this paper . 3 GENDER REPRESENTATION AT ICLR . Tran et al . ( 2020 ) identified that women are highly under-represented at ICLR ; in 2019 , women made up 23.2 % of all CS PhD students in the US ( Zweben & Bizot , 2020 ) , while only 12.1 % of first authors at ICLR from US universities were women . We find that this disparity widened in 2020 , with women compromising just 10.62 % of first authors and 10.77 % of last authors . For comparison , women comprised 23.4 % of enrollees in doctoral CS programs and 24.8 % of computer science , computer engineering , and information programs combined ( Zweben & Bizot , 2021 ) . There are many complex social and economic factors that likely contribute to the underrepresentation of women in computer science , including confidence gaps between men and women in STEM classrooms ( Lubienski et al. , 2013 ) , stereotypes of skill levels and gender roles in academics ( Lubienski et al. , 2013 ) , differences in workplace treatment between men and women ( Funk & Parker , 2018 ) , and even differences in income after graduation ( Xu , 2015 ) . However , the representation disparity at ICLR is strikingly more lopsided than what is observed in the computer science community at large , which indicates community-specific factors that influence representation . In this section , we consider various hypotheses for why such a strong disparity in representation might exist at ICLR . While there may be a range of sociological factors at play that are outside the scope of our study , we focus on factors that can be analyzed through publicly available data . Throughout this section , we focus on the likelihood of women returning to ICLR after appearing once . We focus only on the return rate of first authors . This is because last authors are unlikely to be first time attendees and often have long professional careers in ML that make them likely to return to the conference regardless of the factors identified below . 3.1 RETENTION . We observe that ICLR has somewhat more difficulty retaining women participants than men . Among first-time conference attendees in 2020 , of the women who appeared as first author on a paper , 19.79 % returned in 2021 , compared to 23.83 % for men . Expanding to account for all 2020 attendees regardless of their number of appearances , this mild discrepancy remains ; the return rate for women was 28.12 % compared to 31.64 % for men . As for why this discrepancy exists , the difference in return rate could be explained in part by outcomes in the review process . We hypothesize that authors receiving strongly negative feedback on their contributions may be less likely to return to the conference and that the relatively lower scores women receive make them less likely to return . Tran et al . ( 2020 ) reported that papers from women first authors on average scored 0.16 points lower than men in 2020 . Our 2021 data shows a present but smaller gender gap with women first authors on average scoring 0.13 points lower than men first authors ( p = 0.03 ) . This gender discrepancy is clearly visible in the score histograms in Figure 1 . There is no discernible gender gap for last authors . ( a ) First Author ( b ) Last Author This hypothesis is indeed supported by the data – review scores strongly correlate with retention rates . A logistic model predicting author return probability as a function of mean reviewer score and gender identifies reviewer scores as a strong factor in predicting author return ( p = 0.015 , Table 1 ) . This trend is a significant factor within both the men and women sub-populations ( see Table 5 in the Appendix ) . In a separate test , women with rejected papers had a return rate of 24.28 % while men with rejected papers had a return rate of 30.28 % ( p = 0.073 ) . Research on small minority peer groups in STEM fields ( Etzkowitz et al. , 1992 ; Rosser & Lane , 2002 ) suggests that the effect of reviewer scores may be stronger for women than for men , however we did not detect this as a significant factor in Table 1 ) . 3.2 AMOUNT OF PROFESSIONAL EXPERIENCE . Negative reviewer scores are commonplace in the machine learning community at large . One may expect experienced authors to be well acquainted with this fact , and therefore less strongly impacted by negative scores than those authors receiving their first ever round of reviews . In this section , we quantify how author experience correlates with the return rate of first authors . We consider the relation between professional experience and first author outcomes , in addition to last author ’ s experience level . We identify “ first-time ” authors as people who did not submit a paper to ICLR in 2017-2019 . Note that we use “ first author ” to refer to author order , and “ first time author ” to refer to lack of prior submissions . We find that first time authors are less likely to return to ICLR than other authors ( p < 0.001 ) . This effect is still highly significant even after controlling for reviewer scores . Women first-time authors return with a rate of 20.36 % , while men first-time authors return with a rate of 24.54 % . The observations in this study are applicable to many attendees , as 65 % of women and 63 % of men are first-time authors . We also consider the number of publications an author has and how it correlates with their return rate . After controlling for reviewer scores1 , we find that the number of publications of the first author was correlated with return rate , but not to a statistically significant degree ( p = 0.143 , Table 6 ) . We find that the number of citations is not a good indicator of return rate , possibly because it varies wildly for young researchers .

Summary: This paper presents a study on the relationship between demographics and the submission and review of papers to ICLR, leveraging publicly available data from OpenReview.  The authors perform a substantial manual annotation effort on this dataset, and then study how paper acceptance and "retention" (i.e., whether authors submit to the conference again the following year) are related to gender and geographical location, by attempting to control for various factors. Topic, industry affiliation, experience level, and the use of theorems are also examined.