Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The is a data analysis study on ICLR submissions from 2017 to 2021. Several variables such as gender, location, prior experience, and topic of the papers is considered. The study is mainly univariate and tries to find meaningful patterns for inferring representation of minorities at ICLR.
main_review: Positive aspects:
-	The observations are interesting, specially for the organizers and senior members of ICLR.
-	The methodology and analysis is clear and easy to follow.


Negative aspects:
-	The paper is not well-motivated and does not seem to fit within the general scope of ICLR (see more below).
-	The analyses are univariate and possibly misleading (see more below).
-	The study is non-conclusive and lacks a coherent structure.
-	No in-depth study of the observations is provided and the results raise more questions than answers.



Main Comments:

-	The main problem with the study is the focus on univariate analysis. The phenomena in social sciences often involve more than one variable and a univariate analysis gives a biased and often misleading answer. A notable example is the work by Blau and Kahn: 
Blau, Francine D., and Lawrence M. Kahn. 2017. "The Gender Wage Gap: Extent, Trends, and Explanations." Journal of Economic Literature, 55 (3): 789-865. 
For drawing meaningful conclusions, I encourage authors to perform a multivariate analysis on all the variables considered in this study and possibly more variables drawn from similar studies in social sciences.
-	The study is not well-motivate and it is not clear (1) how it fits withing the scope of ICLR, and (2) what conclusion should be drawn from the multiple correlations found in the study. The analysis done in this paper is narrowed to ICLR data only and does not relate to representation learning and broader applications. The paper is not conclusive and is built as a set of seemingly independent correlation-based analysis without a coherent structure.
-	There are no deeper analysis of the observations. For example, when a certain bias exists (assuming a correct analysis), what are the underlying dynamics that lead to these biases. Considering the anonymity of the reviewing process, in many cases where the authors found there is a gender bias in the accepted papers (per topic), the question arises that how can a gender “bias” appear in this kind of reviewing process. Isn’t this just a byproduct of the small dataset and even smaller subset of female authors within the dataset?
-	I find the justification behind considering only the first and last authors for the mentorship analysis rather poor. The papers submitted in CS venues are more often than not the product of more than one team’s effort in which the senior heads of each team do not necessarily have mentorship relationship with the first author of the paper (often times a student). I think authors should consider the affiliations, experience, and gender of all authors of a paper and draw conclusion on the mentorship. Also, there can be more than one mentor (e.g., two senior authors with the same affiliation as the first author). The same goes for attributing a location to a paper; last author is not a good representation necessarily. A better approach is to make a new class of “multi-national” papers. 
-	In Section 3.1, authors study data from 2020 and 2021 to infer the statistics for first authors who came back the next year. The conference in these two years was virtual and, quite possibly, an anomaly in terms of the behavioral patterns. Authors should consider previous years as well. 

summary_of_the_review: I evaluate the paper as a reject for the following reasons:
-	The paper is not well-motivated and does not seem to fit within the general scope of ICLR (see more below).
-	The analyses are univariate and possibly misleading (see more below).
-	The study is non-conclusive and lacks a coherent structure.
-	No in-depth study of the observations is provided and the results raise more questions than answers.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims to quantify to some degree how gender, country of origin, and paper topic impact the review process, decision to submit, and paper review outcomes at ICLR, using data from 2017 to 2021. The authors report several interesting outcomes, including a lower return/retention rate for women attendees of ICLR (as well as authors with low review scores), higher scores for papers of Western origin, the increased likelihood of theorem-containing papers to be accepted, and the higher rate of acceptance for industry papers.
main_review: STRENGTHS:
- This paper studies an important topic regarding how conference (here specifically ICLR) submission/score/acceptance is correlated with various aspects that are in and out of authors' control, including their choice of topic, gender, location, etc. This has implications for how conferences can be designed to be more inclusive, guide scholars toward more friendly/specific venues, and how as a community we may re-think the assignment of prestige/importance to conferences.
- The authors examine several cross-variable impacts, including gender/topic, experience/geographic region, and gender/experience. It would be interesting to measure how impactful such variables are compared to each other, perhaps via some sort of ANOVA or recursive elimination feature importance assessment, or the proposal of a method for measuring confounding/combinatorial impacts of each of the studied variables.

WEAKNESSES:
- It would have been interesting to see how gender representation could have changed if non-first/last authors were also labeled with their gender identity, although I recognize this may have been too expensive to label.
- It is very difficult in general to make strong comments about the impact of various demographic factors on conferences and the review process, and while the authors propose several important insights and hypotheses to explain differences in e.g. return rate of women, it is difficult to establish how likely these hypotheses are (or to establish causality)
- It would be interesting to see how the authors account for the self-selection bias of the fields/organizations that submit to or have members who attend ICLR, and perhaps compare this to other conferences (including those that are more specialized to specific fields like ACL, as well as other more general conferences like NeurIPS).
- There is not much discussion of possible reasons for the scoring difference among geographic regions; it would be nice to see more clarity around what causes could be hypothesized and what data the authors believe is missing/can be collected to test such hypotheses.
summary_of_the_review: This paper touches on the very important topic of investigating explanatory effects of demographic and meta-factors in conference attendance, submission, and acceptance. This is an extraordinarily wide mandate, and the paper attempts to cover a fair amount of ground - this also results in several intuitive, unanswered questions when reading. Despite the opportunity for deeper analysis, I believe this paper is valuable in understanding representation at conferences (specifically ICLR) and provides some starting points for improving the same.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a study on the relationship between demographics and the submission and review of papers to ICLR, leveraging publicly available data from OpenReview.  The authors perform a substantial manual annotation effort on this dataset, and then study how paper acceptance and "retention" (i.e., whether authors submit to the conference again the following year) are related to gender and geographical location, by attempting to control for various factors. Topic, industry affiliation, experience level, and the use of theorems are also examined.
main_review: Strong points:

-The peer review process in machine learning, and diversity and/or bias issues in CS/AI/ML, are both crucially important issues that are in dire need of research.  This work addresses both of these issues simultaneously, and is therefore likely to obtain substantial attention.

-This work studies not only the impact of gender and geographic location, but also industry affiliation, the use of theorems, topics, experience level, etc.

-Substantial and labor-intensive manual annotation of ICLR authors' gender was performed in order to execute the study. The authors chose to perform this work to get more reliable results, instead of simply relying on automatic tools which are known to have a bias toward performing better on Western names.

-Attempting to control for various factors, due to the observational nature of the dataset, makes the analysis much more rigorous.

-There are several very interesting findings in the research, which could have real-world impact on the peer review process, and on the academy at large. For example: ICLR authors are more likely to return or have their paper accepted if their mentor is the same gender, larger ICLR teams have higher review scores on average, women tend to have larger co-author teams than men.

Weak points:

-The precise relationship to the work of Tran et al. (2020) is unclear, e.g. what are the methodological differences and the novel findings? (Note, if this paper turns out to be by the same authors as the Tran et al. arXiv paper, and is actually a conference version of that same paper, its significance would be higher. I cannot assess this due to the double blind.)

-It is a complicated question as to whether this paper is a good "fit" for ICLR.  On one hand, this work does not have the methodological novelty we would expect of an accepted ICLR paper, and this type of research is not normally published at this venue.  On the other, since it studies the ICLR conference itself, its findings are clearly of interest to the ICLR community.  Furthermore, it would behoove the ICLR community to be open to submissions such as this which are "outside the box" but still insightful.

-Arguably, a journal format would be more appropriate for this work, as it would provide more space for a thorough analysis. Then again, there is no journal which is directly affiliated with the ICLR conference, so this would have disconnected the paper from the ICLR community.

-I would like to have seen a more sophisticated level of analysis which leverages techniques developed by the ICLR community, and from machine learning at-large.  For example, NLP methods including topic models, word embeddings, and/or BERT could have contributed to the analysis.  This work only scratches the surface of the analyses on this dataset which are possible when leveraging the authors' gender annotations.

-The paper could be strengthened by a deeper investigation into whether the disparities identified are due to factors relating to systemic bias, sexism, racism, etc.  Similarly, the paper could have delved deeper into the extent to which English language capabilities are the reason for the disparities, e.g. by reporting results when controlling for the percentage of English language speakers per country.

-The potential impact of COVID-19 on the population under study should have been discussed.

-Since much of the analysis hinges on annotations, it would be appropriate to report inter-annotator agreement metrics.



Questions to the authors:

-Why is it particularly important to perform this study on the ICLR conference (other than the fact that the data are available on OpenReview)?



Additional feedback / minor suggestions:

-Pg 3, the phrase "limited evidence" is a bit ambiguous as it is unclear whether authors are saying there is evidence (the data suggest the statement is true, but the evidence is limited), or there isn't evidence (the data suggest the statement is false, but the evidence is limited)

-Pg 3, use parenthetical citations (\citep) for "Etzkowitz et al. (1992); Rosser & Lane (2002)"

-Pg 5, "increase monotonically with number of authors" (missing "the")

-Pg 6, "papers from East and South Asian" (Asia)

-Pg 7, "women did doing better than men" (remove "doing")

-Pg 9, "as a function 3 industry indicators" (of)

-Pg 9, "10.62%" (missing parentheses around the number)

-Pg 9, "mentorship is import" (important)

-The authors should mention that this work assumes a gender binary, and that future work could and should study nonbinary and transgender populations.

-Once this work is published, it would be extremely valuable if the authors could make their annotated dataset publicly available.
summary_of_the_review: This paper does not fit the typical mold of an ICLR paper, in that it is mainly empirical work rather than presenting novel methods.  As always, more analysis could have been done.  However, its findings regarding participation and review in ICLR are valuable to the ICLR community, and it would likely obtain substantial attention in the literature and have an impact on the machine learning community at large.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper conducts an observational study of disparities in the ICLR 2017-2021 reviewing process. The work centers around the analysis of disparities in reviewing scores and acceptance rates across gender, number of authors, countries, papers topics, and industry/academia. The authors present several findings that are likely of interest to the community and provide some potential explanations behind some of such findings.  
main_review: The exploratory data analysis and regression results presented in the paper contain some interesting results. Although the authors do not provide explanations behind each of these results, it might be helpful to raise awareness of these patterns in the community. At the same time, most of the findings from this exploratory analysis overlap with those of (Tran et al., 2020), thus I am unsure of how novel the authors' contributions are, besides presenting an updated analysis.

As the authors correctly note in the Ethics Statement, this is an observational study. All regression models control only for the reviewers' scores, thus in many instances it's unclear what kind of effect, if any, these models are actually capturing. The paper contains statements such as "probe the effects of paper topic on the review process", "review score impacts retention", "how it impacts their return rate", "we remove the effect of reviewer score by controlling for this variable, there is still a small residual effect of gender" etc. These regressions simply uncover associational patterns and thus causal statements should be avoided.

About Section 3.1: 
* In Section 3.1, the histogram on the left of Figure 1 would suggest a large difference in scores between men and women, so I found the fact that the difference was only 0.13 surprising. Is this difference of 0.13 based on the data whose distribution is displayed in the histogram?
* Was any residual diagnostics analysis for the logistic regression models conducted? All models are simple (i.e., they include only a few variables) so it should be rather straightforward to check whether the assumptions are met (e.g., by using the diagnostic plots presented here https://arxiv.org/pdf/1612.03257.pdf)
* In the regression of retention ~ reviewers scores+gender, how were observations of individuals that appeared in multiple papers as first authors handled? These observations are dependent and thus it might be a good idea to cluster the standard errors by the first author
* Are these results robust to changes in the way that reviewers scores are taken into account? e.g., if the most negative score instead of the mean score is accounted for, do the results still hold? 
* Past work has shown that type I error may not be controlled for when running logistic regression on peer-reviewed data (https://arxiv.org/abs/1912.13188), so this issue should be acknowledged
* Based on Table 1, we cannot really say much about whether the difference between the coefficients in the two regression models is statistically significant. The most straightforward way to make some inference on that would be to fit a single model of retention ~ intercept + I(is woman) + reviewer score + reviewer score * I(is woman) and do a Wald test on the last coefficient. That said, the difference in the  probabilities predicted by the current models are very small
* some of the concerns and suggestions raised above apply also to the remaining sections of the paper


About Section 3.2:
* "we find that the number of publications of the first author had a positive impact on return rate (p = 0.143, Table 6)". This p-value is above any of the significance levels that are generally considered (0.001, 0.01, 0.05, 0.1), so I would interpret this coefficient (and not "impact") as not being statistical significant. I read that "borderline p-value 0.11", so I assume that the significance level the authors adopted is 0.1.

Minor details:
* some of the summary statistics that are reported contain two decimal digits, while others only contain one
* the manuscript contains some typos (e.g., "These data was")
* Section 1: In (Zweben & Bizot, 2021) I could not find the information about women comprising 23.6% of the enrollees in computation graduate programs
* Section 2: Perhaps it might useful to mention that the papers included in the dataset represent almost (but not) all of the submissions to ICLR in the period considered
* Captions of figures 3-7 need to be improved. Table 1 needs to be improved too, e.g., by adding an additional column on the left containing the labels "men" and "women"
* It would be helpful if the authors could provide the code to replicate these results
summary_of_the_review: The analysis presented in the paper contains some interesting results. However, many of the findings overlap with those of (Tran et al., 2020), which conducted a similar analysis. In addition, I raised several concerns and questions on the methodology used in the paper.
review_title: null

Meta-review: In this paper, the authors present an investigation of the impact of demographics on the peer review outcomes of ICLR. This is an important topic, as the demographics of ICLR and similar conferences are seriously skewed and may cause some people to feel excluded. The authors look into this complex problem with extensive manual annotations and analyses. 

The main weakness of this paper is that it is observational, and while the results are interesting, it is difficult to take away a clear and convincing message for the future. Part of the reason is that the whole problem is quite complex, and the hypotheses that are presented and tested in this paper reveal relatively shallow findings. Compared to the NeurIPS experiments which are carefully designed, these are not causal (see one of the reviewers' comments), so it is difficult to draw conclusions beyond correlations.

In summary, the results are interesting, and despite some of the reviewers' concerns, I would not exclude this paper because of the topic being irrelevant to the cfp, but I think the paper needs a more clear and convincing message.
Decision: Reject