Evaluating multiple candidates simultaneously reduces racial disparities in promotion and tenure

Domain: chemistry
Published: 2026-02-23
Editorial outcome: major_revision

Abstract
Black and Hispanic faculty–underrepresented minorities (URMs) within academia–face career barriers that come to a crux in promotion and tenure decisions. Leveraging a natural experiment in choice architecture within a dataset of 1804 promotion and tenure decisions across six universities, we find that joint (906 faculty) vs. separate (898 faculty) evaluation reduces racial disparities in faculty outcomes. Specifically, in joint evaluation, an analysis of the simple slopes finds that Black and Hispanic faculty receive, on average, 9% fewer negative votes at the department level than in separate evaluations when controlling for research productivity, school, gender, rank, discipline, department size, and grant acquisition. Using moderated mediation analyses, we calculate that this translates into a 16.2% increase in the likelihood of a Black/Hispanic faculty member receiving a promotion. In a survey of 289 professors who have served on promotion and tenure committees (i.e., the key P&T decision-makers), we find that only 17% of faculty expect joint evaluation to improve underrepresented minority faculty outcomes and, conversely, 43% expect separate evaluation to improve underrepresented minority faculty outcomes. This natural experiment suggests that altering evaluation mode or simulating joint evaluation mode could help address academia’s underrepresentation problem, but not in the way decision-makers expect.

Introduction
Black and Hispanic individuals comprise 31% of the US population but only 11% of faculty1 and are thus defined as underrepresented minorities (URMs) within academia. This number decreases as you go up the ranks, with URM faculty more widely represented as Assistant Professors (13%) than in Associate (11%) or Full Professor ranks (8%). The low levels of URM representation in tenured faculty roles have been linked to racial disparities in the promotion and tenure process and to the notion of gatekeeping2–4, where groups of high-powered decision-makers decide on the career progression of URMs. In academia, these gatekeepers are senior faculty who serve on promotion and tenure (P&T) committees and wield high power over junior faculty members’ scientific careers and contributions. Although these gatekeeping decisions present as meritocratic, recent research has shown that - while accounting for scholarship, discipline, grant funding, and more - URM faculty receive a significantly lower proportion of positive votes relative to their non-URM (White or Asian) peers2. These racial disparities are damaging to science and, thus, society, as a more diverse academy is linked with increased innovation5 and research productivity6. The goal of this paper is to examine how choice architecture could provide a pathway to addressing these disparities.

Policy makers and academics have noted the need for policy development and implementation evaluation to be grounded in rigorous empirical studies7,8. To accomplish this, there is a need for more experimental and quasi-experimental field research to test interventions that can help mitigate inequities. Moreover, these interventions need to be tailored to the context in which they are being employed. Existing policies to address racial disparities in organizations face a number of challenges in the context of promotion decisions. Some policies are grounded in research that suggests removing race from the equation by conducting a blind-evaluation process in which the race of the applicant is hidden from the evaluators9. These approaches appear to be supported by evidence but are not feasible in the majority of promotion cases where the decision-makers are already aware of the candidate’s identity. Another policy has been to balance the systemic disadvantages faced by URM candidates through identity-conscious policies such as affirmative action, in which identity characteristics are acknowledged to reduce discrimination against URM candidates and promote equal educational and employment opportunities10–13. However, despite being effective in increasing representation, such approaches have been subjected to legal and ethical challenges-particularly in academia-and are challenging to implement and sustain. Another widely used group of policies seeks to broaden diversity among decision-makers, again a practice grounded in empirical evidence14 but limited in application since it requires URM faculty to partake in significant institutional service. Yet other approaches aim to address the decision makers’ perceptions of race through diversity or debiasing training15. Although such interventions have been implemented broadly across management, a recent meta-analysis of 492 studies on implicit bias training concluded that “changes in implicit measures are possible, but those changes do not necessarily translate into changes in explicit measures or behavior”(p.1)16. In sum, although numerous approaches that could inform policy change and address racial disparities have been tried out in the real world, in academic settings, the effective approaches appear to be unfeasible, and the feasible approaches appear to be ineffective.

Through this paper, we propose to leverage choice architecture principles as a foundation for a novel set of policy recommendations. Such approaches have been effective in tackling gender bias in organizations17,18, particularly when compared to the mixed effects of more frequently used approaches like diversity training15. Choice architecture refers to the design of decision-making environments, with nudges referring to when small contextual changes impact how choice options are evaluated. Leveraging choice architecture, in the form of nudges, has been discussed as an easy-to-implement policy intervention to address racial disparities in P&T; however, these ideas have yet to be tested empirically19,20. In this paper, we examined whether evaluation mode - if the URM promotion candidates were evaluated in isolation (separate evaluation) or simultaneously with other candidates seeking promotion (joint evaluation) - can mitigate racial disparities in the P&T process.

Racial minorities face disadvantages at multiple stages in their career. Partially, these disadvantages can be linked to persistent negative stereotypes about URMs in the workplace21. According to the theory of shifting standards22,23, these negative stereotypes lead to double standards, where URM employees are judged to a higher standard compared to non-URM employees2,22,24. This can result in candidates with similar attributes being evaluated differently based on their race22,23; for example, for a URM employee, receiving a poor performance review one year may mean they do not get a promotion, but for a non-URM employee, they may be given the benefit of the doubt. This is exactly what has been observed in past research on P&T decisions in academia2. URM faculty are held to a higher standard when it comes to performance on traditional research metrics (h-index), with non-URM faculty being given the benefit of the doubt for low performance, but the same standard is not being applied to URM faculty. The question this paper investigates is whether the negative stereotypes and double standards are amplified or mitigated in joint evaluation.

To do so, we draw from research on behavioral decision theory. This stream of literature has examined joint vs. separate evaluation modes in forced-choice paradigms-i.e., selecting between two options-predominantly in the domain of marketing25. However, the central hypothesis is very relevant to promotion decisions. According to this theory, in separate evaluations, individuals rely more on surface-level information-i.e., information that is easy to interpret-and less on deep-level information26,27. To demonstrate this, consider that you are evaluating a female candidate on their leadership skills based on their past work performance. Without any comparisons to anchor performance standards (i.e., separate evaluation), it might be difficult to determine if this employee’s performance is below average, average, or above average on leadership because evaluators lack a reference point. As such, evaluators are more likely to rely on surface-level information, such as a candidate’s gender, and implicitly use gender-based stereotypes related to leadership28. Alternatively, in joint evaluation, evaluators do have a reference point because they can compare the two (or more) options to each other. This makes it easier to conceptualize the hard-to-evaluate information and be more deliberative in the evaluation process. This is what was found in the laboratory work by Bohnet, van Geen, and Bazerman26. Specifically, when predicting the future performance of a hypothetical employee, in a separate evaluation, decision makers relied more on gender stereotypes (surface-level information), whereas in a joint evaluation, they relied more on past performance (deep-level information).

Through reducing the role of stereotypes in evaluations, we expect that joint evaluation will disrupt the use of double standards. A strong motivation in decision-making-and cognition more broadly-is to be logically consistent29,30. However, as is evidenced from double standards in the P&T process2 and many other cases22,23,31,32, decision-makers do not apply the same process consistently. Nevertheless, we argue that decision-makers are more likely to be consistent when evaluating two candidates jointly than when making these evaluations separately. This is because, in joint evaluation, the discrepancies in decision processes (i.e., the shifting standards22) will be more salient than in separate evaluation. Thus, from this theoretical perspective, we derive the hypothesis that joint evaluation will reduce racial disparities in promotion decisions.

That said, we also acknowledge the competing perspective that joint evaluation would worsen outcomes for URM faculty. This perspective is aligned with general social psychological theories of perception, that race is more salient when individuals are in a group (2 or more) than when they are alone33,34. For example, if a decision maker is evaluating a P&T candidate individually, their most prominent social feature might be that they are a faculty member; however, if they are evaluated with someone of a different race, it will become more salient that they are a Hispanic faculty member and that the other candidate is a White faculty member. Note, in a previous version of the manuscript, we formally hypothesized this competing perspective; however, on the suggestion of a reviewer, we decided to focus on a single hypothesis, which, based on the stronger theoretical rationale for the role of evaluation mode in the decision-making literature, was that joint evaluation would reduce racial disparities.

We examine this hypothesis in the context of P&T decisions. Due to the extremely low representation of other racial minority groups (N=2; e.g., Native Alaskan, Pacific Islander, and Native American), we operationalized URMs as Black and Hispanic faculty and non-URMs as White and Asian faculty. The dataset includes a diverse set of six US public and private institutions, varying in size (~10,000 to ~80,000 students), from across four states, and ranging in their US News national university ranking (rounded to the nearest ten universities rank as 50th, 50th, 130th, 170th, 170th, and 390th in the nation). Promotion policies can vary across institutions35,36, however, in general, they follow a similar approach. Faculty are required to go up for promotion after a certain number of years (i.e., tenure clock), though they can go up early if they elect to, and they can receive an extension if their university permits (e.g., birth of a child). Once faculty have chosen to go up for promotion, most universities’ faculty affairs units follow a rigid timeline determining, for example, when the candidate must submit their promotion portfolio and by when external review letters must be completed. Within this context, we define joint evaluation as when more than one candidate is being evaluated at the department-level within this allotted window. In other words, there is more than one P&T candidate going up for promotion to the same rank, in the same department, in the same promotion cycle. The exact timeline for P&T varied across the six institutions in our sample; however, they each allotted around one month for P&T decisions to be made at the department level (see Table S1). Therefore, when multiple candidates were evaluated in the same year, they will both need to be considered within this time window. Notably, this is a less rigid designation of joint evaluation compared to past laboratory studies where joint evaluation occurred at exactly the same time and was a forced-choice between two options25, that said, the proximity of the evaluations still suggests that decision makers will use the candidates as a reference point for each other26,37.

Results
Results: Forecasting study
Prior to examining the role of evaluation mode in P&T decision making, we conducted a forecasting survey to examine gatekeepers’ (i.e., professors’) lay beliefs about how evaluation mode would influence racial disparities in the P&T process. After all, for policy to be implemented, it is important to know where key decision-makers stand on the issue to see if they need validation or convincing.

This project surveyed professors (N=285) who had served on promotion and tenure committees before to determine whether their beliefs about evaluation modes’ relationships with unfavorable evaluations of URM candidates were correct. A histogram of the results is presented in Fig. 1. As seen in the graph, the modal response was that the evaluation mode had “no effect”. However, the data was skewed towards the left-hand side of the scale (i.e., that separate evaluation would reduce racial disparities), with 42.81% of participants predicting that separate evaluation would be “slightly” or “substantially” better; alternatively, only 17.19% predicted joint evaluation would be “slightly” or “substantially” better. In support of this, running a one-sample two-tailed Welch’s t-test, we found that the mean of 2.56 (SD=1.13; 95% CI [2.43, 2.69]) was significantly below the midpoint of 3, (t[284]=−6.59, p<0.001, d=0.78), supporting the conclusion that on average professors held the belief that separate evaluation would be better.

In sum, gatekeepers’ lay beliefs are misaligned with our central Hypothesis with only a minority (17.19%) of participants expecting that joint evaluation would reduce racial disparities, and the vast majority of participants (82.81%) either believing that evaluation mode would have no effect or that separate evaluation would reduce racial disparities. It is particularly notable that the most common response from gatekeepers was in line with the competing hypothesis, that separate evaluation would reduce racial disparities. The contrast between this lay hypothesis and our initial hypothesis drawn from behavioral decision theory sets up the natural experiment as an empirical test to resolve the two contrasting perspectives.

Results: Natural experiment
No evidence of self selection
Given the high-stakes nature of promotion and tenure (P&T) decisions in academia, it is not feasible, or arguably ethical, to randomly assign candidates to either a joint or separate evaluation process. Therefore, we observed the effect of natural variation in evaluation mode across candidates to test the proposed competing hypotheses. We define this variation as quasi-random, meaning that the allocation to condition was not random but mirrors that of an experimental design, as the allocation to condition is unrelated to the dependent variable38. Quasi-experiments are powerful designs because they allow tests of causality in real-world settings39–46. In order to justify our classification of this as a quasi-experiment, we delve into what determines evaluation mode in the P&T process.

One threat to the integrity of a natural experiment is if candidates or decision-makers could self-select into an evaluation mode44. For example, if URM candidates believed there would be benefits of joint or separate evaluation and self-selected into one of these groups. However, given the structure of the P&T process, this is very unlikely. Within the university system, the time a candidate is granted before being evaluated for P&T is largely predetermined by university policy. For example, many universities in the US have a 6-year tenure clock, and by the end of this time period, faculty must go up for P&T. Decision makers in the P&T process have no control over this and thus cannot exert influence over a candidate’s evaluation mode. Candidates also have limited influence over the length of these time periods. A notable exception is that candidates could ask for an extension to this period due to extenuating circumstances (e.g., the birth of a child) or decide to go up for tenure early47. However, based on the results of a two-tailed Welch’s t-test, we see no evidence in our data that candidates were actively using these policies to influence evaluation mode, as individuals who were evaluated jointly vs. separately spent a similar amount of time in their previous rank (Joint: 6.06 years, Separate: 6.06 years; t(1744)=0.02, p=0.983, d=0.001, 95% CI [−0.30, 0.31]) and took extensions at a similar rate (Joint: 7.06%, Separate: 7.35%; t(1800)=0.23, p=0.815, d=0.01, 95% CI [−0.02, 0.03]). In fact, based on the results of two-tailed Welch’s t-tests, faculty in joint and separate evaluation were very similar in terms of gender (t(1802)=0.92, p = 0.356, d=0.04, 95% CI [−0.02, 0.07]), URM status (t(1739)=0.92, p=0.359, d = 0.04, 95% CI [−0.02, 0.04]), h-index (t(1212)=−1.43, p = 0.152, d=0.08, 95% CI [−0.19, 0.03]), university ranking (t(1800)=1.32, p=0.186, d=0.06, 95% CI [−2.19, 0.11.25]), and grants (t(1578)=−0.85, p=0.398, d=0.04, 95% CI [−1.05, 0.42]); see Table 2. The only significant difference across faculty observed was in total votes, with candidates in joint evaluation having more department voters (t(1780)=−3.92, p<0.001, d=0.18, 95% CI [−2.45, −0.82]). However, mathematically, this is to be expected as it is more likely that a large department will jointly evaluate a candidate than a smaller department because there are more untenured faculty each year. Note, this variable (along with all the others mentioned) was included as a control. Moreover, evidence from the forecasting survey shows that the modal perspective from faculty is that the evaluation mode has no effect on P&T outcomes (40.13%). In sum, within the confines of the P&T system, self-selection from the decision makers or candidates into evaluation mode is challenging, and examining the one exception (candidates seeking extensions), we find no evidence of self-selection in our sample nor clear expectation that one evaluation mode would improve their P&T outcomes.

Another threat to quasi-experiments is differences in environment between the two conditions44. For example, it is possible that joint or separate evaluation is more common at certain schools in our sample. To assess this, we examined the distribution of joint or separate evaluation cases across the six institutions in our sample. In the overall sample, the total number of candidates that were evaluated separately was 898 (49.78), and the total that were evaluated jointly was 906 (50.22%). Separated by institution, there are cases of joint and separate evaluation in all schools, with all schools having at least 43% of their decisions made in joint evaluation mode; see Table S2 for means and SDs of all study variables by institution. Further, separated by academic discipline, there are also cases of joint and separate evaluation in all disciplines with more than 3 cases. This distribution suggests that there were no major differences in the environment in which joint vs. separate evaluations took place. In addition, addressing another threat to the quasi-experimental design, the roughly even distribution of cases across institutions and disciplines means that there are no systematic differences in P&T policy across the conditions.

To conclude, based on the definition of Shadish and colleagues38 and past quasi-experimental work by Talhelm and Dong44, we classify the distribution of evaluation mode across P&T cases as quasi-random. As such, analyses will provide a causal test of the role of evaluation mode in P&T decisions38–43,45,46,48. Further, following the best practice recommendations for quasi-experiments38,40,45,46,48, we strengthen internal validity by running a model including control variables to account for potential confounding variables49. These controls are detailed below, and justification for each variable is provided in the method section35.

Hypothesis tests
The central hypothesis of this paper is that joint evaluation will reduce racial disparities in voting outcomes for faculty, with the competing hypothesis derived from the forecasting survey that separate evaluation will reduce racial disparities. To do so, we use two operationalizations of voting outcomes at the department level. The first is the negative vote percentage (continuous variable), and the second is whether the vote was unanimous or not (binary variable). Results are reported for both sets of analyses below. Descriptive statistics are shown in Table 1, along with the subset demographics for candidates being evaluated jointly vs. separately (see Table 2) and URM vs. non-URM candidates (see Table 3).

Negative vote percentage. We regressed negative vote percentage on URM status, evaluation mode, and their interaction, with the interaction providing the critical test of the hypothesis. As shown in Table 4, analyses found evidence for an interaction effect (b=−0.10, SE=0.02, p=0.003, 95% CI [−0.16, −0.03]). Specifically, evident from Fig. 2, the difference in negative votes between URM and non-URM candidates (i.e., racial disparities), was greater in separate than joint evaluation.

To further examine the effect on negative vote percentage, additional models were run in a stepwise manner to rule out alternative explanations39; note that due to missing data (see Table S3), these models also reduced sample size and, thus, statistical power. First, as shown in Table 4, a model (N=1535) was run that incorporated candidate gender, school code, CIP Code, promotion rank, and years in present rank, which replicated the original finding (b=−0.09, SE = 0.03, p=0.005, 95% CI [−0.16, −0.03]). Second, as shown in Table 5, results also replicated when candidates’ productivity (h-index and number of grants as PI) was included as control variables (N=1027), addressing the possibility that candidates in joint evaluation differ in terms of scholarly ability (b=−0.13, SE=0.04, p=0.003, 95% CI [−0.21, −0.04]). Third, as shown in Table 5, to address concerns that the observed effect may be due to differences in department size between joint and separate evaluation-e.g., joint evaluation and positive P&T outcomes are more likely in large departments-we controlled for the number of total votes as a proxy for department size (N=1535) and replicated results (b=−0.09, SE = 0.03, p = 0.005, 95% CI [−0.16, −0.03]). Finally, as shown in Table 5, we ran a model including all control variables in the same model (N=1027) and again replicated the results (b=−0.13, SE=0.04, p=0.004, 95% CI [−0.21, −0.04]).

To probe the shape of the interaction, a post hoc analysis of the simple slopes was conducted using the model with all the controls included (see Table S4). For URM faculty, going up for P&T jointly—as compared to separately—led to a 9% reduction in negative votes received at the department level (t(983)=-2.23, b=−0.09, SE=0.04, p = 0.026, 95% CI [−0.17, −0.01]). For non-URM faculty, there was a significant relationship in the opposing direction (t(983)=2.62, b=0.03, SE = 0.01, p = 0.009, 95% CI [0.01, 0.06]). Consistent with our hypothesis, joint evaluation reduced racial disparities leading to a reduction in negative votes for URM faculty.

Unanimous Votes. Similar analyses were conducted to investigate the effect of evaluation mode on whether the department level vote was unanimously positive (1) or not (0). A model without control variables found no interaction between URM status and evaluation mode (Wald χ²(1)=1.13, OR=1.08, p=0.289, 95% CI [0.94, 1.23]; see Table S5). The model including all the controls also did not find a significant interaction (Wald χ²(1)=1.88, OR=1.14, p=0.171, 95% CI [0.95, 1.36]; see Table S5). Similar results were found using a generalized linear model (GLM) with a logit-link (Wald χ²(1)=0.04, OR=0.39, p=0.084, 95% CI [0.13, 1.14]; see Table S6). In sum, although the pattern of results was in a consistent direction, the results do not replicate when using unanimous votes as the dependent variable.

Provost Vote. Next, we examined the downstream effect of evaluation mode at the department level on the provost vote. At the institutions in this dataset the Provost vote represents the ultimate decisions on whether a candidate will receive a promotion (1) or not (0). Therefore, to contextualize our result, we calculated the conditional indirect effect of URM status on provost vote via department-level negative vote percentage, depending on whether candidates were evaluated jointly or separately. In other words, how did the benefits of joint evaluation for URM faculty at the department level translate into likelihood of receiving a positive provost vote. Moderated mediation analyses found support for a conditional indirect effect (Index of moderated mediation=0.61, p=0.025, 95% boot CI [0.15, 1.21]); see Table S7), meaning that the interaction between URM status and evaluation mode had downstream effects on provost vote via department level voting. We also examined the conditional main effect of URM status, i.e., regressing provost vote on the interaction between URM status and evaluation mode. The interaction term from these analyses was in the expected direction but was not statistically significant (b=−0.03, SE=0.04, p=0.460, 95% CI [−0.12, 0.05]). Post-hoc tests found that, for a URM faculty, being evaluated jointly lead to a 16.2% increase in the likelihood of receiving a positive provost vote.

Robustness checks
We conducted a series of analyses to ensure our results were robust to different model specifications. First, we conducted analyses with slight model modifications. We started by clustering the standard errors, replicating results when standard errors were clustered on university (b=−0.13, SE=0.04, p=0.002, 95% CI [−0.21, −0.05]) and department (b=−0.13, SE=0.05, p=0.009, 95% CI [−0.22, −0.03]; see Table S8). Testing another slight model modification, we ran a model with university-by-CIP fixed effects. We identified 103 University-by-CIP cells and ran the fixed effect finding results consistent with the OLS results (b=−0.12, SE=0.04, p=0.037, 95% CI [−0.23, −0.01]; see Table S9). Next, we ran a model relaxing our exclusion criteria. In the analysis plan we excluded Native American, Native Alaskan, Pacific Islander from analyses. Our initial rationale for not including these individuals was due to the small sample (N=2) and the precedent set by other research by other teams using this dataset (e.g., Masters-Waage et al. 2). As a robustness check we conducted analyses including these individuals and replicated our results (b=−0.12, SE=0.04, p=0.004, 95% CI [−0.21, −0.04]; see Table S10). Finally, due to the high amounts of missing data for h index and grants (see Table S3), we conducted analyses using a missing indicator approach by creating dummy variables for external grants and h index with missing values. Again, the interaction term between URM status and evaluation mode was significant (b=−0.10, SE=0.03, p=0.003, 95% CI [−0.16, −0.03]; see Table S11).

A second set of robustness checks focused on addressing model assumption violations. First, the data is highly-skewed with 75.5% of candidates receiving no negative votes at the department level. We also conducted a Breusch-Pagan test to examine whether the regression violated the homoscedasticity, and results showed that the data has a heteroscedasticity issue (BP index=37.106, df=9, p<0.001). However, VIFs (ranging between 1.04 and 2.23) and Durbin-Watson analyses (DW=1.97, p=0.324) showed that the data were free from multicollinearity and autocorrelation issues. To statistically address both the skew and heteroscedasticity issues we use a 5000 times bootstrap resample method to test the robustness of the results. These results replicated support for our hypothesis (b=−0.12, SE=0.03, p= 0.004, 95% boot CI [−0.20, −0.04]; see Table S12). In addition, to further address the issues related to skew, we ran a model including only candidates who received at least one negative vote (24.5% of the data). These analyses again replicated support for our hypothesis (b=−0.30, SE=0.11, p=0.011, 95% CI [−0.52, −0.07]; see Table S13), and increased the r-squared to 27%, which was higher than the model including candidates with no negative votes (r-squared=9%).

A third set of robustness checks sought to account for the unique structure of the data. Decisions in our dataset were made at the department-level, with the vast majority of departments making multiple decisions across the seven-year window of data collection. In the primary analyses we controlled for department (i.e., discipline) and treated each of these decisions independently. However, an alternative approach would be to treat the data as a panel by grouping candidates by department and looking at decisions made over time. To conduct these analyses, we transposed the current dataset into a panel data with departments within each University as ID and year as the time variable. The panel data includes 326 unique departments in 7 years. The first set of analysis examined whether there was significant difference in joint evaluation across universities and departments. The results showed that there was no significant difference in joint evaluation across universities (F(5,1798)=2.15, p=0.057). The Tukey post-hoc analysis showed that none of the pairwise differences between schools are statistically significant at the 0.05 level. Following on from this, we replicated the analyses using a panel data approach. We first conducted a Hausman test to determine the appropriate analysis to perform. The results of the Hausman test suggest using the random effect model (χ2=4.59, df=3, p=0.204). The results from these analyses replicated support for our hypothesis in a model including all the control variables (b=−0.12, SE=0.04, p=0.003, 95% CI [−0.21, −0.04]) and across all models conducted in a stepwise manner (see Table S14).

Supplemental Analyses
We ran a series of supplemental analyses to test the parameters of our theory. First, our theory generally proposes that joint evaluation reduces racial disparities at both the promotion to associate and full professor level, hence our decision to group both types of decisions together in our analysis. However, to test whether the effect differs across decisions, we conducted subset analyses focusing on promotion to associate and full professor separately. Results replicated at the promotion to the associate professor level (b=−0.13, SE=0.06, p= 0.040, 95% CI [−0.25, −0.01]; see Table S15) and the full professor level (b=−0.15, SE=0.07, p=0.025, 95% CI [−0.28, −0.02]; see Table S16), with a consistent interaction pattern between URM status and evaluation mode compared to the full-set analysis.

Another assumption of our theory was that the beneficial effect of evaluation mode for URM faculty would be consistent irrespective of the URM status of the faculty member they were being evaluated jointly with. To test this, we conducted analyses focusing on only cases in which joint evaluation occurred when URM candidates were being evaluated alongside non-URM candidates. Results from these analyses are in line with the conclusion above, finding that joint evaluation led to a lower negative vote percentage for URM faculty and an overall reduction in racial disparities (b=−0.12, SE=0.05, p=.006, 95% CI [-0.21, −0.04]; see Table S17). Next, we sought to conduct a similar analysis focusing only on cases in which joint evaluation occurred when URM candidates were being evaluated alongside other URM candidates. Unfortunately the occurrence of this was very infrequent (N=28), due to the underrepresentation of URM faculty in academia (roughly 8%) making it very unlikely two candidates would be evaluated simultaneously. Nevertheless we conducted the analyses and saw a similar pattern of results despite the interaction not being significant (b=−0.14, SE=0.08, p=0.069, 95% CI [−0.29, −0.01]; see Table S18).

A final supplemental analysis we conducted focused on the interaction between evaluation mode and gender (see Tables S19 and S20). Given that past research has found that evaluation mode can reduce biases experienced by women in the workplace26, we sought to replicate the results reported above using gender as the moderator instead of URM status. Analyses revealed no evidence for an interaction between evaluation mode and gender in terms of negative votes (b= 0.03, SE = 0.03, p=0.284, 95% CI [−0.02, 0.08]; see Table S19) nor unanimous votes (Wald χ²(1)=0.07, OR=0.99, p=0.793, 95% CI [0.88–0.1.10]; see Table S20). However, it is also important to note that analyses of the main effect gender of voting outcomes found no evidence for gender disparities for P&T voting outcomes at the department level (negative vote percentage, b=−0.01, SE=0.01, p=0.456, 95% CI [−0.03, 0.01]; unanimous votes, Wald χ²(1)=1.11, OR=1.02, p=0.292, 95% CI [0.98, 1.07]; see Tables S19 and S20). Therefore, the failure to replicate past work finding that evaluation mode reduces gender bias is likely due to the lack of evidence for a gender bias in P&T decision-making in the current dataset.

Methods
Forecastingsurvey
Participants. This project surveyed professors who had served on promotion and tenure committees before to determine whether their beliefs about evaluation modes and unfavorable evaluations of URM candidates were correct. The sample was collected from across 11 universities in the United States. As a convenience sample, no statistical method was used to predetermine sample size, and no exclusions were used. A total of 285 participants completed the survey, of which 77 were women, with the following race/ethnic breakdown: White (72.60%), Asian (11.74%), Hispanic (7.12%), Black/African American (3.91%), Native American, Native Alaskan, Pacific Islander (.71%), or Other (3.91%).

Ethics and Consent. This study was approved by the University of Houston Institutional Review Board (IRB400004725) and complied with ethical regulations. Consent took place online prior to participants answering the online survey questions.

Data collection. Participants were recruited via their institution's email address. Participants were informed that previous research has identified racial disparities in the promotion and tenure process and were then asked to provide feedback on an intervention to address this disparity. Participants saw the following question prompt:
“Within the promotion and tenure process, for about half of P&T cases, there is a group of two or more candidates seeking a promotion to the same rank in the same department in the same year (instead of candidates being the only ones in a department seeking a promotion to that rank during that year).
Do you think being evaluated as a group will lead to better P&T voting outcomes for URM faculty (right side of scale), do you think being evaluated individually will lead to better outcomes (left side of scale), or do you believe it doesn’t matter?”(middle of scale)
Participants then responded to this question on a 5-point scale ranging from “1” individual evaluation will be substantially better, to “3” evaluation mode will have no effect, to “5” joint evaluation will be substantially better.

Natural experiment
Participants. This project uses data from the Center for Equity in Faculty Advancement (CEFA) database66 which is a product of a collaborative consortium of ten American universities that are actively interested in studying promotion and tenure processes; used previously by Masters-Waage et al. 2. The present study uses data gathered on P&T decisions for 1,804 P&T candidates from 2015–2022. Data was collected from across six research-intensive institutions (with the other four participating institutions not contributing data or data not being usable due to reporting formats) and included all available data; therefore, no statistical method was used to predetermine sample size. The dataset includes decisions to promote faculty to associate professor with tenure and full professor 2. Most candidates were men (63%) and Caucasian/White (59%), and 48% of candidates were seeking Promotion to Full Professor.

Consortium Information. Our team built a consortium of universities to study promotion and tenure processes. In building this consortium, the goal was to bring together a diverse set of institutions in terms of size, research status, and ranking. In terms of student size, the consortium includes two smaller institutions (5000–15,000 students), three medium-sized institutions (40,000-50,000), and one larger institution (60,000+ students). The consortium primarily includes universities classified as Carnegie R1 institutions (i.e., very high research activity) but also one university classified as a Carnegie R2 institution (i.e., high research activity), providing variation of research activity. Further, whilst all institutions rank in the top 500 universities in the US according to the US News ranking, their position varies from 40th to 450th, with the median position being 157th. In addition, the set of universities includes two minority serving institutions, one designated as a Hispanic-Serving Institution (HSI) and another as a Historically Black College or University (HBCU). Finally, the six institutions span four US states providing geographical diversity.

All institutions were allotted approximately one month for P&T decisions to be made at the department level. Therefore, when multiple candidates were evaluated in the same year, they were both considered within this time window. For a full breakdown of the P&T timeline at each institution, see Table S1; note, the identity of the university was masked to preserve anonymity.

Ethics and Consent. This research complied with ethical regulations and, after review by the University of Houston Institutional Review Board (IRB), was declared to be non-human-subjects research (STUDY00002463; MOD00003374). IRB reliance agreements were then obtained from each consortium university. As described in the Data Collection section, each university’s Academic Affairs leadership coded and then de-identified data that was shared with the leadership team. Consent was not obtained directly from participants, as de-identified archival data was used.

Data Collection. The data used in this study were collected to answer a host of research questions around P&T decision-making by the Center for Excellence in Faculty Advancement (CEFA). A subset of the analyses was previously reported by Masters-Waage et al. 2 (see paper for data collection details). Due to the sensitivity of the data collected, our team worked with members of the provost’s office at each consortium partner institution to identify coders who already had access to confidential P&T data as part of their routine job responsibilities. These individuals, who followed a coding protocol developed by our team, coded institutional data relevant to the P&T process (e.g., voting outcomes) and candidate characteristics (e.g., discipline, race/ethnicity). To ensure the coding quality was high, we held bi-weekly meetings with coders to answer questions. In addition, senior administrators at each partner university cross-checked 2% of cases for coding errors. Note, neither the coders nor the administrators were directly involved in the P&T decision-making process at their institution. Anonymized coding sheets were then merged into a dataset by the research team with an identifier used to scrape additional productivity data from online repositories (i.e., publication statistics, grants, and patents). The identifier used for scraping is stored in a database separate from the dataset used in the analyses, ensuring that the final dataset includes no identifiable information. For more details on the privacy steps taken in the coding process, see Masters-Waage et al. 2.

Candidate URM status. Candidate demographics were recorded in candidates’ HR profiles (or an equivalent) and shared with us by the institution. Our partner institutions based their records on self-reported data that candidates provided (generally during the institution’s onboarding process), meaning race/ethnicity was self-identified. A dummy variable was created indicating if participants were from an underrepresented minority (coded as “1”) or not (coded as “0”), using the definition from the National Science Foundation67. The low number of individuals identifying as Native American/Native Hawaiian (0.4%; N=8) and Other (2.7%; N=48) makes it difficult to interpret the results for these ethnic groups; thus, they were not included in the analyses. Therefore, candidate URM status was coded as “1” for Black or Hispanic faculty and “0” for White or Asian faculty.

Joint (vs. Separate) evaluation. Joint evaluation is defined as when more than one P&T candidate is seeking promotion to the same role, in the same department, in the same academic discipline, and in the same year. A separate evaluation was any case where this was not occurring. The total number of candidates that were evaluated separately was 898 (49.78%), and the total number that were evaluated jointly was 906 (50.22%). The joint evaluation variable was converted into a dummy variable indicating “1” if the candidates were evaluated jointly and “0” if they were evaluated separately.

Voting outcomes. Promotion and tenure voting takes place within subsequent committee votes with department, college, and university-level committees, although some universities do not have separate college committees. The voting outcome variable used in this paper was committee voting at the department level (i.e., within the candidates’ department). Although the dataset does include voting at the college and provost level, the joint evaluation occurred at the department level; for the college and provost level, it is likely that there are people being evaluated at the same time every year, given that they review candidates from a much larger pool.

As part of the voting process, committee members review candidate materials and convene to discuss and vote on P&T candidates, recording the number of votes for (“yes”) or against (“no”) a candidate, as well as the number of abstentions. From “yes” and “no” votes, two voting outcome variables were used for each of the six institutions investigated within the consortium: negative vote percentage and unanimous vote. Both these outcome variables are found to correlate with the provost vote, which represents the ultimate vote of whether a candidate receives a promotion or not.

Negative voting percentage. At the department level, negative voting percentage refers to the percentage of “no” votes that a faculty member received during each voting level. Overall, negative vote percentage is a more sensitive measure of committee support and allows for committee decisions to be compared across candidates beyond a binary variable about whether promotion was granted.

Unanimous Votes. A candidate was marked as having received a unanimous vote (1) if they received all “yes” votes. If a candidate received any “no” votes, their voting outcome was marked as a non-unanimous vote (0). Abstentions were not factored into the coding of this dummy variable. Although unwarranted, unanimous votes are considered a gold standard to the extent that a non-unanimous decision can raise red flags when a candidate is reviewed by subsequent committees. Therefore, regardless of whether or not a promotion was ultimately granted, unanimous versus non-unanimous voting outcomes are a recognized distinction with real-world consequences.

Control variables. Given that the design of this study is quasi-random, the role of other confounding factors cannot be ruled out. Therefore, several control variables were included in the analyses to account for alternative explanations. As a baseline, analyses controlled for candidates’ (a) institution, (b) discipline, (c) gender, (d) rank they are being promoted to (associate/full), and (e) years in current rank; note, all these variables were coded by individuals in the Provost office at each institution. In addition, other theoretically relevant variables, including scholarly productivity and department size, were included as controls.

Scholarly productivity. To account for the possibility that there are differing levels of scholarly productivity in candidates who go up for tenure separately as opposed to jointly, we controlled for two research metrics. The first was the h-index, which, despite being an imperfect measure of research impact, is used widely as a metric for judging academics in nearly all fields68; note, this variable was scraped from Google Scholar profiles at the time of the candidate's P&T decision. The second is the external grants, which refer to the number of external grants awarded as the principal investigator. This variable was coded by individuals in the Provost's office from candidates’ CVs.

Department size. Another factor that is likely to vary across joint and separate evaluations is department size. This is because, mathematically, it is more likely that a large department will jointly evaluate a candidate than a smaller department because there are more untenured faculty each year. To calculate this variable, we used the total number of votes cast at the department level.

Analysis plan. All analyses were conducted separately for negative voting percentage and unanimous voting outcomes. Stepwise regressions were run to test hypothesis 1. Model 1 included only the key independent variables (URM status, joint evaluation, and their interaction term). Model 2 included the baseline controls (institution, discipline, gender, rank they are being promoted to [associate/full], and years in current rank). Model 3 included the baseline controls and scholarly productivity metrics (h-index and external grants). Model 4 included the baseline controls and department size (total department votes). Model 5 included all the control variables simultaneously. Additionally, we included a Model 0, in which we did not include the interaction term and only the two independent variables (URM status, joint evaluation). For analyses on negative vote percentage (continuous variable), OLS regression was used, reporting unstandardized coefficients. For unanimous votes (binary variable), logistic regression was used, reporting odds ratios.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "The article is well written, incorporates a novel dataset, and covers an important topic. However, I do see some opportunities for improvement. They are described below in no particular order.\nPut the sample size for the forecasting study in Table 1's notes.\nRegarding Table 3, to determine if the key independent variables are indeed statistically significant, post-regression tests of joint significance should be performed (and disclosed) on the coefficients of URM, Joint, and URM x Joint.\nEspecially given that all 6 sampled institutions are research institutions, it is surprising that the research productivity variables (h-index & grants) are always statistically insignificant. While outside the scope of this study, this may be surprising enough to warrant increased investigation and discussion within the manuscript.\nWhile not an overly important diagnostic, that the R^2s are very small (0.09 or less) is somewhat noteworthy. The models appear to explain very little of the variation in promotion decisions. At least for the full model (the one with the full set of control variables), it would be interesting to see the partitioned R^2s for at least the 3 key independent variables (URM, Joint, and URM x Joint). This would estimate an answer to how much of the variation is being explained specifically by these key variables. Given overall R^2 = 0.09, the answer will of course be \"very little,\" but a quantitative estimate would be helpful.\nI recommend adding this control variable: the percent of the department that is URM. Interactions with this new control may be warranted and enlightening.\nThe abstract states that 6 institutions were sampled. Then, on p.9, it says that 10 were. Finally, on p.18, it says that 6 institutions were chosen from a consortium of 10, clearing up the confusion. Still, the confusion is unnecessary.\nThe discussion of the supplemental analysis (joint evaluation cases where URM candidates were being evaluated alongside other URM candidates are excluded) is overly brief. These results should be presented in a new table. Results from the opposite subsample (only those cases included) should also be presented, perhaps in panel B of this new table. Further, it should be more clearly disclosed how this subsample was identified (Was it simply URM cases > 1?)\nDescriptive statistics across URM status should be presented, as they were across Joint Evaluation status in Table 2.\nEven if the results mirror one another, showing models across the subsamples of Assoc. Prof. promotions and Full Prof. promotions would be enlightening.\nWhile it is perhaps unnecessary to tabulate all these additional coefficients, it would help the reader to know whether any other interaction terms with URM (e.g., h-index x URM) are statistically significant.", "reviewer_2": "This paper presents a field test of the established finding that when candidates are evaluated jointly, this leads to more positive outcomes for populations that face negative stereotypes (e.g., women, URMs) than when candidates are evaluated separately. The test is conducted in a highly consequential setting: promotion and tenure decisions at universities. It involves the analysis of 1,804 such decisions at six universities and shows that when two or more candidates are under consideration for promotion or tenure by the same department in the same year, the positive vote share for minority candidates increases.\nI have reviewed this paper (anonymously) previously for another journal, and it has been updated only slightly since my last review. Therefore, many of my comments will be similar to those the authors have received from me before. In addition to revisiting my previous points, I have also incorporated feedback from other reviewers that I believe would further enhance the paper, which I became privy to thanks to reviewing a prior draft of this paper for another journal .\nWhile I think the paper is already very strong, I have a number of suggestions that I hope will help the authors improve it.\nCOMMENTS ON LITERATURE REVIEW AND THEORY\n(1) I found the inclusion of Hypothesis 2 on page 7 and the paragraph that preceded it a bit misguided. We have strong theoretical support and empirical support for Hypothesis 1 from decades of research on joint vs. separate decision making but absolutely no evidence for Hypothesis 2. I don’t see a tension and suggest removing Hypothesis 2 and the short and uncompelling arguments presented to support it. Perhaps the logic articulated prior to the presentation of Hypothesis 2 could be mentioned when discussing the poor performance of lay forecasters who sought to predict the paper’s findings? For non-experts who are unfamiliar with the large literature on joint versus separate decision making, perhaps the weak arguments made to support Hypothesis 2 came to mind. But the researchers writing this paper should focus exclusively on the well-established and proven theory behind Hypothesis 1 to justify their study.\n(2) The paper would benefit from greater specificity and contextualization regarding its operationalization of “joint evaluation.” While the authors define joint evaluation broadly—considering professors evaluated in the same calendar year as jointly evaluated—this deviates from traditional approaches that emphasize simultaneous assessments (i.e., within the same meeting). The analysis could be deepened by exploring whether temporal proximity of evaluations strengthens the “joint evaluation” effect; for example, are the effects more pronounced when evaluations occur in the same quarter, month, or meeting? If the authors do not have the ability to analyze this with their data, then I believe it is at least worth mentioning in the paper’s discussion (and/or introduction) that an additional contribution of the paper is to show that it is not necessary for two cases to be evaluated simultaneously for the “joint evaluation” results from past studies to occur.\n(3) I found it surprising that the paper only focuses on joint vs. separate P&T decisions for URM faculty and does not analyze joint vs. separate P&T decisions for women. The literature on joint vs. separate decisions has primarily focused on how women in stereotypically male domains benefit from joint evaluation. This should be acknowledged openly, and the authors should also run additional analyses with a focus on the promotion of women (instead of URMs) and report those results, too (even briefly). Whether an analysis of joint vs. separate evaluations of women faculty shows the same pattern or not, it would be interesting and informative to know this in light of past research (the analyses could be relegated to an appendix and mentioned briefly in the discussion, but they belong somewhere in the narrative).\n(4) The paper would benefit from a more detailed discussion of the tenure process. For instance, they should define what constitutes a positive tenure vote (unanimous support? or merely a majority? And is this the same across institutions studied?). Moreover, the authors should offer a concrete interpretation of the observed 11% reduction in negative votes for URM candidates evaluated jointly. For instance, by making some broad assumptions about typical tenure and promotion policies, how might this reduction translate into an actual increase in the number of URM professors securing tenure or promotion? Addressing these points would significantly enhance the practical significance and clarity of the study’s findings.\n(5) I would also encourage the authors to explicitly position the contribution of the paper as extending beyond academia. The question of how evaluation mode affects outcomes by race likely applies beyond the specific context of university tenure committees to decision-making more broadly. Highlighting the generalizability of this phenomenon would significantly broaden the contribution of your paper.\nCOMMENTS ON EMPIRICS\n(6) I appreciate the work that surely went into obtaining this remarkable, unique dataset. One challenge is that the distribution of negative votes is highly skewed. The median number of negative votes is zero in both conditions, and key result appears to be driven by a very small group of URM candidates who received a non-zero number of negative votes. Specifically, only 48 URM candidates received any negative votes. Assuming that it is not straightforward to obtain additional data from 2023 and 2024 to increase the sample size somewhat, I would encourage the authors to transparently note this limitation of the current analysis (that it hinges on these small number statistics.).\n(7) Why not control for how many other candidates have been considered so far, in total, at the time of the candidate’s vote? Or how far into the P&T season (e.g. in days/weeks) the candidate was considered?\n(8) I found the justification for only looking at Black and Hispanic URMs and excluding Native Americans somewhat confusing. With an N of 2, why not include Native Americans at least in a robustness check? They are certainly URMs! The exclusion of the “Other” race category seems more justifiable since it is unclear what “Other” means.\n(9) To enhance reproducibility and transparency, I strongly encourage the authors to upload all analysis scripts, materials (for the forecasting experiment), and more detailed documentation of variable creation on their OSF.", "reviewer_3": "I co-reviewed this manuscript with one of the reviewers who provided the listed reports. This is part of the Nature Communications initiative to facilitate training in peer review and to provide appropriate recognition for Early Career Researchers who co-review manuscripts.", "reviewer_4": "Comments on Joint Evaluation Reduces Racial Disparities in Promotion and Tenure: Evidence from a Natural Experiment\nSummary:\nThe authors use variation in the “mode” of tenure voting, joint vs. single, across several institutions and departments using remarkable data on candidates’ attributes combined with department-by-institution information on the voting process. They begin with a survey asking whether those gatekeepers think this would improve or worsen disparities in URM tenure cases. Framed within a cross-disciplinary literature on choice architecture the authors conclude that in fact URM candidates receive fewer negative votes when evaluated in the joint, as opposed to separate, case, contrary to what the gatekeepers thought would happen.\nComments:\nI very much appreciate framing the paper in terms of choice architecture and anchoring – in fact I think there is a little room to expand on the role anchoring plays in this particular case. I appreciated the discussion here as it sets up the mechanisms for the experiment very nicely, this was excellent framing and I appreciated that you framed the potential for it to go either way. Starting with a survey of gatekeepers’ beliefs enriched the study, in particular given the results.\nI was unclear on how “treatment” is “assigned” here (using the terms lightly). You say in the back end of the paper that, “Joint evaluation is defined as when more than one P&T candidate is seeking promotion to the same role, in the same department, in the same academic discipline, and in the same year. Separate evaluation was any case when this was not occurring (p. 20)”. How was the mode determined? Does it vary over time within department? Does it vary across departments within a university? I think yes to both of these but the extent of this is important for understanding where the variation is coming from.\nPage 11 partially discusses this, but this is an important aspect to be clear on. I took time to download and see the data (thanks for providing this, it is a great practice). I did not see a year indicator in the data, so it seems as if this is all running as a pooled analysis. If you have the data by year (I assume you don’t or you would have included it) I suggest treating this as a panel.\nTables 1 and 2 could be expanded to provide a clearer picture of the populations. I suggest having table 1 as only a table of means (and SD’s) of the universe of schools (or departments). I will assume that the pool is balanced (each school is in the data in each year). We might have statistics about the universities (size, category, possibly some measure of “rank” or competitiveness) separately by single vs. joint evaluations. I understand that there are 10 universities but later you say four universities had unusable data later on in the paper.\nIn the main model, you have institution and CIP code controls, meaning that you have what I believe are university and separately department (or field) fixed effects (from the table notes). This means that the variation could be coming from changes within university-departments over time, or is coming simply off of differences across departments within universities, possibly over time or not, it is unclear which, but is important. The data available for download show that there is within department variation, implying that either there were changes over time or the variation is simply random across years, most likely the former. One would have to assume that the switch across years was uncorrelated with any other changes that are related to the vote share, which is possible but unlikely. I might also suggest a first regression where the dependent variable is the treatment itself – joint eval, such that the reader can understand which universities/departments are more likely to feature this mode. Without a time element however, you can only learn so much from this.\nI believe you’re using a linear probability model (i.e. OLS) which is probably just fine, but because the dependent variable is a percentage (0-1) and is therefore bounded, using a generalized linear model (GLM) with a logit-link might improve precision. I also suggest clustering standard errors on universities (or departments), given the marginal results I think it’s important to take inference seriously here. I tried this with the data and code provided and the standard errors shrink, which is surprising, but shows there is more precision to be had. Likewise, given there is variation within university-CIP cells in joint evaluation you can actually have a university-by-CIP fixed effects and results are similar.\nI’d also suggest putting the sample sizes in the results table. You note some drop out due to missing data – you could remedy that by setting the covariates for those observations (assuming they’re not the dependent variable) to zero and including an additional dummy variable indicating that the control was missing to get some sample size back and rule out any issues from attrition across models. You lose about half the sample from the first to the last regression. Unless that attrition is random, it is worth figuring out which variables are causing the issue – it’s difficult to determine how much the sample change affects inference. At the very least the reader should be aware of this.\nAs an even smaller note, I’d suggest making the dependent variable positive votes as opposed to negative, it’s just easier to interpret the effect of joint evaluation as a “positive” effect. I apologize for what might seem like nit-picking here, please do not hear me saying I don’t believe (or like) the results, the opposite is true – I want to make sure they are as strong, reliable, and clear as possible. Now on to actual comments.\nThe dependent variable has a pretty limited distribution. The data suggest that more than three-quarters of the votes were unanimous, meaning there is very little variation in the dependent variable to begin with. Possibly this explains the very small R2, and why the unanimous variable outcome does not show effects. This is pretty important for the analysis, and for the reader.\nI may be reading the table wrong, but my interpretation is that on net, URM candidates receive the same share of votes. The main effect (e.g. Model 1 of Table 3) has a positive coefficient of 0.10 for URMs and the interaction effect is -0.09. I take this to mean that in that model (1) in joint evaluations, URM’s receive the same share of negative votes but when evaluated separately, they receive 9 percentage points fewer votes. I believe percentage points is the right interpretation here as the dependent variable is the percent of votes. I think you could add the baseline negative vote share to the table and then calculate what 0.09 is off of the base share of negative votes. On line 266 and in the abstract you say that the result is a 9% reduction in negative votes, which I believe is incorrect.\nI was also struck by the fact that H-index and grants have no relationship to vote shares, and by the lack of negative effect for female candidates. Do these fit with other results from the literature? You might consider interacting these with URM (and with joint), but I am not clear on what the story to tell would be. To add, the R2 is quite low considering you have CIP and university FE in the model (unless you reported the adjusted R2, which is fine but just say so). It’s hard to believe that all of those explain only a tiny fraction of the overall variance is explained by all of that information. Possibly this is due to the lack of variation in the dependent variable.\nI think you can learn something from the vote share vs. the unanimous vote vis-a-vis your hypotheses, in particular the anchoring effect. That it is not statistically significant does not imply the result is not meaningful. Nonetheless, I think Figure 2 is worth highlighting more - non-URM votes are unaltered by the switch, but negative URM votes decline by two-thirds (10 percentage points off of a base of 15 percent)! You could do this for unanimous vs. not as well. I think this says something about the mechanism – opinions about non-URM’s don’t change, URM’s are simply seen in a less negative light, which I think points more to the double-standard mechanism you mentioned in the framing section more than the negative stereotyping, though it is tough to pin the mechanism down."}, "author_response": "We appreciate your words of support and the in-depth comments you have provided.\nBelow, find a point-by-point response to each comment.\nPut the sample size for the forecasting study in Table 1's notes.\nThank you for noting this. We have made this change - please see Figure 1, which is where we present the results of the forecasting study. Table 1 in the document is the means and SDs for the promotion and tenure study.\nRegarding Table 3, to determine if the key independent variables are indeed statistically significant, post-regression tests of joint significance should be performed (and disclosed) on the coefficients of URM, Joint, and URM x Joint.\nTo address this comment we performed two types of post-regression tests, both aimed at strengthening the robustness of the IV, moderator, and the interaction term. We hope one or both of these analyses help address your query.\nFirst, we conducted a test to determine the joint significance of both independent variables and their interaction term. Given our original model was OLS we used an F- Test to conduct this analysis and did so while including all control variables. The analysis found support for the joint significance of these three variables (F(3,883) = 2.63, p = .049). We now include these results in a footnote of the paper (see p13).\nSecond, we conducted a 5,000 times bootstrap resample method to check the robustness of the results for the model with all control variables in Table 3. The results are shown in Table S12 below and replicate the results of the paper. We mention these results on p15, in a new “Robustness Checks” section of the manuscript.\nTable S12. Bootstrapped OLS regression results for the effect of joint evaluation on department negative vote percentage with 5,000 iterations.\nVariable Department Negative Vote %\n3\nb SE LLCI ULCI\nURM status .07* .03 .01 .13\nJoint evaluation .03** .01 .01 .06\nURM x Joint evaluation -.12** .03 -.21 -.04\nWomen -.01 .02 -.03 .02\nTenure in rank .001*** .002 .002 .01\nPromotion rank .03*** .01 .001 .06\nH-index -.00 .001 -.001 .001\nExternal grants -.001 .002 -.002 .001\nTotal department votes -.001 .001 -.002 .001\nR2 .09\nEspecially given that all 6 sampled institutions are research institutions, it is surprising that the research productivity variables (h-index & grants) are always statistically insignificant.\nAlthough outside the scope of this study, this may be surprising enough to warrant increased investigation and discussion within the manuscript.\nThank you for noting this; we agree it is an interesting point and now pick up on it in the discussion (p. X). We think this speaks to the intense complexity of P&T decisions, and the importance of a factor such as evaluation mode over specific metrics. Interestingly, past research has shown that factors like h-index may have an indirect effect on P&T outcomes via content of external review letters (Madera et al., 2024) and influencing perceptions of minority candidates (Masters-Waage et al., 2024), but it appears it is not a direct predictor of voting outcome. Nevertheless, given the relevance of research output in the P&T process, we still think it is appropriate to include these controls.\nIn the discussion we also highlight that the lack of direct relationship between scholarly productivity and voting outcomes is also interesting as it documents that faculty members evaluate scholarship to a large degree independent of scholarly productivity metrics, potentially relying more on their qualitative assessment of the candidates’ contributions than on cumulative metrics describing impact (citations, h-index).\nFor ease of reference, we provide the added section to the general discussion below (p. 22-23):\n“...[M]easures of scholar productivity (h index and grants obtained) are included in the model and neither of them significantly predicted voting outcomes. Although past research has identified indirect effects of scholarly productivity on voting outcome2,62 the lack of a direct relationship demonstrates that P&T decision makers derive evaluations of research impact and quality through means other than citation metrics, such as relying more on their qualitative assessment of the candidates’ contributions rather than on cumulative metrics66. Instead, it appears that P&T decision making is a more complex process, highlighting the need for more research into this topic in order to identify the factors driving these high-stakes decisions.”\nWhile not an overly important diagnostic, that the R^2s are very small (0.09 or less) is somewhat noteworthy. The models appear to explain very little of the variation in promotion decisions. At least for the full model (the one with the full set of control variables), it would be interesting to see the partitioned R^2s for at least the 3 key independent variables (URM, Joint, and URM x Joint). This would estimate an answer to how much of the variation is being explained specifically by these key variables. Given overall R^2 = 0.09, the answer will of course be \"very little,\" but a quantitative estimate would be helpful.\nWe agree that the R^2s are noteworthy and believe there are both conceptual and practical reasons for this. First, while P&T decisions are sometimes presented as objective meritocratic decisions, in practice, these decisions are made anonymously without a clear rubric to follow. Therefore, in line with our previous comment, we would argue that it is very likely that a wide range of factors influence a P&T member's decisions, including idiosyncratic subjective assessments of a candidate. This suggests why objective metrics (e.g., h-index, grants, years in present rank, etc.) do not explain a large amount of the variance in P&T decisions.\nIn addition to this, it is also likely that the nature of the P&T voting data is also contributing to the R^2s. There is a significant proportion of faculty who receive no negative votes (75.5%), meaning that there is no variance here for the model to predict. However, on suggestion of another reviewer, we ran a model excluding individuals who received zero negative votes. Using this subsample (see Table S13), we replicated our results and found that the R^2 increases to 27%. Therefore, it appears that the skew in the data contributed to the low R^2s.\nFinally, we would also stress that in behavioural science an R^2 of below 10% is not surprisingly low. This is largely attributable to the high degree of noise observed in human decision making and human behaviour more generally. These challenges are also amplified when studying decision making in real-world environments. Therefore, when raising this point in the discussion we highlight this issue as a challenge as opposed to a limitation. We have included an added section to the discussion highlighting these issues and the supplementary table displaying the results for the subset analysis:\n“Another challenge associated with the use of a real world dataset is the complex inputs to decisions. As shown in Table 3, the model including all of the control variables accounts for 8% of the variance in negative vote percentage at the department level. An r-squared below 10% is not uncommon in applied psychological science59,60, given the inherent complexity and noise in the psychological process61. Further, as shown in the robustness checks, the r-squared increases to 27% when addressing skew in the dataset by excluding participants with no negative votes. Nevertheless, it does highlight the challenges in predicting P&T voting behavior.”\nTable S13. OLS regression results for the interactive effect of URM status and joint evaluation on department negative vote percentage excluding candidates who received zero negative votes.\nDepartment Negative Vote %\nVariable b SE\nURM status .17* .08\nJoint evaluation .03 .03\nURM x Joint evaluation -.30* .11\nWomen -.04 .03\nTenure in rank .01* .004\nPromotion in rank -.001 .04\nH-index -.001 .001\nExternal grants -.004 .002\nTotal department votes -.01 .001\nR2 .27\nN = 375.\nFinally, we also provide for you below the partitioned R^2s (Table S21); note, to calculate these we used the method outlined by Lindeman, Merenda, and Gold (1980).\nTable S21. Partitioned R for each variable in Table 3, model including all control variables.\nDepartment Negative Vote %\nPartitioned R2\nVariable\nURM status .008\nJoint evaluation .008\nURM x Joint evaluation .02\nWomen .0003\nTenure in rank .02\nPromotion in rank .03\nH-index .008\nExternal grants .008\nTotal department votes .003\nI recommend adding this control variable: the percent of the department that is URM. Interactions with this new control may be warranted and enlightening.\nUnfortunately, while we do control for discipline in our analyses, we do not have a variable indicating the percentage of the candidate’s department who were URM, a variable not usually recorded in P&T files or archived in institutional records. The P&T data was coded by members in the provost's office at each institution using highly protected institutional data. Further, given that department composition changes over time and turnover particularly among URM faculty can be significant, this would require that URM representation was coded in each department for each year. Typically, universities do not have easily accessible data on department demographics separated by each year.\nTherefore, while we would like to include this as a control variable, we do not think it is feasible given the nature of the dataset. In the revised manuscript, we highlight this limitation in the general discussion (p. 23).\nThe abstract states that 6 institutions were sampled. Then, on p.9, it says that 10 were. Finally, on p.18, it says that 6 institutions were chosen from a consortium of 10, clearing up the confusion. Still, the confusion is unnecessary.\nThank you for allowing us to clarify. Initially, the consortium that came together to create this dataset included 10 institutions. However, two institutions did not complete all required elements of the coding process and (after construction of the consortium) we learned that two of the smaller institutions had non-traditional ways of completing P&T voting, e.g., they did not clearly structure P&T committees or record votes. Therefore, these four institutions are not included in analyses as none of them have data on voting outcomes; the focal dependent variable. In the paper, to improve clarity, we now only refer to six institutions throughout and have included a footnote on page 9 explaining the lack of data for four institutions (this footnote is provided below):\n“The Center for Excellence in Faculty Advancement (CEFA) dataset includes a total of ten institutions, however, only six institutions provided usable data for this paper. At the time of submission to the journal, two institutions had not completed the coding process, so we cannot include their data. Further, another two institutions had non-traditional ways of completing P&T voting - e.g., they did not clearly structure P&T committees or record votes - so we could not include this data in the analyses. In sum, these four institutions are not included in analyses as none of them have data on voting outcomes.”\nThe discussion of the supplemental analysis (joint evaluation cases where URM candidates were being evaluated alongside other URM candidates are excluded) is overly brief. These results should be presented in a new table. Results from the opposite subsample (only those cases included) should also be presented, perhaps in panel B of this new table. Further, it should be more clearly disclosed how this subsample was identified (Was it simply URM cases > 1?)\nWe appreciate the encouragement to further develop the supplemental analyses (p16-18). In the revised manuscript we now include supplemental analyses for a) the associate and full professor level separately, b) an interaction with gender, and c) when URM faculty are being evaluated compared to non-URM faculty vs. only URM faculty.\nFor the final supplementary analyses we have followed your guidance. First, we conducted analyses focusing on only cases where a URM faculty was compared to a non- URM faculty. These analyses replicate the findings from the primary analyses (b = -.12, SE = .05, p = .006; see Table S17). Second, we conducted analyses focusing on cases where URM faculty are only compared to URM faculty. Note, this is a rare case as the underrepresentation of URM faculty (8% across the US) makes it particularly unlikely that two or more URM faculty will be evaluated in the same P&T year; in our analysis there were 28 such cases before listwise deletions. Nevertheless, we find results that are a similar pattern to those seen across the paper though not reaching statistical significance (b = -.14, SE = .08, p = .07; see Table S18). These analyses are reported in the Supplementary Analyses section (p17)\nDescriptive statistics across URM status should be presented, as they were across Joint Evaluation status in Table 2.\nGreat point, we now include that in the table (see Table 2).\nEven if the results mirror one another, showing models across the subsamples of Assoc. Prof. promotions and Full Prof. promotions would be enlightening.\nWe have added these subsample results to the supplementary materials and mention them in the manuscript (see section title “Supplementary Analyses”, p16-17).\nIn a model including all of the control variables, find a similar pattern of results at both levels of analysis which is consistent with the overall finding, i.e., joint evaluation reduces racial disparities. Using the smaller subsamples, the result is statistically significant at the associate professor level (b = -.13, SE = .06, p = .04) and the full professor level (b = -.15, SE = .07, p = .02).\nFor ease of reference we include both of these supplementary tables below:\nTable S15. OLS regression results for the interactive effect of URM status and joint evaluation on department negative vote percentage for candidates seeking promotion to associate professor.\nDepartment Negative Vote %\nVariable b SE\nURM status .10 .05\nJoint evaluation .01 .02\nURM x Joint evaluation -.13* .06\nWomen -.01 .02\nTenure in rank .03*** .01\nH-index -.00 .001\nExternal grants -.003 .002\nTotal department votes -.00 .001\nR2 .11\nTable S16. OLS regression results for the interactive effect of URM status and joint evaluation on department negative vote percentage for candidates seeking promotion to full professor.\nDepartment Negative Vote %\nVariable b SE\nURM status .07 .05\nJoint evaluation .06** .02\nURM x Joint evaluation -.15* .07\nWomen -.01 .02\nTenure in rank .004 .002\nH-index .001 .001\nExternal grants .00 .001\nTotal department votes -.002 .002\nR2 .12\nWhile it is perhaps unnecessary to tabulate all these additional coefficients, it would help the reader to know whether any other interaction terms with URM (e.g., h-index x URM) are statistically significant.\nThank you for this suggestion and we agree that there are many interesting avenues to explore with this dataset. That said, we are hesitant to include these analyses for a couple reasons. First, we would like to keep this paper’s narrative focused on joint vs separate evaluation as a moderating factor. The interaction effect between URM and h-index is documented in our prior published work (2024) and published in Nature Human Behaviour.\nIn other words, we have focused this paper on, from a behavioral decision making perspective, an important phenomenon with significant policy implications: joint and separate evaluations. However, if there is another interaction you are interested in that concerns evaluation mode, we would be happy to test it and include it as a supplemental analysis.\nNote, in the Supplementary Analyses section we now also model an interaction between gender and evaluation mode but the result is not significant.\nReviewer #1 (Remarks on code availability):\nn/a\nReviewer #2 (Remarks to the Author):\nThis paper presents a field test of the established finding that when candidates are evaluated jointly, this leads to more positive outcomes for populations that face negative stereotypes (e.g., women, URMs) than when candidates are evaluated separately. The test is conducted in a highly consequential setting: promotion and tenure decisions at universities. It involves the analysis of 1,804 such decisions at six universities and shows that when two or more candidates are under consideration for promotion or tenure by the same department in the same year, the positive vote share for minority candidates increases.\nI have reviewed this paper (anonymously) previously for another journal, and it has been updated only slightly since my last review. Therefore, many of my comments will be similar to those the authors have received from me before. In addition to revisiting my previous points, I have also incorporated feedback from other reviewers that I believe would further enhance the paper, which I became privy to thanks to reviewing a prior draft of this paper for another journal.\nWhile I think the paper is already very strong, I have a number of suggestions that I hope will help the authors improve it.\nAuthors’ Response\nThank you for your encouraging words and support for this project. Below please find a point-by-point response to each of your comments.\nCOMMENTS ON LITERATURE REVIEW AND THEORY\n(1) I found the inclusion of Hypothesis 2 on page 7 and the paragraph that preceded it a bit misguided. We have strong theoretical support and empirical support for Hypothesis 1 from decades of research on joint vs. separate decision making but absolutely no evidence for Hypothesis 2. I don’t see a tension and suggest removing Hypothesis 2 and the short and uncompelling arguments presented to support it. Perhaps the logic articulated prior to the presentation of Hypothesis 2 could be mentioned when discussing the poor performance of lay forecasters who sought to predict the paper’s findings? For non-experts who are unfamiliar with the large literature on joint versus separate decision making, perhaps the weak arguments made to support Hypothesis 2 came to mind. But the researchers writing this paper should focus exclusively on the well-established and proven theory behind Hypothesis 1 to justify their study.\nThank you for raising this issue. The original motivation of us developing this study was past judgement and decision making (JDM) work identifying the benefits of joint evaluation, so hypothesis 1 was our central hypothesis. However, when discussing the project in early stages with social psychologists and colleagues in other disciplines, we found that many people predicted the opposite results, that separate evaluation improves outcomes for URMs. This was our motivation to include hypothesis 2 as a competing hypothesis and the associated theoretical arguments. However, we agree that these theoretical arguments are more convoluted, therefore, we appreciate the encouragement to remove hypothesis 2 and more tightly focus the paper on the JDM perspective.\nThat said, while we have removed hypothesis 2, we still maintain parts of the theorizing but have moved it to the discussion section for the forecasting study (p8-9). As you note, the forecasting survey shows that a significant number of faculty intuitively side with this hypothesis, so we think it is important to discuss why individuals may have this lay belief.\nOverall, we believe this improves the flow of the manuscript and keeps the paper more tightly focused on evaluation mode as a mechanism. With this more localized focus we have also been able to streamline the introduction.\n(2) The paper would benefit from greater specificity and contextualization regarding its operationalization of “joint evaluation.” While the authors define joint evaluation broadly—considering professors evaluated in the same calendar year as jointly evaluated—this deviates from traditional approaches that emphasize simultaneous assessments (i.e., within the same meeting). The analysis could be deepened by exploring whether temporal proximity of evaluations strengthens the “joint evaluation” effect; for example, are the effects more pronounced when evaluations occur in the same quarter, month, or meeting? If the authors do not have the ability to analyze this with their data, then I believe it is at least worth mentioning in the paper’s discussion (and/or introduction) that an additional contribution of the paper is to show that it is not necessary for two cases to be evaluated simultaneously for the “joint evaluation” results from past studies to occur.\nWe agree this is an important point. We have included in the methods a new section describing how we operationalize and distinguish this from past research on this construct.\nWe do not have precise data on the temporal proximity of evaluations. However, the majority of institutions (including all of the ones in our dataset) have very tight windows for P&T decisions to be made, meaning that decisions are typically made in close proximity to one another. In the revised submission we include Table S1, which details the P&T timeline at the six institutions in this dataset and shows that in each case departments committees must complete P&T voting roughly within a one-month period. Therefore, when multiple candidates were evaluated in the same year they will both need to be considered within this time window. Therefore, critically, when making decisions in the joint evaluation condition it is likely that P&T committees were mentally aware of and considering both candidates simultaneously. We now highlight this in a footnote in the introduction (p7), provide more details in the methods sections (p26), and include the new table for reference (Table S1).\nIn addition, in the review manuscript we also identify the lack of temporal specificity as a limitation and an interesting contribution to research on joint and separate evaluations in the broader judgment and decision making literature. As you suggest, the slight temporal distance highlights that the benefits of joint evaluation are not dependent on simultaneous evaluation as an additional contribution of this research (p19). For ease of reference we provide this section below:\n“A fourth contribution concerns temporal proximity. Past work on evaluation mode has focused on contexts in which two options are compared simultaneously. However, in this research that was not a requirement. First of all, although P&T decisions at each institution need to be made within a short time window (e.g., one month), they were not forced to be made in the same meeting. Further, even if they were conducted in the same meeting the discussions of each candidate were likely to take place sequentially. On the one hand, the lack of temporal specificity of when these decisions occurred is a limitation of this dataset, however, it also provides a more general test of the benefits of joint evaluation more akin to how these decisions are made in the real world. Therefore, finding that the benefits of joint evaluation replicate despite relaxing temporal proximity demonstrates the generalizability of the “evaluation nudge” theory.”\n(3) I found it surprising that the paper only focuses on joint vs. separate P&T decisions for URM faculty and does not analyze joint vs. separate P&T decisions for women. The literature on joint vs. separate decisions has primarily focused on how women in stereotypically male domains benefit from joint evaluation. This should be acknowledged openly, and the authors should also run additional analyses with a focus on the promotion of women (instead of URMs) and report those results, too (even briefly). Whether an analysis of joint vs. separate evaluations of women faculty shows the same pattern or not, it would be interesting and informative to know this in light of past research (the analyses could be relegated to an appendix and mentioned briefly in the discussion, but they belong somewhere in the narrative).\nThank you for raising this important point. Although the focus of our work is on URM faculty, we examined gender in more detail based on your suggestion. Interestingly, we find no significant interaction between gender and evaluation mode in terms of negative votes (b = .03, SE = .03, p = .28) nor unanimous votes (OR = 0.99, 95% CI [.88-.1.10]). A reason for this could be that unlike for URM faculty, in our dataset, gender is not associated with differential voting outcomes. Therefore, the lack of gender difference in terms of voting outcomes means that there is no gender inequity for evaluation mode to mitigate. As you suggest, we mention this in the supplemental analyses for the paper. For ease of reference we provide this section below (p17-18):\n“A final supplemental analysis we conducted focused on the interaction between evaluation mode and gender. Given that past research has found that evaluation mode can reduce biases experienced by women in the workplace27, we sought to replicate the results reported above using gender as the moderator instead of URM status. Analyses revealed no evidence for an interaction between evaluation mode and gender in terms of negative votes (b = .03, SE = .03, p = .28; see Table S19) nor unanimous votes (OR = 0.99, 95% CI [.88-.1.10]; see Table S20). However, it is also important to note that analyses of the main effect gender of voting outcomes found no evidence for gender disparities for P&T voting outcomes at the department level (negative vote percentage, b = -.01, SE = .01, p = .46; unanimous votes, OR = 1.02, 95% CI [.98-1.07]). Therefore, the failure to replicate past work finding that evaluation mode reduces gender bias, is likely due to the lack of evidence for a gender bias in P&T decision making in the current dataset.”\n(4) The paper would benefit from a more detailed discussion of the tenure process. For instance, they should define what constitutes a positive tenure vote (unanimous support? or merely a majority? And is this the same across institutions studied?). Moreover, the authors should offer a concrete interpretation of the observed 11% reduction in negative votes for URM candidates evaluated jointly. For instance, by making some broad assumptions about typical tenure and promotion policies, how might this reduction translate into an actual increase in the number of URM professors securing tenure or promotion? Addressing these points would significantly enhance the practical significance and clarity of the study’s findings.\nWe have taken this advice on board in the revision of the paper, making changes throughout the manuscript.\nFirst, in the introduction we provide a clearer description of the tenure process, pasted below for ease of reference (see p7):\n“We examine this hypothesis in the context of P&T decisions made within US institutions. Promotion policies can vary across institutions34,35, however, in general they follow a similar approach. Faculty are required to go up for promotion after a certain number of years (i.e., tenure clock), though they can go up early if they elect to and they can receive an extension if their university permits (e.g., birth of a child). Once faculty have chosen to go up for promotion, most universities’ faculty affairs units follow a rigid timeline determining, for example, when the candidate must submit their promotion portfolio and by when external review letters must be completed. The particular part of the P&T process we focus on is the meeting of the P&T committee at the department-level, to make their decision on the candidate. Department P&T committee decisions are influential since department P&T committee members are direct colleagues and usually in the same or adjacent disciplines as the P&T candidate.\nIn this paper, we define joint evaluation as when more than one candidate is being evaluated within this alloted window. In other words, there is more than one P&T candidate going up for promotion to the same rank, in the same department, in the same promotion cycle. Notably, this is a less rigid designation of joint evaluation compared to past laboratory studies where joint evaluation occurred at exactly the same time and was a forced-choice between two options26, that said, the proximity of the evaluations still suggests that decision makers will use the candidates as a reference point for each other27,36.”\nSecond, in the methods, we provide a longer and more precise description of the tenure voting process.\n“Promotion and tenure voting takes place within subsequent committee votes with department, college, and university level committees, although some universities do not have separate college committees. The voting outcome variable used in this paper was committee voting at the department level (i.e., within the candidates’ department). Although the dataset does include voting at the college and provost level, the joint evaluation occurred at the department level; for the college and provost level, it is likely that there are people being evaluated at the same time every year, given they review candidates from a much larger pool.”\nAs part of the voting process, committee members review candidate materials and convene to discuss and vote on P&T candidates, recording the number of votes for (“yes”) or against (“no”) a candidate, as well as the number of abstentions. From “yes” and “no” votes, tTwo voting outcome variables were used for each of the six institutions investigated within the consortium: negative vote percentage and unanimous vote.”\nThird, we liked your suggestion to conceptualize how an the shift in negative votes at the department level could affect the overall tenure process. To do so, we examined the downstream effect of evaluation mode at the department level on the provost vote. At the institutions in this dataset the Provost vote represents the ultimate decision on whether a candidate will receive a promotion (1) or not (0). Therefore, to contextualize our result, we calculated the conditional indirect effect of URM status on provost vote via department-level negative vote perception, depending on whether candidates were evaluated jointly or separately. In other words, how did benefits of joint evaluation for URM faculty at the department level translate into likelihood of receiving a positive provost vote. Moderated mediation analyses found support for a conditional indirect effect (Index of moderated mediation = .65, 95% bootCI = [.18, 1.25]); see Table S7), meaning that the interaction between URM status and evaluation mode had downstream effects on provost vote via department level voting. Post-hoc tests found that, for a URM faculty, being evaluated jointly lead to a 16.2% increase in the likelihood of receiving a positive provost vote. We provide these analyses in the results section (p14).\nTable S7. Moderated mediation results for condition indirect effect of URM status and joint evaluation on Provost vote via department negative vote percentage.\nDepartment Negative Vote % Provost vote\nVariable b b\nURM status .08* .66\nJoint evaluation .04**\nURM x Joint evaluation -.13**\nWomen -.01 -.26\nTenure in rank .003* .38\nH-index -.000 .001\nExternal grants -.0002 -.01\nTotal department votes -.0003 .02\nDepartment Negative -5.00***\nVote %\nR2 .05 .36\nEffect Bootstrap CI\nSeparate evaluation -.39 (-.94, .06)\nJoint evaluation .26** (.06, .49)\nIndex of moderated .65 (.18, 1.25)\nmediation\n(5) I would also encourage the authors to explicitly position the contribution of the paper as extending beyond academia. The question of how evaluation mode affects outcomes by race likely applies beyond the specific context of university tenure committees to decision-making more broadly. Highlighting the generalizability of this phenomenon would significantly broaden the contribution of your paper.\nThank you for encouraging us to broaden the discussion of our paper, we have followed this advice and think the discussion section is greatly improved. In particular, we focus on the experiences of URM employees in other professional occupations which have a comparably high-stakes promotion system.\nIn addition, we also expand on the contribution of this finding to research on group decision making in the discussion (p18-19). As we mention in the manuscript, the majority of work on evaluation mode has focused on individual decision making. Finding that the benefits extend to group decision making broadens the potential effect that evaluation mode could have on organizational processes.\nWhile we make these additions, we have also tried to stay close to Nature Communication’s requirements that authors do not “go beyond the findings of the data”, therefore, we present these expansions as a direction for future research as opposed to a direct implication of our study.\nFor ease of reference the relevant section of the discussion is pasted below (p21):\n“Beyond universities, this paper also has practical implications to organizations more generally. As a case study, the university promotion system shares many similarities with other organizational systems, with promotion candidates being evaluated by a team of decision makers on a variety of key performance indicators. Moreover, within organizations, there is also variation in evaluation mode. For example, during annual or quarterly reviews employees are more-or-less jointly evaluated, as there is typically an entire cohort/department being evaluated within a given period. However, there are also cases where separate evaluations are very common, such as time-in-grade promotion decisions which occur at varying intervals depending on employees’ length of employment. Such decisions are common, including in the United States government, and can be mechanisms for systemic biases. For example, administrative data concerning the U.S. Patent Office, which uses time-in-grade promotions, demonstrates promotion delays and gaps for Black patent examiners54. Based on the results of this paper, further applied research should examine whether joint evaluation within the U.S. Patent Office could help reduce these racial disparities. The same also applies to the myriad of private organizations using separate evaluations, that also benefit from a more top-down organizational structure than the higher education system55, allowing them to implement joint evaluation mode universally for promotion decisions.”\nCOMMENTS ON EMPIRICS\n(6) I appreciate the work that surely went into obtaining this remarkable, unique dataset. One challenge is that the distribution of negative votes is highly skewed. The median number of negative votes is zero in both conditions, and key result appears to be driven by a very small group of URM candidates who received a non-zero number of negative votes. Specifically, only 48 URM candidates received any negative votes. Assuming that it is not straightforward to obtain additional data from 2023 and 2024 to increase the sample size somewhat, I would encourage the authors to transparently note this limitation of the current analysis (that it hinges on these small number statistics.).\nThank you for raising this important point. We have added a section to the limitations section in the discussion where we highlight this (p23). Even in our data (which to our knowledge is the largest dataset of P&T decisions obtained), the low number of URM candidates is an unavoidable feature of studying racial disparities in academia, where URM faculty make up less than 10% of our sample. Notably, in this paper, the central hypothesis focuses on the interaction between URM status and evaluation mode. Therefore, we are not simply theorizing the positive effect of joint evaluation on URM faculty outcomes, we are also theorizing that this effect will not be present for non-URM faculty as majority group members, which we also find support for.\nIn addition, to address the skew in distribution of negative votes, we conducted robustness check with (subsample removing unanimous votes) and clustered standard errors based on Universities and departments.\nTable S13. OLS regression results for the interactive effect of URM status and joint evaluation on department negative vote percentage excluding candidates who received zero negative votes.\nDepartment Negative Vote %\nVariable b SE\nURM status .17* .08\nJoint evaluation .03 .03\nURM x Joint evaluation -.30* .11\nWomen -.04 .03\nTenure in rank .01* .004\nPromotion in rank -.001 .04\nH-index -.001 .001\nExternal grants -.004 .002\nTotal department votes -.01 .001\nR2 .27\nN = 375.\n(7) Why not control for how many other candidates have been considered so far, in total, at the time of the candidate’s vote? Or how far into the P&T season (e.g. in days/weeks) the candidate was considered?\nWe agree this would be an interesting variable but unfortunately, as described above, we do not have temporal data on when these decisions were made. That said, as described in response to your Comment 2, we looked into the P&T policy at each institution in our dataset and found that in all cases they required candidates to be evaluated within a relatively short time window to meet university timelines for completing multiple levels of required reviews (roughly one month). Therefore, the P&T season for all candidates was a short time window and thus it is likely (to save time and avoid scheduling conflicts) that departments arranged for all candidates to be evaluated in the same meeting. Further, even if they are not, the ordering would be tightly arranged.\nAs mentioned in comment 2, we now highlight the temporal structure of the voting in a footnote in the introduction (p7), provide more details in the methods sections (p26), include the new table for reference (see Table S1), and address the limitations (and contribution) of a lack of temporal specificity in the discussion (p19).\n(8) I found the justification for only looking at Black and Hispanic URMs and excluding Native Americans somewhat confusing. With an N of 2, why not include Native Americans at least in a robustness check? They are certainly URMs! The exclusion of the “Other” race category seems more justifiable since it is unclear what “Other” means.\nThank you for noting this. Our initial rationale for not including these individuals was due to the small sample and the precedent set by other research by other teams using this dataset (e.g., Masters-Waage et al., 2024). Nonetheless, we ran a robustness check including Native Americans in the analyses and found the same result, (see Table S10), and documented the replicated finding in a new section title “Robustness Checks” (see p14-15).\n(9) To enhance reproducibility and transparency, I strongly encourage the authors to upload all analysis scripts, materials (for the forecasting experiment), and more detailed documentation of variable creation on their OSF.\nThank you, we now provide all analysis scripts and materials on our OSF page, the documentation of how variables were created is provided in the methods section.\nReviewer #3 (Remarks to the Author):\nI co-reviewed this manuscript with one of the reviewers who provided the listed reports. This is part of the Nature Communications initiative to facilitate training in peer review and to provide appropriate recognition for Early Career Researchers who co-review manuscripts.\nAuthors’ Response:\nThank you for your contributions to the review process.\nReviewer #4 (Remarks to the Author):\nComments on Joint Evaluation Reduces Racial Disparities in Promotion and Tenure: Evidence from a Natural Experiment\nSummary:\nThe authors use variation in the “mode” of tenure voting, joint vs. single, across several institutions and departments using remarkable data on candidates’ attributes combined with department-by-institution information on the voting process. They begin with a survey asking whether those gatekeepers think this would improve or worsen disparities in URM tenure cases. Framed within a cross-disciplinary literature on choice architecture the authors conclude that in fact URM candidates receive fewer negative votes when evaluated in the joint, as opposed to separate, case, contrary to what the gatekeepers thought would happen.\nAuthors’ Response:\nThank you for the support of our dataset and the manuscript. Below we provide a point-by-point response to each of your comments.\nComments:\n1. I very much appreciate framing the paper in terms of choice architecture and anchoring – in fact I think there is a little room to expand on the role anchoring plays in this particular case. I appreciated the discussion here as it sets up the mechanisms for the experiment very nicely, this was excellent framing and I appreciated that you framed the potential for it to go either way. Starting with a survey of gatekeepers’ beliefs enriched the study, in particular given the results.\nThank you for your supportive words, in the new manuscript we have taken further steps to focus the paper on choice architecture. Reading through the totality of the reviewer comments we decided to focus the paper more closely on evaluation mode from the judgment and decision making perspective.\nOn the encouragement of another reviewer we did decide to remove the formal hypothesis 2, i.e., that separate evaluation will improve outcomes for URM faculty. Doing so helps us by providing a stronger theoretical proposition that this paper is investigating. That said, we also agreed that highlighting that results could “go either way” - and that a significant number of forecasters believed that separate evaluation will improve outcomes for URM faculty - demonstrates the importance of this research question. Therefore, we have moved the discussion of the competing hypothesis to the end of the forecasting study. This positions our theoretical hypothesis (that joint evaluation will improve outcomes) against the lay hypothesis of P&T decision makers (that joint evaluation will worsen or have no effect on outcomes).\nFor ease of reference we pasted this new section from the discussion of the forecasting study below (p8-9):\n“It is particularly notable that the most common response from gatekeepers was that separate evaluation would reduce racial disparities. Although this perspective contrasts our hypothesis, it is somewhat aligned with general social psychological theories of perception, that race is more salient when individuals are in a group (2 or more) than when they are alone37,38. For example, if a decision maker is evaluating a P&T candidate individually, their most prominent social feature might be that they are a faculty member; however, if they are evaluated with someone of a different race, it will become more salient that they are a Hispanic faculty member and that the other candidate is a White faculty member. In sum, the results of the forecasting survey present a competing hypothesis based on faculty lay beliefs, that separate evaluation will reduce racial disparities in the P&T process. The contrast between this lay hypothesis and our initial hypothesis drawn from behavioral decision theory sets up the natural experiment as an empirical test to resolve these two contrasting perspectives.”\n2. I was unclear on how “treatment” is “assigned” here (using the terms lightly). You say in the back end of the paper that, “Joint evaluation is defined as when more than one P&T candidate is seeking promotion to the same role, in the same department, in the same academic discipline, and in the same year. Separate evaluation was any case when this was not occurring (p. 20)”. How was the mode determined? Does it vary over time within department? Does it vary across departments within a university? I think yes to both of these but the extent of this is important for understanding where the variation is coming from. Page 11 partially discusses this, but this is an important aspect to be clear on. I took time to download and see the data (thanks for providing this, it is a great practice). I did not see a year indicator in the data, so it seems as if this is all running as a pooled analysis. If you have the data by year (I assume you don’t or you would have included it) I suggest treating this as a panel.\nIn the revised manuscript we have provided more information on the assignment process in the introduction (p7). To answer your two specific questions: yes, it does vary over time within departments; yes, it does vary across departments within a university. Ultimately, joint evaluation is a function of when two candidates were hired in the same year, or if not, two candidates hired in different years happen to go up together (e.g., one took an extension or went up early). Therefore, department or university has little effect on this, except that larger departments are more likely to have multiple candidates going up (which we control for).\nWe have pasted the new paragraph in the introduction in which we provide more details on the “assignment” process (p7):\n“We examine this hypothesis in the context of P&T decisions made within US institutions. Promotion policies can vary across institutions34,35, however, in general they follow a similar approach. Faculty are required to go up for promotion after a certain number of years (i.e., tenure clock), though they can go up early if they elect to and they can receive an extension if their university permits (e.g., birth of a child). Once faculty have chosen to go up for promotion, most universities’ faculty affairs units follow a rigid timeline determining, for example, when the candidate must submit their promotion portfolio and by when external review letters must be completed. The particular part of the P&T process we focus on is the meeting of the P&T committee at the department-level, to make their decision on the candidate. Department P&T committee decisions are influential since department P&T committee members are direct colleagues and usually in the same or adjacent disciplines as the P&T candidate.\nIn this paper, we define joint evaluation as when more than one candidate is being evaluated within this allotted window. In other words, there is more than one P&T candidate going up for promotion to the same rank, in the same department, in the same promotion cycle. Notably, this is a less rigid designation of joint evaluation compared to past laboratory studies where joint evaluation occurred at exactly the same time and was a forced-choice between two options26, that said, the proximity of the evaluations still suggests that decision makers will use the candidates as a reference point for each other27,36.”\nAs to whether we have the data by year, we do. We have also taken your advice and conducted a robustness check treating this data as a panel. To do so, we transposed the current dataset into a panel data with departments within each University as ID and year as the time variable. The panel data includes 326 unique departments in 7 years. We report these analyses in response to your more specific comment (C4) below concerning the time element.\n3. Tables 1 and 2 could be expanded to provide a clearer picture of the populations. I suggest having table 1 as only a table of means (and SD’s) of the universe of schools (or departments). I will assume that the pool is balanced (each school is in the data in each year). We might have statistics about the universities (size, category, possibly some measure of “rank” or competitiveness) separately by single vs. joint evaluations. I understand that there are 10 universities but later you say four universities had unusable data later on in the paper.\nThank you for this suggestion, we have created a new table that provides the means (and SDs) for each of the study variables broken down by school. Given that the primary analyses are conducted collapsing across schools, we decided to keep Table 1 in the main manuscript and add this new table in a supplement (see Table S2).\nOn the request of another reviewer we have also expanded Table 2 to include a breakdown by URM status (see Table 2)\nDue to the sensitivity of the data we intend to keep the identity of the universities as concealed as possible, therefore, we are hesitant to include detailed statistics about the universities. However, we have included a new section of the methods in which we go into more general details about the universities in our sample (p25-26; see below).\n“Our team built a consortium of universities to study promotion and tenure processes. In building this consortium, the goal was to bring together a diverse set of institutions in terms of size, research status, and ranking. In terms of student size the consortium includes two smaller institutions (5,000-15,000 students), three medium-sized institutions (40,000-50,000), and one larger institution (60,000+ students). The consortium primarily includes universities classified as Carnegie R1 institutions (i.e., very high research activity) but also one university classified as Carnegie R2 institutions (i.e., high research activity), providing variation of research activity. Further, whilst all institutions rank in the top 500 universities in the US according to the US news ranking, their position varies from 40th to 450th with the median position being 157th. In addition, the set of universities includes two minority serving institutions, one designated as a Hispanic- Serving Institution (HSI) and another as a Historically Black College or University (HBCU). Finally, the six institutions span four US states providing geographical diversity.\nAll institutions were allotted approximately one month for P&T decisions to be made at the department level. Therefore, when multiple candidates were evaluated in the same year they were both considered within this time window. For a full breakdown of the P&T timeline at each institution see Table S1; note, the identity of the university was masked to preserve anonymity.”\nAlso, thank you for allowing us to clarify about the institutions in our data vs. those in the consortium. Initially, the consortium that came together to create this dataset included 10 institutions. However, for a variety of reasons not all of the data is available for analysis. First, two institutions have not completed the coding process, so we cannot include their data. Second, after construction of the consortium we learned that two of the smaller institutions had non-traditional ways of completing P&T voting, e.g., they did not clearly structure P&T committees or record votes. So, we could not include this data in the analyses. Therefore, these four institutions are not included in analyses as none of them have data on voting outcomes; the focal dependent variable. We now explain this more clearly in a footnote (see p9).\n4. In the main model, you have institution and CIP code controls, meaning that you have what I believe are university and separately department (or field) fixed effects (from the table notes). This means that the variation could be coming from changes within university-departments over time, or is coming simply off of differences across departments within universities, possibly over time or not, it is unclear which, but is important. The data available for download show that there is within department variation, implying that either there were changes over time or the variation is simply random across years, most likely the former. One would have to assume that the switch across years was uncorrelated with any other changes that are related to the vote share, which is possible but unlikely. I might also suggest a first regression where the dependent variable is the treatment itself – joint eval, such that the reader can understand which universities/departments are more likely to feature this mode. Without a time element however, you can only learn so much from this.\nWe used a panel data analysis approach to verify the results. The panel data consisted with 7 unique years’ P&T decisions: ranging from 2016 - 2022 and 326 unique departments. First, we examined whether there is significant difference in joint evaluation across Universities and departments. The results showed that there was no significant difference in joint evaluation across Universities (F(5,1798) = 2.15, p = .06). The Tukey post-hoc analysis showed that none of the pairwise differences between schools are statistically significant at the 0.05 level.\nIn addition, we replicated the analyses using a panel data approach. We first conducted a Hausman test to determine the appropriate analysis to perform. The results of the Hausman test suggest using the random effect model (χ2 = 4.59, df = 3, p = .20). The panel data results are shown in Table S14.\nTable S14. Stepwise random effect models for the interactive effect of URM status and joint evaluation (also termed Joint Eval.) on department negative vote percentage.\nVariable Department Negative Vote %\nControls\nb SE b SE b SE b SE b SE\nURM status .10*** .02 .09*** .02 .07* .03 .09*** .02 .07* .03\nJoint evaluation .04*** .01 .04*** .01 .03** .01 .04*** .01 .03** .01\nURM x Joint eval. -.10** .03 -.09** .03 -.13** .04 -.09** .03 -.13** .04\nWomen -.003 .01 -.01 .01 -.004 .01 -.004 .01\nPromotion rank .02* .01 .04** .01 .02 .01 .04* .01\nTenure in rank .01** .002 .01*** .002 .01** .002 .01*** .002\nH-index -.001 .001 -.001 .001\nExternal grants -.00 .00 .00 .001\nTotal department votes -.00 .001 -.00 .001\nR2 .02 .07 .08 .07 .08\n5. I believe you’re using a linear probability model (i.e. OLS) which is probably just fine, but because the dependent variable is a percentage (0-1) and is therefore bounded, using a generalized linear model (GLM) with a logit-link might improve precision. I also suggest clustering standard errors on universities (or departments), given the marginal results I think it’s important to take inference seriously here. I tried this with the data and code provided and the standard errors shrink, which is surprising, but shows there is more precision to be had. Likewise, given there is variation within university-CIP cells in joint evaluation you can actually have a university-by-CIP fixed effects and results are similar.\nWe appreciate the suggestions and have followed your advice.\nWe first conducted the GLM analysis with logit-link by coding 1 = any negative vote, 0 = unanimous for the logit model. The results were shown in Table S6 below. The result patterns were consistent with those of OLS model with the p value of the interaction shrinking to .08. Note, in the paper we found that there was no interactive effect on unanimous votes so this result is consistent. This result is reported in the manuscript on p13.\nDepartment Negative Vote (1 = any negative vote, 0 = unanimous)\nb SE\nURM status .43 .37\nJoint evaluation .37* .17\nURM x Joint eval. -.95 .55\nWomen .03 .17\nPromotion rank .82 .19\nTenure in rank .04* .02\nH-index -.001 .007\nExternal grants .01 .01\nTotal department votes .07*** .001\n2 .41\nMcFadden R\n2 .50\nCox and Snell R\nWe also clustered the standard error on University and department to check the robustness of the results. The results are shown in Table S8 and reported in the paper on p14-15.\nTable S8. OLS regression models for the interactive effect of URM status and joint evaluation (also termed Joint Eval.) on department negative vote percentage using clustered standard errors for universities (Model 1) and departments (Model 2).\nCluster SE on Universities Cluster SE on Departments\nb SE b SE\nJoint evaluation .03*** .003 .03 .01\nURM x Joint eval. -.12* .04 -.12* .05\nWomen -.01 .01 -.01 .01\nPromotion rank .04*** .01 .04** .01\nTenure in rank .01*** .001 .01* .002\nH-index -.00001 .001 -.00001 .001\nExternal grants .001 .0004 .001 .001\nTotal department votes .0004 .0003 .0004 .004\n2 .08 .08\nR\nLastly, we ran the university-by-CIP fixed effects. We identified 103 University-by-CIP cells and ran the fixed effect using fixest package (see Table S9). The results are consistent with OLS results. We report these results on p14-15 in the manuscript.\nTable S9. OLS regression model for the interactive effect of URM status and joint evaluation (also termed Joint Eval.) on department negative vote percentage using university-by-CIP fixed effects\nDepartment Negative Vote %\nb SE\nURM status .06** .02\nURM x Joint eval. -.12** .03\nWomen -.0004 .001\nPromotion rank .04** .01\nTenure in rank .006*** .001\nH-index .0002 .001\nExternal grants -.001 .001\nTotal department votes .001 .001\nR2 .06\n6. I’d also suggest putting the sample sizes in the results table. You note some drop out due to missing data – you could remedy that by setting the covariates for those observations (assuming they’re not the dependent variable) to zero and including an additional dummy variable indicating that the control was missing to get some sample size back and rule out any issues from attrition across models. You lose about half the sample from the first to the last regression. Unless that attrition is random, it is worth figuring out which variables are causing the issue – it’s difficult to determine how much the sample change affects inference. At the very least the reader should be aware of this.\nBelow we provided a missing data summary. There is significant missing data in controls such as h index. This is because h index was retrieved from faculty google scholar pages, however, not all faculty in the data set had google scholars, meaning that their data was missing.\nThe presence of missing data is a major reason why we performed our analyses in a stepwise fashion, demonstrating that the results replicate when adding more controls and by virtue reducing the sample size. In the manuscript we now make this decision more explicit and highlight to the reader the decreasing sample size by indicating the N for each model and include the N for each model in the tables.\nBelow for your reference is a Table S3 which provides a count of the missing data.\nTable S3. Count of missing data for each variable used in analyses.\nVariable # Missing\nJoint Evaluation 0\nCandidate Gender 0\nNumber of Department Votes 0\nURM Status 58\nGrants Awarded as PI 196\nDepartment Negative Vote Percentage 209\nCandidate H Index 573\nWe also conducted a missing indicator approach by creating dummy variables for external grants and h index with missing values. The results are shown in Table S11 below and mentioned on p15.\nTable S11. Interactive effect of URM status and joint evaluation (also termed Joint Eval.) on department negative vote percentage with missing value dummy variables.\nDepartment Negative Vote %\nb SE\nURM status .09*** .02\nURM x Joint eval. -.10** .03\nWomen -.01 .01\nPromotion rank .02 .01\nTenure in rank .004* .001\nH-index .0004 .001\nH-index_missing .01 .01\nExternal grants -.001 .001\nExternal grants_missing .01 .02\nTotal department votes -.001 .001\n7. As an even smaller note, I’d suggest making the dependent variable positive votes as opposed to negative, it’s just easier to interpret the effect of joint evaluation as a “positive” effect. I apologize for what might seem like nit-picking here, please do not hear me saying I don’t believe (or like) the results, the opposite is true – I want to make sure they are as strong, reliable, and clear as possible. Now on to actual comments.\nThank you for this comment, certainly not nit-picking as it is a question we have thought about a lot as a team. We agree referring to positive vote percentage could make some sentences in this paper more intuitive to understand. However, given that the majority of votes at the department level are positive and many candidates can receive positive unanimous votes, we focused on negative votes as they are a deviation from the modal response. Further, just one negative vote can have a significant effect as it makes a vote non-unanimous.\nBeyond this rationale, we are also aware that this dataset has been worked on previously by other scholars (and planned for future scholars) and the standard has been set that researchers refer to “negative vote percentage” as opposed to “positive vote percentage”.\n8. The dependent variable has a pretty limited distribution. The data suggest that more than three-quarters of the votes were unanimous, meaning there is very little variation in the dependent variable to begin with. Possibly this explains the very small R2, and why the unanimous variable outcome does not show effects. This is pretty important for the analysis, and for the reader.\nThanks for suggesting this. The table below shows the distribution of the department negative votes. It shows that 75.5% of the votes were unanimous.\nDistribution of the Department negative vote %\nData category Percentage\n0 75.5%\n0-0.1 12.3%\n0.1-0.2 6.5%\n0.2-0.5 4.0%\n0.5-1 1.7%\nThank you for the suggestion that the relatively small variation in the dependent variable may explain the small R2 as this appears to be the case. We ran the same model with a subsample without unanimous votes, including 375 candidates. The results were consistent with these of the whole sample and R2 increased to around 0.27. These results are reported on p15 of the manuscript.\nTable S13. OLS regression results for the interactive effect of URM status and joint evaluation on department negative vote percentage excluding candidates who received zero negative votes.\nDepartment Negative Vote %\nVariable b SE\nURM status .17* .08\nURM x Joint evaluation -.30* .11\nWomen -.04 .03\nTenure in rank .01* .004\nPromotion in rank -.001 .04\nH-index -.001 .001\nExternal grants -.004 .002\nTotal department votes -.01 .001\nR2 .27\nN = 375.\n9. I may be reading the table wrong, but my interpretation is that on net, URM candidates receive the same share of votes. The main effect (e.g. Model 1 of Table 3) has a positive coefficient of 0.10 for URMs and the interaction effect is -0.09. I take this to mean that in that model (1) in joint evaluations, URM’s receive the same share of negative votes but when evaluated separately, they receive 9 percentage points fewer votes. I believe percentage points is the right interpretation here as the dependent variable is the percent of votes. I think you could add the baseline negative vote share to the table and then calculate what 0.09 is off of the base share of negative votes. On line 266 and in the abstract you say that the result is a 9% reduction in negative votes, which I believe is incorrect.\nThank you for providing us with the opportunity to clarify. What is agreeably a bit confusing is that the positive coefficient of 0.10 means that URM faculty are receiving more negative votes. The interaction term being negative indicates that this decreases in joint evaluation. The 9% number is coming for the generation of the simple slopes (see p13), which indicate that for URM faculty specifically that the relationship between evaluation mode and department negative vote percentage has a coefficient of 0.09.\nTo help make this clearer we have taken a number of steps. First we provide a model without the interaction (see Table 3), to indicate that URM status is associated with a higher negative vote percentage overall.\nFinally, on the suggestion of another reviewer, we have also conducted analyses to conceptualize the downstream effect of a shift in negative vote percentage on the Provost vote. At the institutions in this dataset the Provost vote represents the ultimate decisions on whether a candidate will receive a promotion (1) or not (0). Therefore, to contextualize our result, we calculated the conditional indirect effect of URM status on provost vote via department-level negative vote perception, depending on whether candidates were evaluated jointly or separately. In other words, how did the benefits of joint evaluation for URM faculty at the department level translate into likelihood of receiving a positive provost vote. Moderated mediation analyses found support for a conditional indirect effect (Index of moderated mediation = .65, 95% bootCI = [.18, 1.25]); see Table S7), meaning that the interaction between URM status and evaluation mode had downstream effects on provost vote via department level voting. Post-hoc tests found that, for a URM faculty, being evaluated jointly lead to a 16.2% increase in the likelihood of receiving a positive provost vote. We provide these analyses in the results section (p14) and mention the result in the abstract.\n10. I was also struck by the fact that H-index and grants have no relationship to vote shares, and by the lack of negative effect for female candidates. Do these fit with other results from the literature? You might consider interacting these with URM (and with joint), but I am not clear on what the story to tell would be. To add, the R2 is quite low considering you have CIP and university FE in the model (unless you reported the adjusted R2, which is fine but just say so). It’s hard to believe that all of those explain only a tiny fraction of the overall variance is explained by all of that information. Possibly this is due to the lack of variation in the dependent variable.\nThank you for noting this, we agree it is an interesting point and now pick up on it in the discussion (p22-23). We think this speaks to the intense complexity of P&T decisions, and the importance of a factor such as evaluation mode over specific metrics.\nInterestingly, past research has shown that factors like h index may have an indirect effect on P&T outcomes via content of external review letters (Madera et al. (2024) External review letters in academic promotion and tenure decisions are reflective of reviewer characteristics. Research Policy, 53(2), 104939) and influencing perceptions of minority candidates (Masters-Waage et al. (2024). Underrepresented minority faculty in the USA face a double standard in promotion and tenure decisions. Nature Human Behaviour, 8(11), 2107-2118) but it appears it is not a direct predictor of voting outcome.\nFor ease of reference provide the added section to the general discussion below (p22-23):\n“Another challenge associated with the use of a real world dataset is the complex inputs to decisions. As shown in Table 3, the model including all of the control variables accounts for 8% of the variance in negative vote percentage at the department level. An r-squared below 10% is not uncommon in applied psychological science59,60, given the inherent complexity and noise in the psychological process61. Further, as shown in the robustness checks, the r-squared increases to 27% when addressing skew in the dataset by excluding participants with no negative votes. Nevertheless, it does highlight the challenges in predicting P&T voting behavior. This is notable given that measures of scholar productivity (h index and grants obtained) are included in the model and neither of them significantly predicted voting outcomes. Although past research has identified indirect effects of scholarly productivity on voting outcome2,62 the lack of a direct relationship demonstrates that P&T decision makers derive evaluations of research impact and quality through means other than citation metrics, such as relying more on their qualitative assessment of the candidates’ contributions rather than on cumulative metrics66. Instead, it appears that P&T decision making is a more complex process, highlighting the need for more research into this topic in order to identify the factors driving these high-stakes decisions. ”\nFurther, in line with our response to Comment 8 above, we also think the lack of variation in the dependent variable also plays a role.\n11. I think you can learn something from the vote share vs. the unanimous vote vis-a-vis your hypotheses, in particular the anchoring effect. That it is not statistically significant does not imply the result is not meaningful. Nonetheless, I think Figure 2 is worth highlighting more - non-URM votes are unaltered by the switch, but negative URM votes decline by two-thirds (10 percentage points off of a base of 15 percent)! You could do this for unanimous vs. not as well. I think this says something about the mechanism – opinions about non-URM’s don’t change, URM’s are simply seen in a less negative light, which I think points more to the double-standard mechanism you mentioned in the framing section more than the negative stereotyping, though it is tough to pin the mechanism down.\nThank you for this suggestion. We did look into the effect for unanimous votes and found the effect not to be significant, we now report this figure in a supplement. We include a new subsection of the results focusing specifically on unanimous votes.\nMore critically, we agree that this finding does provide support inline with the double standard mechanism. In essence the double standard hypothesis is somewhat related to stereotyping, as it is one’s prior beliefs about a group (i.e., stereotypes) that lead to the double standard. That said, your comment encouraged us to give more attention to the double-standard mechanisms in the introduction and the discussion. This also provided us with the opportunity to discuss figure 2 in more depth (p19). Note, due to the updates in the analysis in response to the reviewers the effect for non-URM faculty is now significant with these individuals being evaluated more harshly in joint evaluation, receiving 3% more negative votes. We now mention this in the results and pick up on it in the discussion (see p19).\nFinally, we would like to thank you for your thorough review and hope our comments have sufficiently addressed them.\n38"}, {"round": 2, "reviewer_comments": {"reviewer_1": "The authors have adequately addressed my concerns.", "reviewer_2": "I thank the authors for their thoughtful and comprehensive response to the reviewers’ comments. I appreciate the point-by-point responses provided in their rebuttal letter.\nThe revised manuscript shows improvements in several areas. Most notably, the authors have streamlined the theoretical framing by removing the competing Hypothesis 2. I also appreciate the detailed information about the tenure process, the expanded analyses (including gender effects and various robustness checks), and the broader positioning of the findings beyond academia.\nI believe that the current version of this manuscript is very strong and I only have a few minor comments at this point\n1. I appreciate the authors’ effort to assess practical significance by linking evaluation mode to provost outcomes through negative vote share using a moderated mediation model. However, this approach restricts the analysis to a single pathway and assumes the effect of evaluation mode on the provost’s decision operates only through department-level votes. While this is useful for isolating that channel, I recommend also presenting the total effect of evaluation mode on the provost decision (e.g., regressing the provost vote directly on the URM x Joint Evaluation interaction). This would provide a complementary and more policy-relevant estimate of the intervention’s overall impact.\n2. I did not find Figure 2 to be a terribly compelling visual display of the key interaction effect and might suggest considering other ways of presenting the effect visually that would be more intuitive to general audiences, perhaps with a simple bar chart. It would also be helpful to state in the Figure 2 notes which regression estimates were used to generate the image presented.\n3. Ideally Table 2 would not only show summary statistics describing faculty and their departments by joint versus separate evaluation but would also present t-tests or two sample proportion tests to assess the balance of these summary statistics across conditions to provide statistical support for the claim that this was a valid quasi-experiment (reporting results in the section entitled “Quasi-Experimental Design” from page 9-12). An overall F-test could be used to assess balance given that some individual covariates are likely to be slightly imbalanced across the many tests conducted.\n4. In my first read of the manuscript, I had not picked up on the fact that these votes took place at universities that are not, for the most part, terribly prestigious. I think providing this context in the abstract and introduction (that this analysis was done at primarily R1 universities ranked outside the top 100 in the United States) would be helpful. The results could plausibly differ at highly selective institutions, so being explicit about where this work was done up front seems important.", "reviewer_3": "I co-reviewed this manuscript with one of the reviewers who provided the listed reports. This is part of the Nature Communications initiative to facilitate training in peer review and to provide appropriate recognition for Early Career Researchers who co-review manuscripts."}, "author_response": "Thank you for your supportive words. We addressed the additional comments, which helped significantly streamline and strengthen the paper. All suggestions were addressed through contained changes in the manuscript. We provide a detailed response to each comment (in bold) indented below (in plain text).\n1. I appreciate the authors’ effort to assess practical significance by linking evaluation mode to provost outcomes through negative vote share using a moderated mediation model. However, this approach restricts the analysis to a single pathway and assumes the effect of evaluation mode on the provost’s decision operates only through department-level votes. While this is useful for isolating that channel, I recommend also presenting the total effect of evaluation mode on the provost decision (e.g., regressing the provost vote directly on the URM x Joint Evaluation interaction). This would provide a complementary and more policy-relevant estimate of the intervention’s overall impact.\nWe appreciate your point here and the policy-relevance of the conditional main effect. We did not conduct this analysis previously as it does not directly test our theory of whether joint evaluation affects evaluation of URM faculty. This is because at the provost level there is almost always joint evaluation, as a provost will see numerous P&T candidate portfolios every year. Therefore, our effect is theoretically confined to the department-level where the evaluation mode varies.\nThat said, from a practical perspective we do see the value of estimating the conditional direct effect as this demonstrates the rates of promotion for candidates in each evaluation mode. Therefore, we have included this estimate in the manuscript (see p15):\n“We also examined the conditional main effect of URM status, i.e., regressing provost vote on the interaction between URM status and evaluation mode. The interaction term from these analyses was in the expected direction but was not statistically significant (b = -.03, SE = .04, p = .46).”\n2. I did not find Figure 2 to be a terribly compelling visual display of the key interaction effect and might suggest considering other ways of presenting the effect visually that would be more intuitive to general audiences, perhaps with a simple bar chart. It would also be helpful to state in the Figure 2 notes which regression estimates were used to generate the image presented.\nWe experimented with different visualizations and are grateful for your suggestion as we decided to go with a bar chart. It is indeed a better graphical representation of results in the manuscript, as shown in the figure below:\nFigure 2. Relationship between URM status and negative vote percentage at the department level based on regression estimates from interaction effect in the base model (i.e., no control variables.\nURM\ne 20\ng\na\nt\nn\ne\nc r 15\ne\nP\ne\nt\no\nV 10\ne\nv\ni\nt\na\ng 5\ne\nN\nEvaluation Mode\n3. Ideally Table 2 would not only show summary statistics describing faculty and their departments by joint versus separate evaluation but would also present t-tests or two sample proportion tests to assess the balance of these summary statistics across conditions to provide statistical support for the claim that this was a valid quasi-experiment (reporting results in the section entitled “Quasi-Experimental Design” from page 9-12). An overall F-test could be used to assess balance given that some individual covariates are likely to be slightly imbalanced across the many tests conducted.\nWe have revised Table 2 to include t-tests for each comparison and reported these results in the section titled “Quasi-Experimental Design”, see p11. We ran separate t-tests as we did expect one variable to differ across the conditions. That was total department votes, as it is expected that larger departments will be more likely to have more joint evaluations occur. However, as shown, this is the only variable that significantly differed across conditions and is included as a control variable. With this addition of a row, we also separated the URM status comparison into a new separate table (Table 3). We provide Table 2 below:\nTable 2. Breakdown of the demographics for separate vs. joint evaluation. The final row reports Welch’s t-test for comparison across conditions for each variable.\nWomen H-Index Extensions Total Years in US News External URM Status\nDepartment Present Rank University Grants as PI\nVotes Ranking\nSeparate evaluation 38.20% 14.96 7.35% 8.72 6.06 138.49 4.84 12.74%\nJoint evaluation 36.09% 15.80 7.06% 10.35 6.06 133.96 5.16 11.31%\nT-test (t) 0.92 -1.43 0.57 -3.92*** 0.02 1.46 -0.85 0.92\n* p < .05; ** p < .01; *** p < .001\n4. In my first read of the manuscript, I had not picked up on the fact that these votes took place at universities that are not, for the most part, terribly prestigious. I think providing this context in the abstract and introduction (that this analysis was done at primarily R1 universities ranked outside the top 100 in the United States) would be helpful. The results could plausibly differ at highly selective institutions, so being explicit about where this work was done up front seems important.\nWe have included a statement highlighting this in the introduction. We do think it is a relevant point and mention in the context of the breadth of our sample which includes a range of universities including public and private, and large, medium, and smaller institutions. In developing our consortium partnership, we intentionally pursued the inclusion of a broad set of institutions that represent Higher Education in the U.S. more broadly than a smaller, more prestigious set of universities. Having that said, while our dataset does not include top 10, highly prestigious institutions, the dataset does contain universities ranked in the top 50 and top 100. We think this variety helps strengthen our conclusions. Below is the revised sentence from the introduction stating this (see p7):\n“We examine this hypothesis in the context of P&T decisions made within a diverse set of six US public and private institutions, varying in size (~10,000 to ~80,000 students), from across four states, and ranging in their US News national university ranking (rounded to the nearest ten universities rank as 50th, 50th, 130th, 170th, 170th, and 390th in the nation)”"}, {"round": 3, "reviewer_comments": {"reviewer_2": "I thank the authors for their responses to my comments, particularly regarding the practical significance analyses, the improved visualization in Figure 2, the statistical tests for balance across conditions, and the clarification of institutional context.\nI believe the current version of the manuscript is very strong and makes an important contribution to the literature. I have no further comments.", "reviewer_3": "I co-reviewed this manuscript with one of the reviewers who provided the listed reports. This is part of the Nature Communications initiative to facilitate training in peer review and to provide appropriate recognition for Early Career Researchers who co-review manuscripts."}, "author_response": ""}]

Decision letter