# Industry Collaborations of Research Teams: Are They Penalized or Rewarded in the Grant Evaluation Process ?

Sila Öcalan-Özel, Patrick Llerena

► To cite this version:

Sila Öcalan-Özel, Patrick Llerena. Industry Collaborations of Research Teams: Are They Penalized or Rewarded in the Grant Evaluation Process?. Frontiers in Research Metrics and Analytics, 2021, 6, 18 p. 10.3389/frma.2021.707278. hal-03571919

HAL Id: hal-03571919

https://hal.univ-lorraine.fr/hal-03571919

Submitted on 4 Jan 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

Check for
updates

# Industry Collaborations of Research Teams: Are They Penalized or Rewarded in the Grant Evaluation Process?

Sila Öcalan-Özel* and Patrick Llerena

BETA, Université de Strasbourg, Strasbourg, France

This paper explores the relationship between the industry collaborations of grant applicant teams and the outcomes of a multistage grant evaluation process. We studied this relationship by focusing on two possible channels of impact of industry engagement—team diversity (or the diversity effect) and prior collaboration experience (or the experience effect)—and examined their influence on the evaluators’ decision by using the proxies of direct industry engagement (i.e., the involvement of a company-affiliated researcher in the grant applicant team) and indirect industry engagement (i.e., joint publications with a company-affiliated researcher prior to the grant application), respectively. We analyzed data extracted from the application and reviewed materials of a multidisciplinary, pan-European research funding scheme—European Collaborative Research (EUROCORES)—for the period 2002–2010 and conducted an empirical investigation of its three consecutive grant evaluation stages at the team level. We found that teams presenting an indirect engagement were more likely to pass the first stage of selection, whereas no significant relationships were found at any of the three evaluation stages for teams presenting a direct engagement. Our findings point to the heterogeneity of the decision-making process within a multistage grant evaluation scheme and suggest that the policy objective of fostering university–industry collaboration does not significantly impact the funding process.

**Keywords:** industry collaboration, diversity, prior experience, grant peer review, research funding

## OPEN ACCESS

**Edited by:**
Fabiana Visentin,
United Nations University—Maastricht
Economic and Social Research
Institute on Innovation and
Technology, Netherlands

**Reviewed by:**
Stefano Baruffaldi,
University of Bath, United Kingdom
Markus Simeth,
Copenhagen Business School,
Denmark

*Correspondence:
Sila Öcalan-Özel
silaocalan@gmail.com
s.ozel@unistra.fr

**Specialty section:**
This article was submitted to
Research Policy and Strategic
Management,
a section of the journal
Frontiers in Research Metrics and
Analytics

**Received:** 09 May 2021
**Accepted:** 10 September 2021
**Published:** 21 October 2021

**Citation:**
Öcalan-Özel S and Llerena P (2021)
Industry Collaborations of Research
Teams: Are They Penalized or
Rewarded in the Grant Evaluation
Process?
Front. Res. Metr. Anal. 6:707278,
doi: 10.3389/frma.2021.707278

## INTRODUCTION

With the emergence of knowledge-based economy, promoting university–industry (U–I) knowledge and technology transfer (KTT) has become one of the main objectives of science, technology, and innovation (STI) policies (Geuna and Muscio, 2009). To achieve this goal of stimulating KTT, a recently published OECD report (2019) on U–I collaboration shows that the organization’s member states deploy different policy mixes, combining financial, regulatory, and soft policy instruments. One of the policy tools that has contributed most to the recent evolution of the institutional framework underpinning the STI policy has been research funding, in the form of both block and competitive grants. However, while the latter are among the most widely used financial instruments across OECD countries, there are marked heterogeneities in terms of the budgets allocated, the duration of these grants, their beneficiaries, and the selection criteria employed (OECD, 2019).
Funding agencies around the world award grants for research, some of which are dedicated directly to promoting U-I collaboration. Such direct collaborative grant schemes aim principally at enhancing business innovation and boosting the social return from public investment in science and international competitiveness (Cunningham and Gök, 2012); at the same time, they are also becoming common in both low- and middle-income countries, as they strive to catch up with their more developed counterparts (Bruhn and McKenzie, 2019). Yet, even in the cases in which a grant program is not explicitly aimed at promoting KTT, those involved in the decision-making process may be inclined to foster U-I collaboration (Banal-Estañol et al., 2019a). Thus, in practice, evaluators who look favorably (or unfavorably) on U-I KTT might be more (or less) likely to take into consideration the industry collaboration activities of an applicant team when allocating public funds.

In this study, we seek to determine whether the industry engagements of applicant teams are penalized or rewarded by evaluators at the various stages of the grant evaluation process. In practice, industry collaborations may be penalized because evaluators can deem them to be of little or no value for scientific output, or they may simply wish to avoid the risk that the collaborative firm will appropriate the publicly funded research result. This latter argument finds some support, as it has been shown that the propensity of academic scientists to share information tends to be lower when there is a risk of appropriation (Haeussler, 2011). Unlike academic science, where conformity to the norms of openness (Merton, 1973) is the widely accepted practice, industrial science is often guided by secrecy given the commercial interests involved (Cohen et al., 2000), resulting in a lower propensity for information-sharing among such scientists. For these reasons, evaluators might penalize industry collaboration.

In contrast, some evaluators might show a greater propensity to favor applicant teams' industry collaborations, especially in certain application-oriented disciplines such as engineering. Funding agencies and evaluators that seek to foster U-I collaboration may opt to award grants to teams that present a clear industry engagement. Thus, they are likely to select applicant teams that show themselves capable of producing either technologies that have the potential for industrial application or original research that can permit the transfer of knowledge, or teams that have previous collaboration experience that has resulted in the development of KTT networks. To the best of our knowledge, there has been no attempt to date to examine the relationship between U-I collaborations and the grant allocation process, with the exception of Banal-Estañol et al.'s (2019a). Yet, even here, the authors do not directly address this specific relationship, and they do not seek to offer an in-depth explanation of the forces at play. Indeed, the study limits itself to the disciplines of the physical sciences and engineering in the United Kingdom and concerns itself solely with the final outcome of the grant application process (i.e., successful or otherwise).

Here, to account for the potential effects of industry collaboration on the research funding process, we examine the role of two possible channels of impact of the U-I collaborations

of applicant teams during the grant allocation process. The first is given the label *diversity effect* and is captured by the *direct* industry engagement of applicant teams, and the second is labelled the *experience effect* and is captured by the *indirect* industry engagement of applicant teams. By using *direct* and *indirect* industry engagements as proxies, we are able to measure the impact of team diversity and prior experience of industry collaboration on the decisions made by different evaluators (namely, the members of an Expert Panel and External Reviewers) in the multistage, multidisciplinary, pan-European grant scheme of EUROCORES (European Collaborative Research) managed by the European Science Foundation (ESF).

We examine the nature of *direct* industry engagement in the light of discussions in the diversity literature, where diverse teams have been shown to more likely exhibit both higher ex-post performance (Williams and O'Reilly, 1998) and produce novel combinations and breakthrough research (Arts and Veugelers, 2015). In this respect, we conjecture that applicant teams having an industry/company-affiliated partner in the grant application are more likely to produce original research, as they are diverse in knowledge and skills, and more likely to produce transferable technology for the industry given that they are in direct proximity to the industry network. We refer to this as *direct* industry engagement or the *diversity effect*.

Additionally, evaluators may select applicant teams on the grounds that they already present certain embedded characteristics, including prior experience of industry collaboration for the transfer of knowledge and technology. In this respect, we conjecture that applicant teams with past experience of industry engagement should be more capable of KTT in the future. As a proxy of prior experience, we consider applicant teams that include a member with a joint-publication with industry/company prior to the present grant application. We refer to this as *indirect* industry engagement or the *experience effect*.

To test these two conjectures concerning the relationships between *direct/indirect* industry engagement and the outcomes of the three consecutive stages in a grant evaluation process, we draw on a sample, providing information about 10,533 EUROCORES grant applicants across all scientific domains, corresponding to 1,642 outline proposals and 886 full proposals for the period 2002–2010. The empirical results suggest that teams with an *indirect* industry engagement are more likely to pass the first stage of selection (outline proposals) conducted by Expert Panel members, whereas no significant link can be established for the subsequent evaluation stages (full proposals) conducted, first by External Reviewers and second by Expert Panel members. More specifically, we are unable to observe a significant link between *direct* industry engagement and the outcome of the grant evaluation process at any of its stages. The presence (or otherwise) of such a correlation may, among other reasons, be attributable to the heterogeneity in the decision-making processes perpetuated by different actors at subsequent stages (e.g., collective vs individual decision making; decision-making subject to time constraints; and having to access detailed information) or to the objective of the funding agency that explicitly recognizes the wish to promote U-I collaboration. In
the case of EUROCORES, the explicit objective of the grant scheme was to stimulate the highest quality innovative, multidisciplinary, and investigator-driven collaborative research, by bringing together different scientists and national research funding organizations; however, fostering U-I collaboration was not an explicitly stated aim.<fnref n="1" /> Nonetheless, as Banal-Estañol et al. (2019a) argue, even in the absence of such an explicit aim, the evaluators may have been implicitly inclined to foster industry collaborations.

The rest of the paper is structured as follows. The *Grant Evaluation Process and Industry Engagement* section presents a general framework in which the grant peer-review process is outlined, the *diversity* and *experience* effects and their respective proxies of *direct* and *indirect* industry engagements are described, and the EUROCORES funding scheme is explained. The *Data and Methodology* section describes the data, variables, and the methodology used. The *Econometric Results* section presents the empirical results for the three consecutive evaluation stages, and the *Discussion and Conclusion* section concludes by discussing these outcomes.

# GRANT EVALUATION PROCESS AND INDUSTRY ENGAGEMENT

## Grant Peer-Review Process

Although some funding agencies award grants on the basis of their own in-house evaluations, a major proportion of competitive grants are allocated using a peer-review process (Stephan, 2012; Roumbanis, 2019). This can have both positive and negative outcomes: on the one hand, a peer-review system encourages researchers to remain productive throughout their careers, on the other hand, the system discourages risk taking, tending to support projects assessed as having a high degree of “doability” (Stephan, 2012), and thus the peer-review process has a marked influence on the direction and orientation of scientific research in general.

Funding organizations typically state that they allocate grants in accordance with a project's scientific merit or excellence. Indeed, there is clearly a need to allocate scarce resources to feasible research projects with the potential to generate high-quality output of notable impact (Tijssen et al., 2002). However, claims have been made that funding decisions are not simply based on scientific merit or excellence but that they are also influenced by, among other factors, the personal attributes of individual applicants (Witteman et al., 2019), the structure of the research teams (Banal-Estañol et al., 2019b), the personal attributes of the peer reviewers evaluating the projects (Gallo et al., 2016), the intellectual distance (Boudreau et al., 2016), and the network ties between the applicants and their evaluators (Ebadi and Schiffauerova, 2015).

The grant peer-review process appears to be prone to different types of bias, being influenced by factors unrelated to a project's

actual quality (see, for example, Wennerås and Wold, 1997, for a discussion of the sexism inherent to the process). Such bias might be both intentional and unintentional, given that the funding process is characterized by a whole series of uncertainties regarding the true quality of a research proposal. A review of the literature reveals the different types of bias with which applicants have to contend. Bromham et al. (2016), for instance, found that high levels of interdisciplinarity reduce the likelihood of attracting funds. The authors claim that while interdisciplinary research is often encouraged at the policy level, funding organizations or reviewers tend to favor more focused research, pointing to what has been called an “interdisciplinarity paradox” (Woelert and Millar, 2013). This interdisciplinarity bias may reflect the fact that evaluators have a highly focused expertise, and hence find it difficult to evaluate all sections of a given project (Woelert and Millar, 2013; Bromham et al., 2016). A similar bias may occur in reaction to novelty. Indeed, novel projects are difficult to evaluate as they involve a high degree of uncertainty and are thus usually perceived as riskier in terms of feasibility (Boudreau et al., 2016; Stephan et al., 2017; Wang et al., 2017).

In this study, among the various factors that might affect a funding decision, we are specifically concerned with the impact of industry engagements or collaborations of research teams on the grant evaluation process. In particular, we examine the effects of a team's *direct* and *indirect* industry engagements in terms of the *diversity effect* of the former and the *experience effect* of the latter.

# Structural Diversity and Experience of Prior Collaborations

In this subsection, we discuss the impact of the industry engagement of research teams on the grant allocation process in relation to two factors: team diversity and prior experience. In a grant allocation process, KTT policy goals can be achieved either by selecting researchers who are able to produce original research outcomes with a high potential for transfer to industry or by selecting researchers with prior experience of KTT and who are, therefore, more likely to continue to foster transfer in the future.

The research performance of teams has been associated with their diversity (Horwitz, 2005; van Knippenberg and Mell, 2016); thus, teams that can draw on different knowledge bases and skills are more likely to produce new combinations and more breakthrough research (Arts and Veugelers, 2015). It should be noted, however, that *diversity* may be used to refer to different attributes of a team. Indeed, the distinction between demographic and structural diversity has been highlighted in this body of literature (see Williams and O'Reilly, 1998; Cummings, 2004), with the former referring to easily observable characteristics, such as gender, age, and race, and the latter referring to cognitive abilities. Diversity in cognitive attributes is a widely accepted condition leading to novel combinations and hence to radical innovations in the Schumpeterian sense (Nelson and Winter, 1982; Fleming, 2007). Cummings (2004) shows that structurally diverse teams—comprising members with different organizational affiliations, roles, or positions and with distinct knowledge bases and skills—present performance benefits,

<footnote n="1">¹For more information, please see http://archives.esf.org/coordinating-research/eurocores/the-scheme.html (Last accessed on 21/06/2021).</footnote>
especially when these members share knowledge outside of the
group. Hence, teaming up with an external partner contributes to
structural diversity and increases the potential to produce original
or novel research.

The potential for producing original research might be a factor
favored by evaluators in the grant peer-review process. By
positively considering the structural diversity introduced by a
project's industry partners, evaluators can target two objectives:
they can increase team diversity and can facilitate KTT to the
industry. Yet, too much diversity can increase communication
and coordination costs between team members (Dahlin et al.,
2005). Moreover, the novel methods and theories that are likely to
be developed by diverse teams—in particular, those considered
frontier research—are subject to greater uncertainty (Lee et al.,
2015). Indeed, the literature points to a “bias against novelty” on
the grounds that the competitive selection procedures of funding
agencies and increasingly risk-averse evaluators prefer safer
projects to risky novel research (Boudreau et al., 2012; Wang
et al., 2017; Wang et al., 2018). Indeed, diverse teams may be
penalized and see their projects rejected as evaluators may find it
difficult to assess this diversity (Lamont, 2009), perceiving their
projects to be riskier and less feasible (Luukkonen, 2012). In a
recent study, Banal-Estañol et al. (2019b) found that structurally
diverse teams exhibiting greater heterogeneity in terms of
knowledge and skills, education, and/or scientific ability are
less likely to receive grants. Evaluators or funding agencies are
thus faced with a dilemma in which they have to decide whether
or not to allocate scarce funds to structurally diverse teams.

In addition to a team's structural diversity, a further
dimension that might prove influential for funding
agencies seeking to foster KTT—either explicitly or
implicitly—is a team's prior experience of collaboration
with the industry. Previous participation in KTT activities
generates greater expectations about continuing transfer
practices in the future (Bercovitz and Feldman, 2003) as a
result of learning by doing or learning by interacting with
industry. For instance, D'Este and Patel (2007) found that
researchers with prior industry collaborations are more likely
to be involved in interactions with industry both in terms of
the frequency and variety of interaction channels used. To
account for previous collaborations, the authors use a number
of joint publications with the industry and the average value
of collaborative grants with industry in order to capture the
effects of a variety of channels. In addition, the authors
suggest that a large number of interactions with industry
in the past may indicate the establishment of a personal
network of relationships which is likely to have generated
enduring mutual trust. Thus, funding agencies that seek to
promote knowledge transfer may favor teams with prior
experience of industry engagement and those that,
therefore, have access to an already established industry
network, making them more likely to transfer knowledge
in the future. Conversely, if collaborative activities are
perceived negatively—because, for instance, evaluators
believe that collaborations interfere with the orientation of
the proposed research—then past collaborations may be
penalized in the grant evaluation process.

Direct and Indirect Industry Engagements

In this subsection, we differentiate between the direct and indirect industry engagements of grant applicant teams in the light of team diversity and their prior experience of industry collaborations.

Empirical KTT research to date has focused primarily on
formal interaction channels—above all, patents—given their
greater data availability and ease of measurement. Yet, there is
a growing stream in the literature that concerns itself with less
formalized channels (Grimpe and Fier, 2010; Perkmann et al.,
2013). Here, we also focus on these less formal channels and
explore the relationship established by two types of industry
collaboration engaged in by applicant teams with grant evaluation
outcomes.

We measure industry engagement by focusing on two types of interactions. First, we examine *direct* engagements (those that manifest during the grant application process), that is, joint or collaborative project applications with industry, whereby an industry/company affiliated member forms a part of the team. If the team members are affiliated to different organizations (i.e., university or industry), the team is likely to exhibit greater diversity in terms of knowledge and skills. As discussed previously, structurally diverse teams are more likely to create novel combinations, which should increase the probability of their producing transferable (or even transferred) output. They can achieve this because of the enhancement of proximity to the market of their research results and the fact that such teams form a part of an established industry network thanks to their ongoing collaboration. Thus, team diversity, proxied here by *direct* engagement, could be a decisive factor in a grant allocation process.

Second, we examine indirect engagements (those made prior
to the grant application), that is, joint publications with industry.
Prior joint publications involving applicant team members and
an industry/company affiliated researcher are used, here, as a
proxy for the prior collaboration experience (Tijssen et al., 2009).
Past collaboration with industry allows teams to enhance their
KTT capacity in the future, exploiting mechanisms of learning by
doing and an already established network with industry
researchers (D'Este and Patel, 2007). Thus, prior experience of
collaboration with industry, proxied here by indirect engagement,
could also be a decider in the allocation of grants. Below, we
discuss in further detail our proxies of direct and indirect industry
engagements and how we propose measuring them.

Direct Engagement: Collaborative Project Applications With Industry

First, to capture the diversity effect, we consider applications to
the funding program in question that include an industry partner.
This we refer to as a team's direct engagement, being indicative of
an ongoing collaborative research project with industry/company
at the time of application.

Large-scale empirical studies at the aggregate level find
complementarities between public and industry collaboration
in R&D projects (David et al., 2000). The synergy between
university scientists' research and firms' innovative efforts to
create new products and processes is found to be the main driver of U-I collaborations (Mansfield and Lee, 1996).

Applying for competitive research grants in collaboration with firms has benefits as well as costs for both parties. On the one hand, collaboration might attract extra funding from industry (in addition to the grants allocated through public funds) and might provide new perspectives for the collaborating parties (Mansfield, 1995; Lee, 2000). On the other hand, collaboration might have a number of distorting effects, such as the appropriation of research results by industry. Collaboration might also result in delays in the publication of outcomes, which can negatively affect the allocation of public funds to such teams if the funding organization or evaluator perceives dissemination through publications to be a priority. On this particular matter, the literature is divided. Some studies report a link between collaboration and a higher number of publications (Fabrizio and Di Minin, 2008; Azoulay et al., 2009), while others observe a negative impact (Murray and Stern, 2007; Huang and Murray, 2009) or an inverted U-shaped relationship (Crespi et al., 2011; Banal-Estañol et al., 2015).

Banal-Estañol et al. (2013) reported that collaborative research projects with firms result in more scientific output than that produced by non-collaborative projects, provided the partner firm is research intensive. However, it has been argued that collaboration can divert the orientation of a university's research agenda (Florida and Cohen, 1999) and be detrimental to the open science culture (Dasgupta and David, 1994). Similarly, while collaborations with academia may not coincide with the rule of secrecy prioritized by firms, they might have positive effects on the productivity of firms' R&D (Cohen et al., 2002), on their patents, products, and sales (Zucker et al., 2002; Cassiman and Veugelers, 2006) and on their absorptive capacity (Cohen and Levinthal, 1990).

An additional dimension affecting the collaboration decision is the type of research in which researchers engage (i.e., basic vs applied). Drawing on the grant application data of the UK's Engineering and Physical Sciences Research Council (EPSRC), Banal-Estañol et al. (2019a) found that applied researchers tend to submit more collaborative projects with firms. Furthermore, they found an inverted U-shaped relationship between team size and collaboration with firms and a lower rate of collaboration with teams based in a Russell Group university. Their findings also reveal that collaborative projects with firms are more likely than non-collaborative projects to be funded, yet these collaborative projects have a lower publication rate.

U-I collaborations strengthen diversity and, hence, provide fertile ground for the proliferation of original research. Yet, diversity, while sometimes rewarded, might also be penalized during the grant allocation process as novel research is associated with higher risks. Thus, evaluators may either look favorably on the collaboration and opt to allocate public grants to structurally diverse teams or, equally, they might opt to deny a collaborative project funding due to the aforementioned costs of industry collaborations.

## Indirect Engagement: Joint Publications With Industry

Joint publications with industry are one of the analytical tools that enable a comparison and assessment of the interactions between universities and industry (Tijssen et al., 2009). Given the confidential nature of contractual engagements, previous studies have used coauthorship as a proxy for collaborative activity (Mindruta, 2013).

Yet, as Katz and Martin (1997) remind us, co-publication data are not without their limitations. For instance, not all joint publications are indicative of an actual U-I collaboration, given that a coauthor might have left the university and affiliated to industry or he or she might have a dual affiliation. Despite these limitations, joint publications are one of the standard means for measuring interorganizational research collaborations, given that the indicator is quantifiable, invariant, and relatively inexpensive to access (Abramo et al., 2009).

An active collaboration may manifest itself in joint U-I publications in peer-reviewed journals and conference proceedings (Melin and Persson, 1996; Tijssen et al., 2009; Tijssen, 2012). Such co-publications include the author's company or industry address, thus allowing the existence and scope of collaborative activity to be identified. Thus, a joint publication provides the information needed for constructing aggregate-level proxy measures for U-I collaboration, both in terms of intensity and magnitude (Tijssen et al., 2009; Tijssen et al., 2016). Abramo et al. (2009) reported a positive influence of joint publications with industry on the research performance of university researchers in Italy. Moreover, a number of studies have found a positive influence of joint publications on the KTT output of universities (D'Este and Patel, 2007; Carayol and Matt, 2004; Wong and Singh, 2013).

Even in such cases where the affiliation of the coauthors may not be immediately visible during the evaluation process (because of either a lack of time or the availability of resources), co-publications may reveal the existence of another type of collaborative activity in the past that applicants explicitly detail in their CVs. Furthermore, forming a part of an industry network as a result of such collaborative activities might be an indicator of greater research visibility which should be readily observable in the review process.

Thus, joint publications with industry/company are useful analytical tools that allow us to proxy for prior industry engagement of a grant applicant team and hence to their ability to transfer knowledge and technology in the future. In the broader research funding framework, we refer to this activity as a team's *indirect* engagement with industry since it is something that has been established prior to the grant application. It is our conjecture that prior experience of collaboration could be decider in a funding process for evaluators that seek to foster U-I relations. Yet, to the best of our knowledge, no study to date has examined the effect of joint publications with industry on the outcomes of the grant evaluation and/or funding allocation process.
<figure><img src="image_3.png" /><figcaption>**FIGURE 1** | EUROCORES multistage evaluation process. CRP: Collaborative Research Project, NFO: National Funding Organization.</figcaption></figure>

In the section that follows, we describe the specific stages making up the grant evaluation and allocation process of the EUROCORES Scheme.

## EUROCORES Grant Evaluation and Allocation Process

European Collaborative Research (EUROCORES) was a broad, complex pan-European grant scheme aimed at supporting collaborative and interdisciplinary research by bringing together national research funding and performing organizations. The program facilitated multinational research collaboration, while funding remained at the national level. EUROCORES was initiated by the ESF in 2001 and included in the European Commission's Sixth Framework Programme (FP6) between 2003 and 2009. Networking and science support activities were later financed by the ESF's member organizations until the EUROCORES scheme was terminated in 2015.

The evaluation of Collaborative Research Project (CRP) proposals in the EUROCORES scheme was a multistage process and included the establishment of an independent, international Expert Panel (see **Figure 1**). In response to an open call for proposals, outline proposals of about three pages in length were submitted by a team of applicants composed of principal investigators (PIs) and associate partners (APs), coordinated by a project leader (PL) (in principle, a minimum of three PIs from at least three different countries were required).<fnref n="2" /> At this outline proposal stage (First Stage), the Expert Panel was responsible for the sifting of proposals prior to the invitation of full proposals. The panel constituted a group of experts in the relevant scientific domains, brought together for a 1- to 2-day face-to-face meeting to select the proposals via a process of consensus.

At the full proposal stage (Second Stage), each proposal was sent for written external assessment to at least three independent reviewers, including reviewers from outside Europe. Reviewers were asked to fill out an evaluation form comprising 8-10 sections. Each section addressed a different criterion, including

the scientific quality of the proposal, the multidisciplinarity of the proposal, the suitability of the PIs and their groups for the project, the complementarity of the groups, the feasibility of the proposal and the budget requested. For each criterion, the reviewers were asked to assign a score on a four- or five-point Likert scale and to add their written comments in the corresponding space. Reviewers were typically given 4 weeks to complete their reports, and the applicants then had the opportunity to reply to the anonymous reviewer reports.<fnref n="3" />

Finally, the written reviewer assessments and the replies submitted by the applicants were considered by the Expert Panel, with scientific quality constituting the main selection criterion. In the last stage of the scientific assessment (Third Stage), the Expert Panel selected the CRPs that were to be recommended to the National Funding Organizations (NFOs) for funding. National eligibility criteria might then have had to be applied to particular applicants from countries that specified “special national requirements” for project applications. Finally, the NFOs were asked to approve funding for the individual teams from their countries in accordance with established national procedures.

## DATA AND METHODOLOGY

### Description of Data

We used research grant data drawn from the EUROCORES funding scheme, provided by the ESF, for the period 2002–2010. The initial data set provides information for 10,533 grant applicants across all scientific domains who teamed up to form a CRP, corresponding to 1,642 outline proposals and 886 full proposals. The process was overseen by a total of 2,246 External Reviewers and 829 panel members. EUROCORES established 47 programs<fnref n="4" /> in a range of disciplines (Life, Earth and Environmental Sciences; Biomedical Sciences; Physical and Engineering Sciences; Humanities and Social

<footnote n="2">²In this study, the participants of a CRP team are considered as being the PIs of an individual project (IP) within a CRP. CRPs are coordinated by a project leader (PL) and might include self-funding APs, who are also PIs of an IP.</footnote>

<footnote n="3">³Note the Second Stage, carried out by the External Reviewers, was not a selection stage. The reviewers' reports and the scores given to the full proposals acted as a guide to the Expert Panel in the last stage of the process.</footnote>

<footnote n="4">⁴Each CRP was included within a corresponding program, and an Expert Panel was set up for each program.</footnote>
Sciences), and about 150M euros was allocated to fund the
selected projects.

For each application submitted, the data contained
information on the name, year of birth, gender, and
institutional affiliation of the applicants and evaluators and
reported the date of the application and the amount of
funding requested. Information concerning the applicants'
*direct* engagement with industry, proxied here by an ongoing
project partnership in a CRP with a company-affiliated
researcher, can therefore be obtained directly from the
EUROCORES database. The data also reported the decisions
of and the scores given by the evaluators at the
corresponding stage.

Bibliometric data on research publications were extracted
from the Web of Science (WoS) Core Collection, a
comprehensive, international multidisciplinary bibliographic
database. As of 2020, the WoS Core Collection contained over
21,100 international peer-reviewed journals (including open
access journals) in 254 WoS journal subject categories, as well
as books and conference proceedings. As with its similar
counterparts, the WoS database presents a number of
drawbacks, including the overrepresentation of English-
language journals to the detriment of other languages and the
underrepresentation of publications in the Social Sciences and
Arts and Humanities (Mongeon and Paul-Hus, 2016; Vera-
Baceta et al., 2019). Nevertheless, the WoS database continues
to be one of the main sources for bibliometric analyses.

The bibliographic record of each publication contains, among
other details, the authors' institutional affiliation, which enables
us to identify joint publications with industry (or company)
affiliated researchers. To identify coauthors affiliated to a
company, we ran a computer code that was able to detect
abbreviations or extensions related to a private entity, such as,
"Itd., Inc., GmbH, SRL, public limited company, limited liability
company". Through this coauthorhip relation, we were able to
detect the applicants' *indirect* engagement with industry prior to
the grant application. In our database, 48,458 out of a total of
707,158 applicant publications include at least one coauthor with
a private company affiliation (i.e., 6.8%).

Econometric Treatment

To capture both the *diversity* and *experience* effects, we examined the influence of the *direct* (project partnership) and *indirect* (coauthorship) industry engagement of the applicant teams on the outcomes of the three consecutive stages in the grant peer-review process: that is, the first Panel Stage (First Stage), the External Reviewer Stage (Second Stage), and the second Panel Stage (Third Stage). In our analyses, we do not consider the final decision of the NFOs to either fund or not fund the project, given that this ultimate stage is only marginally different from the Third Stage of selection. In the majority of cases, the NFOs adhered to the funding recommendations of the Expert Panels.<fnref n="5" />

In the First Stage, we estimate the probability of passing to the
second stage, in which applicants were invited to submit the full
proposal. As the dependent variable in this stage is binary (i.e.
passed or not), we use a probit model. In the Second Stage, we
used an OLS model since the dependent variable is defined as the
average review score awarded by the External Reviewers. In the
analysis reported here, we consider the average reviewer score
calculated over the following five questions<fnref n="6" />: Q1—scientific
quality of the proposed research, Q2—multidisciplinarity of
the proposal, Q3—suitability of the PIs for the project,
Q4—the level of collaborative interaction between the groups
in the project, and Q5—feasibility of the project and
appropriateness of methodology. We had to standardize the
review scores because in some of the years, scores were
awarded on a 4-point Likert scale, while in others, these were
awarded on a five-point scale. Finally, in the Third Stage,
regarding recommendation for funding, we again used a
probit estimate since the dependent variable is binary
(i.e., passed or not).

We corrected for any potential sample selection bias by
following Heckman's two-stage estimation procedure
(Heckman, 1979), using the inverse Mills ratio (IMR)<fnref n="7" />. The
IMR generated from the first probit estimation is included in
the OLS estimation as an additional explanatory variable.
Regressions were estimated with heteroscedasticity-consistent
standard errors clustered at the program-level and include
year, scientific field, and program fixed effects.

Variable Specification

Main Explanatory Variables

Our main explanatory variables concern the *direct* and *indirect* industry engagements of the applicant teams, which shed light on the diversity and experience effects. Direct involvement is proxied by the existence of team members in a CRP with a company affiliation during the EUROCORES grant application. Thus, *private partner* is a binary variable taking a value of 1 if the applicant team has at least one team member with a company affiliation and 0 if otherwise. Indirect involvement is proxied by evidence of a team's joint publication with industry prior to the EUROCORES grant application. Thus, *joint publication* is a binary variable taking a value of 1 if the applicant team has at least one member with a publication coauthored with a company-affiliated researcher and 0 if otherwise.

Control Variables

We included several control variables measured at the applicant
team-level and evaluator-level and which may affect the outcome
of the evaluation process (see **Table 1** for variable descriptions).

<footnote n="5">⁵With the exception of 40 projects, all the projects selected by the Expert Panel in the Third Stage were funded by the respective NFOs.</footnote>

<footnote n="6">⁶Recall, reviewers were asked to fill out an evaluation form comprising 8-10 sections, each addressing a different criterion. The five specific questions used in our analysis, unlike the others, did not vary over the years and, hence, are used here to establish the average review score.</footnote>

<footnote n="7">⁷Inverse Mills ratio (IMR) is the ratio of the probability density function to the complementary cumulative distribution function.</footnote>
<table><caption>TABLE 1 | Variable descriptions.</caption><thead><tr><th>Variable</th><th>Description</th></tr></thead><tbody><tr><td>Dependent variables</td><td></td></tr><tr><td>Passed First Stage</td><td>Dummy equal to 1 if the project passed the first stage of the selection process (from outline to full proposal stage)</td></tr><tr><td>Reviewer score</td><td>Average reviewer score to questions Q1–Q5 (z-scores)</td></tr><tr><td>Passed Third Stage</td><td>Dummy equal to 1 if the project passed the third stage of the selection process and is thus recommended for funding to NFOs</td></tr><tr><td>Applicant Team Characteristics</td><td></td></tr><tr><td>Joint publication</td><td>Dummy equal to 1 if the team includes at least one member having a co-publication with a company-affiliated researcher prior to the grant application</td></tr><tr><td>Private partner</td><td>Dummy equal to 1 if the team includes at least one member having a private sector affiliation</td></tr><tr><td>Productivity</td><td>Average number of publications of the team members (in logs)</td></tr><tr><td>Star scientist</td><td>Dummy equal to 1 if the team includes at least one of the most cited applicants based on the discipline-year citation distribution (top 1%)</td></tr><tr><td>Cognitive proximity</td><td>Share of overlapping WoS subject categories between team members and Panel Members/Reviewers (254 WoS subject categories clustered into 21)</td></tr><tr><td>Research diversity</td><td>Average Blau index of team members (1 - Σs<sub>i</sub><sup>2</sup>) where s<sub>i</sub> is the share of publications in clustered WoS subject category i)</td></tr><tr><td>Network proximity</td><td>Dummy equal to 1 if the team and panel members have at least one coauthor in common</td></tr><tr><td>Female ratio</td><td>Share of female members within a team</td></tr><tr><td>Age</td><td>Average age of team members (in logs)</td></tr><tr><td>Shanghai ranking</td><td>Dummy equal to 1 if the team includes at least one member affiliated to a university in the Shanghai Ranking top 100 (base year 2005)</td></tr><tr><td>Past EUROCORES grant</td><td>Dummy equal to 1 if the team includes at least one member awarded a EUROCORES project in previous years</td></tr><tr><td>Team size</td><td>Number of researchers within a team composed of a Project Leader, Principal Investigators, and Associate Partners (in logs)</td></tr><tr><td>Number of countries</td><td>Number of distinct participating countries within a team (in logs)</td></tr><tr><td>Budget requested</td><td>Total budget requested for the project normalized by team size (Euro, in logs)</td></tr><tr><td>Expert Panel Characteristics</td><td></td></tr><tr><td>Joint publication panel</td><td>Dummy equal to 1 if the Expert Panel includes at least one member having a co-publication with a company-affiliated researcher</td></tr><tr><td>Female ratio panel</td><td>Share of female members within a Panel</td></tr><tr><td>Productivity panel</td><td>Average number of publications of Panel Members (in logs)</td></tr><tr><td>Age panel</td><td>Average age of Panel Members (in logs)</td></tr><tr><td>Shanghai ranking panel</td><td>Dummy equal to 1 if the Panel includes at least one member affiliated to a university in the Shanghai Ranking top 100 (base year 2005)</td></tr><tr><td>Panel size</td><td>Number of members within a Panel (in logs)</td></tr><tr><td>Panel workload</td><td>Number of assessed projects normalized by panel size (in logs)</td></tr><tr><td>External Reviewer Characteristics</td><td></td></tr><tr><td>Joint publication reviewer</td><td>Dummy equal to 1 if the External Reviewer has a co-publication with a company-affiliated researcher</td></tr><tr><td>Female reviewer</td><td>Dummy equal to 1 if the External Reviewer is female</td></tr><tr><td>Productivity reviewer</td><td>Number of publications of the External Reviewer (in logs)</td></tr><tr><td>Age reviewer</td><td>Age of the External Reviewer (in logs)</td></tr><tr><td>Shanghai ranking reviewer</td><td>Dummy equal to 1 if the External Reviewer is affiliated to a university in the Shanghai Ranking top 100 (base year 2005)</td></tr></tbody></table>

Bibliometric indicators are constructed through the individual data retrieved from the WoS database. We measured Productivity by the number of total publications of the team members normalized by the team size (in logs). Star scientist indicates whether the team includes at least one of the most cited researchers, corresponding to the top 1% of a specific discipline and the year of citation distribution. To measure Research diversity, we used the Blau index, given by $1 - \sum s_i^2$, where $s_i$ is the share of publications in a WoS subject category i, calculated with respect to other team members. We clustered 254 WoS subject categories into 21 categories in line with the literature (Glänzel and Schubert, 2003; Leydesdorff and Rafols, 2009). We also accounted for different proximity measures to the evaluator. Cognitive proximity is the share of overlapping clustered WoS subject categories between team members and the evaluator (either the Expert Panel members or the External Reviewers depending on the stage of evaluation). Network proximity is a dummy variable taking a value of 1 if the team

and the panel members share at least one coauthor in common prior to the grant application<sup>8</sup>.

In addition, we include the following demographic characteristics of the team: Female ratio controls for the gender composition of a team calculated as the share of females in the applicant team. Age is the average age of team members (in logs). Shanghai ranking is a binary variable that accounts for institutional prestige and takes a value of 1 if the team includes at least one member affiliated to a university included in the Shanghai Ranking top 100 (base year taken as 2005).

<footnote n="8"><sup>8</sup>We were unable to find any common coauthorship relations between the applicant team members and the External Reviewers. This is consistent with the ESF guidelines aimed at avoiding any conflict of interest between the applicant and reviewer. Therefore, the variable Network proximity is not included in the Second Stage of analysis.</footnote>
Moreover, we consider a dummy called *Past EUROCORES grant* which takes a value of 1 if the team includes at least one member who has obtained a EUROCORES grant in the past. This variable accounts for a phenomenon similar to the so-called Matthew Effect (Merton, 1968) and also for the 'learning by doing' effect. *Team size* is the number of PIs within the applicant team (in logs) who lead an individual project within a CRP<fnref n="9" />. We control also for *Number of countries*, which refers to the number of distinct participant countries within a team (in logs), since country diversity is one of the prerequisites of EUROCORES. Finally, concerning applicant team characteristics, we consider the *Budget requested* normalized by team size (in logs). For the Third Stage, we also include *Reviewer scores mean as an explanatory variable* since the final Expert Panel decision is contingent on the scores given by the reviewers. To calculate this, standardized average review scores were divided by the number of reviewers corresponding to the CRP. To account for the divergence of scores given by different reviewers to the same CRP, we include the variable *Reviewer scores variance*.

A number of studies show that the grant decision process is not solely influenced by applicant characteristics but also by the characteristics of the evaluators (Mutz et al., 2012; van Arensbergen et al., 2014; Gallo et al., 2016). For the First and Third Stages of evaluation, we therefore take into consideration Expert Panel characteristics. Given that the panel comprises a group of scientists, the measure has to be aggregated. First, we control for the panel's *indirect* engagement by considering the variable *Joint publication panel*.<fnref n="10" /> Then, we control for the panel's gender by means of the *Female ratio panel*. Next, we introduced the variables of *Productivity panel*, *Age panel*, and *Shanghai ranking panel*. All these controls are calculated in the same way as for the applicant teams. In addition, we consider *Panel size*, that is, the number of members sitting on a panel (in logs). Finally, we take into account *Panel workload*, which is the number of assessed projects normalized by *Panel size* (in logs).

For the Second Stage of evaluation, we consider External Reviewer characteristics, measured in this instance at the individual level. Following the same procedure, we account for gender by means of the variable *Female reviewer*, and include variables considering *Productivity reviewer*, *Age reviewer*, and *Shanghai ranking reviewer*.

Finally, we include scientific discipline, year, and program fixed effects using dummies. In line with the ESF's classification, we cluster CRPs into three scientific disciplines, namely "physics and engineering", "life and biomedical sciences," and "social sciences and humanities," to obtain a more even distribution of the disciplines. We control for the project application year as

well as for the EURCORES program representing the relevant CRPs.

## Descriptive Statistics

**Tables 2, 3, 4, 5, 6** show the descriptive statistics for the applicant teams and evaluators. In this subsection, we present the descriptive statistics for some of the variables outlined above. The percentage of project applications that passed the First Stage of selection (from outline proposal to full proposal) is about 39%. At the beginning of this stage, 53% of teams present at least one team member with a joint publication with a company. Around 4% of the teams have at least one team member with a company affiliation. On average, the number of publications were 11 and 6.5% of teams contain a star scientist. As for the teams' demographics, on average, the share of females in the applicant teams was 19.5% and the average age of applicants was 47 years. The average number of members in a team was 6.8, and the budget requested per capita was 271,765 euros. Of all the outline proposals, about 48% are in life and biomedical sciences, 26% in physics and engineering, and 26% in social sciences and humanities.

At the beginning of the Second Stage, following the initial sifting by the Expert Panel, teams with *indirect* engagement increased to 60%, and the share of teams with *direct* engagement dropped to 2.4%. The average productivity increased to 12.4 publications per applicant. At this stage, we observed a decline in the share of females to 16.6%.

At the end of the Third Stage of selection, 61% of full proposals were recommended for funding to the NFOs. Teams with *indirect* engagement increased to 62%, and the share of teams with *direct* engagement remained at 2.4%. The average productivity increased to 13 publications.

As for the reviewers, the share with joint publications was about 14%, while Expert Panels including at least one member with a joint publication with a company stood at about 11%. On average, the Expert Panels contained 12 members, and the number of projects evaluated per panel member was about three.

The correlation matrix presented in **Table 7** shows the correlation coefficients between the variables concerning the applicant team at the First Stage.

## ECONOMETRIC RESULTS

We investigated the impacts of team diversity and prior collaboration experience of EUROCORES applicants on the decisions of the evaluators at consecutive stages of the evaluation process. The results of the estimations are reported in **Tables 8, 9, 10**. We used different model specifications by introducing variables related to the applicant team and the evaluators stepwise. All regressions were estimated with heteroscedasticity-consistent standard errors clustered at the program-level and included year, scientific field and program fixed effects. As a cautionary note, the following results allowed us to observe the correlation between the dependent and independent variables; they cannot be confidently interpreted as causal.

<footnote n="9">⁹In our analysis, recall, an applicant team refers to a team of leaders comprising PIs. These teams are coordinated by a project leader (PL) who is also a PI of an individual project within a CRP. Teams may include self-funding associate partners (APs) from within or outside Europe.</footnote>

<footnote n="10">¹⁰We did not control for the panel's *direct* involvement, as there are no private company-affiliated panel members in our database. Likewise, we decided to exclude the reviewers' *direct* involvement because only a very few (20) present private company affiliation.</footnote>
<table><caption>**TABLE 2** | Descriptive statistics of applicants—First stage.</caption><thead><tr><th>Statistic</th><th>Observations</th><th>Mean</th><th>Std dev</th><th>Min</th><th>Max</th></tr></thead><tbody><tr><td>Passed First Stage</td><td>1,378</td><td>0.388</td><td>0.487</td><td>0</td><td>1</td></tr><tr><td>Joint publication</td><td>1,378</td><td>0.533</td><td>0.499</td><td>0</td><td>1</td></tr><tr><td>Private partner</td><td>1,378</td><td>0.038</td><td>0.191</td><td>0</td><td>1</td></tr><tr><td>Productivity</td><td>1,378</td><td>11.033</td><td>19.604</td><td>0</td><td>186</td></tr><tr><td>Star scientist</td><td>1,378</td><td>0.065</td><td>0.246</td><td>0</td><td>1</td></tr><tr><td>Cognitive proximity</td><td>1,378</td><td>0.368</td><td>0.348</td><td>0</td><td>1</td></tr><tr><td>Research diversity</td><td>1,378</td><td>0.440</td><td>0.317</td><td>0</td><td>1</td></tr><tr><td>Network proximity</td><td>1,378</td><td>0.025</td><td>0.155</td><td>0</td><td>1</td></tr><tr><td>Female Ratio</td><td>1,378</td><td>0.195</td><td>0.208</td><td>0</td><td>1</td></tr><tr><td>Age</td><td>1,378</td><td>46.746</td><td>4.585</td><td>34</td><td>66,333</td></tr><tr><td>Shanghai Ranking</td><td>1,378</td><td>0.577</td><td>0.494</td><td>0</td><td>1</td></tr><tr><td>Past EUROCORES grant</td><td>1,378</td><td>0.134</td><td>0.341</td><td>0</td><td>1</td></tr><tr><td>Team size</td><td>1,378</td><td>6.800</td><td>4.572</td><td>3</td><td>51</td></tr><tr><td>Number of countries</td><td>1,378</td><td>4.389</td><td>1.894</td><td>2</td><td>16</td></tr><tr><td>Budget requested</td><td>1,378</td><td>271,765,000</td><td>211,800,400</td><td>10,417</td><td>4,448,720</td></tr><tr><td>Life and biomedical sciences</td><td>1,378</td><td>0.478</td><td>0.500</td><td>0</td><td>1</td></tr><tr><td>Physics and engineering</td><td>1,378</td><td>0.260</td><td>0.439</td><td>0</td><td>1</td></tr><tr><td>Social sciences and humanities</td><td>1,378</td><td>0.262</td><td>0.440</td><td>0</td><td>1</td></tr></tbody></table>

<table><caption>**TABLE 3** | Descriptive statistics of applicants—Second Stage.</caption><thead><tr><th>Statistic</th><th>Observations</th><th>Mean</th><th>Std dev</th><th>Min</th><th>Max</th></tr></thead><tbody><tr><td>Average reviewer score</td><td>1,930</td><td>0.003</td><td>0.793</td><td>-2.704</td><td>1.066</td></tr><tr><td>Joint publication</td><td>1,938</td><td>0.602</td><td>0.490</td><td>0</td><td>1</td></tr><tr><td>Private partner</td><td>1,938</td><td>0.024</td><td>0.154</td><td>0</td><td>1</td></tr><tr><td>Productivity</td><td>1,938</td><td>12.467</td><td>19.470</td><td>0</td><td>145</td></tr><tr><td>Star scientist</td><td>1,938</td><td>0.084</td><td>0.278</td><td>0</td><td>1</td></tr><tr><td>Cognitive proximity</td><td>1,938</td><td>0.331</td><td>0.434</td><td>0</td><td>1</td></tr><tr><td>Research diversity</td><td>1,938</td><td>0.480</td><td>0.308</td><td>0</td><td>1</td></tr><tr><td>Female ratio</td><td>1,938</td><td>0.166</td><td>0.188</td><td>0</td><td>1</td></tr><tr><td>Age</td><td>1,938</td><td>46.897</td><td>4.596</td><td>34</td><td>66,333</td></tr><tr><td>Shanghai ranking</td><td>1,938</td><td>0.685</td><td>0.465</td><td>0</td><td>1</td></tr><tr><td>Past EUROCORES grant</td><td>1,938</td><td>0.214</td><td>0.410</td><td>0</td><td>1</td></tr><tr><td>Team size</td><td>1,938</td><td>7.126</td><td>4.329</td><td>3</td><td>51</td></tr><tr><td>Number of countries</td><td>1,938</td><td>4.678</td><td>1.846</td><td>2</td><td>15</td></tr><tr><td>Budget requested</td><td>1,938</td><td>276,555,400</td><td>173,829,500</td><td>10,417</td><td>2,557,539</td></tr><tr><td>Life and biomedical sciences</td><td>1,938</td><td>0.377</td><td>0.485</td><td>0</td><td>1</td></tr><tr><td>Physics and engineering</td><td>1,938</td><td>0.313</td><td>0.464</td><td>0</td><td>1</td></tr><tr><td>Social sciences and humanities</td><td>1,938</td><td>0.310</td><td>0.463</td><td>0</td><td>1</td></tr></tbody></table>

## First Stage

Table 8 reports the results of the probit regression corresponding to the First Stage of selection by the Expert Panel. Coefficients are the marginal effects calculated by setting all variables at their mean values.

In the case of our main variables of interest, the variable capturing the teams' prior experience of collaboration is significant at the First Stage, whereas the variable accounting for the teams' structural diversity is not significant. The *Joint publication* coefficient (indirect engagement) is significantly positive across all models. More specifically, the coefficient of *Joint publication* in Model 3 is 0.05 ($p < 0.1$), implying that the presence in a team of at least one member with a joint publication with a company-affiliated researcher increases the probability of passing the first selection stage by 0.05 compared to a team with no members presenting a joint publication with a company-

affiliated researcher. In other words, teams incorporating at least one member with a joint publication with a company are more likely to pass from the outline to the full proposal stage, supporting the argument that teams having prior collaboration experience with industry are more likely to be “rewarded.” On the other hand, we do not observe a significant effect of the *Private partner* variable (direct engagement), which seems to indicate that the diversity argument is not supported.

Overall, the variables reflecting applicants' scientific productivity (i.e. the total number of publications and star scientists) are not significant. At this initial stage, the panel characteristics and the more readily observable characteristics of the applicants, such as research diversity, proximity to the panel, and being affiliated to a top-academic institution, matter the most.
<table><caption>**TABLE 4** | Descriptive statistics of applicants—Third Stage.</caption><thead><tr><th>Statistic</th><th>Observations</th><th>Mean</th><th>Std dev</th><th>Min</th><th>Max</th></tr></thead><tbody><tr><td>Passed Third Stage</td><td>533</td><td>0.612</td><td>0.488</td><td>0</td><td>1</td></tr><tr><td>Joint publication</td><td>533</td><td>0.623</td><td>0.485</td><td>0</td><td>1</td></tr><tr><td>Private partner</td><td>533</td><td>0.024</td><td>0.154</td><td>0</td><td>1</td></tr><tr><td>Productivity</td><td>533</td><td>13.193</td><td>20.092</td><td>0</td><td>145</td></tr><tr><td>Star scientist</td><td>533</td><td>0.081</td><td>0.273</td><td>0</td><td>1</td></tr><tr><td>Cognitive proximity</td><td>533</td><td>0.445</td><td>0.360</td><td>0</td><td>1</td></tr><tr><td>Research diversity</td><td>533</td><td>0.487</td><td>0.305</td><td>0</td><td>1</td></tr><tr><td>Network proximity</td><td>533</td><td>0.032</td><td>0.176</td><td>0</td><td>1</td></tr><tr><td>Female ratio</td><td>533</td><td>0.170</td><td>0.193</td><td>0</td><td>1</td></tr><tr><td>Age</td><td>533</td><td>46.943</td><td>4.607</td><td>34</td><td>66.333</td></tr><tr><td>Shanghai ranking</td><td>533</td><td>0.685</td><td>0.465</td><td>0</td><td>1</td></tr><tr><td>Past EUROCORES grant</td><td>533</td><td>0.205</td><td>0.404</td><td>0</td><td>1</td></tr><tr><td>Team size</td><td>533</td><td>7.024</td><td>4.237</td><td>3</td><td>51</td></tr><tr><td>Number of countries</td><td>533</td><td>4.689</td><td>1.869</td><td>2</td><td>15</td></tr><tr><td>Budget requested</td><td>533</td><td>276,625.500</td><td>176,882.300</td><td>10,417</td><td>2,557,539</td></tr><tr><td>Reviewers scores mean</td><td>525</td><td>-0.010</td><td>0.547</td><td>-2.233</td><td>0.998</td></tr><tr><td>Reviewers scores variance</td><td>524</td><td>0.498</td><td>0.535</td><td>0</td><td>3.700</td></tr><tr><td>Life and biomedical sciences</td><td>533</td><td>0.375</td><td>0.485</td><td>0</td><td>1</td></tr><tr><td>Physics and engineering</td><td>533</td><td>0.319</td><td>0.467</td><td>0</td><td>1</td></tr><tr><td>Social sciences and humanities</td><td>533</td><td>0.306</td><td>0.461</td><td>0</td><td>1</td></tr></tbody></table>

<table><caption>**TABLE 5** | Descriptive Statistics of Expert Panel [41 panel groups].</caption><thead><tr><th>Statistic</th><th>Observations</th><th>Mean</th><th>Std dev</th><th>Min</th><th>Max</th></tr></thead><tbody><tr><td>Joint publication panel</td><td>41</td><td>0.110</td><td>0.169</td><td>0</td><td>1</td></tr><tr><td>Female ratio panel</td><td>41</td><td>0.184</td><td>0.126</td><td>0</td><td>0.455</td></tr><tr><td>Productivity panel</td><td>41</td><td>11.011</td><td>26.871</td><td>0</td><td>166.333</td></tr><tr><td>Age panel</td><td>41</td><td>51.024</td><td>3.383</td><td>43.231</td><td>58.200</td></tr><tr><td>Shanghai ranking panel</td><td>41</td><td>0.927</td><td>0.264</td><td>0</td><td>1</td></tr><tr><td>Panel size</td><td>41</td><td>12.293</td><td>4.360</td><td>4</td><td>30</td></tr><tr><td>Panel workload</td><td>41</td><td>2.921</td><td>2.649</td><td>0.385</td><td>14.286</td></tr></tbody></table>

<table><caption>**TABLE 6** | Descriptive statistics of external reviewers.</caption><thead><tr><th>Statistic</th><th>Observations</th><th>Mean</th><th>Std dev</th><th>Min</th><th>Max</th></tr></thead><tbody><tr><td>Joint publication reviewer</td><td>1,938</td><td>0.138</td><td>0.345</td><td>0</td><td>1</td></tr><tr><td>Female reviewer</td><td>1,938</td><td>0.166</td><td>0.372</td><td>0</td><td>1</td></tr><tr><td>Productivity reviewer</td><td>1,938</td><td>6.569</td><td>15.995</td><td>0</td><td>177</td></tr><tr><td>Age reviewer</td><td>1,938</td><td>48.496</td><td>10.181</td><td>30</td><td>85</td></tr><tr><td>Shanghai ranking reviewer</td><td>1,938</td><td>0.269</td><td>0.443</td><td>0</td><td>1</td></tr><tr><td>Average reviewer score</td><td>1,930</td><td>0.003</td><td>0.793</td><td>-2.704</td><td>1.066</td></tr></tbody></table>

An examination of the bibliometric indicators in the full model specification shows that the coefficient measuring the applicant team's *Cognitive proximity* to a panel member is significantly positive, whereas the coefficients of *Network proximity* and *Research diversity* are significantly negative. However, the team *Productivity* coefficient is nonsignificant at this stage.

The coefficient measuring the share of females is significantly negative, indicating that higher shares of females within teams reduce the likelihood of passing the First Stage. This finding is in line with the literature concerning gender disparities in the grant peer review process (van der Lee and Ellemers, 2015; Shannon

et al., 2019; Witteman et al., 2019).<fnref n="11" /> On the other hand, having at least one team member affiliated to a prestigious university, having obtained a EUROCORES grant in the past, and the team size and number of countries represented in the team increase the likelihood of success at this stage, as their coefficients are significantly positive.

<footnote n="11">¹¹This result is also in line with a previous study undertaken by the authors [Bianchini et al., 2021 (manuscript submitted for publication)], as part of an ANR GIGA project, investigating gender bias in the EUROCORES evaluation process.</footnote>
<table><caption>TABLE 7 | Pairwise correlation matrix.</caption><thead><tr><th></th><th></th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th><th>8</th><th>9</th><th>10</th><th>11</th><th>12</th><th>13</th></tr></thead><tbody><tr><td>1</td><td>Joint publication</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>2</td><td>Private partner</td><td>0.02</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>3</td><td>Female ratio team</td><td>-0.04</td><td>-0.06*</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>4</td><td>Productivity team</td><td>0.38**</td><td>0.01</td><td>-0.04</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>5</td><td>Star scientist team</td><td>0.21**</td><td>0.03</td><td>-0.03</td><td>0.44**</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>6</td><td>Cognitive proximity team</td><td>0.32**</td><td>-0.03</td><td>-0.02</td><td>0.32**</td><td>0.08**</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>7</td><td>Research diversity team</td><td>0.39**</td><td>-0.03</td><td>0.00</td><td>0.22**</td><td>0.05</td><td>0.60**</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>8</td><td>Network proximity</td><td>0.14**</td><td>-0.03</td><td>-0.02</td><td>0.15**</td><td>-0.00</td><td>0.11**</td><td>0.10**</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>9</td><td>Team size</td><td>0.22**</td><td>0.08**</td><td>-0.01</td><td>0.09**</td><td>0.17**</td><td>0.17**</td><td>0.23**</td><td>0.11**</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>10</td><td>Age team</td><td>0.10**</td><td>-0.03</td><td>-0.09**</td><td>0.08**</td><td>-0.00</td><td>0.06*</td><td>0.06*</td><td>0.04</td><td>-0.06*</td><td></td><td></td><td></td><td></td></tr><tr><td>11</td><td>Shanghai ranking team</td><td>0.10**</td><td>0.01</td><td>-0.04</td><td>0.03</td><td>0.05</td><td>0.11**</td><td>0.12**</td><td>0.05</td><td>0.15**</td><td>0.04</td><td></td><td></td><td></td></tr><tr><td>12</td><td>Past EUROCORES grant team</td><td>0.12**</td><td>-0.04</td><td>-0.11**</td><td>0.11**</td><td>0.04</td><td>0.12**</td><td>0.19**</td><td>0.07**</td><td>0.09**</td><td>0.03</td><td>0.06*</td><td></td><td></td></tr><tr><td>13</td><td>Budget requested</td><td>0.03</td><td>-0.01</td><td>0.04</td><td>0.05*</td><td>-0.03</td><td>-0.02</td><td>0.04</td><td>-0.01</td><td>-0.16**</td><td>0.10**</td><td>0.03</td><td>0.03</td><td></td></tr><tr><td>14</td><td>Number of countries</td><td>0.25**</td><td>0.02</td><td>-0.01</td><td>0.11**</td><td>0.10**</td><td>0.22**</td><td>0.28**</td><td>0.14**</td><td>0.69**</td><td>0.07**</td><td>0.22**</td><td>0.14**</td><td>-0.07**</td></tr></tbody></table>

Pairwise correlation matrix of applicant team characteristics for the First Stage of evaluation; * indicates p < 0.05; ** indicates p < 0.01.

Finally, in the case of the panel-level characteristics, we find a negative impact of *Panel size* and *Panel workload*, the latter reflecting the level of competition within a given program, that is, the higher the number of projects per panel, the lower the probability of passing this stage.

## Second Stage

Table 9 reports the results of the OLS regression corresponding to the Second Stage of evaluation by the External Reviewers. As explained above, we control for potential sample selection bias by using a two-step Heckman procedure in some model specifications (i.e., Models 2, 4, and 6). Consequently, the results are reported without and with IMRs.

At this stage, External Reviewers are insensitive to variables reflecting industry engagement both in terms of a team's prior collaboration experience and the involvement of an industry partner in a team. When we correct for potential selection bias in the full model specification, we do not observe a significant relationship between the variables *Joint publication* or *Private partner* with the scores awarded by the reviewers.

Moreover, unlike the First Stage, at this stage of the evaluation, the variables reflecting scientific productivity (i.e., team productivity, star scientist, and reviewer productivity) have a significant relation with the reviewers' scores. Across all models, the variable *Productivity* presents a positive significant coefficient suggesting that a higher number of publications is associated with higher reviewer scores. On the other hand, having a *Star scientist* in the team, *Cognitive proximity* to the reviewer, and a higher *Female ratio* lower the likelihood of obtaining higher review scores as their coefficients are significantly negative. Here, our finding regarding cognitive distance is in line with that of Gallo et al. (2016), suggesting that a closer intellectual distance between the reviewer and the applicant leads to less favorable evaluations.

Finally, we find a negative relation between a reviewer's productivity and the scores awarded to the proposals, that is, the higher a reviewer's productivity, the lower the score given.

## Third Stage

Table 10 reports the results of the probit regression corresponding to the Third Stage of selection by Expert Panel members. Coefficients are the marginal effects calculated by setting all variables at their mean values.

The variables concerning industry engagement (i.e., *Joint publication* and *Private partner*) are not significant. The negative impact of industry engagement as it relates to diversity is only observed in limited specifications. In the full model specification, having a private partner in a team does not significantly affect success in the Third Stage of selection (i.e., obtaining a funding recommendation).

Overall, in this last stage of the process, our findings are in line with the “old boy” network effect (Rose, 1989; Travis and Collins, 1991), since the network proximity of teams to the evaluators is highly determinant of being selected. Likewise, the review outcome of the previous stage is also an important determinant of passing the last stage of selection.

At this stage, a team's *Productivity* is no longer significant while the variable *Star scientist* becomes positively significant. In the Third Stage, the *Reviewer scores mean* has, as expected, a significant positive influence on the outcome, given that the Expert Panel adheres primarily to the reviewer scores awarded in the previous stage. *Network proximity* also has a significant positive impact suggesting that having a coauthor in common with a panel member increases the likelihood of acceptance in this last stage of selection. We also observe a positive impact of the *Budget requested*.

Finally, when we consider panel-level characteristics, we find significant negative effects of *Age panel*, *Panel size*, and *Panel workload*.

## DISCUSSION AND CONCLUSION

This paper has examined the determinants of a multistage grant allocation system as implemented by different sets of evaluators (i.e., Expert Panel and External Reviewers) active at different points in the process. Specifically, we analyzed the impacts of the
<table><caption>TABLE 8 | First Stage—Probit regression results.</caption><thead><tr><th colspan="4">Dependent variable: Passed First Stage</th></tr><tr><th></th><th>(1)</th><th>(2)</th><th>(3)</th></tr></thead><tbody><tr><td>Joint publication</td><td>0.086***<br>(0.026)</td><td>0.082***<br>(0.025)</td><td>0.050*<br>(0.030)</td></tr><tr><td>Private partner</td><td>-0.060<br>(0.079)</td><td>-0.077<br>(0.078)</td><td>-0.092<br>(0.071)</td></tr><tr><td>Productivity</td><td>0.035<br>(0.033)</td><td>0.021<br>(0.030)</td><td>0.021<br>(0.032)</td></tr><tr><td>Star scientist</td><td>0.059<br>(0.057)</td><td>0.064<br>(0.061)</td><td>0.051<br>(0.056)</td></tr><tr><td>Cognitive proximity</td><td>0.123***<br>(0.047)</td><td>0.126***<br>(0.047)</td><td>0.116**<br>(0.049)</td></tr><tr><td>Research diversity</td><td>-0.036<br>(0.052)</td><td>-0.061<br>(0.053)</td><td>-0.139***<br>(0.043)</td></tr><tr><td>Network proximity</td><td>-0.032<br>(0.103)</td><td>-0.046<br>(0.106)</td><td>-0.185*<br>(0.096)</td></tr><tr><td>Female ratio</td><td></td><td>-0.187**<br>(0.074)</td><td>-0.197***<br>(0.071)</td></tr><tr><td>Age</td><td></td><td>-0.162<br>(0.167)</td><td>-0.211<br>(0.172)</td></tr><tr><td>Shanghai ranking</td><td></td><td>0.126***<br>(0.031)</td><td>0.096***<br>(0.032)</td></tr><tr><td>Past EUROCORES grant</td><td></td><td></td><td>0.082*<br>(0.050)</td></tr><tr><td>Team size</td><td></td><td></td><td>0.120***<br>(0.045)</td></tr><tr><td>Number of countries</td><td></td><td></td><td>0.138***<br>(0.050)</td></tr><tr><td>Budget requested</td><td></td><td></td><td>0.030<br>(0.020)</td></tr><tr><td>Joint publication panel</td><td></td><td></td><td>-0.337<br>(0.353)</td></tr><tr><td>Female ratio panel</td><td></td><td></td><td>-0.103<br>(0.215)</td></tr><tr><td>Productivity panel</td><td></td><td></td><td>0.113<br>(0.105)</td></tr><tr><td>Age panel</td><td></td><td></td><td>0.438<br>(0.418)</td></tr><tr><td>Shanghai ranking panel</td><td></td><td></td><td>-0.005<br>(0.177)</td></tr><tr><td>Panel size</td><td></td><td></td><td>-0.125**<br>(0.061)</td></tr><tr><td>Panel workload</td><td></td><td></td><td>-0.216***<br>(0.048)</td></tr><tr><td>Program fixed effects</td><td>yes</td><td>yes</td><td>no</td></tr><tr><td>Year dummies</td><td>yes</td><td>yes</td><td>yes</td></tr><tr><td>Domain dummies</td><td>yes</td><td>yes</td><td>yes</td></tr><tr><td>AIC</td><td>1,574.516</td><td>1,552.338</td><td>1,547.474</td></tr><tr><td>BIC</td><td>1830.707</td><td>1824.215</td><td>1714.782</td></tr><tr><td>Log likelihood</td><td>-738.258</td><td>-724.169</td><td>-741.737</td></tr><tr><td>Deviance</td><td>1,476.516</td><td>1,448.338</td><td>1,483.474</td></tr><tr><td>Num. obs</td><td>1,378</td><td>1,378</td><td>1,378</td></tr></tbody></table>

Robust standard errors in parentheses, clustered at program-level. ***, **, * indicate significance at the 1, 5, and 10% levels, respectively.

applicant teams' *direct* industry engagement (diversity effect) and *indirect* industry engagement (experience effect) on the outcomes at each stage in the evaluation.

Our results indicate that scientific productivity is not always the main selection criteria in the grant allocation process. This is in line with previous studies suggesting that the funding outcome may be influenced by the individual characteristics of the

applicants (unrelated to their scientific productivity) (Wennerås and Wold, 1997; Grimpe, 2012) or by the attributes of the project evaluators (Gallo et al., 2016). Given that the knowledge transfer goals of STI policy can be achieved, in part, by means of competitive grants, it is our conjecture that the KTT-related attributes of applicant teams might also affect the grant allocation process. Here, we have argued that this can be operationalized by considering two indicators related to industry engagement: 1) whether the applicant team exhibits diversity (*direct* engagement), proxied by the presence of an industry/company-affiliated member in the collaborative research project—in this respect, structurally diverse teams in terms of their knowledge and skills are more likely to produce novel combinations with a high potential for transfer due to direct proximity with the industry network; and 2) whether the team has past experience of collaboration (*indirect* engagement), proxied by the presence of a member in the applicant team with joint publications with industry prior to the grant application. We argue that prior collaboration with industry enhances the potential to transfer in the future, something that can be guaranteed or facilitated, for instance, by means of the informal networks established with industry researchers during the collaborative relation or by means of learning by doing.

Although we found some evidence of *indirect* engagement to support the experience effect, we are unable to identify any significant impact of *direct engagement* to support the diversity effect on the grant allocation process. The reason why prior experience might be more valued than having an industry partner in an applicant team appears to be that it does not constitute a direct threat of the appropriation of research results by industry. Moreover, occupying a position in an established industry network thanks to prior collaborations (proxied here by co-publications) might be perceived by evaluators as a more reliable indicator of future knowledge transfer activities. Such network visibility can be readily captured by the Expert Panel when screening the applicants' CVs in the first stage of selection, at a point in the process when the project content remains somewhat limited because only outline, and not full, proposals are being evaluated. This means that panel members might, at this stage, be basing their selections on the more visible attributes of an applicant team. Yet, despite threats of industry appropriation, direct engagement may also be valued on the grounds that diverse teams are more likely to produce novel research (Arts and Veugelers, 2015). The trade-off between novelty and risk of appropriation of transferable research outcomes might go some way to explain why we do not observe any significant effect of *direct* engagement.

The study reported here contributes to the literature by providing an examination of the impact of industry engagement on the grant allocation process with a particular focus on the effects of an applicant team's *diversity* and *experience*. To the best of our knowledge, the only study to date that has attempted to examine the effects of industry collaboration is Banal-Estañol et al.'s (2019a), and here in the present study, only in part, as it does not constitute the primary focus of the authors' analysis. Our study, therefore, builds on and extends this earlier investigation by incorporating the dimension
<table><caption>TABLE 9 | Second Stage—OLS regression results.</caption><thead><tr><th></th><th colspan="6">Dependent variable: Reviewer score</th></tr><tr><th></th><th>(1)</th><th>(2)</th><th>(3)</th><th>(4)</th><th>(5)</th><th>(6)</th></tr></thead><tbody><tr><td>Joint publication</td><td>-0.024<br>(0.045)</td><td>-0.099**<br>(0.048)</td><td>-0.031<br>(0.045)</td><td>-0.079<br>(0.049)</td><td>-0.047<br>(0.047)</td><td>-0.070<br>(0.052)</td></tr><tr><td>Private partner</td><td>-0.062<br>(0.123)</td><td>-0.058<br>(0.122)</td><td>-0.061<br>(0.122)</td><td>-0.058<br>(0.122)</td><td>-0.095<br>(0.124)</td><td>-0.062<br>(0.128)</td></tr><tr><td>Productivity</td><td>0.169***<br>(0.038)</td><td>0.152***<br>(0.038)</td><td>0.149***<br>(0.039)</td><td>0.141***<br>(0.039)</td><td>0.160***<br>(0.040)</td><td>0.145***<br>(0.042)</td></tr><tr><td>Star scientist</td><td>-0.159**<br>(0.068)</td><td>-0.201***<br>(0.068)</td><td>-0.154**<br>(0.068)</td><td>-0.180***<br>(0.068)</td><td>-0.164**<br>(0.068)</td><td>-0.178**<br>(0.069)</td></tr><tr><td>Cognitive proximity</td><td>-0.148***<br>(0.051)</td><td>-0.168***<br>(0.051)</td><td>-0.151***<br>(0.051)</td><td>-0.162***<br>(0.051)</td><td>-0.125**<br>(0.053)</td><td>-0.129**<br>(0.053)</td></tr><tr><td>Research diversity</td><td>0.047<br>(0.072)</td><td>0.018<br>(0.072)</td><td>0.032<br>(0.072)</td><td>0.024<br>(0.072)</td><td>-0.019<br>(0.075)</td><td>0.009<br>(0.080)</td></tr><tr><td>Female ratio</td><td></td><td></td><td>-0.351***<br>(0.104)</td><td>-0.242**<br>(0.112)</td><td>-0.328***<br>(0.105)</td><td>-0.247*<br>(0.130)</td></tr><tr><td>Age</td><td></td><td></td><td>0.108<br>(0.191)</td><td>0.188<br>(0.193)</td><td>0.062<br>(0.193)</td><td>0.142<br>(0.207)</td></tr><tr><td>Shanghai ranking</td><td></td><td></td><td>0.140***<br>(0.040)</td><td>0.076<br>(0.047)</td><td>0.125***<br>(0.041)</td><td>0.084<br>(0.056)</td></tr><tr><td>Past EUROCORES grant</td><td></td><td></td><td></td><td></td><td>0.087<br>(0.057)</td><td>0.040<br>(0.072)</td></tr><tr><td>Team size</td><td></td><td></td><td></td><td></td><td>0.014<br>(0.065)</td><td>-0.023<br>(0.074)</td></tr><tr><td>Number of countries</td><td></td><td></td><td></td><td></td><td>0.080<br>(0.082)</td><td>0.025<br>(0.097)</td></tr><tr><td>Budget requested</td><td></td><td></td><td></td><td></td><td>0.004<br>(0.031)</td><td>-0.009<br>(0.034)</td></tr><tr><td>Joint publication reviewer</td><td></td><td></td><td></td><td></td><td>-0.018<br>(0.061)</td><td>-0.017<br>(0.061)</td></tr><tr><td>Female reviewers</td><td></td><td></td><td></td><td></td><td>-0.044<br>(0.048)</td><td>-0.040<br>(0.048)</td></tr><tr><td>Productivity reviewer</td><td></td><td></td><td></td><td></td><td>-0.050**<br>(0.023)</td><td>-0.050**<br>(0.023)</td></tr><tr><td>Age reviewer</td><td></td><td></td><td></td><td></td><td>0.087<br>(0.085)</td><td>0.086<br>(0.085)</td></tr><tr><td>Shanghai ranking reviewer</td><td></td><td></td><td></td><td></td><td>0.016<br>(0.039)</td><td>0.015<br>(0.039)</td></tr><tr><td>Inverse Mills Ratio</td><td></td><td></td><td>-0.449***<br>(0.093)</td><td>-0.299**<br>(0.117)</td><td></td><td>-0.252<br>(0.238)</td></tr><tr><td>Program fixed effects</td><td>yes</td><td>yes</td><td>yes</td><td>yes</td><td>yes</td><td>yes</td></tr><tr><td>Year dummies</td><td>yes</td><td>yes</td><td>yes</td><td>yes</td><td>yes</td><td>yes</td></tr><tr><td>Domain dummies</td><td>yes</td><td>yes</td><td>yes</td><td>yes</td><td>yes</td><td>yes</td></tr><tr><td>Observations</td><td>1,930</td><td>1,930</td><td>1,930</td><td>1,930</td><td>1,930</td><td>1,930</td></tr><tr><td>R2</td><td>0.143</td><td>0.154</td><td>0.155</td><td>0.157</td><td>0.161</td><td>0.162</td></tr><tr><td>Adjusted R2</td><td>0.123</td><td>0.133</td><td>0.133</td><td>0.136</td><td>0.136</td><td>0.136</td></tr><tr><td>Residual Std. Error</td><td>0.742</td><td>0.738</td><td>0.738</td><td>0.737</td><td>0.737</td><td>0.737</td></tr><tr><td>F Statistic</td><td>7.005***</td><td>7.435***</td><td>7.163***</td><td>7.170***</td><td>6.308***</td><td>6.219***</td></tr></tbody></table>

Robust standard errors in parentheses, clustered at program-level. ***, **, * indicate significance at the 1, 5, and 10% levels, respectively.

of indirect engagement in an effort to account for collaborations established in the past.

Another important contribution made by the present study is the fact that our original and particularly rich database has allowed us to observe a multistage, multidisciplinary evaluation process at a more granular level within a pan-European grant scheme. Previous studies have tended to concentrate their research on a single country and a single discipline and have restricted themselves to a single stage in an evaluation process. However, grant allocation is rarely a single-step process in

which funding organizations decide whether or not to award public research funds; rather, different actors are engaged in the process, and their decisions are likely to exhibit a range of heterogeneities, as shown in our study. In terms of the methodology employed, this study contributes to the wider literature by developing an aggregated approach and conducting its analysis at the team level, whereas most previous studies of grant allocation consider individual-level data. It should be noted, however, that Banal-Estañol et al. (2019b) is an exception in this respect as their study also focuses on the team dimension.
<table><caption>TABLE 10 | Third Stage—Probit regression results.</caption><thead><tr><th colspan="4">Dependent variable: Passed Third Stage</th></tr><tr><th></th><th>(1)</th><th>(2)</th><th>(3)</th></tr></thead><tbody><tr><td>Joint publication</td><td>-0.050<br>(0.057)</td><td>-0.054<br>(0.056)</td><td>-0.015<br>(0.064)</td></tr><tr><td>Private partner</td><td>-0.354***<br>(0.123)</td><td>-0.378***<br>(0.120)</td><td>-0.165<br>(0.234)</td></tr><tr><td>Productivity</td><td>0.102**<br>(0.048)</td><td>0.098**<br>(0.047)</td><td>-0.047<br>(0.040)</td></tr><tr><td>Star scientist</td><td>-0.011<br>(0.090)</td><td>-0.007<br>(0.083)</td><td>0.164**<br>(0.083)</td></tr><tr><td>Cognitive proximity</td><td>0.023<br>(0.090)</td><td>0.029<br>(0.089)</td><td>-0.043<br>(0.115)</td></tr><tr><td>Research diversity</td><td>0.092<br>(0.103)</td><td>0.071<br>(0.096)</td><td>0.246*<br>(0.146)</td></tr><tr><td>Network proximity</td><td>0.209***<br>(0.055)</td><td>0.202***<br>(0.060)</td><td>0.323***<br>(0.052)</td></tr><tr><td>Female ratio</td><td></td><td>-0.089<br>(0.144)</td><td>0.040<br>(0.161)</td></tr><tr><td>Age</td><td></td><td>-0.186<br>(0.315)</td><td>-0.220<br>(0.390)</td></tr><tr><td>Shanghai ranking</td><td></td><td>0.074<br>(0.072)</td><td>-0.073<br>(0.101)</td></tr><tr><td>Past EUROCORES grant</td><td></td><td></td><td>0.024<br>(0.068)</td></tr><tr><td>Team size</td><td></td><td></td><td>-0.087<br>(0.083)</td></tr><tr><td>Number of countries</td><td></td><td></td><td>0.090<br>(0.112)</td></tr><tr><td>Budget requested</td><td></td><td></td><td>0.094**<br>(0.045)</td></tr><tr><td>Reviewers scores mean</td><td></td><td></td><td>0.887***<br>(0.136)</td></tr><tr><td>Reviewers scores variance</td><td></td><td></td><td>0.074<br>(0.067)</td></tr><tr><td>Joint publication panel</td><td></td><td></td><td>0.531<br>(0.549)</td></tr><tr><td>Female ratio panel</td><td></td><td></td><td>-0.126<br>(0.301)</td></tr><tr><td>Productivity panel</td><td></td><td></td><td>-0.153<br>(0.156)</td></tr><tr><td>Age panel</td><td></td><td></td><td>-1.142**<br>(0.559)</td></tr><tr><td>Shanghai ranking panel</td><td></td><td></td><td>0.024<br>(0.080)</td></tr><tr><td>Panel size</td><td></td><td></td><td>-0.405***<br>(0.084)</td></tr><tr><td>Panel workload</td><td></td><td></td><td>-0.257***<br>(0.077)</td></tr><tr><td>Program fixed effects</td><td>yes</td><td>yes</td><td>no</td></tr><tr><td>Year dummies</td><td>yes</td><td>yes</td><td>yes</td></tr><tr><td>Domain dummies</td><td>yes</td><td>yes</td><td>yes</td></tr><tr><td>AIC</td><td>696.119</td><td>698.808</td><td>504.598</td></tr><tr><td>BIC</td><td>897.210</td><td>912.734</td><td>649.489</td></tr><tr><td>Log likelihood</td><td>-301.060</td><td>-299.404</td><td>-218.299</td></tr><tr><td>Deviance</td><td>602.119</td><td>598.808</td><td>436.598</td></tr><tr><td>Num. obs</td><td>533</td><td>533</td><td>524</td></tr></tbody></table>

Robust standard errors in parentheses, clustered at the program level. ***, **, * indicate significance at the 1, 5; and 10% levels, respectively.

Despite the conjectures we draw above, our empirical analyses are unable to provide a definitive answer to understand why *direct* and *indirect* engagements lead to different outcomes at different stages of the grant application process. Further qualitative

investigation is needed to understand what goes in this black box. Additional conjectures might be made to explain stage-specific heterogeneities, including such factors as the differences in group vs individual decision-making processes (Olbrecht and Bornmann, 2010), and the impact of limitations of the information and time made available to evaluators when having to examine applicant profiles and project proposals. Here, the differences between the evaluators engaged at distinct stages of the process in terms of their characteristics, objectives, constraints, and access to relevant information have meant that different evaluation mechanisms have been operational in consecutive stages. The Panel Stage is representative of a group decision-making process under the constraints of time, whereas the External Reviewers operated alone when assessing the proposals and were given more time to evaluate projects and applicants. Additionally, at the first stage of application, panel members evaluate only the outline proposal, and thus, they do not have access to the information that will be included in the full proposal. This contrasts with the information available to the reviewers, which is considerably more detailed with regard to both the project content and the applicants. This explains why at the early stages of selection, more visible quality signals, including university affiliation (Shanghai Ranking) and previous grants held, are significant determinants of selection, whereas in later stages, when the full proposal contents are available, specific quality measures, such as the number of publications and reviewer scores, acquire greater weightage.

A further limitation of our study relates to the different mechanisms that might impact industry collaboration. Here, the set of mechanisms affecting the evaluators' perceptions of the *direct* and *indirect* collaborations of applicant teams could be much broader. To measure *direct* engagement, we used ongoing collaborations with an industry partner during the grant application and related this attribute to a team's structural diversity. Yet, it is evident that in doing so, we employ a rather narrow approach to diversity, given that structural diversity might be measured by using other team attributes as the diversity literature illustrates. Moreover, *direct* industry involvement might capture aspects other than diversity that is related to the scope of a proposed project, i.e., whether the project is more application oriented or whether it seeks to develop solutions for industrial use. Likewise, to account for *indirect* engagement, and hence for the experience effect, different proxies of prior industry engagement, such as patents and know-how licenses, consultancy agreements, spin-off formations, and researcher mobility, can be used in addition to co-publications.

Our research has a number of implications for policy makers and research funding agencies. In a knowledge-based economy, STI policies aim to foster KTT by employing different instruments, including competitive grants. Although, the development of U-I collaboration was not one of the explicit aims of the EUROCORES funding scheme, the transfer of research outcomes was a dimension that was sometimes referred to in reviewer reports. Here, an in-depth analysis of these reports might provide further insights into how the evaluators perceive the
allocation of public money to teams with industry engagement. Funding organizations (or the organizations managing the funding process on their behalf) need to develop guidelines to inform evaluators of the potential benefits and costs of industry engagement and, thus, ensure, the effective allocation of public funds so as to optimize U-I collaboration to the benefit of both the organizations and society as a whole. This objective should be made more explicit in calls for project proposals and in evaluator guidelines and on their evaluation forms as interorganizational collaboration can only enhance the frontiers of knowledge and technology.

DATA AVAILABILITY STATEMENT

The dataset used in this article is not publicly available since the
data is confidential. Authorization from the data provider
(European Science Foundation) is required for the access.
Inquiries concerning the access to the datasets could be
directed to Patrick Llerena (plerena@unistra.fr), the principal
investigator of the project GIGA.

REFERENCES

Abramo, G., D'Angelo, C. A., Di Costa, F., and Solazzi, M. (2009). University–industry Collaboration in Italy: A Bibliometric Examination. *Technovation* 29 (6-7), 498–507. doi:10.1016/j.technovation.2008.11.003

Arts, S., and Veugelers, R. (2015). Technology Familiarity, Recombinant novelty, and Breakthrough Invention. *Ind. Corp Change* 24 (6), 1215–1246. doi:10.1093/icc/dtu029

Azoulay, P., Ding, W., and Stuart, T. (2009). The Impact of Academic Patenting on the Rate, Quality and Direction of (Public) Research Output. *J. Ind. Econ.* 57 (4), 637–676. doi:10.1111/j.1467-6451.2009.00395.x

Banal-Estañol, A., Jofre-Bonet, M., and Lawson, C. (2015). The Double-Edged Sword of Industry Collaboration: Evidence from Engineering Academics in the UK. *Res. Pol.* 44 (6), 1160–1175. doi:10.1016/j.respol.2015.02.006

Banal-Estañol, A., Macho-Stadler, I., and Pérez-Castillo, D. (2019a). *Funding Academic Research: grant Application, Partnership, Award, and Output*. Mimeo.

Banal-Estañol, A., Macho-Stadler, I., and Pérez-Castrillo, D. (2019b). Evaluation in Research Funding Agencies: Are Structurally Diverse Teams Biased against?. *Res. Pol.* 48 (7), 1823–1840. doi:10.1016/j.respol.2019.04.008

Banal-Estañol, A., Macho-Stadler, I., and Pérez-Castrillo, D. (2013). Research Output from University-Industry Collaborative Projects. *Econ. Dev. Q.* 27 (1), 71–81. doi:10.1177/0891242412472535

Bercovitz, J., and Feldman, M. (2003). Technology Transfer and the Academic Department: Who Participates and Why?. Summer Conference, Copenhagen, June 12-14, 2003 DRUID. https://users.nber.org/~confer/2003/si2003/papers/ prio/feldman.pdf.

Boudreau, K., Guinan, E. C., Lakhani, K. R., and Riedl, C. (2012). The novelty Paradox & Bias for normal Science: Evidence from Randomized Medical grant Proposal Evaluations. *Harv. Business Sch. Working*. Paper Series #13-053 http:// nrs.harvard.edu/urn-3:HUL.InstRepos:10001229.

Boudreau, K. J., Guinan, E. C., Lakhani, K. R., and Riedl, C. (2016). Looking across and Looking beyond the Knowledge Frontier: Intellectual Distance, novelty, and Resource Allocation in Science. *Manage. Sci.* 62 (10), 2765–2783. doi:10.1287/mnsc.2015.2285

Bromham, L., Dinnage, R., and Hua, X. (2016). Interdisciplinary Research Has Consistently Lower Funding success. *Nature* 534 (7609), 684–687. doi:10.1038/ nature18315

AUTHOR CONTRIBUTIONS

All authors listed have made a substantial, direct, and
intellectual contribution to the work and approved it for
publication.

FUNDING

This study is realized under the framework of the Project GIGA (Gender Bias in Grant Allocation). GIGA is financed by the Agence Nationale de la Recherche (ANR)-AAPG Grant ANR-19-CE26-0014. We acknowledge that EUROCORES data used in this study were provided by the European Science Foundation (ESF).

ACKNOWLEDGMENTS

We are indebted to other GIGA project members, Stefano
Bianchini and Emre Özel, for their support in data
preparation and variable construction.

Bruhn, M., and McKenzie, D. (2019). Can Grants to Consortia Spur Innovation and Science-Industry Collaboration? Regression- Discontinuity Evidence from Poland. *World Bank Econ. Rev.* 33 (3), 690–716. doi:10.1093/wber/ lhx014

Carayol, N., and Matt, M. (2004). Does Research Organization Influence Academic Production?. *Res. Pol.* 33 (8), 1081–1102. doi:10.1016/j.respol.2004.03.004

Cassiman, B., and Veugelers, R. (2006). In Search of Complementarity in Innovation Strategy: Internal R&D and External Knowledge Acquisition. *Manage. Sci.* 52 (1), 68–82. doi:10.1287/mnsc.1050.0470

Cohen, W. M., and Levinthal, D. A. (1990). Absorptive Capacity: A New Perspective on Learning and Innovation. *Administrative Sci.* Q. 35, 128–152. doi:10.2307/2393553

Cohen, W. M., Nelson, R. R., and Walsh, J. P. (2002). Links and Impacts: The Influence of Public Research on Industrial R&D. *Manage. Sci.* 48 (1), 1–23. doi:10.1287/mnsc.48.1.1.14273

Cohen, W. M., Nelson, R. R., and Walsh, J. P. (2000). Protecting Their Intellectual Assets: Appropriability Conditions and Why US Manufacturing Firms Patent (Or Not) (No. W7552). National Bureau of Economic Research.

Crespi, G., D'Este, P., Fontana, R., and Geuna, A. (2011). The Impact of Academic Patenting on university Research and its Transfer. *Res. Pol.* 40 (1), 55–68. doi:10.1016/j.respol.2010.09.010

Cummings, J. N. (2004). Work Groups, Structural Diversity, and Knowledge Sharing in a Global Organization. *Manage. Sci.* 50 (3), 352–364. doi:10.1287/mnsc.1030.0134

Cunningham, P., and Gök, A. (2012). *The Impact and Effectiveness of Policies to Support Collaboration for R&D and Innovation*. Compendium Of Evidence On the Effectiveness Of Innovation Policy Intervention, Vol. 12 (06). London, United Kingdom: NESTA.

Dahlin, K. B., Weingart, L. R., and Hinds, P. J. (2005). Team Diversity and Information Use. *Academy of Manage. Journal*. 48 (6), 1107–1123. doi:10.5465/amj.2005.19573112

Dasgupta, P., and David, P. A. (1994). Toward a New Economics of Science. *Res. Pol.* 23 (5), 487–521. doi:10.1016/0048-7333(94)01002-1

David, P. A., Hall, B. H., and Toole, A. A. (2000). Is Public R&D a Complement or Substitute for Private R&D? A Review of the Econometric Evidence. *Res. Pol.* 29 (4-5), 497–529. doi:10.1016/s0048-7333(99)00087-6

D'Este, P., and Patel, P. (2007). University-industry Linkages in the UK: What Are the Factors Underlying the Variety of Interactions with Industry?. *Res. Pol.* 36 (9), 1295–1313. doi:10.1016/j.respol.2007.05.002
Ebadi, A., and Schiffauerova, A. (2015). How to Receive More Funding for Your Research? Get Connected to the Right People! *PloS one* 10 (7), e0133061.
doi:10.1371/journal.pone.0133061

Fabrizio, K. R., and Di Minin, A. (2008). Commercializing the Laboratory: Faculty Patenting and the Open Science Environment. *Res. Pol.* 37 (5), 914–931.
doi:10.1016/j.respol.2008.01.010

Fleming, L. (2007). Breakthroughs and the “Long Tail” of Innovation. *MIT Sloan Manage. Rev.* 49 (1), 69.

Florida, R., and Cohen, W. (1999). "Engine or Infrastructure? the university Role in Economic Development. From Industrializing Knowledge," in *Industrializing Knowledge: University-Industry Linkages in Japan and the United States*. Editors L. M. Branscomb, F. Kodama, and R. Florida (Cambridge, MA: MIT Press), 589–610.

Gallo, S. A., Sullivan, J. H., and Glisson, S. R. (2016). The Influence of Peer Reviewer Expertise on the Evaluation of Research Funding Applications. *PloS one* 11 (10), e0165147.
doi:10.1371/journal.pone.0165147

Geuna, A., and Muscio, A. (2009). The Governance of university Knowledge Transfer: A Critical Review of the Literature. *Minerva* 47 (1), 93–114.
doi:10.1007/s11024-009-9118-2

Glänzel, W., and Schubert, A. (2003). A New Classification Scheme of Science fields and Subfields Designed for Scientometric Evaluation Purposes. *Scientometrics* 56 (3), 357–367.
doi:10.1023/a:1022378804087

Grimpe, C. (2012). Extramural Research grants and Scientists' Funding Strategies: Beggars Cannot Be Choosers?. *Res. Pol.* 41 (8), 1448–1460.
doi:10.1016/j.respol.2012.03.004

Grimpe, C., and Fier, H. (2010). Informal university Technology Transfer: a Comparison between the United States and Germany. *J. Technol. Transf* 35 (6), 637–650.
doi:10.1007/s10961-009-9140-4

Haeussler, C. (2011). Information-sharing in Academia and the Industry: A Comparative Study. *Res. Pol.* 40 (1), 105–122.
doi:10.1016/j.respol.2010.08.007

Heckman, J. J. (1979). Sample Selection Bias as a Specification Error. *Econometrica* 47, 153–161.
doi:10.2307/1912352

Horwitz, S. K. (2005). The Compositional Impact of Team Diversity on Performance: Theoretical Considerations. *Hum. Resource Dev. Rev.* 4 (2), 219–245.
doi:10.1177/1534484305275847

Huang, K. G., and Murray, F. E. (2009). Does Patent Strategy Shape the Long-Run Supply of Public Knowledge? Evidence from Human Genetics. *Amj* 52 (6), 1193–1221.
doi:10.5465/amj.2009.47084665

Katz, J. S., and Martin, B. R. (1997). What Is Research Collaboration?. *Res. Pol.* 26 (1), 1–18.
doi:10.1016/s0048-7333(96)00917-1

Lamont, M. (2009). *How Professors Think: Inside the Curious World of Academic Judgment*. Cambridge, MA: Harvard University Press.

Lee, Y.-N., Walsh, J. P., and Wang, J. (2015). Creativity in Scientific Teams: Unpacking novelty and Impact. *Res. Pol.* 44 (3), 684–697.
doi:10.1016/j.respol.2014.10.007

Lee, Y. S. (2000). The Sustainability of university-industry Research Collaboration: An Empirical Assessment. *The J. Technol. Transfer* 25 (2), 111–133.
doi:10.1023/a:1007895322042

Leydesdorff, L., and Rafols, I. (2009). A Global Map of Science Based on the ISI Subject Categories. *J. Am. Soc. Inf. Sci.* 60 (2), 348–362.
doi:10.1002/asi.20967

Luukkonen, T. (2012). Conservatism and Risk-Taking in Peer Review: Emerging ERC Practices. *Res. Eval.* 21 (1), 48–60.
doi:10.1093/reseval/rvs001

Mansfield, E. (1995). Academic Research Underlying Industrial Innovations: Sources, Characteristics, and Financing. *Rev. Econ. Stat.* 77, 55–65.
doi:10.2307/2109992

Mansfield, E., and Lee, J.-Y. (1996). The Modern university: Contributor to Industrial Innovation and Recipient of Industrial R&D Support. *Res. Pol.* 25 (7), 1047–1058.
doi:10.1016/s0048-7333(96)00893-1

Melin, G., and Persson, O. (1996). Studying Research Collaboration Using Co-authorships. *Scientometrics* 36 (3), 363–377.
doi:10.1007/bf02129600

Merton, R. K. (1968). The Matthew Effect in Science: The Reward and Communication Systems of Science Are Considered. *Science* 159 (3810), 56–63.
doi:10.1126/science.159.3810.56

Merton, R. K. (1973). *The Sociology of Science: Theoretical and Empirical Investigations*. Chicago: University of Chicago Press.

Mindruta, D. (2013). Value Creation in university-firm Research Collaborations: A Matching Approach. *Strat. Mgmt. J.* 34 (6), 644–665.
doi:10.1002/smj.2036

Mongeon, P., and Paul-Hus, A. (2016). The Journal Coverage of Web of Science and Scopus: a Comparative Analysis. *Scientometrics* 106 (1), 213–228.
doi:10.1007/s11192-015-1765-5

Murray, F., and Stern, S. (2007). Do formal Intellectual Property Rights Hinder the Free Flow of Scientific Knowledge?. *J. Econ. Behav. Organ.* 63 (4), 648–687.
doi:10.1016/j.jebo.2006.05.017

Mutz, R., Bornmann, L., and Daniel, H.-D. (2012). Does Gender Matter in Grant Peer Review?. *Z. für Psychol.* 220 (2), 121–129.
doi:10.1027/2151-2604/a000103

Nelson, R. R., and Winter, S. G. (1982). The Schumpeterian Tradeoff Revisited. *Am. Econ. Rev.* 72 (1), 114–132.

OECD (2019). *University-industry Collaboration: New Evidence and Policy Options*. Paris: OECD Publishing.
doi:10.1787/e9c1e648-en

Olbrecht, M., and Bornmann, L. (2010). Panel Peer Review of grant Applications: what Do We Know from Research in Social Psychology on Judgment and Decision-Making in Groups?. *Res. Eval.* 19 (4), 293–304.
doi:10.3152/095820210x12809191250762

Perkmann, M., Tartari, V., McKelvey, M., Autio, E., Broström, A., D'Este, P., et al. (2013). Academic Engagement and Commercialisation: A Review of the Literature on university-industry Relations. *Res. Pol.* 42 (2), 423–442.
doi:10.1016/j.respol.2012.09.007

Rose, S. (1989). Women Biologists and the "old Boy" Network. *Women's Stud. Int. Forum*. Pergamon, 12 (3), 349–354.
doi:10.1016/s0277-5395(89)80011-1

Roumbanis, L. (2019). Peer Review or Lottery? A Critical Analysis of Two Different Forms of Decision-Making Mechanisms for Allocation of Research grants. *Sci. Technol. Hum. Values* 44 (6), 994–1019.
doi:10.1177/0162243918822744

Shannon, G., Jansen, M., Williams, K., Cáceres, C., Motta, A., Odhiambo, A., et al. (2019). Gender equality in Science, Medicine, and Global Health: where Are We at and Why Does it Matter?. *The Lancet* 393 (10171), 560–569.
doi:10.1016/s0140-6736(18)33135-0

Stephan, P. E. (2012). *How Economics Shapes Science*, Vol. 1. Cambridge, MA: Harvard University Press.

Stephan, P., Veugelers, R., and Wang, J. (2017). Reviewers Are Blinkered by Bibliometrics. *Nature* 544 (7651), 411–412.
doi:10.1038/544411a

Tijssen, R. J. W., Van Leeuwen, T. N., and Van Wijk, E. (2009). Benchmarking university-industry Research Cooperation Worldwide: Performance Measurements and Indicators Based on Co-authorship Data for the World's Largest Universities. *Res. Eval.* 18 (1), 13–24.
doi:10.3152/095820209x393145

Tijssen, R. J. W., Visser, M. S., and Van Leeuwen, T. N. (2002). Benchmarking International Scientific Excellence: Are Highly Cited Research Papers an Appropriate Frame of Reference?. *Scientometrics* 54 (3), 381–397.
doi:10.1023/a:1016082432660

Tijssen, R. J. W., Yegros-Yegros, A., and Winnink, J. J. (2016). University-industry R&D Linkage Metrics: Validity and Applicability in World university Rankings. *Scientometrics* 109 (2), 677–696.
doi:10.1007/s11192-016-2098-8

Tijssen, R. (2012). *R&D Globalization Processes and university-industry Research Cooperation: Measurement and indicators Centre for Science and Technology Studies (CSWTS) Working Paper Series*. Leiden: CSWTS.

Travis, G. D. L., and Collins, H. M. (1991). New Light on Old Boys: Cognitive and Institutional Particularism in the Peer Review System. *Sci. Technol. Hum. Values* 16 (3), 322–341.
doi:10.1177/016224399101600303

Van Arensbergen, P., van der Weijden, I., and van den Besselaar, P. (2014). The Selection of talent as a Group Process. A Literature Review on the Social Dynamics of Decision Making in grant Panels. *Res. Eval.* 23 (4), 298–311.
doi:10.1093/reseval/rvu017

Van der Lee, R., and Ellemers, N. (2015). Gender Contributes to Personal Research Funding success in The Netherlands. *Proc. Natl. Acad. Sci. USA* 112 (40), 12349–12353.
doi:10.1073/pnas.1510159112

Van Knippenberg, D., and Mell, J. N. (2016). Past, Present, and Potential Future of Team Diversity Research: From Compositional Diversity to Emergent Diversity. *Organizational Behav. Hum. Decis. Process.* 136, 135–145.
doi:10.1016/j.obhdp.2016.05.007

Vera-Baceta, M.-A., Thelwall, M., and Kousha, K. (2019). Web of Science and Scopus Language Coverage. *Scientometrics* 121 (3), 1803–1813.
doi:10.1007/s11192-019-03264-z

Wang, J., Lee, Y.-N., and Walsh, J. P. (2018). Funding Model and Creativity in Science: Competitive versus Block Funding and Status Contingency Effects. *Res. Pol.* 47 (6), 1070–1083.
doi:10.1016/j.respol.2018.03.014
Wang, J., Veugelers, R., and Stephan, P. (2017). Bias against novelty in Science: A Cautionary Tale for Users of Bibliometric Indicators. *Res. Pol.* 46 (8), 1416–1436. doi:10.1016/j.respol.2017.06.006

Wennerås, C., and Wold, A. (1997). Nepotism and Sexism in Peer-Review. *Nature* 387 (6631), 341–343. doi:10.1038/387341a0

Williams, K. Y., and O'Reilly, C. A., III (1998). Demography and Diversity in Organizations: A Review of 40 Years of Research. *Res. Organizational Behav.* 20, 77–140.

Witteman, H. O., Hendricks, M., Straus, S., and Tannenbaum, C. (2019). Are Gender Gaps Due to Evaluations of the Applicant or the Science? A Natural experiment at a National Funding agency. *The Lancet* 393 (10171), 531–540. doi:10.1016/s0140-6736(18)32611-4

Woelert, P., and Millar, V. (2013). The 'paradox of Interdisciplinarity' in Australian Research Governance. *High. Educ.* 66, 755–767. doi:10.1007/s10734-013-9634-8

Wong, P. K., and Singh, A. (2013). Do co-publications with Industry lead to Higher Levels of university Technology Commercialization Activity?. *Scientometrics* 97 (2), 245–265. doi:10.1007/s11192-013-1029-1

Zucker, L. G., Darby, M. R., and Armstrong, J. S. (2002). Commercializing Knowledge: university Science, Knowledge Capture, and Firm Performance in Biotechnology. *Manage. Sci.* 48 (1), 138–153. doi:10.1287/mnsc.48.1.138.14274

**Conflict of Interest:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The handling editor declared a shared research group ARN project Gender bias In Grant Allocation (GIGA) with the authors at the time of review.

**Publisher's Note:** All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors, and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Copyright © 2021 Öcalan-Özel and Llerena. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.