# Measuring interdisciplinarity of research grant applications An indicator developed to model this selection criterion in the ERC's peer-review process

Ivana Roche, Dominique Besagni, Claire François, Marianne Hörlesberger, Edgar L. Schiebel, Dirk Holste

<figure><img src="image_1.png" /></figure>

## To cite this version:

Ivana Roche, Dominique Besagni, Claire François, Marianne Hörlesberger, Edgar L. Schiebel, et al. Measuring interdisciplinarity of research grant applications An indicator developed to model this selection criterion in the ERC's peer-review process. 14th ISSI, Jul 2013, Vienne, Austria. hal-00960072

HAL Id: hal-00960072

https://hal.science/hal-00960072

Submitted on 26 Mar 2014

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Measuring interdisciplinarity of research grant applications.
An indicator developed to model this selection criterion
in the ERC's peer-review process

Ivana Roche¹, Dominique Besagni¹, Claire François¹, Marianne Hörlesberger², Edgar Schiebel², Dirk Holste²

¹ivana.roche@inist.fr; dominique.besagni@inist.fr; claire.francois@inist.fr

CNRS, Institut de l'Information Scientifique et Technique, UPS 76, 2 allée du Parc de Brabois, Vandœuvre-lès-
Nancy, F-54519 Cedex (France)

²marianne.hoerlesberger@ait.ac.at; edgar.schiebel@ait.ac.at; dirk.holste@ait.ac.at

AIT, Austrian Institute of Technology GmbH, Donau-City-Strasse 1, 1220 Vienna (Austria)

**Introduction**

Scientific experts having to evaluate the interest of research projects submitted for grant to the European Research Council (ERC) must lean on the four “frontier research” criteria among which: “... it pursues questions irrespective of established disciplinary boundaries, involves multi-, inter- or trans-disciplinary research that brings together researchers from different disciplinary backgrounds, with different theoretical and conceptual approaches, techniques, methodologies and instrumentation, perhaps even different goals and motivations”, according to the EC’s High Level Expert Group report (2005) assertions and that we defined in our DBF project (Hörlesberger et al., 2013) as interdisciplinarity.

**Background**

Interdisciplinarity is used as a proxy to infer self-
consistently the presence and proportions of
characteristic terminology associated with
individual ERC panels, thereby revealing the intra
or inter-panel character of a proposal. It was built
upon a previously successfully tested approach that
the frequency of occurrence and distribution of
research field specific keywords can classify and
characterize research fields. While the core of the
approach has been retained, the computation has
been adopted and fine-tuned to the grant scheme
under study. The underlying basic hypothesis is that
the larger the proportion of inter-panel keywords,
the more interdisciplinary is the proposal. To this
end, each keyword is labeled according to its
statistical frequency across all panels, filters are
applied to distinguish relevant from irrelevant
keywords, and the tallying of keywords with their
assigned panels is assessed to classify each
proposal with respect to its share of inter-panel
keywords.

**Data and Methodology**

Primary data are directly obtained from the
documents supplied by the researcher. Each project
proposal is indexed by keywords (KW) extracted
from its textual information. After validation, we
apply the diffusion model approach. First, the so-
called Home Panel (H-P) terms are defined. We
assumed KW which are specific for a panel
occurred with a higher probability in that panel
rather than in others. This probability is defined by
the frequency of one KW in a panel divided by the
number of proposals in this panel, namely the
relative term frequency (rtf<sub>KW/panel</sub>). Then, for a KW,
we calculate its rtf<sub>KW/panel</sub> in each panel and the
panel with the highest probability is declared to be
its H-P. So, we obtain for each H-P the list of its
H-P terms. Therefore the complete terminology
associated to a panel consists of the union of its H-P
term list and the set of terms imported from the
other panels and we refer to as “abroad terms”.

In order to make the lists of H-P terms more
consistent, we use the Gini index, a measure of the
inequality of a distribution, varying from 0 to 1, a
value of 0 expressing total equality and a value of 1
maximal inequality. It is commonly used in
economics as a measure of inequality of the income
or wealth. It is, in this study, employed as a
measure of the dispersion of a KW. A Gini index of
1 tells us that the KW is very specifically limited to
only one panel. Conversely, a Gini index equal to 0
means a completely uniform distribution and
indicates that the term occurs in all the considered
panels. We define a cut-off threshold to discard
KW which dispersion will be considered excessive.
Finally, the tallying of KW with their newly
assigned H-P is assessed to calculate an index for
the share of abroad terms in the list of KW
representing the content of each project proposal.
The underlying assumption is that the larger the
proportion of abroad terms, the more
interdisciplinary a project proposal is.
<figure><img src="image_3.png" /><figcaption>Figure 1. Methodological schema of the evaluation process of the interdisciplinarity of research grant applications</figcaption></figure>

## The Case Study

We applied our methodology to a case study coming from project proposals submitted to the ERC 2009 Starting Grant Call.

<table><caption>Table 1. Distribution of successful and non- successful proposals submitted to the ERC 2009 Starting Grant Call</caption><thead><tr><td></td><td>ERC StG 2009</td><td>Dataset (6 panels)</td></tr></thead><tbody><tr><td>Proposals</td><td>2.503</td><td>198</td></tr><tr><td>Successful</td><td>244</td><td>41</td></tr><tr><td>Non-successful</td><td>2.259</td><td>157</td></tr></tbody></table>

Among the 19 ERC panels representing Life Sciences (LS) and Mathematics, Physics, Chemistry, Engineering & Earth Sciences (PE), we chose 6 panels with a balance between LS and PE as well as between basic and applied fields.

The analysis shows the results for ERC panel PE1 (Mathematics and Mathematical foundations) which received 39 submissions, 11 of them successful. The value of Interdisciplinarity in this example is the share of abroad terms calculated with a Gini index cut-off threshold of 0.1.

In the DBF project, we introduced a statistical discrete choice model (DCM) to estimate the decision probability for a proposal to be accepted on the basis of measured attributes of “frontier research” (Scherngellet al., 2013).

We studied in a proof-of-concept approach the influence of those attributes on the success probability and conducted an initial analysis of the *ex-post* comparison between the indicator-based scientometric evaluation and the empirical peer-review process. Interdisciplinarity is one of the five indicators we developed in the context of this project and it is an element that proves to influence significantly the selection probability of the projects.

The results obtained with Interdisciplinarity are encouraging and we are experimenting with it on a different dataset from e-Corda.

We also have to introduce another dimension to that indicator: the diversity of the sources (H-P) of abroad terms, on top of just counting them.

## Acknowledgments

Work accomplished in the context of the DBF (Development and Verification of a Bibliometric Model for the Identification of Frontier Research) project within the European Research Council's CSA of the EU's 7th Framework Programme.

## References

* Hörlesberger M., Roche I., Besagni D., Scherngell T., Francois C., Cuxac P., Schiebel E., Zitt M., and Holste D. (2013). A concept for inferring “frontier research” in grant proposals, Scientometrics (to appear).

* Scherngell T., Roche I., Hörlesberger M., Besagni D., Züger M.-E., and Holste D. (2013). Initial comparative analysis of model and peer-review process for ERC starting grant proposals, Research Evaluation (submitted).