An update of the research excellence indicator.
Date: 2015-08-11
Institutions: Joint Research Centre
Subjects: EU research policy, innovation, organisation of research, patent, research policy, research report, research results, scientific research

An update of the Research Excellence 
Indicator 

Deliverable for WP2 of the 
INNOVA MEASURE Project 

Sjoerd Hardeman and Daniel Vertesy 
2 0 1 5  

Report EUR 27374 EN

European Commission 
Joint Research Centre 

Contact information 
Daniel Vertesy and Sjoerd Hardeman 
Address: Joint Research Centre, Via Enrico Fermi 2749, TP 361, 21027 Ispra (VA), Italy 
E-mail: daniel.vertesy@jrc.ec.europa.eu 
Tel.: +39 0332 783556 
Fax: +39 0332 785733 
JRC Science Hub  
https://ec.europa.eu/jrc 

Legal Notice 
This publication is a Technical Report by the Joint Research Centre, the European Commission’s in-house science service.  
It aims to provide evidence-based scientific support to the European policy-making process. The scientific output 
expressed does not imply a policy position of the European Commission. Neither the European Commission nor any person 
acting on behalf of the Commission is responsible for the use which might be made of this publication. 

All images © European Union 2015, except: cover image by Christian Ferrari 

JRC96817 

EUR 27374 EN 

ISBN 978-92-79-36869-1 (PDF) 

ISSN 1831-9424 (online) 

doi:10.2760/770103 

Luxembourg: Publications Office of the European Union, 2015 

© European Union, 2015 

Reproduction is authorised provided the source is acknowledged. 

Abstract 

This report describes the 2015 update of the composite indicators on scientific and technological research excellence, and 
presents the outcomes of a sensitivity analysis addressing the key underlying modelling assumptions. We mostly retain 
the  structure  of  the  indicator  introduced  in  2013  and  measure  research  excellence  in  terms  of  top  level  outcomes  of 
scientific  and  technological  research  of  countries  by  highly  cited  publications,  PCT  patents,  world  class  universities  and 
research institutes and high-value research grants (ERC), while introducing a few minor changes, concerning the alignment 
of indicators to the reference year, and the use of a single source for the indicator on universities and research institutes 
in order to improve consistency over time. As a result, we increase the coverage of the indicator to more years, and are 
better  able  to  measure  trends.  We  also  revisit  some  key  points  of  criticism  concerning  the  selection  of  indicators,  their 
scale normalization and the geographical and historical coverage. The uncertainty and sensitivity analyses reveal that the 
two key sources of volatility in country ranks are the selection of the two strong indicators only (highly cited publications 
and PCT patents, and the often overlooked choice for denominators used for scale-normalization. The findings transcend 
the statistical properties of the indicator, and underline the importance of making normative choices with regards to the 
context of use of the composite indicators. 

An  update  of  the  Research  Excellence 
Indicator 

Sjoerd Hardeman & Daniel Vértesy 

Unit of Econometrics & Applied Statistics, Joint Research Centre, European Commission 

June 2015 

Executive Summary 

This report describes the 2015 update of the composite indicators on scientific and technological 
research  excellence,  and  presents  the  outcomes  of  a  sensitivity  analysis  addressing  the  key 
underlying  modelling  assumptions.  We  mostly  retain  the  structure  of  the  indicator  introduced  in 
2013  and  measure  research  excellence  in  terms  of  top  level  outcomes  of  scientific  and 
technological  research  of  countries  by  highly  cited  publications,  PCT  patents,  world  class 
universities and research institutes and high-value research grants (ERC), while introducing a few 
minor  changes,  concerning  the  alignment  of  indicators  to  the  reference  year,  and  the  use  of  a 
single  source  for  the  indicator  on  universities  and  research  institutes  in  order  to  improve 
consistency over time. As a result, we increase the coverage of the indicator to more years, and are 
better able to measure trends. We also revisit some key points of criticism concerning the selection 
of  indicators,  their  scale  normalization  and  the  geographical  and  historical  coverage.  The 
uncertainty and sensitivity analyses reveal  that the two key sources of volatility in country ranks 
are the selection of the two strong indicators only (highly cited publications and PCT patents, and 
the often overlooked choice for denominators used for scale-normalization. The findings transcend 
the  statistical  properties  of  the  indicator,  and  underline  the  importance  of  making  normative 
choices with regards to the context of use of the composite indicators. 

Acknowledgement:  this  research  was  funded  by  the  Directorate-General  of  Research  and 
Innovation  of  the  European  Commission  (DG  RTD),  in  the  framework  of  the  INNOVA  MEASURE 
project (FP7-Adhoc-2007-13, contract nr. 632806). 

1 

Figure E1 Component and Composite scores (“2015”) 

2 

"2015" Scores by indicatorCountrieshicit_pubpct_poptopinst_gerderc_goverdherdAustriaAT7554195245BelgiumBE8040205543BulgariaBG4211105422CyprusCY59121010029Czech RepublicCZ3214102418GermanyDE6468373448DenmarkDK8864434056EstoniaEE6119101621GreeceEL5612104623SpainES6420624243FinlandFI7786154546FranceFR6139384344CroatiaHR1913101012HungaryHU4316108227IrelandIE6729104230ItalyIT5924433839LithuaniaLT4211101015LuxembourgLU5938101022LatviaLV3811101014MaltaMT5214101016NetherlandsNL8756896372PolandPL2412101815PortugalPT6513103123RomaniaRO3110101214SwedenSE7189206052SloveniaSI5027101421SlovakiaSK3212101014United KingdomUK7434477154EU-28EU5935394544EU-28 (non-ERA comparison)EUx59353943SwitzerlandCH100896210086IcelandIS86321002551NorwayNO7349423046TurkeyTR3812101014IsraelIL66681610052SerbiaRS26131001024BrazilBR38121615ChinaCN_X_HK28102319IndiaIN37851032JapanJP47601031Rep. of KoreaKR11111011RussiaRU11111011UkraineUA12111011United StatesUS72483449Research Excellence CompositeNote: Indicator scores shown above are normalized between 10-100. Arrows indicate whether the growth of the composite score was faster or slower than the growth of the respective EU-28 benchmark scores in past 4 years.% HighlyCited PublicationsPCT applicationsper populationTop Universities & PRO'sper GERDERC Grants per Public R&DCN 
 
 
 
Contents 

Executive Summary .............................................................................................................................................................................. 1 

Contents ...................................................................................................................................................................................................... 3 

1. Introduction ......................................................................................................................................................................................... 4 

2. Recapitulating the Research Excellence Indicator: structure and criticism ..................................................... 4 

3. Revisiting the Research Excellence Indicator: 5 challenges ..................................................................................... 6 

Challenge #1: how to solve the issue of some countries not being eligible for ERC grants? ............... 6 

Challenge  #2:  how  to  construct  a  full  data  set  covering  many  countries  for  many  consecutive 
years? ...................................................................................................................................................................................................... 6 

Challenge #3: which numerators (not) to consider for capturing Research Excellence? ......................... 7 

Challenge #4: which denominators (not) to consider for capturing Research Excellence? .................... 9 

Challenge #5: how to aggregate and weigh components? ...................................................................................... 9 

4. Revised structure and main results of the Research Excellence Indicator ................................................... 11 

5. Discussion and concluding remarks ................................................................................................................................... 17 

References .............................................................................................................................................................................................. 19 

Appendix A: overview of country performance .................................................................................................................. 20 

Appendix B: Descriptive statistics .............................................................................................................................................. 22 

3 

 
 
 
 
1. Introduction

This note presents the results of the indicator on research excellence in science & technology at the 
country  level.  As  such,  this  report  builds  on  previous  work  and  experience  of  the  European 
Commission related to the measurement of research excellence. The Report of the Expert Group on 
the  Measurement  of  Innovation  (Barré  et  al.,  2011)  proposed  a  very  broad  set  of  indicators  of 
research  excellence  that  encompassed 
indicators  of  research  funding,  collaboration  and 
specialization alongside output-oriented indicator. A follow-up feasibility study on the development 
of a composite indicator of research excellence by the JRC (Vértesy and Tarantola, 2012) showed 
statistically that the indicators proposed by the Expert Group cover more than a single underlying 
phenomenon.  Finally,  a  previous  study  presented  two  composite  indicators  measuring  research 
excellence  in  science  &  technology  for  a  restricted  number  of  countries  and  covering  a  limited 
number  of  years  (Hardeman  et  al.,  2013;  European  Commission,  2014).  The  current  note  then 
follows  from  a  project  initiated  by  the  Directorate-General  for  Research  and  Innovation  of  the 
European Commission (DG RTD) within the context of refining indicators for the Innovation Union. 

The aim of the present note is to offer an update of the 2013 study (Hardeman et al, 2013) in light 
of  some  of  the  criticism  raised  after  publication  of  the  earlier  study.  First,  we  recapitulate  the 
indicator  of  research  excellence  in  science  &  technology  (section  2).  Here,  we  focus  on  the  main 
structure and the criticisms that followed from its introduction. Second, we outline the  challenges 
met in revising the indicator in the face of the criticisms raised (section 3). Here, we focus on the 
number  and  kind  of  countries  that  now  included,  the  range  of  (consecutive)  years  covered,  the 
variables  included  (both  numerators  and  denominators),  and  the  choice  of  weighting  and 
aggregating components. Apart from some minor changes in the methodology of the index (slightly 
more countries, alignment of consecutive years included, transformation of variables), the structure 
of the updated index remains virtually the same. The main issue raised throughout this note is that 
no “optimal” Research Excellence Indicator can be constructed; thus emphasizing the need to report 
on and discuss volatility in the results of this composite as can be addressed using uncertainty and 
sensitivity analysis.  Fourth, following on the main structure of the indicator and differences therein 
as compared with the previous edition of the Research Excellence Indicator, we present the main 
empirical outcomes (section 4). After presenting the baseline results we proceed with the outcomes 
of  the  sensitivity  analysis  and  show  some  contrasting  results  focusing  on  two  alternative 
methodological set-ups. Finally, we conclude with a brief discussion of the main implications that 
can be derived from the results of this study (section 5).   

2. Recapitulating the Research Excellence Indicator: structure and criticism

As mentioned in the introduction to this note, the Research Excellence indicator has initially been 
proposed  in  2013  and  builds  upon  4  methodological  choices  (Hardeman  et  al.,  2013).  First,  the 
indicator built on 4 components, namely: (i) highly cited publications (source: Science-Metrix via DG 
RTD),  (ii)  PCT  patents  (source:  OECD),  (iii)  ERC  grants  (expressed  in  total  EURO  value;  source:  DG 
RTD), and (iv) top research institutes (source: Scimago). Second, these 4 components were rescaled 
using respectively (i) total number of publications, (ii) total number of inhabitants, (iii) total amount 
of expenditures in R&D, and (iv) total amount of public expenditures in R&D as denominators. Third, 
the rescaled components were normalized using a min-max approach into a common scale ranging 

4 

from  10-100.  Fourth,  the  rescaled  and  normalized  components  were  subsequently  aggregated 
using equal weights and geometric averaging.  

The update of the indicator measuring research excellence in S&T by and large follows the same 
structure as the one outlined  in Hardeman et al. (2013). However, anticipating on the request  as 
outlined in the introduction to this note and following on some of the issues that were brought to 
the  fore  after  the  introduction  of  the  previous  indicator  (cf.  S rensen  et  al.,  2015),  we  take  into 
account the effects of: 

1. 

Including  or  excluding  the  component  on  ERC  grants.  Researchers  outside  the  European 
Research  Area  are  not  eligible  for  being  funded  by  grants  on  offer  by  the  European 
Research Council. For those countries in which researchers are not eligible for ERC grants, 
this  4th  component  has  not  been  included  in  the  overall  index;  rather,  the  index  is  then 
simply based upon the remaining 3 components. Obviously, whenever statistical patterns of 
the ERC component are not in line with those found in the other 3 components, the choice 
of including 3 or 4 components for countries in which researchers are eligible to ERC grants 
might bias the outcomes of the analysis. 

2.  Covering  a  larger  set  of  countries  for  multiple  consecutive  years.  The  previous  indicator 
presented results for two (non-consecutive) years only. The aim of the update is to come up 
with an indicator that covers a wider range of consecutive years. In addition, the previous 
indicator  covered  a  limited  range  of  countries  only  (n  =  40)  and  especially  few  countries 
outside the EU were included. Hence, the aim of the update is to cover more countries by 
putting extra emphasis on including non-EU countries. 

3.  Alternative  variables  for  the  numerators  of  the  core  components.  Although  the  4 
components will remain in principle the same (i.e. still based on publications, patents, ERC 
grants, and top research institutes data), the exact specification of these variables will put 
under closer scrutiny (esp. regarding ERC grants and top research institutes). 

4.  Alternative  variables  for  the  denominators  of  the  core  components.  As  outlined  in  the 
previous  report  (Hardeman  et  al.,  2013),  the  results  from  the  indicator  on  research 
excellence in S&T were particularly sensitive to the choice of denominators to re-scale the 
components. Hence we consider to revisit the denominators of the 4 components. 

5.  Alternative  weighting  and aggregation  schemes  to  aggregate  the  components.  Composite 
indicators in general (cf. Ravallion, 2012a) and composite indicators measuring innovation 
and  research  in  science  and  technology  in  particular  (cf.  Grupp  &  Mogee,  2004;  Grupp  & 
Schubert, 2010; Saisana et al., 2011) have often been criticized for their subjectivity in the 
methodological  choices  concerning  the  underlying  weighting  and  aggregation  scheme.  On 
the  one  hand,  subjectivity  is  unavoidable  in  any  metric  system  in  the  socio-economic 
domain;  composite  or  otherwise  (cf.  Saltelli,  2007),  Yet,  on  the  other  hand,  this  does  not 
mean  that  this  subjectivity  cannot  be  accounted  for  (Saisana  et  al.,  2005;  Paruolo  et  al., 
2013).  

Overall,  these  5  issues  bring  along  an  equal  amount  of  challenges  for  constructing  a  revised 
Research  Excellence  Indicator.  In  the  next  section  we  will  address  these  challenges,  and  how  we 
propose to tackle them, in turn. 

5 

 
3. Revisiting the Research Excellence Indicator: 5 challenges 

Challenge  #1:  how  to  solve  the  issue  of  some  countries  not  being  eligible  for 
ERC grants? 
By  including  a  variable  based  on  ERC  grant  data,  the  Research  Excellence  Indicator  is  difficult  to 
construct  for  all  countries  in  the  same  way.  To  resolve  this  issue,  the  previous  version  of  the 
Research Excellence Indicator (Hardeman et al.,  2013)  simply replaced the missing value for ERC 
grants  of  non-eligible  countries  by  their  average  across  the  other  3  components.  Obviously, 
however,  such  a  replacement  might  lead  to  severe  disturbances  in  country  scores  and  rankings 
whenever  countries  that  are  eligible  for  ERC  grants  receive  disproportionally  more  or  less  grants 
relative to their scores in other components. Previously (Hardeman et al., 2013) such disturbances 
only  occurred  for  a  highly  restrictive  number  of  countries.  However,  this  is  not  to  say  that  such 
disturbances are to rule out for a new version of the indicator as well. 

More  in  general,  the  validity  of  including  the  ERC  variable,  as  with  including  other  variables  (see 
challenge #3), is an open issue and depends upon one’s conceptualization of Research Excellence to 
begin  with.  Indeed,  it  has been  argued  before (see Hardeman  et  al.,  2013)  that  the  ERC  variable 
constitutes a so-called “weak” indicator of Research Excellence as it can be considered an input to 
rather than an output of excellent research activities. Overall,  we do not believe there is a single 
best  Research  Excellence  Indicator;  at  least  not  one  that  can  be  objectively  defined  based  on 
statistical  reasoning  only.  More  important  then,  what  we  choose  to  do  is  to  bring  to  the  fore 
disturbances, including those due to including or excluding the ERC variable, using uncertainty and 
sensitivity  analysis.  In  so  doing,  we  can  still  come  up  with  a  single  composite  indicator  for  all 
countries,  whilst  not  hiding  any  of  the  volatility  that  might  be  there  due  to  the  particular 
methodological set-up of our proposed Research Excellence Indicator. 

Challenge  #2:  how  to  construct  a  full  data  set  covering  many  countries  for 
many consecutive years? 
When  constructing  a  composite  indicator  there  is  always  a  trade-off  between  including  more 
countries and years on the one hand and introducing more and more missing values in the data on 
the  other.  The  problem  of  missing  values,  though,  can,  at  least  partially,  be  resolved  using 
advanced statistical techniques that allow making an informed guess about those values that are 
missing.  Giving  the  missing  value  profiles  of  our  variables  of  interest  we  deem  imputation  of 
missing values feasible for some but not all variables.1 That is to say, for those variables that have 
missing  values  for  more  than  50%  of  the  total  number  of  observations  we  do  not  consider 
imputation  feasible.  As  a  consequence  we  can  only  construct  a  composite  indicator  covering  4 
years. As to the range of countries included, we first considered covering and collected raw data for 

1 Full details of the imputation of missing values are available upon request. 

6 

 
 
 
                                                 
60 countries.2 However, some countries are poorly represented across the board; that is, for some 
countries  missing  values  pop  up  for  (nearly)  all  variables.  Hence,  we  choose  to  exclude  those 
countries. In addition, it has been explicitly noted by some data providers (in particular the OECD on 
PCT patents and Science-Metrix on highly cited publications) that data are relatively unreliable for 
those countries reporting extreme low values. Whereas the exact threshold of extreme low values 
has  not  been  specified,  we  decide  to  exclude  those  countries  that  report  values  belonging to  the 
lowest 10% of the distribution. Overall, we are left with a complete data set on all variables for 43 
countries  (including  the  EU)  covering  4  years.  Note  however,  that  for  some  variables  (i.e.  those 
having  missing  value  profiles  <  50%)  we  have  longer  time  series  as  our  imputation  method  has 
been applied on a time series of 9 years. 

An additional, though rather specific, issue here concerns the set-up of the 4th component; that is, 
the  component  measuring  top  universities  and  research  institutes.  Before,  this  component  was 
based  upon  including  both  Scimago  and  CWTS  data.  However,  as  the  methodology  of  the  data 
collected by CWTS has not remained the same over the years, these data cannot be collected for a 
set  of  consecutive  years  in  a  systematic  way.3  Hence,  we  choose  to  include  Scimago  data  only 
instead. 

Challenge  #3:  which  numerators  (not)  to  consider  for  capturing  Research 
Excellence? 
As explained in the previous report (Hardeman et al., 2013), we collected two sets of variables  – 
strong and weak – based on 3 distinct criteria (see Figure 1). We believe these 3 criteria and the 
numerators  thus  selected  by  following  the  flow  chart,  are  still  valid  in  principle.  First,    including 
highly  cited  publications  as  a  numerator  is  valid  as  (i)  these  data  concern  research  (albeit  fairly 
restricted to scientific research only), (ii) these data represent high-end quality aspects of research 
(albeit  perhaps  biased  towards  mainstream  high-end  quality  only),  and  (iii)  these  data  represent 
research outputs (albeit in codified form only). Admittedly, highly cited publications do not capture 
all aspects of Research Excellence, However, this is much more a legitimation for including other 
variables as well, instead of a reason not to take into account this particular variable. In fact, given 
that highly cited publications meet all criteria deemed important, we consider it a strong indicator 
of Research Excellence. 

Second,  a  similar  story  holds  for  data  on  PCT  patents.  In  other  words,  PCT  patents  (i)  are  clearly 
related  to  research  (albeit  technological  research  only),  (ii)  represent  high-end  quality  aspects  of 
research  (albeit  perhaps  biased  towards  multinational  research  activities  only),  and  (iii)  are 

2  The  total  amount  of  raw  data  we  collected  in  all  covers  14  years  (2000-2013).  As we  finalized  our data 
collection  as  of  December  2014,  this  latter  year  has  not  been  included.  Concerning  country  coverage,  we 
collected data for 60 countries. The countries covered involve: (i) EU member states, (ii) n on-EU (potential) 
ERA members (as the European Research Area stretches beyond the set of EU member states, we also collect 
data for all non-EU ERA members), (iii) non-EU OECD member states, and (iv) BRICS countries (Brazil, Russia, 
India, China, and South-Africa) 
3  In  fact,  if  we  look  at  the  patterns  in  the  CWTS  data  across  the  years,  for  each  country  we  find  a  “boat-
shaped” curve indicating that each country would have had relatively many excellent institutes in the earlier 
years of data collection, witness a sudden drop thereafter, and then again increase in the number of excellent 
institutes for the latest years. Given that we find this pattern for most countries, we do not trust these results 
to capture real developments in the number of excellent institutes per country. 

7 

 
 
                                                 
concerned  with  outputs  instead  of  inputs  (albeit  again  in  codified  form  only).  Like  highly  cited 
publications,  focusing  on  PCT  patents  only  is  not  enough  for  capturing  Research  Excellence  as  a 
whole. Hence, again, a composite indicator approach seems legitimate.  Some have argued, though, 
that variables are available (such as – highly – cited EPO patents) that capture Research Excellence 
better  than  PCT  patents.  Here,  we  respond  that  while  alternatives  are  not  available  in  a  timely 
fashion (e.g. triadic patents), others suffer from other biases (such as a home country bias for EPO 
patents). Overall, then, like highly cited publications, we consider PCT patents a strong indicator in 
measuring Research Excellence. 

Third, though in contrast to highly cited publications and PCT patents, the variable measuring top 
universities and research institutes constitutes a weak instead of a strong indicator. Although this 
variable  is  both  about  research  and  high-end  quality  research,  it  can  be  questioned  whether  this 
variable is merely  about outputs or also includes an input (mass) component. On the other hand, 
the same aspect (mass) that renders this variable being susceptible of capturing inputs rather than 
outputs, also puts emphasis on some of the human (as opposed to codified) aspects of research 
outputs; that is, those embodied in people as they work in universities or research institutes. Hence, 
albeit a weak indicator, we do believe that this variable captures aspects of research Excellence not 
grasped by any of the other strong indicators of Research Excellence. 

Figure 1 Flow chart on the selection of variables on Research Excellence 

Finally, the same can be said about the inclusion of ERC grants data as an indicator of Research 
Excellence. Albeit weak (i.e. focusing on inputs rather than outputs), it captures aspects of research 
not  being  captured  by  any  of  the  other  strong  indicators  of  Research  Excellence.  That  is  to  say, 
whilst the other indicators offer an ex-post assessment of Research Excellence, ERC grants tell us 

8 

 
 
 
something about Research Excellence “in-the-making”. All in all, and perhaps superfluously, we do 
not consider the proposed set of numerators undebatable let alone perfect. Apart from the problem 
of  ERC  grants  data  not  being  available  for  all  countries  (challenge  #1),  we  take  it  as  a  valid 
criticism  that  the  proposed  Research  Excellence  Indicator  includes  both  “strong”  and  “weak” 
indicators. Framed differently, however, one could also argue that whereas most assessment only 
focus  on  a  limited  set  of  “conservative”  indicators,  the  proposed  Research  Excellence  Indicator 
offers a first attempt to broaden the scope of indicators to include less conventional variables. Off 
course, as with including or excluding ERC grants data, it remains open for a thorough uncertainty 
and  sensitivity  analysis  whether 
including  weak/non-conventional  variables  alongside 
strong/conservative variables makes a substantial difference in terms of scores and rankings. 

Challenge  #4:  which  denominators  (not)  to  consider  for  capturing  Research 
Excellence? 
Choosing  the  right  denominators  is  a  tricky  issue,  as  different  denominators  imply  a  different 
interpretation of Research Excellence altogether. That is to say, whilst denominating by investments 
in  research  offers  a  value-for-money  interpretation  to  research  excellence,  denominating  by  the 
number  of  researchers  in  a  country  renders  a  labor  productivity  interpretation  of  research 
excellence. On a similar account, denominating by the total population of a country or its economic 
size (as measured by for example total GDP) offers yet other interpretations to research excellence. 
Answering  the  question  of  which  denominator  is  the  right  one  thus  crucially  depends  upon  our 
preference for either one of these different interpretations.  

We  do  not  favor  either  one  of  the  different  interpretations  of  Research  Excellence,  although  we 
admit that, for political (i.e. normative) reasons one or the other interpretation might be preferred. 
In the end, we stick to the structure of the previously proposed Research Excellence Indicator; that 
is, one that includes a mix of different denominators. Offering a mix of different denominators in a 
way does justice to these different interpretations simultaneously. Nevertheless, as with the other 
methodological choices discussed throughout the challenges discussed here, we deem it of crucial 
importance to interpret its results in light of alternative methodological set ups. To facilitate such 
comparisons,  we  perform  uncertainty  and  sensitivity  analysis  taking  into  the  volatility  of  the 
denominators  included  in  the  proposed  Research  Excellence  Indicator  vis-à-vis  alternative 
composites that include different denominators.    

Challenge #5: how to aggregate and weigh components? 
Choosing an appropriate scheme to aggregate and weigh components is not an exact science. First, 
as  to  aggregating  components,  the  main  issue  concerns  the  amount  of  compensability  among 
components  allowed  for.  Whilst  arithmetic  averaging  allow  for  full  compensability  among 
components  (i.e.  bad  performance  in  one  component  can  be  fully  compensated  by  optimal 
performance in another component), geometric averaging only allow for partial compensability (i.e. 
good performance in one component can partially but not fully substitute for bad performance in 
another  component).  The  issue  of  compensability  is,  arguably,  most  pressing  in  the  context  of 
measuring human development (Klugman et al., 2011; Ravallion, 2012b) and less so in the context 
of measuring Research Excellence. Nevertheless, it might be argued that in order to be excellent, a 

9 

 
 
 
country should perform relatively well on all components, not just perform excellent in a single one. 
All  in  all,  we  propose  to  take  into  account  the  choice  of  aggregating  components  as  one  of  our 
ingredients in the set-up of our uncertainty and sensitivity analysis. 

Second,  concerning  weighting  different  components  in  an  overall  index,  a  distinction  needs  to  be 
made  between  nominal  and  effective  weights  (Paruolo  et  al,  2013).  Nominal  weights  are  the 
weights that have been imposed by the developer of a composite a priori, whilst effective weights 
tell  us  something  about  the  actual  importance  of  the  roles  played  by  different  components. 
Although we do not know how important each component should be vis-à-vis all other components, 
we do have a clear take on the importance of some components vis-à-vis some other components. 
That is to say, preferably we would have the strong indicators to turn out more important than the 
weak  indicators.  In  addition,  though  not  backed  up  by  clear  theoretical  reasoning,  we  would 
preferably  have  the  strong  indicators  approximately  equally (effectively)  important  and  the  weak 
indicators equally (effectively) important.  

10 

 
 
 
 
4. Revised structure and main results of the Research Excellence Indicator 

As becomes clear from Table 1, the structure of the revised Research Excellence Indicator remains 
largely  intact.  Apart  from  the  particular  components  being  transformed,  the  main  changes  thus 
applied for this edition of the Research Excellence Indicator concern the addition of countries and 
years.  Concerning  the  inclusion  of  years  it  has  to  be  noted  that  for  this  edition  years  among  all 
variables (numerators and denominators) have been aligned as to allow for comparisons over time 
more  appropriately.  The  reason  for  transforming  different  components  for  this  edition  is  that  by 
including  a  set  of  different  countries  and  years,  distributions  have  changed  in  a  way  such  that 
whilst some components (i.e. PCT patents over population) are no longer extremely skewed, others 
(i.e. Value of ERC grants over GERD) now are. Note, however, that, given that we add new years and 
countries, also the minimums and maximums per component differ vis-à-vis the previous edition. 

Table 1 The revised structure of the Research Excellence Indicator 

Numerator 

Denominator 

Outlier 
Treatment 

Normalization 

Weighting and 
Aggregation 

Data Source 

Nr. of highly-cited 
publications (top 10%) 
PCT patents 

Nr. of 
publications 
Population 

- 

- 

Min-Max  
(10-100) 

Geomeric average 
using nominally 
equal weights 

Science Metrix 
(Scopus) 
OECD, Eurostat 
DG RTD 

Value of ERC grants  GOVERD + HERD  Winsorization 
Top universities & 
research organizations 
*)  Note:  The  indicator  of  ‘Top  universities  and  research  organizations’  is  constructed  using  ranking  data  based  on 
normalized Impact scores from Scimago Institute Ranking only to allow coherent comparison over time, differently from 
previous editions, which combined Leiden ranking as well as data from Scimago Institute Ranking. 

Winsorization 

Scimago IR* 

GERD 

Figure  2  presents  the  main  results  of  the  research  excellence  indicator  for  the  four  most  recent 
years. Blue bars refer to ERA countries, red bars to non-ERA countries. The EU-28 is considered to 
be a special actor and is shown by two scores for better comparison: the ‘EU’ score was computed 
by including all four indicators of research excellence, the ‘EUx’ score excludes ERC grants and thus 
should  be  used  to  compare  with  non-ERA  countries. This  helps  avoid  misleading  interpretation  of 
trends over time, since ‘ERC’ is the indicator responsible for most growth over time due to the fact 
that ERC grant value has been gradually increasing.   

Timeliness is  a  problematic  issue  when  it  comes  to measuring  research  excellence,  most  notably 
because  the  quality  of  research  is  assessed  by  peers.  For  instance,  in  the  case  of  highly  cited 
publications,  we  apply  a  citation  window  of  3  years.  Thus,  from  bibliometric  data,  we  can  only 
identify excellent publications of 2010 after the end of 2013. Given our aim to align indicators to 
the same year so that comparisons over time – and studies on links between policy measures and 
excellence – can be done in a proper manner, we build the composite from seemingly less recent 
data. When labelling years, we refer to the year of edition in double quotation marks (i.e., “2015” 
for the most recent year), but in the Appendix table refer to the data corresponding to the year of 
measurement. 

11 

 
 
 
 
Figure 2 Research Excellence Composite Scores 

90

80

70

60

50

40

30

20

10

0

"2015" - Non-ERA

"2015" - ERA

"2012"

"2013"

"2014"

H
C

L
N

K
D

K
U

E
S

L
I

S

I

S
U

E
D

I

OF
N

T
A

R
F

U
E

E
B

x
U
E

S
E

T
I

P
J

R
K

E
I

Y
C

U
H

S
R

L
E

T
P

G
B

U
L

E
E

I

S

N
C

N

I

Z
C

T
M

R
B

L
P

T
L

R
T

V
L

K
S

O
R

R
H

A
U

U
R

Source: Authors’ calculations as described in the text.  
Note: ERC scores are excluded from Non-ERA country (including EUx) scores  

The results from the uncertainty and sensitivity analysis are also very instructive. The construction 
of the composite indicator involved taking various decisions amidst some degree of uncertainty, so 
final scores – and ranks – are loaded with all these accumulated uncertainties. Table 2 shows the 
different  methodological  choices  taken  into  account  for  our  uncertainty  and  sensitivity  analysis. 
First, although our proposed Research Excellence Indicator uses geometric averaging to aggregate 
the  4  components,  we  compare  this  methodological  with  using  arithmetic  averaging.  This  allows 
more  compensation  between  country  scores  in  the  different  dimensions  of  excellence.  Second, 
although our proposed Research Excellence Indicator includes 4 components (scale-normalized with 
4 different numerators), for our uncertainty and sensitivity analysis we also considered (i) focusing 
on strong indicators of Research Excellence only (hence including highly cited publications and PCT 
patents  as  numerators  only)  and  (ii)  focusing  on  those  variables  for  which  in  principle  we  could 
have data on all countries (hence excluding ERC grants data from the analysis). This is particularly 
important to assess the validity of comparing countries of the European Research Area with non-
ERA countries. Finally, we considered as alternatives a set of different denominators. The choice of 
denominator  for  scale-normalization  is  often  down-played,  however  each  denominator  has  a 
different meaning, and selecting a ‘best one’ for making large and small countries comparable is 
not self-evident. Often the purpose of the analysis (i.e., efficiency in spending or producing) defines 
the best denominator. Here we compare our proposed ‘baseline’ framework which includes a mix of 
denominators with a set of composites that include the same denominator across all components 
(GDP,  population,  the  number  of  researchers,  gross  R&D  expenditure,  or  public  and  private  R&D 
spending). In a way,  these alternatives offer a set  of composite indicators offering a much more 
restricted  view  on  what  Research  Excellence  stands  for.  That  is  to  say,  whilst  our  proposed 
Research Excellence Indicator seeks to offer an encompassing view on Research Excellence by not 

12 

 
 
favoring for example either a value-for-money perspective or a labor-productivity perspective, the 
alternatives considered here offer a much more singular view on the issue.   

Table 2 Overview of modelling choices considered for the robustness analysis 

Reference (baseline scenario) 

Alternatives 

Uncertainty in the aggregation formula 
geometric average 

Uncertainty in the selection of component indicators 

HICIT, PCT, TOPINST, ERC 

arithmetic average 

Strong indicators only (HICIT, PCT) 
Exclude ERC for all countries (HICIT, PCT, TOPINST) 

Uncertainty in the selection of denominator for rescaling components 

Indicator-specific denominators  
(HICIT / Total Publication; PCT/Population; 
TOPINST/Population; ERC/Public R&D) 

All components denominated by GDP 
All components denominated by Population 
All components denominated by Nr. Researchers 
All components denominated by GERD 
Components denominated by PCT/BERD, all others divided 
by Public R&D (GOVERD+HERD) 

All  combined,  we  compute  scores  and  ranks  for  36  scenarios  for  the  sensitivity  analysis.4  Figure 
3presents the outcomes of the sensitivity analysis in a way that makes clear which methodological 
choices are most important in designing the Research Excellence Indicator. Each box plot shows the 
differences between the average ranks obtained after taking a selected alternative with respect to 
the average ranks obtained from all other alternatives, considering the 4-year pooled country data. 
This way, all potential sources of uncertainties are considered simultaneously, so the box plots can 
be compared with one another. The median is represented by a line subdividing the box, the length 
of the box represents the interquartile range (IQR), while whiskers extend to include all data within 
1.5  x  IQR  from  the  upper  and  lower  quartiles.  That  is  to  say,  those  alternative  methodological 
choices  showing  the  largest  spread  in  rank  shifts  are  those  that  render  the  Research  Excellence 
scores  most  volatile.  Rank  shifts  may  exaggerate  marginal  score  changes  especially  in  middle 
ranges, so it is advisable to compare the uncertainty results in terms of score shifts as well. Scores 
obtained  for  the  various  scenarios,  of  course,  need  to  be  re-normalized  in  order  to  make  them 
comparable  compare.  Thus,  ranks  in  Figure  3  should  be  considered  together  with  score  shifts 
plotted in Figure 4.  

Clearly, some methodological choices are more influential than others. The choice of aggregation in 
terms  of  which  average  to  use  makes  little  difference  (first  box),  but  the  selection  of  the  set  of 
indicators  and  the  choice of  denominators  have  a  clear  impact.  In  particular, two  methodological 
choices  stand  out.  One  concerns  in  the  inclusion  or  exclusion  of  weak  indicators  as  numerators 
constituting  the  Research  Excellence  Indicator  (second  box).  Since  correlation  between  strong 
indicators  and  weak indicators  is  relatively  weaker,  (see  Table  5  of  Appendix  B),  focusing  just  on 
strong indicators renders vastly different rankings than including weak indicators as well. This can 
be attributed to the large number of 0’s in the TOPINST data and the sensitivity to the threshold 
with regards to how many top institutions are to be counted therein. When only the ERC indicator is 
excluded  from  the  aggregation  (third  box),  the  rank  shifts  are  more  limited.  The  other 

4 These result from simultaneously assessing  2 alternatives for the aggregation formula, 3 alternatives for 
the set of components, and 6 alternatives for the uncertainty in the denominators. 

13 

 
 
                                                 
methodological  choice  that  stands  out  concerns  the  choice  of  denominator;  in  particular  whether 
one takes a singular view on Research Excellence focusing on a value-for-money perspective or not. 
Using either total  expenditures in R&D (GERD – seventh box)  or a mix of tailored  expenditures in 
R&D (BERD for patents and GOVERD + HERD for all other numerators; eighth box) matters a great 
deal in the rankings of countries as compared to the proposed mixed set of numerators.  

Figure 3 Uncertainty in country rankings due to changing assumptions: rank shifts 

Figure 4 Uncertainty in country rankings due to changing assumptions: score changes 

14 

-150-100-50050100Rank Shifts, 4 yearsArithmetic AverageStrong indicators onlyExcluding ERC for allAll Denoms: GDPAll Denoms: POPAll Denoms: #ResearchersAll Denoms: GERDDenoms: Public or Private R&DNote: Rank shifts resulting from simulation runs for 36 scenarios for 43 countries, 4 years-.50.5Score shiftsArithmetic AverageStrong indicators onlyExcluding ERC for allAll Denoms: GDPAll Denoms: POPAll Denoms: #ResearchersAll Denoms: GERDDenoms: Public or Private R&DNote: re-normalized scores (0-1) resulting from simulation runs for 36 scenarios for 43 countries, 4 years 
 
 
Overall,  rankings  differ  substantially  when  comparing  our  proposed  (i.e.  baseline)  Research 
Excellence  Indicator  with  those  using  alternative  methodological  set-ups.  Table  3  presents  the 
baseline ranking and compares it with the median rank across all different methodological set-ups 
considered  throughout  the  sensitivity  analysis  and  its  corresponding  50%  confidence  interval.  As 
becomes  especially  clear  by  considering  the  confidence  intervals,  rankings  differ  considerably  by 
considering one or the other methodological set-up. 

Table 3 Robustness of country ranks with respect to all uncertainties considered (“2015”) 

Country 
Switzerland 
Netherlands 
Denmark 
United Kingdom 
Sweden 
Israel 
Iceland 
United States 
Germany 
Norway 
Finland 
Austria 
France 
EU-28 
Belgium 
EU-28* 
Spain 
Italy 
Japan 
Rep. of Korea 
Ireland 
Cyprus 

Code  Baseline  Median 
CH 
NL 
DK 
UK 
SE 
IL 
IS 
US 
DE 
NO 
FI 
AT 
FR 
EU 
BE 
EUx 
ES 
IT 
JP 
KR 
IE 
CY 

1 
2 
6 
10 
5.5 
6 
9.5 
13 
10 
14 
11 
13.5 
16 
16 
14 
18 
17 
19.5 
21 
21 
25 
21 

1 
2 
3 
4 
5 
6 
7 
8 
9 
10 
11 
12 
13 
14 
15 
16 
17 
18 
19 
20 
21 
22 

50% 
confidence 
interval 

[1, 3] 
[2, 4] 
[4, 11] 
[8, 14] 
[5, 15] 
[3, 10] 
[5, 16] 
[10, 27] 
[7, 16] 
[9, 19] 
[7, 24] 
[11, 25] 
[13, 20] 
[14, 19] 
[12, 19] 
[16, 19] 
[13, 23] 
[10, 23] 
[19, 35] 
[18, 31] 
[20, 33] 
[8, 26] 

Country 
Hungary 
Serbia 
Greece 
Portugal 
Bulgaria 
Luxembourg 
Estonia 
Slovenia 
P.R. China 
India 
Czech Rep. 
Malta 
Brazil 
Poland 
Lithuania 
Turkey 
Latvia 
Slovakia 
Romania 
Croatia 
Ukraine 
Russia 

Code  Baseline  Median 
HU 
RS 
EL 
PT 
BG 
LU 
EE 
SI 
CN 
IN 
CZ 
MT 
BR 
PL 
LT 
TR 
LV 
SK 
RO 
HR 
UA 
RU 

1 
2 
6 
10 
5.5 
6 
9.5 
13 
10 
14 
11 
13.5 
16 
16 
14 
18 
17 
19.5 
21 
21 
25 
21 

23 
24 
25 
26 
27 
28 
29 
30 
31 
32 
33 
34 
35 
36 
37 
38 
39 
40 
41 
42 
43 
44 

50% 
confidence 
interval 

[1, 3] 
[2, 4] 
[4, 11] 
[8, 14] 
[5, 15] 
[3, 10] 
[5, 16] 
[10, 27] 
[7, 16] 
[9, 19] 
[7, 24] 
[11, 25] 
[13, 20] 
[14, 19] 
[12, 19] 
[16, 19] 
[13, 23] 
[10, 23] 
[19, 35] 
[18, 31] 
[20, 33] 
[8, 26] 

Note: * EUx scores computed for comparison with non-ERA countries (exclude ERC scores) 

Zooming  in  on  the  two  alternative  methodological  set-ups  that  show  the  highest  volatility  as 
compared  to  our  proposed  (i.e.  baseline)  Research  Excellence  Indicator,  Figure  5  and  Figure  6 
present differences in scores and ranks when comparing the baseline Research Excellence Indicator 
with  respectively  including  strong  indicators  only  (Figure  5)  and  using  expenditures  in  R&D  as 
denominators only (Figure 6). Considering the strong indicators scenario, some of the countries that 
improve  their  performance  significantly  (by  9  positions  or  more)  include  Finland,  Japan  and 
Luxembourg, while Serbia, the United Kingdom and India fall back by 8 positions or more. Should 
one  choose  a  scenario  where  GERD  is  used  to  scale-normalize  all  indicators,  countries  such  as 
Cyprus,  Serbia,  Bulgaria,  Hungary,  Estonia,  Romania  and  Ukraine  would  gain  more  than  15 
positions,  while  the  US,  Finland  and  Luxembourg  would  drop  15  positions  – evidently,  due  to  the 
relatively lower and stronger R&D spending, respectively.  

Clearly, using either conservative, strong indicators only and/or taking a singular, value-for-money 
perspective  on  Research  Excellence  matters  a  great  deal  for  the  scores  and  rankings  thus 

15 

 
 
generated. This is not to say that one or another set-up of a Research Excellence Indicator is better 
or worse. Rather, it does point out that the particular view one has on Research Excellence should 
be considered of crucial importance for how Research Excellence gets measured. 

Figure 5 Score and Rank differences: strong indicators only vs. the baseline scenario  

100

90

80

70

60

50

40

30

20

10

0

Strong Indicators Only

Baseline

0

9
-

2
-

1

3

0

2
-

4
-

0
1
-

0

4
-

4

6

0
1

5
-

3

1
1
-

3
-

5

4

4

4

6
-

2
-

5
-

4

2

6
-

6

7
-

4

1

5
-

5
-

2

4
-

2

3
-

5
1

8

5

0

0

0

I

HF
C

E
S

K
D

L
N

L
I

E
D

T
A

P
J

O
N

E
B

S
U

S

I

K
U

R
K

R
F

U
L

E
I

U
E

x
U
E

S
E

T
I

E
E

T
P

I

S

Y
C

L
E

T
M

U
H

T
L

G
B

N
C

R
T

V
L

Z
C

K
S

R
B

O
R

S
R

N

I

L
P

R
H

A
U

U
R

Note:  Rank  differences  shown  in  black  boxes;  negative  values indicate rank  improvements  with regards  to  the  baseline 
scenario.  

Figure  6  Score  and  Rank  differences:  all  indicators  scale-normalized  by  GERD  vs.  the  baseline 
scenario 

100

90

80

70

60

50

40

30

20

10

0

Denominator = GERD

Baseline

3
2
-

0

9
1
-

3

1
-

1
2
-

3

9
-

4
1
-

3

4
1
-

6
-

1
-

5
1
-

0
1

3
1

7

9

6

5

1
1
-

9
1
-

7

7
-

3
1

5
1

6
1
-

0
2

9

1
1

6
-

0
1
-

6
-

3
1

1
-

2

7

2
1

6

0

3

2
-

5
1

9

S
R

L
N

Y
C

H
C

L
I

G
B

K
U

S
E

U
H

S

I

L
E

T
I

U
E

E
E

E
S

K
D

O
N

E
D

R
F

E
B

N

I

O
R

x
U
E

N
C

T
A

I
F

A
U

S
U

R
K

P
J

T
L

R
H

V
L

E
I

L
P

T
M

I

S

T
P

Z
C

K
S

R
T

U
R

U
L

R
B

Note:  Rank  differences  shown  in  black  boxes;  negative  values indicate rank  improvements  with regards  to  the  baseline 
scenario. 

A final result to report upon here concerns the variance of the composite scores explained by each 
of the components. Using sensitivity indices, we aim at understanding whether the strong indicators 
and  the  weak  indicators  have  an  effective  weight  that  reflects  statistically  their  conceptual 
importance.  Table  4  presents  correlations  of  the  various  components  with  respect  to  each  other 
and the overall proposed index, and sensitivity indices computed as correlation squared. The highly 
cited  publications  over  total  publications  and  the  PCT  patents  indicators  explain  the  strongest 
variance,  thus  have  the  strongest  effective  weight,  while  the  two  weaker  indicators  have  less 

16 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
influence  on  total  variance.  This  is  very  much  in  line  with  the  expectations  and  thus  makes  any 
‘manual’  adjustment  of  weights  (scaling  coefficients)  according  to  Paruolo  et  al  (2013)  less 
relevant. 

Table 4 Sensitivity Indices: correlation between the components and the composite 

Component 

Correlation with composite  

r2 (sensitivity indices) 

hicit_pub   (‘strong’) 

pct_pop   (‘strong’) 
topinst_gerd   (‘weak’) 

erc_goverdherd   (‘weak’) 

0.87 

0.78 
0.68 

0.55 

0.76 

0.61 
0.46 

0.30 

5. Discussion and concluding remarks 

This note offered an update of the Research Excellence Indicator comparing EU member states with 
its main comparators (other ERA countries, OECD countries, and BRICS countries) in terms of their 
performance in producing excellent research. Apart from revisiting the main methodological set-up 
of the indicator to accommodate more countries and (consecutive) years, we presented an update 
of  the  main  results  in  terms  of  scores  and  rankings.  More  importantly,  the  main  result  of  this 
update  holds  that  any  indicator  measuring  Research  Excellence  crucially  depends  upon  the  basic 
conceptual framework underlying it. Herein, two methodological choices stand out.  

First, when it comes to the choice of numerators it matters a great deal whether one restricts the 
kind of numerators to include strong, conservative indicators only or to expand this set by including 
less conventional albeit perhaps weaker ones. Opting for the latter choice might become ever more 
relevant in light of new trends in research such as the emergence of Mode 2 knowledge production 
(Nowotny  et  al.,  2001)  and  Science  2.0  (Waldrop,  2008).  Such  and  related  trends  call  for  a  re-
thinking of the meaning of Research Excellence beyond the “usual  suspects”.  Although in shifting 
from scientific excellence to (scientific and technological) research excellence, arguably, we already 
extended our scope, most indicators are still largely based upon evaluations typically taking place 
with  reference  to  academia’s  peer  review  system  and  therewith  do  take  on  board  alternative 
research evaluations (S rensen et al.,  2015). One potential consequence of such an expansion of 
indicators  might  be  that  the  different  components  capturing  Research  Excellence  become  ever 
more  misaligned;  not  because  they  do  not  all  capture  Research  Excellence  but  because  what  is 
considered  research  and  excellent  research  therein  expands  vastly  different  activities  and  value 
assessments.  

Second, when it comes to the choice of denominators it matters a great deal whether one takes a 
singular  (e.g.  value-for-money)  or  more  encompassing  view  upon  Research  Excellence  has  an 
important impact on resulting scores and rankings. This is not to say that one or the other is better 
or worse. All depends on the purpose for using one or another methodological set-up. For example, 
if one is to use such an indicator for allocating research funds, it matters a great deal whether one 
is interested in raising labor productivity of researchers in terms of excellent outcomes or whether 
one is interested in raising (monetary) returns on investments. What is clear is that the results of a 
Research  Excellence  Indicator  are  not  univocally  clear  among  the  various  interpretations  that  are 
around.  

17 

 
 
Overall, our main message holds that any indicator measuring Research Excellence should be used 
with  great  care.  Given  that  research  systems  are  complex  dynamic  systems,  inputs  often  do  not 
translate into (excellent) outputs linearly. That is to say, even if we were to be interested in returns-
on  investments  in  research,  keeping  denominators  fixed  (i.e.  using  GERD  across  all  components) 
might still offer a deceiving picture whereas investments made in research today do not translate 
into excellent research outcomes today as well.   

18 

 
 
 
 
References 

Barré,  R.,  Hollanders,  H.  &  Salter,  A.  2011.  Indicators  of  research  excellence.  Retreived  from: 
http://ec.europa.eu/commission_2010-2014/geoghegan-quinn/hlp/documents/20120309-hlp-
research-excellence_en.pdf.  

European  Commission  (2014).  Innovation  Union  Competitiveness  Report  2013.  Luxembourg: 
Publications Office of the European Union. 

Grupp, H., & Mogee, M. E. (2004). Indicators for national science and technology policy: how robust 
are composite indicators?. Research Policy, 33(9), 1373-1384. 

Grupp, H., & Schubert, T. (2010). Review and new evidence on composite innovation indicators for 
evaluating national performance. Research Policy, 39(1), 67-78. 

Hardeman,  S.;  Van  Roy,  V.;  Vertesy,  D.  (2013).  An  analysis  of  national  research  systems  (I):    a 
composite  indicator  for  scientific  and  technological  research  excellence.  JRC  Scientific  and  Policy 
Reports, Luxembourg: Publications Office of the European Union. 

Klugman,  J.,  Rodríguez,  F.,  &  Choi,  H.  J.  (2011).  The  HDI  2010:  new  controversies,  old  critiques. 
Journal of Economic Inequality, 9(2), 249-288. 

Nowotny H., Scott P., Gibbons M. (2001) Rethinking science: knowledge and the public in an age of 
uncertainty. Cambridge: Polity. 

Paruolo, P., Saisana, M., & Saltelli, A. (2013).  Ratings and rankings: voodoo or science?. Journal of 
the Royal Statistical Society: Series A (Statistics in Society), 176(3), 609-634. 

Ravallion, M. (2012a). Mashup Indices of Development. The World Bank Research Observer, 27(1), 
1-32. 

Ravallion,  M.  (2012b).  Troubling  tradeoffs  in  the  human  development  index.  Journal  of 
Development Economics, 99(2), 201-209. 

Saisana,  M.,  d’Hombres,  B.,  &  Saltelli,  A.  (2011).  Rickety  numbers:  Volatility  of university  rankings 
and policy implications. Research Policy, 40(1), 165-177. 

Saisana,  M.,  Saltelli,  A.,  & Tarantola,  S.  (2005).  Uncertainty  and  sensitivity  analysis  techniques  as 
tools  for  the  quality  assessment  of  composite indicators.  Journal  of  the  Royal  Statistical  Society: 
Series A (Statistics in Society), 168(2), 307-323. 

Saltelli, A. (2007). Composite indicators between analysis and advocacy. Social Indicators Research, 
81(1), 65-77. 

Sørensen, M. P., Bloch, C., & Young, M. (2015). Excellence in the knowledge-based economy: from 
scientific to research excellence. European Journal of Higher Education, (ahead-of-print), 1-21. 

Vértesy, D. & Tarantola, S. 2012. Composite indicators of research excellence. Ispra: Joint Research 
Centre of the European Commission. 

Waldrop, M. M. (2008). Science 2.0. Scientific American, 298(5), 68-73. 

19 

 
Appendix A: overview of country performance 

Group 
EU 

EFTA 

Country 
AT 
BE 
BG 
CY 
CZ 
DE 
DK 
EE 
EL 
ES 
FI 
FR 
HR 
HU 
IE 
IT 
LT 
LU 
LV 
MT 
NL 
PL 
PT 
RO 
SE 
SI 
SK 
UK 
EU28 
CH 
IS 
LI* 
NO 

Top 10% most highly cited 
publications per total pub’s. 
2007  2008  2009  2010 
15.6 
17.0 
6.0 
14.5 
9.0 

14.4 
16.1 
5.7 
12.2 
7.1 
13.5 
18.0 
11.0 
11.6 
11.5 
14.3 
12.3 
5.1 
8.5 
13.8 
12.4 
7.0 
10.8 
4.7 
12.6 
17.4 
5.5 
11.7 
6.1 
15.1 
10.1 
7.0 
14.8 
11.6 
18.4 
16.0 
14.7 
14.2 

14.0 
16.4 
5.5 
13.7 
8.3 
13.7 
17.7 
11.9 
11.5 
12.2 
14.3 
12.5 
5.0 
7.6 
14.6 
12.7 
7.8 
15.5 
6.5 
12.4 
17.8 
5.4 
12.1 
5.3 
15.5 
9.5 
6.0 
15.3 
11.7 
19.0 
16.7 
19.5 
15.0 

13.8 
16.3 
6.0 
11.3 
8.0 
13.8 
17.9 
11.9 
11.7 
12.5 
14.2 
12.7 
5.4 
8.5 
14.7 
12.9 
7.6 
16.0 
6.1 
8.1 
18.0 
5.9 
12.4 
4.8 
15.7 
9.8 
7.4 
15.5 
11.8 
19.4 
17.9 
11.9 
14.6 

Nr. PCT Patent applications 

2007 
1,326 
1,161 
29 
9 
219 
14.5  18,744 
1,353 
18.8 
46 
14.0 
118 
12.1 
1,539 
12.9 
1,600 
15.1 
6,819 
13.2 
61 
6.2 
248 
9.4 
438 
15.1 
3,360 
13.8 
18 
8.6 
38 
16.5 
24 
5.9 
10 
10.4 
3,578 
18.9 
165 
6.4 
114 
11.8 
37 
5.1 
3,165 
16.4 
120 
10.4 
48 
7.5 
16.0 
6,417 
12.2  50,805 
2,456 
19.6 
28 
20.1 
16 
21.5 
660 
14.7 

2008 
1,190 
1,089 
28 
10 
213 
17,053 
1,253 
46 
109 
1,629 
1,503 
6,903 
47 
222 
426 
3,215 
29 
55 
25 
11 
3,570 
196 
134 
42 
3,004 
141 
33 
6,029 
48,205 
2,257 
26 
21 
667 

2009 
1,281 
1,144 
28 
12 
179 
17,301 
1,128 
48 
110 
1,775 
1,502 
7,012 
44 
234 
379 
3,134 
14 
52 
31 
2 
3,247 
257 
139 
43 
2,849 
131 
36 
5,686 
47,803 
2,226 
35 
12 
749 

2010 
1,414 
1,254 
30 
6 
155 
18,492 
1,145 
50 
91 
1,905 
1,567 
7,224 
48 
244 
340 
3,147 
20 
54 
13 
7 
2,944 
279 
135 
43 
2,849 
135 
48 
5,760 
49,396 
2,360 
27 
26 
734 

20 

Value of ERC grants (Mln. EURs) 
2010 
2007 
14.4 
1.4 
14.8 
2.2 
0.5 
0.1 
0.9 
0.2 
1.4 
0.3 
78.1 
6.5 
9.2 
0.8 
0.1 
0.0 
4.4 
1.1 
30.7 
5.1 
9.9 
1.2 
70.7 
8.4 
0.0 
0.0 
4.3 
1.2 
4.1 
0.4 
32.8 
5.2 
0.0 
0.0 
0.0 
0.0 
0.0 
0.0 
0.0 
0.0 
41.9 
5.6 
2.1 
0.1 
3.9 
0.5 
0.1 
0.0 
25.7 
2.9 
0.1 
0.0 
0.0 
0.0 
115.0 
12.3 
465.7 
55.3 
45.2 
4.3 
0.4 
0.0 
0.0 
0.0 
6.4 
0.0 

2008 
4.7 
5.7 
0.4 
0.6 
0.8 
19.7 
3.0 
0.1 
2.3 
11.4 
5.5 
22.1 
0.0 
2.0 
1.2 
11.9 
0.0 
0.0 
0.0 
0.0 
14.7 
0.8 
1.4 
0.1 
9.5 
0.0 
0.0 
35.6 
153.5 
15.5 
0.4 
0.0 
1.4 

2009 
9.6 
9.9 
0.4 
0.6 
0.9 
43.2 
5.6 
0.1 
2.7 
19.9 
6.8 
45.8 
0.0 
2.6 
1.7 
19.8 
0.0 
0.0 
0.0 
0.0 
26.3 
1.7 
2.6 
0.1 
17.0 
0.0 
0.0 
69.4 
286.6 
30.8 
0.4 
0.0 
3.0 

Nr. of universities and research 
institutes in world top 500 

2007 
3 
6 
0 
0 
1 
46 
5 
0 
0 
15 
1 
38 
0 
0 
1 
10 
0 
0 
0 
0 
18 
0 
0 
0 
2 
0 
0 
45 
192 
16 
1 
0 
4 

2008 
3 
4 
0 
0 
0 
47 
6 
0 
0 
20 
1 
33 
0 
0 
1 
12 
0 
0 
0 
0 
17 
0 
2 
0 
2 
0 
0 
41 
190 
16 
1 
0 
4 

2009 
3 
4 
0 
0 
0 
49 
5 
0 
0 
23 
2 
33 
0 
0 
1 
14 
0 
0 
0 
0 
22 
0 
2 
0 
3 
0 
0 
38 
199 
17 
1 
0 
5 

2010 
2 
2 
0 
0 
0 
51 
6 
0 
0 
20 
1 
33 
0 
0 
0 
17 
0 
0 
0 
0 
23 
0 
0 
0 
3 
0 
0 
36 
194 
17 
1 
0 
4 

 
  
  
 
Group 
Other 
ERA 

Others 
BRICS & 
OECD 

Country 
AL* 
AM* 
BA* 
FO* 
IL 
MD* 
ME* 
MK* 
RS 
RU 
TR 
UA 
BR 
CN 
IN 
JP 
KR 
US 

Top 10% most highly cited 
publications per total pub’s. 
2007 
2007  2008  2009  2010 
0 
4.8 
7 
10.7 
12 
4.3 
0 
19.0 
2,119 
14.4 
7 
7.7 
0 
9.4 
3 
4.4 
0 
7.8 
848 
4.4 
380 
7.2 
128 
4.0 
531 
6.5 
5,980 
7.4 
6.6 
1,312 
7.9  29,001 
10.3 
7,257 
15.2  50,044 

4.1 
9.4 
4.4 
10.5 
13.6 
10.3 
6.0 
5.5 
7.6 
4.0 
7.7 
3.5 
6.5 
7.3 
6.9 
8.2 
10.1 
15.1 

8.2 
9.0 
4.2 
23.4 
14.0 
8.9 
15.1 
5.4 
6.9 
4.1 
7.6 
3.8 
6.4 
7.3 
6.8 
8.2 
9.7 
15.0 

5.4 
10.8 
4.1 
27.8 
13.9 
7.1 
1.2 
3.3 
5.9 
4.4 
7.4 
3.9 
6.4 
7.0 
7.0 
8.2 
9.1 
14.9 

Nr. PCT Patent applications 

2008 
0 
3 
8 
0 
1,793 
3 
0 
3 
0 
778 
388 
86 
552 
6,384 
1,359 
28,082 
7,188 
44,654 

2009 
0 
6 
9 
0 
1,690 
2 
0 
3 
0 
834 
462 
105 
547 
10,186 
1,536 
31,002 
8,728 
42,887 

2010 
0 
5 
6 
0 
1,701 
2 
0 
0 
0 
976 
560 
161 
600  n.a. 
13,558  n.a. 
1,878  n.a. 
37,126  n.a. 
9,548  n.a. 
45,232  n.a. 

Value of ERC grants (Mln. EURs) 
2010 
2007 
0.0 
0.0 
0.0 
0.0 
0.0 
0.0 
0.0 
0.0 
30.1 
5.2 
0.0 
0.0 
0.0 
0.0 
0.0 
0.0 
0.0 
0.0 
0.0 
0.0 
0.0 
0.0 
0.0 
0.0 

2008 
0.0 
0.0 
0.0 
0.0 
11.6 
0.0 
0.0 
0.0 
0.0 
0.0 
0.0 
0.0 

2009 
0.0 
0.0 
0.0 
0.0 
19.2 
0.0 
0.0 
0.0 
0.0 
0.0 
0.0 
0.0 

n.a. 
n.a. 
n.a. 
n.a. 
n.a. 
n.a. 

n.a. 
n.a. 
n.a. 
n.a. 
n.a. 
n.a. 

n.a. 
n.a. 
n.a. 
n.a. 
n.a. 
n.a. 

Nr. of universities and research 
institutes in world top 500 

2007 
0 
0 
0 
0 
1 
0 
0 
0 
0 
0 
1 
0 
0 
0 
3 
2 
0 
223 

2008 
0 
0 
0 
0 
1 
0 
0 
0 
0 
0 
0 
0 
0 
0 
3 
2 
0 
217 

2009 
0 
0 
0 
0 
1 
0 
0 
0 
1 
0 
0 
0 
0 
0 
1 
2 
0 
208 

2010 
0 
0 
0 
0 
1 
0 
0 
0 
1 
0 
0 
0 
0 
0 
3 
1 
0 
208 

Notes: Countries marked with asterisk (*) were excluded from the calculation of the composite indicators due to their volatile or outlier scores. 
Years refer to year of publication or patent filing, after which citation windows. Data was frozen in Jan 2015. 
Sources: Highly cited publications: science Metrix (Scopus); PCT patent applications: OECD PATSTAT database; ERC grants: Special tabulation from CORDA database (DG-RTD); Universities 
and research institutes: Scimago Institutes Ranking, various editions. 

21 

 
  
  
 
  
 
Appendix B: Descriptive statistics 

Table 5 Descriptive statistics and Correlation Table 

Descriptive statistics of raw data, 4 years pooled data 

% of highly cited 
publications per total 
publications  
[hicit_pub] 

PCT patent 
applications per 
million population 
[pct_pop] 

Value of ERC grants 
per Public R&D 
(GOVRD+HERD) 
(thousand PPPs) 
[erc_goverdherd] 

Nr. Top Universities 
and Research Inst’s 
within world top 500 
per GERD (billion 
PPPs) 
[topinst_gerd] 

Nr. Observations 
Min 
Max 
Mean 
Standard Deviation 

172 
3.5 
19.6 
11.0 
4.1 

172 
1.1 
347.3 
87.0 
96.3 

172 
0.0 
4.5 
0.4 
0.6 

Correlations between min-max normalized indicators, 4 years pooled data 

(1) hicit_pub 
(2) pct_pop 
(3) erc_goverdherd 
(4) topinst_gerd 

(1) 

1 
0.691 
0.376 
0.578 

(2) 

0.691 
1 
0.343 
0.302 

(3) 

0.376 
0.343 
1 
0.137 

136 
0.0 
10.9 
2.2 
2.8 

(4) 

0.578 
0.302 
0.137 
1 

22 

 
 
 
 
 
 
 
 
 
 
 
 
Europe Direct is a service to help you find answers to your questions about the European Union 
Freephone number (*): 00 800 6 7 8 9 10 11 

(*) Certain mobile telephone operators do not allow access to 00 800 numbers or these calls may be billed. 

A great deal of additional information on the European Union is available on the Internet. 
It can be accessed through the Europa server  http://europa.eu. 

How to obtain EU publications 

Our publications are available from EU Bookshop (http://bookshop.europa.eu), 
where you can place an order with the sales agent of your choice. 

The Publications Office has a worldwide network of sales agents. 
You can obtain their contact details by sending a fax to (352) 29 29-42758. 

European Commission 
EUR 27374 EN – Joint Research Centre 

Title: An update of the Research Excellence Indicator 

Author(s): Sjoerd Hardeman, Daniel Vertesy 

Luxembourg: Publications Office of the European Union 

2015 – 22+4 pp. – 21.0 x 29.7 cm 

EUR – Scientific and Technical Research series – ISSN 1831-9424 (online) 

ISBN 978-92-79-36869-1 (PDF) 

doi:10.2760/770103 

K
J
-
N
A
-
2
7
3
7
4
-
E
N
-
N

JRC Mission 

As the Commission’s  
in-house science service,  
the Joint Research Centre’s 
mission is to provide EU  
policies with independent,  
evidence-based scientific  
and technical support  
throughout the whole  
policy cycle. 

Working in close  
cooperation with policy  
Directorates-General,  
the JRC addresses key  
societal challenges while  
stimulating innovation  
through developing  
new methods, tools  
and standards, and sharing 
its know-how with  
the Member States,  
the scientific community  
and international partners. 

Serving society  
Stimulating innovation 
Supporting legislation 

doi:10.2760/770103 

ISBN 978-92-79-36869-1

24