The adjusted research excellence index 2018 : methodology report.
Date: 2018-12-19
Institutions: Joint Research Centre
Subjects: EU research policy, data collection, data processing, innovation, organisation of research, research and development, research report, scientific cooperation, scientific research

The Adjusted  
Research Excellence 
Index 2018 

Methodology Report 

Vertesy, D. 

2018 

EUR 29567 EN 

This publication is a Technical report by the Joint Research Centre (JRC), the European Commission’s science 
and knowledge service. It aims to provide evidence-based scientific support to the European policymaking 
process. The scientific output expressed does not imply a policy position of the European Commission. Neither 
the European Commission nor any person acting on behalf of the Commission is responsible for the use that 
might be made of this publication. 

Contact information 
Name: Daniel Vertesy 
Address: Via E. Fermi 2749 Ispra (VA) Italy, 21027 
Email: daniel.vertesy@ec.europa.eu  
Tel.: +39 0332 783556 

EU Science Hub 
https://ec.europa.eu/jrc 

JRC114345 

EUR 29567 EN 

PDF 

ISBN 978-92-79-98383-2 

ISSN 1831-9424 

doi:10.2760/126300 

Luxembourg: Publications Office of the European Union, 2018 

© European Union, 2018 

The reuse policy of the European Commission is implemented by Commission Decision 2011/833/EU of 12 
December 2011 on the reuse of Commission documents (OJ L 330, 14.12.2011, p. 39). Reuse is authorised, 
provided the source of the document is acknowledged and its original meaning or message is not distorted. The 
European Commission shall not be liable for any consequence stemming from the reuse. For any use or 
reproduction of photos or other material that is not owned by the EU, permission must be sought directly from 
the copyright holders. 

All content © European Union, 2018 

How to cite this report: Vertesy, D., The Adjusted Research Excellence Index 2018. Methodology Report, 
EUR 29567 EN, Publications Office of the European Union, Luxembourg, 2018, ISBN 978-92-79-98383-2, 
doi:10.2760/126300, JRC114345 

Contents 

Acknowledgements ................................................................................................ 1 

Abstract ............................................................................................................... 2 

1  Introduction ...................................................................................................... 3 

2  Methods and comparability with previous editions .................................................. 4 

3  Results ............................................................................................................. 6 

3.1  Data reliability ............................................................................................. 8 

3.2  Robustness ................................................................................................. 8 

4  Conclusions .................................................................................................... 11 

References ......................................................................................................... 12 

List of abbreviations and definitions ....................................................................... 13 

List of figures ...................................................................................................... 14 

List of tables ....................................................................................................... 15 

Appendix: Original and Normalized Indicator scores ................................................. 16 

i 

Acknowledgements 

The  authors  are  grateful  for  comments  received  from  colleagues  at  the  Competence 
Centre on Composite Indicators and Scorebaords at the JRC as well as from DG RTD. 

Author 

Daniel Vertesy 

1 

 
  
Abstract 

The  Adjusted  Research  Excellence  Index  is  a  composite  indicator  selected  by  the 
European  Research  Area  Council  (ERAC)  as  the  headline  measure  to  monitor  country 
performance  with  respect  to  ERA  Roadmap  Priority  1,  ‘Effective  national  research 
systems’  (European  Commission,  2017).  It  combines  four  dimensions  that  characterize 
effective  research  systems,  in  terms  of  scientific  and  technological  research  excellence 
(the  “production”  of  high-impact  publications  and  valued  patents),  and  the  ability  of 
institutes to attract outstanding research grants and participate in researcher exchanges 
to pave the way for future excellence and develop efficient research capacity. This report 
describes  the  methodology  used  to  compute  the  latest  scores  for  43  countries  (EU 
Member States and Horizon 2020 Associated Countries) and the EU28 for 4 time points: 
2016,  2013,  2011  and  2010.  The  results  show  that  Switzerland  excels  in  having  the 
most effective national research system, followed by Denmark, the Netherlands, Sweden 
and the United Kingdom. Most of the countries show a welcome growth of the composite 
score over the last period, however it is important to keep in mind that growth is, to a 
large  extent,  driven  by  the  overall  increase  in  the  value  of  ERC  grants.  Country  ranks 
were  found  to  be  rather  robust,  with  uncertainty  in  the  modelling  choices  having  only 
limited impact (3-4 rank positions shifts) on the ranks of the majority of the countries. 

2 

 
 
 
1  Introduction 

The  Adjusted  Research  Excellence  Index  is  a  composite  indicator  selected  by  the 
European  Research  Area  Council  (ERAC)  as  the  headline  measure  to  monitor  country 
performance  with  respect  to  ERA  Roadmap  Priority  1,  ‘Effective  national  research 
four  dimensions  that 
systems’  (European  Commission,  2017a,b).  It  combines 
characterize effective research systems, in terms of scientific and technological research 
excellence  (the  “production”  of  high-impact  publications  and  valued  patents),  and  the 
ability  of  institutes  to  attract  outstanding  research  grants  and  participate  in  researcher 
exchanges to pave the way for future excellence and develop efficient research capacity. 
The four dimensions are defined [and henceforth referred to] as:  

  The  share  of  top  10  %  most  highly  cited  publications  per  total  publications 

[HICIT] 

  PCT patent applications per population [PCT] 
  ERC grants per public R&D [ERC], and  
  Participation in Marie Skłodowska-Curie actions: the number of MSCA researchers 

hosted per national MSCA researcher) [MSCA]; 

This  brief  report  presents  the  latest  scores  for  43  countries  (EU  Member  States  and 
Horizon 2020 Associated Countries) and the EU28 total for the purposes of the upcoming 
ERA  Progress  Report.  In  order  to  monitor  progress  over  time,  scores  were  produced 
using the most recently available data (2016), as well as using historical data for earlier 
time points (2010 and 2013). It thus updates the scores and trends published in the ERA 
Progress Report 2016. 

The  focus  of  this  report  are  the  methodological  aspects  of  computing  the  Adjusted 
Research Excellence Index (AREI), which should be seen as complementary to the more 
comprehensive discussions of research performance at the level of countries and the EU, 
which  is  provided  in  the  ERA  Progress  Report.  The  definition  and  quantification  of  the 
concept  of  scientific  and  technological  research  excellence,  or  its  impact  –  which 
continues  to  be  at  the  focus  of  scholarly  attention  –  remains  outside  the  scope  of  this 
methodology report.1 

The AREI,  as any composite indicator, aims to  provide the “big picture” with respect to 
the  phenomenon  of  interest  in  the  form  of  a  summary  score.  Users  of  the  index  are 
invited  to  explore  not  merely  the  aggregate  scores  and  ranking  of  countries,  but  also 
scores  of  the  underlying  components  (as  two  countries  may  achieve  a  similar  score  by 
having  a  different  distribution  of  strengths  and  weaknesses)  as  well  as  the  measurable 
uncertainty  in  the  modelling  choice  that  may  influence  the  aggregate  ranking.  This 
report,  therefore,  goes  beyond  merely  presenting  the  aggregation  methods  and 
aggregate  scores;  the  robustness  section  (3.2)  also  presents  the  outcomes  of  the 
uncertainty  analysis,  which  should  help  users  a  richer  understanding  of  how  countries 
compare  with  one  another  in  terms  of  scientific  and  technological  research  excellence, 
and what are the trends over time.  

1  For  a  more  detailed  discussion  on  the  complexities  related  to  the  conceptualization  and  measurement  of 
scientific  and  technological  research  excellence,  see,  among  others,  Tijssen  (2003),  Hardeman  et  al 
(2013), Sorensen et al (2016), or Ferretti et al (2018). 

3 

 
 
 
                                           
2  Methods and comparability with previous editions 

The  methodology  to  compute  the  Adjusted  Research  Excellence  Index  is  presented  in 
Vértesy  (2015)  and  is  rooted  in  that  of  a  previous  version  of  the  Research  Excellence 
Index  developed  by  the  JRC.  As  some  refinements  were  necessary  due  to  data 
availability and changes in definition of the source variable, the steps taken to compute 
the indicator (following OECD-JRC 2008) are described below. 

Table 1 Collection of data from multiple sources and multiple time points 

Component  Numerator 
HICIT 

Nr. of publications 
among the top 10% 
most highly cited (field 
normalized) 
PCT patent applications 
by inventor country 
and priority year 

Total value of ERC 
grants received by host 
organization in a 
country; (multi-year 
programmes spread 
equally between start 
and end dates) 
Nr. of MSCA 
researchers hosted  

Denominator 
Total nr. of 
publications 

Sources  Sources Detailed 
CWTS 

CWTS, Leiden (and SciVal 
for AM, GE, TN) 

Total Population  OECD, 
World 
Bank 

Public R&D 
(GOVRD+HERD, 
constant Eurs 
PPS) 

DG RTD, 
Eurostat, 
OECD 

OECD_MSTI, OECD_PATS, 
OECD REGPAT microdata 
(AL, ME, RS), World Bank 
World Development 
Indicators 
Special tabulation from 
Horizon 2020 and FP7 
Corda data; 
rd_e_gerdfund (OECD 
data for IL) 

Nr. nationals 
participating in 
MSCA  

DG EAC  Marie Skłodowska-Curie 

Action statistics / country 
fiches, multiple editions2 

PCT 

ERC 

MSCA 

1.  Data  collection:  The  data  sources  are  described  in  Table  1.  In  case  data  was 
missing,  figures  available  for  the  same  country  from  a  neighboring  year  was 
used. Data was initially also collected for OECD and BRICS countries, however, it 
was  not  used  in  this  calculation  due  to  issues  of  comparability  (ERC  and  MSCA 
only meaningful for countries that are part of ERA or associated to Horizon 2020). 
As  a  result,  scores  could  be  computed  for  all  EU  Member  States  and  H2020 
Associated  countries,  with  the  exception  of  the  Feroer  Islands  and  Liechtenstein 
(due to restrictions of data availability and comparability).  

Considering the joint availability of data for the four components across time, four 
time  points  were  identified  for  which  scores  could  be  computed  to  meet  a  fair 
balance between principles of timeliness and alignment of indicators to the same 
year. The two main issues addressed were 1) the lag in the HICIT and PCT scores 
due to citation window and administrative issues of patenting (last available year: 
2015); and the 2) unavailability of comparable MSCA scores between country fact 
sheets released in 2014 and mid-2018. As a compromise, HICIT and PCT scores 
were  “shifted  ahead”  by  1  year  (so  that  actual  2015  scores  represent  the  2016 
“nominal” values, and so forth), and MSCA figures were shifted back by 1.5 years 
(thus  mid-2018  scores  represent  2016).  Considering  also  the  fact  that  ERC  and 
MSCA were introduced with FP7 and the frequency of MSCA data availability, the 
four  time  points  for  which  comparable  scores  could  be  computed  were  identified 
as:  2010,  2011,  2013  and  2016.  The  Appendix  provides  a  detailed  overview  of 
the  original  indicator  values,  as  well  as  the  actual  and  nominal  years  used. 
Descriptive statistics on the variables are available in Table 3. 

2 URL: https://ec.europa.eu/research/mariecurieactions/msca-numbers_en (retrieved: Jul 2018 and Apr 2016) 

4 

 
 
 
 
                                           
2.  Treatment  of  outliers:  outliers  were  detected  for  indicators  that  showed 
absolute  skewness  and  kurtosis  for  the  pooled  dataset  of  220  country-year 
observations  (including  OECD  and  BRICS  countries  discarded  subsequently) 
above the threshold values of 2.2 and 3.5, respectively – in the case of ERC and 
MSCA.  Outliers  were  treated  by  winsorization,  a  method  identified  as  the  one 
causing  the  least  intervention  to  actual  data  points.  As  a  result,  6  values  were 
adjusted for  ERC (affecting IL, CY, UK, CH at  various time points), and 4 values 
for MSCA (LU and CH). 

3.  Normalization of data: for each variable, the pooled dataset of all country-year 
combinations were normalized using the min-max method to obtain a comparable 
range  of  10-100  using  the  formula  below.  (The  minimum  was  set  to  10  rather 
than 0 in order to accommodate the geometric aggregation.) Data was pooled for 
multiple years in order to be able to measure trend over time. 

𝐼𝑞𝑐 =

𝑥𝑞𝑐 − min𝑐⁡(𝑥𝑞)
max𝑐(𝑥𝑞) − min𝑐⁡(𝑥𝑞)

∗ 90 + 10 

(1) 

Where xqc is the value of a component variable q for a country c. The normalized 
scores for each country and indicator are available in the Appendix. 

4.  Multivariate  statistical  analyses  were  performed  to  test  the  correlation 
between the components and the presence of a single latent dimension. A strong, 
positive  correlation  and  the  presence  of  a  single  latent  dimension  were  in  fact 
found,  similar  to  past  years’  results.  The  details  of  the  findings  of  the  principal 
component analysis (PCA) and correlation table are available in  the ‘Robustness’ 
section 3.2 below. 

5.  Aggregation of country scores: the 2018  Adjusted Research Excellence  Index 
scores were computed by taking the geometric mean (with equal weights) of the 
four  normalized  components.  This  method  does  not  allow  full  compensation  of 
relatively  weaker  performance  in  one  component  with  strong  performance  in 
another one, but encourages countries to address bottlenecks.  

6.  Finally,  sensitivity  and  robustness  analyses  were  carried  out  to  identify  the 
statistical  relationship  between  the  components  and  the  composite,  and  to  test 
the impact of modelling choices on country rankings.  

5 

 
 
 
 
 
3  Results 

The  resulting  composite  scores  are  reported  in  Figure  1  for  44  countries  (EU  Member 
States,  Horizon  2020  Associated  Countries  and  the  EU28)  for  three  time  points,  and  in 
Table  2  for  all  time  points.  Results  show  that  Switzerland  excels  in  having  the  most 
effective national research system,  followed by Denmark, the Netherlands, Sweden and 
the United Kingdom.  

Most  of  the  countries  show  a  welcome  growth  of  the  composite  score  over  the  last 
period,  however,  it  is  important  to  keep  in  mind  that  growth  is  largely  driven  by  the 
overall increase  in  the value  of  ERC  grants.  Three  of  the  countries,  Greece,  Bosnia  and 
Herzegovina and Montenegro show a slight decline in scores the last  period. (Note that 
the decline for Switzerland should be read with caution and not considered as a warning 
sign, as this may be affected by the outlier treatment.) 

Scores for the individual components are shown in the Appendix. 

6 

 
 
 
 
 
Table 2 Adjusted Research Excellence Scores, “2018” 

Group 
EU28 

Country 
EU-28 
Austria 
Belgium 
Bulgaria 
Cyprus 
Czech Republic 
Germany 
Denmark 
Estonia 
Greece 
Spain 
Finland 
France 
Hungary 
Croatia 
Ireland 
Italy 
Lithuania 
Luxembourg 
Latvia 
Malta 
Netherlands 
Poland 
Portugal 
Romania 
Sweden 
Slovenia 
Slovakia 
United Kingdom 

Associated  Albania 
Armenia 
Bosnia and Herzegovina 
FYRO Macedonia 
Georgia 
Iceland 
Israel 
Moldova 
Montenegro 
Norway 
Serbia 
Switzerland 
Tunisia 
Turkey 
Ukraine 

Code  2010  2011  2013  2016 
45.0 
36.5 
EU28 
54.9 
41.6 
AT 
55.8 
42.5 
BE 
16.6 
17.2 
BG 
37.1 
29.2 
CY 
23.2 
21.1 
CZ 
50.3 
42.8 
DE 
78.6 
53.9 
DK 
30.4 
25.4 
EE 
25.2 
23.9 
EL 
35.1 
29.0 
ES 
54.9 
44.9 
FI 
46.6 
38.0 
FR 
31.4 
26.0 
HU 
19.1 
14.3 
HR 
52.9 
35.9 
IE 
34.4 
28.6 
IT 
16.7 
17.3 
LT 
57.5 
38.7 
LU 
19.6 
15.5 
LV 
27.8 
16.8 
MT 
74.5 
52.3 
NL 
18.7 
15.6 
PL 
30.1 
23.7 
PT 
16.8 
14.1 
RO 
70.3 
55.4 
SE 
25.9 
25.7 
SI 
19.5 
15.9 
SK 
68.7 
57.0 
UK 
11.9 
12.1 
AL 
20.7 
15.2 
AM 
13.2 
11.1 
BA 
14.3 
11.9 
MK 
18.2 
15.6 
GE 
37.9 
36.5 
IS 
59.9 
57.5 
IL 
15.4 
12.7 
MD 
12.8 
12.3 
ME 
50.2 
47.6 
NO 
15.2 
14.6 
RS 
97.5 
86.9 
CH 
15.0 
13.5 
TN 
16.7 
16.9 
TR 
14.3 
12.4 
UA 

33.7 
39.3 
37.8 
16.7 
27.6 
20.8 
38.2 
49.5 
22.7 
23.4 
26.7 
41.9 
35.2 
24.0 
14.5 
32.0 
26.5 
16.2 
27.7 
16.0 
13.6 
50.6 
15.5 
22.3 
14.3 
52.0 
25.9 
16.1 
52.5 
10.0 
14.9 
11.7 
11.7 
13.5 
34.4 
50.8 
12.5 
13.3 
44.1 
14.0 
80.3 
13.7 
17.2 
12.5 

41.0 
43.5 
51.1 
16.7 
33.7 
21.1 
46.5 
64.0 
26.6 
26.6 
31.7 
50.5 
42.3 
28.3 
15.5 
43.1 
30.9 
14.7 
43.5 
17.8 
17.2 
65.1 
16.6 
25.3 
15.1 
61.1 
26.0 
16.5 
66.6 
10.3 
18.0 
13.9 
13.9 
17.9 
38.0 
59.5 
12.6 
13.7 
39.6 
14.0 
98.9 
13.5 
16.8 
12.0 

CAGR over  
past 3 years 

3% 
8% 
3% 
0% 
3% 
3% 
3% 
7% 
5% 
-2% 
4% 
3% 
3% 
4% 
7% 
7% 
4% 
4% 
10% 
3% 
17% 
5% 
4% 
6% 
4% 
5% 
0% 
6% 
1% 
5% 
5% 
-2% 
1% 
1% 
0% 
0% 
7% 
-2% 
8% 
3% 
0% 
4% 
0% 
6% 

Source: JRC Calculations. Note: CAGR = Compound annual average growth rate  

7 

 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
 
 
 
 
 
 
 
 
Figure 1 Adjusted Research Excellence index scores, 2018 edition 

Source: JRC calculations; Notes: The Research Excellence index is a composite of four components: share of 
top 10% most highly cited publications per total publications (data source: CWTS); PCT patent applications 
per population (OECD, World Bank); Participation in Marie Skłodowska-Curie Actions (DG-EAC); ERC grants 
per public R&D (DG-RTD, Eurostat, OECD). 

3.1  Data reliability 

We note that countries on the ERA periphery with a low number of publications and PCT 
patents  show  a  high  degree  of  fluctuation  for  the  HICIT  and  PCT  components.  We 
noticed that PCT data should be considered unreliable for some or all of the years for the 
following  countries:  AL,  AM,  BA,  CY,  EE,  GE,  LT,  LV,  MD,  ME,  MK,  MT,  TN.  It  is  also 
important  to  note  that  the  trend  of  AREI  score  growth  over  time  is  driven,  to  a  large 
extent,  by  the  expansion  of  the  ERC  program.  This  gives  reason  to  treat  growth  over 
time with caution and use as a benchmark the compound average growth (CAGR) figures 
of the EU28.  

3.2  Robustness 

A strong, positive correlation was found between the components (Pearson’s coeff. 0.52-
0.72).  Principal  Component  Analysis  (PCA)  confirms  the  presence  of  one  single  latent 
dimension  that  explains  72%  of  variance  in  the  data  (4  components,  44  countries,  4 
time points). The PCA  shows a relatively balanced loading for all components,  although 
the  ERC  component  a  slightly  weaker.  (This  is  also  apparent  from  the  pairwise 
correlation  across  the  variables,  presented  in  Table  3.)  We  do  note  at  the  same  time 
that over time, the ERC component is showing the greatest increase, driving most of the 
overall observed growth of the composite scores.  

8 

0102030405060708090100Adjusted Research Excellence Index CH DK NL SE UK IL LU BE FI AT IE DE NO FR EU28 IS CY ES IT HU EE PT MT SI EL CZ AM LV SK HR PL GE RO TR LT BG MD RS TN MK UA BA ME AL'2018' (2016), EU Member States'2018' (2016), Associated Countries'2015' (2013)'2012' (2010) 
 
 
 
 
Table 3 Descriptive statistics and Correlations for the components 

Descriptive Statistics 
(min-max normalized dataset) 

Mean 

S.D. 

Min 

Max 

  Correlation table (N=176) 
PCT 
  HICIT 

MSCA 

ERC 

HICIT 
PCT 
MSCA 
ERC 
Source: JRC calculations 

0.08 
72.2 
1.16 
5,375.3 

0.04 
92.3 
1.42 
7,458.2 

0.0 
0.0 
0.0 
0.0 

0.16 
329.8 
6.30 
28,290.0 

1 
0.71 
0.72 
0.58 

1 
0.66 
0.58 

1 
0.52 

1 

Similar conclusions can be drawn from the sensitivity analysis that was carried out using 
two methods to test the extent to which a component explains variation in the aggregate 
scores,  which  is  a  measure  of  the  importance  of  a  component.3  The  sensitivity  indices 
obtained  from  the  linear  method  (Pearson  correlation  squared)  and  a  non-linear  spline 
fitting  method  (Table  4)  shows  that  composite  scores  are  driven  by  each  of  the 
component  in  a  fairly  balanced  manner;  with  the  strongest  indices  shown  by  the  PCT 
component  and  the  weakest  by  the  ERC  component.  It  is  important  to  keep  in  mind, 
nevertheless, that while the ERC is relatively the weakest contribution to the variance in 
the  results,  it  does  have  an  important  contribution  to  change  over  time,  given  the  fact 
that  ERC  scores  show  a  greater  increasing  trend  over  time  than  any  of  the  other 
components. 

Table 4 Sensitivity indices 

Components 
HICIT 
PCT 
MSCA 
ERC 

Si 
Linear method 
0.714 
0.776 
0.723 
0.630 
Note:  JRC  calculation;  components  were  min-max  method,  MSCA  and  ERC:  outliers  winsorized.  Sensitivity 
indices  (Si)  range  between  0  and  1  and  show  the  extent  to  which  a  component  drives  the  variation  in 
composite scores. 

Si 
Polynomial method 
0.738 
0.811 
0.801 
0.704 

In  the  uncertainty  analysis,  we  tested  the  impact  of  uncertainty  in  the  composite 
indicator  development  methodology  on  the  ranking  of  countries.  We  considered 
uncertainty in three of the modelling choices, as described below:  

 

 

 

the  weighting  of  the  four  components:  AREI  components  are  equally  weighted; 
our tests considered a perturbation of these weights by +/- 25%; 
the  average  applied:  AREI  is  a  geometric  average  of  components,  but  many 
composite indicators apply the fully compensatory arithmetic average; , 
finally,  the  treatment  of  outliers  in  the  ERC  component:  as  an  alternative  to  the 
established  method  of  treating  outliers  by  winsorization,  one  may  consider  log-
normalizing  this  component  given  the  increasing  trend  over  time  in  this 
component,  which  is  suppressed  by  the  winsorization  for  the  best  performing 
country.  

3 For details, see Paruolo et al, 2013 and Becker et al, 2017. 

9 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
                                           
Figure 2 Robustness of Research Excellence index ranks 

Source: JRC calculations; Notes: Ranks for 2016 obtained from 1,000 Monte-Carlo simulation runs 
addressing uncertainty in weights (+/-25% vs. equal), average used to aggregate scores (arithmetic vs. 
geometric), treatment of outliers in ERC (log-normal vs. winsorization).  

We  considered  these  uncertainties  simultaneously,  following  i.e.  Saisana  et  al  (2011), 
and  computed  1,000  simulated  country  rankings  for  the  latest  time  point.  The 
simulations  help  quantify  the  uncertainty  by  effectively  adding  confidence  intervals  to 
the baseline country rankings, as shown Figure 2. The uncertainty analysis reveals that 
the  baseline  rank  corresponds  to  the  median  obtained  from  1,000  Monte-Carlo 
simulation  runs  in  16  of  the  44  countries  ranked,  and  the  difference  is  less  than  3 
positions in all country cases but Georgia and Tunisia. In 93% of the cases (exc. MD, TN 
and MK), the baseline rank falls within the 90% confidence interval. Users of any index 
are  discouraged  to  take  ranks  at  face  value  given  the  modelling  uncertainties,  it  is 
nevertheless  important  to  note  that  the  90%  interval  range  of  the  simulated  ranks  for 
about  19  countries  is  not  more  than  3  positions,  and  for  another  18  countries  it  is  not 
more  than  4  positions.  Ranks  are  relatively  most  volatile  for  Armenia,  Malta,  Georgia, 
Cyprus, Bulgaria and Serbia.  

10 

051015202530354045Country ranksCHDKNLSEUKILLUBEFIATIEDENOFREU28ISCYESITHUEEPTMTSIELCZAMLVSKHRPLGEROTRLTBGMDRSTNMKUABAMEAL90% interval of simulated ranksmedian of simulated ranksbaseline rankRanks for 2016 obtained from 1000 Monte-Carlo simulationsRobustness of ranks 
 
 
 
 
 
4  Conclusions 

This report presented the adjusted research excellence index scores, which updates the 
scores  published  in  the  2016  ERA  Progress  Report.  The  existing  methodology  remains 
applicable for the 2018 update, no further methodological adjustments were necessary.  

The  multivariate  analysis  showed  that  the  indicator  continues  to  reflect  a  single  latent 
dimension,  sensitivity  analysis  shows  a  fairly  balanced  contribution  of  the  four 
component  to  the  variation  of  the  final  scores.  For  analysing  score  changes  over  time, 
users are reminded that most of the score increases are due to the growing trend of the 
ERC component.  

Results  show  that  Switzerland  continues  to  excel  in  having  the  most  effective  national 
research  system,  and  that  Europe  shows  a  Northwest  –  Southeast  divide,  with  also 
strong  performance  shown  by  Denmark,  the  Netherlands,  Sweden  and  the  United 
Kingdom.  Country  ranks  were  found  to  be  rather  robust,  with  uncertainty  in  the 
modelling choices having only limited impact (3-4 rank positions shifts) on the ranks of 
the majority of the  countries. Nevertheless, users  of the aggregate  ranks are reminded 
that when a country rank is taken at face value, the inherent uncertainty in  how it was 
obtained is not considered. The confidence intervals obtained in the uncertainty analysis 
after  running  1,000  Monte-Carlo  simulations  serve  the  purpose  of  showing  the  possible 
range of research excellence ranks a country can obtain should some of the assumptions 
change. 

11 

 
 
 
References 

Becker,  W.,  Saisana,  M.,  Paruolo,  P.,  &  Vandecasteele,  I.  (2017).  Weights  and 
importance in composite indicators: Closing the gap. Ecological Indicators, 80(2017), 12-
22. [doi:10.1016/j.ecolind.2017.03.056] 

European Commission (2017a) ERA Progress Report 2016.  Report from the Commission 
to the Council and the Parliament, COM(2017)35  

European  Commission  (2017b)  ERA  Progress  Report  2016.  Technical  Report.  Data 
gathering and information for the 2016 ERA monitoring. Luxembourg: Publications Office 
of the European Union. [doi: 10.2777/352654] 

Ferretti,  F.,  Pereira,  Â.  G.,  Vértesy,  D.,  Hardeman,  S.  (2018)  Research  excellence 
indicators: time to reimagine the ‘making of’? Science and Public Policy, 45(5), 731-741 
[doi: 10.1093/scipol/scy007] 

Hardeman, S., Van Roy V., Vertesy, D., and Saisana M. (2013): An analysis of national 
research  systems  (I):  A  Composite  Indicator  for  Scientific  and  Technological  Research 
[doi: 
Excellence.  Luxembourg:  Publications  Office  of 
10.2788/95887] 

the  European  Union. 

OECD-JRC  (2008)  Handbook  on  constructing  composite  indicators:  methodology  and 
user guide. Paris: OECD publishing [doi: 10.1787/533411815016.] 

Paruolo  P.,  Saisana  M.,  Saltelli  A.,  2013.  Ratings  and  Rankings:  Voodoo  or  Science?, 
Journal  Royal  Statistical  Society,  A176(3), 
[doi:10.1111/j.1467-
985X.2012.01059.x] 

609-634 

Saisana,  M.,  d’Hombres,  B.,  &  Saltelli,  A.  (2011).  Rickety  numbers:  Volatility  of 
university  rankings  and  policy  implications.  Research  policy,  40(1),  165-177.  [doi: 
10.1016/j.respol.2010.09.003] 

Sorensen,  M.,  Bloch,  C.  and  Young,  M.  (2016).  Excellence  in  the  knowledge-based 
economy:  from  scientific  to  research  excellence.  European  Journal  of  Higher  Education, 
6(3), 217-236. [doi: 10.1080/21568235.2015.1015106] 

Tijssen, R. (2003). Scoreboards of research excellence. Research Evaluation, 12(2), 91-
103. [doi: 10.3152/147154403781776690] 

Vértesy, D. (2015).  Technical note on the  robustness of a  modified research  excellence 
index:  A  Feasibility  Study  for  the  European  Research  Area  Committee  (Technical  report 
No.  EUR  27669  EN).  Luxembourg:  Publications  Office  of  the  European  Union  [doi: 
10.2760/81642] 

12 

 
List of abbreviations and definitions 

AREI 

Adjusted Research Excellence Index 

CAGR 

compound average growth rate 

ERA 

European Research Area 

ERAC  European Research Area Council  

ERC 

European  Research  Council;  also  the  abbreviation  used  for  a  component  of  the 
Adjusted  Research  Excellence  Index  where  it  refers  to  ERC  grants  per  public 
R&D 

HICIT  A component of the Adjusted Research Excellence Index, referring to the share 

of top 10 % most highly cited publications per total publications 

JRC 

Joint Research Centre 

MSCA  Marie Skłodowska-Curie Actions; also the abbreviation used for a component of 
the Adjusted Research Excellence Index where it refers to the number of MSCA 
researchers hosted per national MSCA researcher 

PCA 

Principal component analysis  

PCT 

Patent  Cooperation  Treaty;  also  the  abbreviation  used  for  a  component  of  the 
Adjusted  Research  Excellence  Index,  where  it  refers  to  PCT  patent  applications 
per population 

(For country abbreviations, please refer to Table 2.) 

13 

 
 
List of figures 

Figure 1 Adjusted Research Excellence index scores, 2018 edition ............................. 8 

Figure 2 Robustness of Research Excellence index ranks ......................................... 10 

14 

 
 
List of tables 

Table 1 Collection of data from multiple sources and multiple time points ................... 4 

Table 2 Adjusted Research Excellence Scores, “2018” ............................................... 7 

Table 3 Descriptive statistics and Correlations for the components ............................. 9 

Table 4 Sensitivity indices ..................................................................................... 9 

15 

 
 
 
Appendix: Original and Normalized Indicator scores 

 Year 

HICIT 

PCT 

MSCA 

ERC 

Nominal  2010 
Actual   2009 

2011 
2010 

2013 
2012 

2016 
2015 

2010 
2009 

2011 
2010 

2013 
2012 

2016  2010  2011  2013  2016 
2017 
2012 
2015 

2011 

2014 

H2020  
Group 
EU28 
EU28 
AT 
EU28 
BE 
EU28 
BG 
EU28 
CY 
EU28 
CZ 
EU28 
DE 
EU28 
DK 
EU28 
EE 
EU28 
EL 
EU28 
ES 
EU28 
FI 
EU28 
FR 
EU28 
HU 
EU28 
HR 
EU28 
IE 
EU28 
IT 
EU28 
LT 
EU28 
LU 
EU28 
LV 
EU28 
MT 
EU28 
NL 
EU28 
PL 
EU28 
PT 
EU28 
RO 
EU28 
SE 
EU28 
SI 
EU28 
SK 
EU28 
EU28 
UK 
Associated  AL 
Associated  AM 
Associated  BA 
Associated  MK 
Associated  GE 
Associated  IS 
Associated  IL 
Associated  MD 
Associated  ME 
Associated  NO 
Associated  RS 
Associated  CH 
Associated  TN 
Associated  TR 
Associated  UA 

9.6% 
6.0% 

3.5% 
5.9% 

7.3% 
3.6% 

7.3% 
3.8% 

3.4% 
4.3% 

7.2% 
3.9% 

7.5% 
8.5% 
9.2% 

6.4% 
8.7% 
9.2% 

7.4% 
9.1% 
9.3% 

4.2% 
8.3% 
6.5% 

3.6% 
7.7% 
6.6% 

4.5% 
7.0% 
6.2% 

10.3%  10.5%  10.5%  10.6% 
10.5%  11.4%  11.0%  11.1% 
12.6%  12.5%  12.9%  12.6% 
4.2% 
9.0% 
6.6% 
11.1%  11.5%  11.3%  11.3% 
13.8%  14.1%  13.7%  13.4% 
8.2% 
9.0% 
9.3% 
10.2%  10.6%  10.4%  10.8% 
10.7%  10.8%  11.2%  11.0% 
6.9% 
4.6% 
11.0%  11.7%  12.5%  12.6% 
9.8%  10.1%  10.4% 
4.3% 
3.3% 
7.1% 
10.5%  13.0%  10.8%  13.1% 
6.2% 
4.1% 
5.2%  10.7% 
14.2%  14.8%  14.8%  14.6% 
5.1% 
9.0% 
4.8% 
11.7%  11.8%  11.6%  12.1% 
8.6% 
6.2% 
13.8%  14.1%  14.2%  15.0% 
2.1% 
1.2% 
2.8% 
1.1% 
7.7%  15.4%  11.6% 
7.0% 
3.3% 
4.1% 
1.8% 
2.3% 
2.2% 
6.1% 
4.9% 
1.8% 
4.7% 
8.5%  15.0%  11.3% 
8.9%  13.6%  10.6%  10.8% 
9.5%  10.0%  10.3%  10.0% 
4.7% 
3.3% 
3.2% 
3.8% 
11.1%  10.9%  11.1%  10.7% 
4.1% 
15.3%  15.1%  15.5%  15.3% 
5.8% 
4.7% 
3.6% 

5.1% 
5.6% 
3.2% 

4.6% 
5.1% 
2.9% 

4.7% 
4.8% 
2.5% 

4.2% 
8.8% 
4.1% 

4.2% 
9.3% 
4.3% 

4.4% 
9.4% 
5.2% 

3.6% 
2.6% 

3.4% 
4.9% 

7.5% 
5.7% 

8.0% 
5.2% 

8.7% 
5.1% 

4.6% 

5.4% 

4.4% 

95.0 
153.7 
106.0 
3.7 
11.0 
17.1 
211.2 
204.2 
37.1 
9.8 
38.3 
281.0 
108.3 
23.4 
9.9 
83.8 
53.1 
4.5 
109.9 
14.7 
5.4 
196.4 
6.7 
13.2 
2.1 
306.2 
64.1 
6.7 
91.2 
0.3 
2.1 
2.3 
1.4 
1.3 
111.8 
225.8 
0.5 
6.5 
155.0 
3.2 
287.8 
0.9 
6.5 
2.3 

97.9 
168.8 
114.9 
4.0 
5.5 
14.6 
226.2 
206.2 
37.3 
8.2 
40.9 
292.6 
110.9 
24.4 
10.8 
74.6 
53.1 
6.4 
111.0 
6.4 
15.7 
177.1 
7.3 
12.8 
2.1 
303.6 
65.7 
9.1 
91.7 
0.3 
1.7 
1.4 
0.0 
2.0 
85.4 
222.5 
0.5 
6.5 
150.6 
2.8 
301.6 
1.5 
7.7 
3.5 

100.1 
166.9 
107.9 
7.8 
7.3 
20.5 
223.0 
206.1 
13.4 
10.5 
37.4 
305.4 
117.8 
24.0 
10.7 
81.2 
55.9 
15.3 
130.2 
16.1 
18.3 
209.0 
9.7 
13.8 
3.2 
329.8 
61.3 
8.4 
90.5 
0.4 
2.7 
2.2 
0.2 
2.8 
103.5 
254.6 
1.0 
1.6 
142.0 
3.9 
328.4 
0.6 
8.8 
3.9 

102.8 
177.2 
108.6 
8.3 
13.6 
23.7 
221.1 
222.6 
20.3 
10.0 
38.3 
236.3 
122.0 
26.8 
9.9 
93.2 
59.9 
16.7 
135.6 
15.2 
32.0 
218.5 
14.0 
21.2 
3.8 
329.1 
40.0 
11.2 
96.5 
0.1 
2.5 
1.7 
0.8 
0.7 
117.4 
267.1 
1.4 
5.1 
125.5 
4.9 
310.9 
1.2 
13.7 
3.3 

16 

1.3 
1.5 
1.6 
0.3 
0.9 
1.0 
1.3 
3.1 
1.5 
0.8 
0.9 
1.1 
1.4 
0.6 
0.2 
1.3 
0.5 
0.4 
0.9 
0.7 
0.0 
2.3 
0.3 
0.7 
0.2 
2.1 
1.3 
0.3 
5.4 
0.0 
0.0 
0.0 
0.2 
0.0 
1.3 
1.0 
0.0 
0.0 
3.4 
0.1 
6.8 
0.0 
0.6 
0.0 

1.3 
1.1 
1.9 
0.3 
0.8 
1.0 
1.5 
3.2 
1.2 
0.7 
1.0 
1.1 
1.4 
0.6 
0.1 
1.2 
0.6 
0.4 
4.1 
0.7 
0.3 
1.8 
0.3 
0.9 
0.2 
2.0 
1.0 
0.3 
4.8 
0.0 
0.0 
0.0 
0.1 
0.0 
1.3 
1.1 
0.0 
0.0 
3.5 
0.1 
7.2 
0.0 
0.6 
0.0 

1.2 
0.8 
2.3 
0.2 
0.8 
0.8 
1.5 
3.7 
1.3 
0.7 
0.9 
1.1 
1.2 
0.5 
0.2 
1.2 
0.5 
0.3 
4.9 
0.4 
0.5 
2.1 
0.3 
0.6 
0.2 
2.0 
0.6 
0.3 
4.8 
0.0 
0.0 
0.2 
0.1 
0.0 
1.5 
1.0 
0.0 
0.0 
0.8 
0.1 
7.6 
0.0 
0.6 
0.0 

1.2 
2.0 
2.2 
0.4 
1.0 
0.7 
1.4 
5.5 
1.1 
0.5 
1.0 
1.5 
1.5 
0.5 
0.4 
1.9 
0.5 
0.5 
7.2 
0.3 
2.6 
2.7 
0.5 
0.7 
0.4 
3.3 
0.7 
0.9 
3.9 
0.2 
0.9 
0.1 
0.0 
0.3 
1.4 
1.0 
0.5 
0.0 
3.2 
0.2 
6.3 
0.2 
0.3 
0.3 

2010 
2010 
3,040.5 
3,639.0 
3,078.6 
1,955.7 
9,950.4 
698.3 
1,868.1 
2,955.7 
0.0 
3,022.5 
2,725.6 
3,626.6 
2,943.8 
4,047.9 
0.0 
1,911.1 
2,647.4 
0.0 
0.0 
0.0 
0.0 
5,341.7 
549.9 
1,512.4 
0.0 
5,165.9 
511.5 
0.0 
5,993.5 
0.0 
0.0 
0.0 
0.0 
0.0 
3,415.6 
16,994.0 
0.0 
0.0 
2,319.2 
0.0 
12,335.9 
0.0 
39.7 
0.0 

2011 
2011 
5,018.7 
5,766.4 
5,180.1 
3,031.1 
13,747.0 
944.6 
3,361.1 
5,076.2 
1,805.6 
4,481.7 
4,278.7 
5,262.2 
4,753.4 
6,424.4 
0.0 
5,534.0 
4,397.4 
0.0 
0.0 
0.0 
0.0 
8,943.3 
627.7 
2,556.9 
0.0 
8,144.0 
824.1 
0.0 
10,449.2 
0.0 
0.0 
0.0 
0.0 
0.0 
3,648.1 
26,926.4 
0.0 
0.0 
4,520.5 
0.0 
17,601.7 
0.0 
36.1 
0.0 

2013 
2013 
9,728.6 
10,049.5 
12,310.4 
3,026.4 
32,612.4 
975.3 
6,344.9 
11,503.2 
5,191.1 
7,377.9 
8,123.2 
9,471.7 
8,995.9 
11,333.4 
1,296.3 
12,463.0 
6,808.2 
0.0 
1,347.4 
1,772.4 
0.0 
19,416.1 
1,247.7 
4,880.9 
0.0 
12,652.3 
1,860.4 
516.2 
22,116.4 
0.0 
0.0 
0.0 
0.0 
0.0 
4,879.1 
49,229.4 
0.0 
0.0 
7,251.1 
0.0 
28,100.0 
0.0 
155.9 
0.0 

2016 
2016 
15,054.1 
14,852.5 
20,019.8 
935.7 
41,690.3 
3,246.5 
9,988.7 
19,915.0 
9,561.8 
6,489.0 
13,299.9 
14,785.2 
12,794.2 
20,114.3 
3,287.4 
20,849.4 
11,225.7 
0.0 
5,746.9 
2,171.0 
0.0 
28,290.0 
1,704.4 
11,134.7 
1,276.0 
14,536.4 
2,959.4 
532.3 
34,606.6 
0.0 
0.0 
0.0 
0.0 
0.0 
4,390.7 
70,615.5 
0.0 
0.0 
8,861.4 
782.6 
35,370.1 
0.0 
677.3 
0.0 

 
  
Appendix (cont’d): Normalized Scores 

HICIT 

PCT 

MSCA 

ERC 

2010 
2009 
67.2 
68.9 
81.4 
31.6 
46.9 
42.1 
72.3 
89.0 
43.4 
57.4 
60.7 
67.0 
70.1 
48.3 
27.3 
71.5 
63.1 
40.4 
68.5 
24.2 
29.9 
91.7 
29.4 
61.3 
29.9 
75.8 
49.8 
38.6 
89.3 
10.0 
46.6 
17.6 
14.4 
32.3 
58.4 
62.5 
23.9 
27.0 
72.5 
32.2 
98.5 
34.8 
38.3 
23.3 

EU28 
AT 
BE 
BG 
CY 
CZ 
DE 
DK 
EE 
EL 
ES 
FI 
FR 
HU 
HR 
IE 
IT 
LT 
LU 
LV 
MT 
NL 
PL 
PT 
RO 
SE 
SI 
SK 
UK 
AL 
AM 
BA 
MK 
GE 
IS 
IL 
MD 
ME 
NO 
RS 
CH 
TN 
TR 
UA 

2011 
2010 
68.4 
74.5 
81.3 
29.4 
54.7 
44.0 
74.8 
91.3 
49.7 
56.1 
60.8 
69.2 
70.6 
48.5 
26.9 
76.3 
64.4 
47.5 
84.1 
25.3 
40.0 
95.5 
29.5 
58.2 
28.9 
76.6 
53.2 
35.4 
91.0 
21.1 
50.9 
14.6 
16.7 
56.2 
87.7 
65.7 
25.6 
19.6 
70.9 
36.9 
97.1 
31.7 
35.0 
21.4 

2013 
2012 
68.6 
71.5 
83.7 
25.9 
51.2 
44.5 
73.8 
88.8 
49.5 
59.9 
61.1 
68.0 
72.8 
49.0 
26.0 
81.3 
66.1 
23.9 
70.7 
28.7 
35.8 
95.6 
30.5 
62.0 
36.0 
75.7 
57.7 
35.3 
91.8 
11.0 
98.9 
28.7 
33.9 
96.6 
69.0 
67.1 
24.2 
33.9 
72.3 
30.6 
100.0 
32.6 
33.3 
18.7 

2016 
2015 
69.0 
72.6 
81.5 
29.4 
59.2 
44.5 
73.8 
86.4 
54.6 
59.5 
61.1 
70.7 
71.7 
46.3 
32.2 
81.4 
68.2 
30.1 
84.5 
41.9 
69.8 
94.0 
34.8 
59.5 
33.2 
78.5 
56.5 
41.8 
96.5 
16.3 
75.1 
23.9 
41.2 
73.5 
70.5 
65.7 
32.4 
23.4 
70.0 
29.0 
98.6 
39.3 
32.6 
25.8 

2010 
2009 
34.9 
50.3 
37.8 
11.0 
12.9 
14.5 
65.4 
63.6 
19.7 
12.6 
20.0 
83.7 
38.4 
16.1 
12.6 
32.0 
23.9 
11.2 
38.8 
13.8 
11.4 
61.5 
11.8 
13.5 
10.6 
90.3 
26.8 
11.8 
33.9 
10.1 
10.6 
10.6 
10.4 
10.3 
39.3 
69.2 
10.1 
11.7 
50.6 
10.8 
85.5 
10.2 
11.7 
10.6 

2011 
2010 
68.4 
74.5 
81.3 
29.4 
54.7 
44.0 
74.8 
91.3 
49.7 
56.1 
60.8 
69.2 
70.6 
48.5 
26.9 
76.3 
64.4 
47.5 
84.1 
25.3 
40.0 
95.5 
29.5 
58.2 
28.9 
76.6 
53.2 
35.4 
91.0 
21.1 
50.9 
14.6 
16.7 
56.2 
87.7 
65.7 
25.6 
19.6 
70.9 
36.9 
97.1 
31.7 
35.0 
21.4 

2013 
2012 
36.3 
53.8 
38.3 
12.1 
11.9 
15.4 
68.5 
64.1 
13.5 
12.8 
19.8 
90.1 
40.9 
16.3 
12.8 
31.3 
24.7 
14.0 
44.2 
14.2 
14.8 
64.8 
12.5 
13.6 
10.8 
96.5 
26.1 
12.2 
33.7 
10.1 
10.7 
10.6 
10.1 
10.7 
37.2 
76.8 
10.3 
10.4 
47.3 
11.0 
96.1 
10.2 
12.3 
11.0 

2016 
2015 
37.0 
56.5 
38.5 
12.2 
13.6 
16.2 
68.0 
68.4 
15.3 
12.6 
20.0 
72.0 
42.0 
17.0 
12.6 
34.5 
25.7 
14.4 
45.6 
14.0 
18.4 
67.3 
13.7 
15.6 
11.0 
96.3 
20.5 
12.9 
35.3 
10.0 
10.7 
10.4 
10.2 
10.2 
40.8 
80.0 
10.4 
11.3 
42.9 
11.3 
91.5 
10.3 
13.6 
10.9 

17 

2010 
2011 
28.1 
31.8 
33.3 
13.9 
23.0 
25.0 
28.2 
54.6 
31.0 
21.3 
22.3 
25.6 
29.4 
18.5 
12.9 
28.6 
17.8 
15.1 
22.2 
19.5 
10.0 
43.0 
14.3 
20.4 
13.1 
40.5 
29.0 
14.7 
86.6 
10.0 
10.0 
10.0 
12.4 
10.0 
29.0 
24.1 
10.0 
10.0 
59.1 
11.1 
100.0 
10.0 
19.1 
10.0 

2011 
2012 
35.7 
54.3 
40.1 
11.0 
11.4 
13.8 
69.3 
64.1 
19.8 
12.1 
20.7 
86.7 
39.1 
16.4 
12.8 
29.6 
23.9 
11.7 
39.1 
11.7 
14.1 
56.5 
11.9 
13.4 
10.5 
89.6 
27.2 
12.4 
34.1 
10.1 
10.4 
10.4 
10.0 
10.5 
32.4 
68.3 
10.1 
11.7 
49.5 
10.7 
89.1 
10.4 
12.0 
10.9 

2013 
2014 
27.6 
22.1 
43.3 
12.6 
21.2 
22.0 
30.7 
63.3 
28.1 
19.7 
23.1 
26.4 
27.8 
17.4 
12.3 
27.4 
17.8 
13.8 
80.3 
15.6 
16.6 
40.5 
14.1 
19.0 
13.4 
38.1 
19.1 
14.8 
79.1 
10.0 
10.0 
12.4 
10.8 
10.0 
31.8 
24.3 
10.0 
10.0 
21.7 
11.4 
100.0 
10.0 
18.4 
10.0 

2016 
2017 
27.8 
38.8 
42.1 
16.4 
23.6 
19.6 
30.6 
88.0 
25.3 
17.5 
23.7 
31.4 
30.9 
16.8 
15.9 
36.5 
17.4 
17.8 
100.0 
14.9 
46.5 
48.6 
16.8 
19.6 
15.4 
57.4 
20.0 
22.7 
65.4 
12.5 
22.9 
12.0 
10.0 
14.8 
30.0 
24.4 
16.6 
10.0 
55.3 
12.9 
100.0 
12.5 
14.4 
14.7 

2010 
2010 
19.7 
21.6 
19.8 
16.2 
41.7 
12.2 
15.9 
19.4 
10.0 
19.6 
18.7 
21.5 
19.4 
22.9 
10.0 
16.1 
18.4 
10.0 
10.0 
10.0 
10.0 
27.0 
11.7 
14.8 
10.0 
26.4 
11.6 
10.0 
29.1 
10.0 
10.0 
10.0 
10.0 
10.0 
20.9 
64.1 
10.0 
10.0 
17.4 
10.0 
49.2 
10.0 
10.1 
10.0 

2011 
2011 
28.1 
26.2 
37.6 
13.7 
21.6 
25.0 
31.1 
55.1 
26.9 
19.7 
23.7 
25.4 
30.0 
18.9 
12.0 
26.5 
18.2 
16.1 
68.3 
19.5 
14.1 
36.0 
14.2 
22.3 
13.1 
38.1 
24.0 
14.6 
78.6 
10.0 
10.0 
10.0 
11.8 
10.0 
29.0 
25.4 
10.0 
10.0 
60.0 
11.6 
100.0 
10.0 
19.3 
10.0 

2013 
2013 
41.0 
42.0 
49.2 
19.6 
100.0 
13.1 
30.2 
46.6 
26.5 
33.5 
35.8 
40.1 
38.6 
46.1 
14.1 
49.6 
31.7 
10.0 
14.3 
15.6 
10.0 
71.8 
14.0 
25.5 
10.0 
50.3 
15.9 
11.6 
80.4 
10.0 
10.0 
10.0 
10.0 
10.0 
25.5 
100.0 
10.0 
10.0 
33.1 
10.0 
99.4 
10.0 
10.5 
10.0 

2016 
2016 
57.9 
57.3 
73.7 
13.0 
100.0 
20.3 
41.8 
73.4 
40.4 
30.6 
52.3 
57.0 
50.7 
74.0 
20.5 
76.3 
45.7 
10.0 
28.3 
16.9 
10.0 
100.0 
15.4 
45.4 
14.1 
56.2 
19.4 
11.7 
100.0 
10.0 
10.0 
10.0 
10.0 
10.0 
24.0 
100.0 
10.0 
10.0 
38.2 
12.5 
100.0 
10.0 
12.2 
10.0 

 
 
 
 
GETTING IN TOUCH WITH THE EU 

In person 

All over the European Union there are hundreds of Europe Direct information centres. You can find the 
address of the centre nearest you at: https://europa.eu/european-union/contact_en 

On the phone or by email 

Europe Direct is a service that answers your questions about the European Union. You can contact this 
service: 

- by freephone: 00 800 6 7 8 9 10 11 (certain operators may charge for these calls), 

- at the following standard number: +32 22999696, or 

- by electronic mail via: https://europa.eu/european-union/contact_en 

FINDING INFORMATION ABOUT THE EU 

Online 

Information about the European Union in all the official languages of the EU is available on the Europa 
website at: https://europa.eu/european-union/index_en 

EU publications 
You can download or order free and priced EU publications from EU Bookshop at: 
https://publications.europa.eu/en/publications. Multiple copies of free publications may be obtained by 
contacting Europe Direct or your local information centre (see https://europa.eu/european-
union/contact_en). 

K
J
-
N
A
-
2
9
5
6
7
-
E
N
-
N

doi: 10.2760/126300 

ISBN 978-92-79-98383-2