Validation of the innovation radar assessment framework.
Date: 2018-03-07
Institutions: Joint Research Centre
Subjects: advanced technology industry, innovation, research and development, research project, research report, start-up

Validation  of  the  Innovation  Radar 
assessment framework 

Vincent Van Roy and Daniel Nepelski 

2018  

EUR 29137 EN  

 
 
 
 
  
 
 
 
 
This publication is a Technical report by the Joint Research Centre (JRC), the European Commission’s science and 
knowledge service.  It aims to provide evidence-based scientific  support to the European policy-making process. 
The  scientific  output  expressed  does  not  imply  a  policy  position  of  the  European  Commission.  Neither  the 
European Commission nor any person acting on behalf of the Commission is responsible for  the use which might 
be made of this publication. 

JRC Science Hub 
https://ec.europa.eu/jrc 

JRC110926 

EUR 29137 EN 

PDF 

ISBN 978-92-79-80362-8 

ISSN 1831-9424 

doi:10.2760/196017 

Luxembourg: Publications Office of the European Union, 2018 
© European Union, 2018 

Reproduction is authorised provided the source is acknowledged. 

How  to  cite:  Van  Roy,  V.  and  Nepelski,  D.  Validation  of  the  Innovation  Radar  assessment  framework. EUR 
29137 EN.  Publications Office of the European Union, Luxembourg, 2018. ISBN 978-92-79-80362-8. 
doi:10.2760/196017. JRC110926.

All images © European Union 2018 

Title Validation of the Innovation Radar assessment framework 

Abstract 
In this report we provide an assessment of the statistical methodology behind the Innovation Radar. In particular 
we  analyse  to  what  extent  the  Innovation  potential  index  and  the  Innovator  capacity  index  are  analytically  and 
statistically sound and transparent. The aim of this report is to evaluate to what extent variables that have been 
included  in  these  composite  indicators  make  sense  from  a  statistical  point  of  view.  Overall,  the  Innovation 
potential  index is found to be statistically sound with particularly room for improvement of the market potential 
dimension. The Innovator capacity index is conceptually sound but can be improved statistically. 

Table of contents 

Foreword.............................................................................................................. 3 

Executive summary ............................................................................................... 4 

1 

2 

3 

Introduction .................................................................................................... 6 

Innovation radar: in a nutshell .......................................................................... 8 

Input: development of the framework .............................................................. 10 

3.1  Questionnaire .......................................................................................... 10 

3.2  Scoring system ........................................................................................ 11 

3.3  Duplication .............................................................................................. 12 

4  Process: construction of the composite indicators .............................................. 13 

4.1  Data coverage ......................................................................................... 13 

4.2  Choice of the aggregation method .............................................................. 18 

4.3  Choice of weighting method ...................................................................... 23 

4.4  Multivariate analyses ................................................................................ 24 

5  Output: assessment of the final indices ............................................................ 40 

5.1 

Innovation potential index across innovation types ....................................... 40 

5.2 

Innovation potential index across research partners ..................................... 41 

5.3 

Innovator capacity index across organisation types ...................................... 43 

6  Synthesis of the assessment ........................................................................... 45 

References ......................................................................................................... 51 

List of figures ...................................................................................................... 53 

List of tables ....................................................................................................... 53 

Appendix ............................................................................................................ 54 

 
 
 
 
Foreword 

This report is prepared in the context of the three-year research project on Research on 
Innovation,  Start-up  Europe  and  Standardisation  (RISES),  jointly  launched  in  2017  by 
JRC  and  DG  CONNECT  of  the  European  Commission.  The  JRC  provides  evidence-based 
support to policies in the domain of digital innovation and start-ups. In particular:  

 

Innovation with the focus on maximising the innovation output of EC funded 
research projects, notably building on the Innovation Radar; 

  Start-ups and scale-ups – providing support to Start-up Europe; and 
  Standardisation and IPR policy aims under the Digital Single Market priorities. 

This research builds on the work and expertise gathered within the EURIPIDIS project.  

In  this  report  we  provide  an  assessment  of  the  statistical  methodology  behind  the 
Innovation Radar. In particular we analyse to what extent the Innovation potential index 
and the Innovator capacity index are analytically and statistically sound and transparent. 
The aim of this report is to evaluate to what extent variables that have been included in 
these composite indicators make sense from a statistical and conceptual point of view. It 
is  supposed  to  serve  as  a  basis  for  a  discussion  on  potential  changes  to  the 
questionnaire and the framework. 

3 

 
 
 
 
Executive summary 

The  European  Commission's  (EC)  Framework  Programme  (FP)  constitutes  an  important 
share  in  R&D  expenditures  in  Europe.  Many  EC-funded  research  projects  produce 
cutting-edge  technologies.  However,  there  is  a  feeling  that  not  all  of  them  reach  the 
market. The question is why? Launched in 2014, the Innovation Radar is a joint DG 
CNECT-JRC  initiative  to  identify  high-potential  innovations  and  innovators  in 
EC-funded  research  projects  and  guide  project  consortia in  terms  of  the  appropriate 
steps  to  reach  the  market.  Its  objective  is  to  maximise  the  outcomes  of  public  money 
spent on research. Following its successful launch, the Innovation Radar is becoming the 
main  source  of  actionable  intelligence  on  innovation  in  publically-funded  research 
projects in Europe. 

Data  of  the  Innovation  Radar  stem  from  a  survey  developed  by  DG  CNECT  which  is 
conducted  during  periodic  reviews  of  FP  projects  with  an  ICT  theme.  Two  indices  have 
been built using the Innovation Radar data: 

  Innovation  potential  index:  it  aims  at  measuring  FP  project's  innovation 

development towards commercialisation; 

  Innovator  capacity  index:  it  aims  at  capturing  the  innovation  capacity  of 

innovators that are behind these innovations. 

The  Innovation  potential  index  captures  information  about  three  dimensions  that  are 
essential  in  the  innovation  development  process:  innovation  readiness,  innovation 
management  and  market  potential.  The  Innovator  capacity  index  captures  information 
about  the  innovator's  ability  and  innovator's  environment  to  determine  the  capacity  of 
innovators  in  developing  successful  innovations.  Both  indices  are  constructed  as 
arithmetic aggregates of their respective dimensions as indicated in Figure 1. 

In this report we analyse to what extent the Innovation potential index and the 
Innovator  capacity  index  are  analytically  and  statistically  sound.  We  follow  the 
methodology  of  the  OECD/JRC  handbook  for  constructing  composite  indicators  and 
perform an evaluation of the following items: 

  Input: questionnaire and the scoring system used for the indices; 
  Process: statistical process to construct the indices; 
  Output: statistical soundness of the indices. 

Figure 1: The Innovation potential index and Innovator capacity index 

Source: European Commission JRC 

4 

 
 
      
 
 
The  main  findings  of  the  current  report  on  the  validation  of  the  Innovation  Radar 
assessment framework can be summarised in the following way: 

Input 

  Questionnaire:  slight  adjustments  could  be  considered  as  to  maximise  a  clear 

alignment of reviewers on how to interpret questions; 

  Scoring system: slight adjustments could be considered as to accentuate project 

differences. 

Process 

  Innovation potential index: statistically sound; 

o  The  innovation  management  and  innovation  readiness  dimensions  are 

statistically well-balanced and show a good internal consistency; 

o  More  room  for  improvement  is  observed  for  the  market  potential 

dimension. 

  Innovator  capacity  index:  conceptually  sound  but  can  be  improved 

statistically; 

o  The index would benefit from a more balanced contribution of indicators; 
o  Hence, the collection of indicators that fit better together from a statistical 

perspective could be considered. 

Output 

  Adjustments to the conceptual framework of both indices could be considered as 
to  account  for  differences  in  the  innovation  process  across  innovation  types  and 
research partners. 

Section  6  provides  more  detailed  summary  tables  that  synthesize  the  main  findings  of 
the  assessment  of  the  Innovation  Radar  framework  that  has  been  conducted  in  this 
report. 

5 

 
 
 
 
 
 
1  Introduction 

The  Innovation  Radar  (IR)  is  an  initiative  supported  by  the  European  Commission 
focussing  on  the  identification  of  high  potential  innovations  and  the  key  innovators 
behind them  in FP7, CIP and Horizon2020 projects with an ICT theme  (De Prato  et al., 
2015).  The  IR  serves  as  a  monitoring  tool  for  policy  makers  and  project  officers  at  the 
European Commission as it provides up-to-date information on the innovative  output of 
these  projects.  The  IR  allows  them  to  characterise  innovations  with  respect  to  their 
technical readiness, innovation management and market potential. For innovators, it can 
deliver  information  on  their  individual  performance  and  ongoing  needs  and  the 
environment in which they innovate. Both the information about the innovation potential 
and  innovator  capacity  has  been  summarised  in  two  indices  called  respectively 
Innovation potential index and Innovator capacity index. 

A business intelligence dashboard has been developed for EU policy makers to help them 
make  use  of  the  Innovation  Radar  data  sets  for  policy  development  and  to  empower  a 
more  data-driven  approach  to  managing  the  Horizon  2020  programme.  While  pilot 
editions  have  been  conducted  for  a  limited  number  of  Framework  Programme  projects, 
the dashboard has been deployed to all projects with an ICT theme. 

The deployment of the dashboard to cover all collaborative projects launched under the 
ICT  theme  calls  for  a  formal  validation  of  the  Innovation  Radar  methodology.  In  this 
report  we  provide  an  assessment  of  the  statistical  methodology  behind  the  Innovation 
Radar. In particular we analyse to what extent the Innovation potential index and 
the  Innovator  capacity  index  are  analytically  and  statistically  sound  and 
transparent.  The  aim  of  this  report  is  to  evaluate  to  what  extent  variables  that  have 
been included in these composite indicators make sense from a statistical and conceptual 
point of view. This assessment consists of a statistical evaluation of the following items: 

  Input: relates to the questionnaire and the scoring system that provide the input 

data that feeds the indices of the Innovation Radar; 

  Process:  relates  to  the  statistical  process  to  construct  the  indices  of  the 

Innovation Radar; 

  Output: relates to the statistical soundness of the final indices of the Innovation 

Radar. 

The  three  items  that  are  presented  in  this  report  closely  follow  the  different 
methodological  steps  suggested  by  the  OECD/JRC  handbook  for  constructing  composite 
indicators (OECD & JRC, 2008).  The construction of indices should ideally be guided by 
the  following  steps:  1.  the  development  of  a  framework  defining  the  concept  and  the 
dimensionality of what is meant to be measured; 2. the gathering of data accompanied 
with general data checks (e.g., data coverage, and choice of aggregation and weighting 
methods);  3  the  statistical  choices  to  ensure  the  coherence  and  robustness  of  the 
composite indicator (e.g. multivariate analyses); and eventually 4. a quality assessment 
from  expert  bodies  in  order  to  get  suggestions  and  reviews  about  the  decisions 
undertaken  in  the  previous  stages  of  analysis.  The  sequence  for  the  construction 
procedure is depicted in Figure 2. 

6 

 
 
 
Figure 2: Methodological steps for the construction of the Innovation Radar 

Source: Based on the OECD/JRC handbook on constructing composite indicators (OECD & JRC, 2008). 

We  use  these  sequential  steps  as  guide  for  the  structure  of  this  report.  Section  2 
provides  a  brief  overview  of  the  Innovation  Radar  methodology  and  presents  the  data 
that is included in the dashboard and that is  employed for the statistical assessment of 
the  Innovation  potential  index  and  Innovator  capacity  index  in  the  current  report. 
Section 3 focuses on the framework of the Innovation Radar. Instead of focusing on the 
theoretical  arguments  for  the  different  dimensions  in  both  indices  that  has  been 
analysed in De Prato et al. (2015), we provide an assessment of the framework from a 
statistical point of view, i.e. measuring to what extent the scoring system is adequate in 
measuring  the  underlying  constructs,  and  providing  some 
insights  about  the 
questionnaire that feeds the data for the composite indicators. 

Section 4 provides an in-depth assessment of the current construction of the indices and 
evaluates to what extent the various steps to construct a composite indicator have been 
followed.  While  this  section  is  mainly  focused  on  the  process  of  obtaining  the  indices, 
section  5  focuses  more  on  the  assessment  of  the  final  indices  in  terms  of  their  results 
and  potential  biases  they  may  have  due  to  methodological  choices  made  during  their 
construction. 

Finally, section 6 summarises the practical recommendations concerning the construction 
of the framework and composite indicators. This way, it is supposed to serve as a basis 
for a discussion on potential changes to the questionnaire and the framework.  

7 

 
 
 
 
 
 
 
2  Innovation radar: in a nutshell 

The Innovation Radar (IR) is an EC support initiative that aims to assess the potential of 
innovations developed within FP research projects and to identify the bottlenecks to their 
commercialisation  (De  Prato  et  al.,  2015).  Data  of  the  Innovation  Radar  stem  from  a 
questionnaire  developed  by  DG  CONNECT.  The  questionnaire  is  conducted  by  external 
experts commissioned by DG CONNECT during periodic reviews of the research projects. 
The  Innovation  Radar  monitors  the  ICT  research  actions  and  the  e-infrastructures 
activity  under  the  seventh  Framework  Programme  2007-2013  (under  cooperation  and 
capacities themes), the policy support actions carried out under the competitiveness and 
innovation  framework  policy  support  programme  (CIP  ICT  PSP)  and  the  ICT-related 
projects in Horizon 2020 (EC, 2014).  

Among others, the Innovation Radar aims to identify high potential innovations and the 
key innovators behind them in FP projects. This information is delivered by means of two 
indices.  The  first  index  provides  a  holistic  view  of  the  innovation  potential  of  FP7 
projects,  while the second one is capturing the innovator's  capacity in conducting high-
potential  innovation  activities.  Both  indices  are  respectively  called  Innovation  potential 
index and the Innovator capacity index. The conceptual framework and scoring systems 
behind these two indices was originally developed as pilot editions in 2015 (De Prato et 
al., 2015) and subsequently revised in 2016 (Pesole and Nepelski, 2016). 

A business intelligence dashboard has been developed for EU policy makers to help them 
make use of these data sets for policy development and to empower a more data-driven 
approach to managing the Horizon 2020 programme. While the pilot editions related to a 
limited  number  of  reviews  conducted  between  October  2014  and  December  2016,  the 
dashboard has been deployed to all projects with an ICT theme and contains information 
from January 2016 onwards. The dashboard has automatized the processing of data and 
uses the most recent approach in terms of scoring system and  questionnaire version to 
construct  the  two  indices.  Both  the  questionnaire  and  the  scoring  system  to  construct 
the indices are presented in appendix. 

Table 1: Overview of innovation projects and organisation types 

Calculations: European Commission JRC 
Data: European Commission DG Connect 
Note:  The  indicator  in  the  database  that  identifies  whether  a  firm  categorises  as  a  SME  or  a  large  firm 
contained 92 missing values. In the table above, these missing values have been treated as large firms. 

8 

 
 
 
 
Data from the dashboard has been used for the statistical assessment of both indices of 
the Innovation Radar. Table 1 provides an overview of the sample of innovation projects 
and innovators that we have used for assessment in the current report. Between January 
2016  and  November  2017,  643  EU-funded  collaborative  research  projects  were 
reveiwed.  As  a  result,  1,777  innovations  were  identified.  This  means  that,  on  average, 
every  project  produces  nearly  3  innovations.  The  number  of  distinct  key  organisations 
active in these projects amounted to 1,398. We distinguished six types of organisations, 
including  universities,  research  centres,  small  –and  medium-sized  enterprises  (SMEs), 
large  firms,  governmental  institutions  and  others.  SMEs  represent  the  highest  share  of 
organisations  with  37  percent.  Universities  and  large  firms  both  account  for  23  percent 
each  of  the  organisations,  while  the  percentage  of  research  centers  is lower  amounting 
to 13 percent. The percentage of both the governmental institutions and other types of 
organisations amounts to 4% together. 

9 

 
 
 
 
 
 
 
 
3  Input: development of the framework 

This section provides an assessment of the questionnaire and scoring systems that feed 
the data for the composite indicators. In particular, it aims to identify some pitfalls and 
drawbacks  in  the  current  questionnaire  and  scoring  system  and  provides  some 
recommendations for improvement. 

3.1  Questionnaire 

Concept 

In  survey  sampling,  one  of  the  main  issues  of  survey  designers  is  limiting  respondent 
errors. Several reasons can lead respondents to provide incorrect or biased information. 
It can be due to a misunderstanding of the question by the respondent or alternatively it 
can  be  caused  by  a  misunderstanding  of  the  response  by  the  surveyor.  In  any  case, 
survey questions should be designed in such a way as to minimise possible bias 
from misunderstanding. 

Assessment outcome 

The  question  about  the  most  impressive  partner  in  terms  of  innovation  potential  is 
clearly  stating  that  reviewers  should  highlight  one  particular  partner  in  each  project. 
Hence,  this  question  calls  for  one  partner  name  per  FP  project.  However,  statistics  are 
telling the opposite as observed in Figure 3. From the 1777 innovations identified in the 
actual  dashboard,  13  percent  of  them  report  several  most  impressive  partner  at  the 
overall project-level.  

Figure 3: Number of most impressive partner per project  

Calculations: European Commission JRC 
Data: European Commission DG Connect 

Recommendation 

The assessment outcome calls for a clear alignment across reviewers on how to interpret 
questions.  

10 

 
 
 
 
 
 
 
3.2  Scoring system 

Concept 

A  scoring  system  has  been  developed  to  allow  for  the  classification  of  projects  along 
their  level  of  innovation  potential  and  innovators  along  their  capacity  to  develop  high-
potential  innovations.  The  scoring  systems  that  have  been  used  as  indicators  for  both 
the  Innovation  potential  index  and  the  Innovator  capacity  index  are  presented  in  the 
appendix.  These  scoring  systems  are  in  line  with  other  types  of  scoreboards  that  have 
been  used  in  the  scientific  literature  as  a  ranking  systems  of  technology  development 
projects (see e.g. Cooper, 2007).  

In  general  all  the  questions  relevant  to  measure  each  dimension  captured  in  the  two 
indices  is  used  as  input  in  the  scoring  system.  Each  answer  is  then  allocated  a  certain 
score  as  defined  in  appendix  in  order  to  determine  the  innovation  potential  and 
innovator capacity. 

Although  the  scoring  systems  aims  to  aggregate  data  from  the  questionnaire  to  reduce 
the  dimensionality  of  the  concept  measured,  in  some  cases  it  can  be  beneficial  to 
apply a more diversified rating score in order to accentuate project differences. 
This would improve the accuracy of identification of the indices in the Innovation Radar. 

Assessment outcome 

Maximisation  of  the  diversity  in  rating  score  is  not  always  applied.  The  question  about 
the  partners'  commitment  to  exploit  their  innovation  outlines  6  levels  of  reviewer 
assessment, while the scoring system reduces this information to 3 levels. As illustrated 
in Table 2, an additional scoring level for the projects' commitment would reward almost 
one fourth of the innovation sample.  

Table 2: Change of scoring system for partners' commitment  

Calculations: European Commission JRC 
Data: European Commission DG Connect 

Recommendation 

Consider changing rating scores to accentuate project differences. 

11 

 
 
 
 
 
 
 
 
3.3  Duplication 

Concept 

In  general,  statisticians  discourage  the  use  of  an  ‘index  within  an  index’  on  two  main 
grounds:  the  distorting  effect  of  the  use  of  different  computing  methodologies  and  the 
risk  of  duplicating  variables  (Saisana  et  al.,  2017).  The  former  issue  is  not  a  major 
problem  when  similar  computing  methodologies  have  been  used  as  is  the  case  for  the 
Innovation Radar indices. However, the risk of duplicating indicators when using an 
'index within an index' remains a major issue. 

Assessment outcome 

The  Innovator  capacity  index  contains  the  Innovation  potential  index  as  one  of  its 
indicators.  However,  it  also  includes  two  indicators  that  were  already  included  in  the 
Innovation  potential  index.  This  leads  to  the  duplication  and  double  counting  of  the 
following indicators in the Innovator capacity index: 

  End-user engagement 
  Commitment to innovate 

Recommendations 

  Recalculation  of  the  Innovation  potential  index  without  the  two  duplicate 
indicators and insertion of this revised index in the Innovator capacity index; 

  Collection of other indicators of innovator's environment. 

12 

 
 
 
 
 
 
 
4  Process: construction of the composite indicators 

This  section  provides  an  assessment  of  the  current  methodology  of  the  Innovation 
Radar.  In  particular,  we  assess  to  what  extent  the  methodology  follows  the  various 
methodological  steps  highlighted  by  the  OECD/JRC  handbook  on  composite  indicators 
(OECD  and  JRC,  2008).  This  section  extensively  builds  on  the  expertise  of  the 
Competence  Centre  on  Composite  Indicators  and  Scoreboards  of  the  Joint  Research 
Centre in Ispra.1  

In particular, the construction of indices can be outlined in the following key steps: 

  Data  coverage:  quality  assessment  of  the  raw  data  in  terms  of  data  availability 

and data imputation decisions; 

  Choice  of  aggregation  method:  selection  of  a  suitable  aggregation  method 

allowing or not for compensability among indicators; 

  Choice  of  weighting  method:  selection  of  a  suitable  weighting  method  favouring 

equal weighting or not; 

  Multivariate  analyses:  assessment  of  the  statistical  coherence  in  terms  of  the 

underlying importance of indicators and sub-dimensions. 

In general the process of construction a composite indicator includes additional steps of 
outlier  treatment  and  normalisation.  Outlier  treatment  relates  to  the  identification  and 
replacement of outliers in the raw data. The normalisation step requires the selection of 
a suitable normalisation method in order to adjust the raw data to a notionally common 
scale. These both steps are not relevant for the Innovation Radar as the data is based on 
a survey and hence do not contain outliers in the data. Normalisation is also not needed 
as  indicators  are  comparable  to  each  other  giving  the  scoring  system  that  has  been 
developed. All other steps will be discussed in detail in the following paragraphs. 

4.1  Data coverage 

Concept 

A  representative  data  coverage  is  key  to  create  a  sound  and  transparent  composite 
indicator.  A  low  data  coverage  for  some  indicators  could  bias  the  final  outcome  of  an 
index.  As  a  rule  of  thumb,  a  data  coverage  of  at  least  75  percent  per  indicator 
should  be  available  to  include  an  indicator  in  a  composite  index.  In  this  section  we 
assess the data coverage for each dimension for both indices of the Innovation Radar. 

4.1.1  Innovation potential index 

Assessment outcome 

Figure  4,  Figure  5  and  Figure  6  presents  the  percentages  of  missing  values  for  the 
various indicators populating the Innovation potential index. 

Market potential 

The  indicators  of  market  potential  are  relatively  well  covered,  where  the 
percentages  of  missing  values  remain  below  3  percent  for  most  indicators.  Market 
dynamics  is  the  only  indicator  with  a  problematic  data  coverage.  Data  for  this 
indicator is missing in nearly one third of the cases. This large number of missing values 
may  indicate  a  difficulty  of  reviewers  in  responding  to  questions  about  the  market 

1   For more information about the construction and audit of composite indicators, we refer to the 
Scoreboards: 
Composite 

Indicators 

and 

Competence 
on 
https://ec.europa.eu/jrc/en/coin.  

Centre 

13 

 
 
 
 
 
                                           
conditions  (e.g.  in  comparison,  the  question  on  market  size  has  a  missing  rate  of  44 
percent).  

Figure 4: Overview of missing data for the dimension of market potential 

Calculations: European Commission JRC 
Data: European Commission DG Connect 

Innovation readiness  

In general, we observe a low data coverage for all innovation steps that project 
consortia  have  undertaken  to  develop  and  commercialise  their  innovations  on  the 
market.  All  indicators  in  the  innovation  readiness  dimension  that  capture  innovation 
steps reflect missing rates between 14 and 24 percent. The indicator called “others” that 
provide  the  possibility  to  reviewers  to  indicate  a  particular  type  of  innovation  step  (i.e. 
not listed in the questionnaire) is even lacking in nearly 70 percent of the cases. 

Figure 5: Overview of missing data for the dimension of innovation readiness 

Calculations: European Commission JRC 
Data: European Commission DG Connect 

14 

 
 
 
 
 
 
 
Innovation management 

A  similar  pattern  is  observed  in  Figure  6  for  the  dimension  of  innovation  management: 
all indicators  related  to innovations  steps  are  missing in  17  to  24  percent  of  the  cases. 
Other indicators have almost no missing values. 

Figure 6: Overview of missing data for the dimension of innovation management 

Calculations: European Commission JRC 
Data: European Commission DG Connect 

Missing values for innovation steps 

Since the lower data coverage  on innovation steps  seems to be a general phenomenon 
we  evaluate  in  more  detail  the  pattern  of  missing  innovation  steps  in  each 
innovation. To this purpose, we select those innovations that have at least one missing 
innovation  step  and  analyse  their  missing  patterns  across  the  twelve  innovation  steps 
that  are  surveyed  in  the  questionnaire.  We  group  them  in  four  different  categories 
according to their number of missing innovation steps: 1) 1 to 2, 2) 3 to 5, 3) 6 to 8 and 
4) 9 to 12.  The distribution of innovations along these groups is presented in  Figure 7.  
From  the  population  of  innovations  that  have  at  least  on  missing  innovation  step,  we 
observe the following: 

  The  large  majority  (66  percent)  only  lacks  information  for  1  or  2  innovation 

steps; 

  Around 8 percent lacks information for up to 12 innovation steps; 
  Almost 22 percent of innovations lacks information for 9 to 12 innovation steps. 

Translating this last point to the full sample of innovations, we observe that 12 percent 
of innovations do not have any information about innovation steps.  

15 

 
 
 
 
 
 
 
Figure 7: Distribution of the number of missing innovation steps  

Calculations: European Commission  JRC 
Data: European Commission DG Connect 

Twelve  percent  of  innovations  for  which  almost  no  information  is  available  about  the 
innovation steps is not  a negligible number  given that they constitute a relatively large 
part  of  the  Innovation  potential  index.  Two  main  reasons  can  be  put  forward  to 
explain the low data coverage for innovation steps: 

 
 

It may reflect the difficulty of reviewers to fill this type of question.  
Innovation steps may be left blank because they are most applicable for product 
innovations  and  less  relevant  for  other  types  of  innovations,  such  as  process  or 
service innovations and new marketing and organisational methods. 

To  address  this  latter  issue,  we  analyse  the  distribution  of  innovations  for  which 
none  of  the  innovation  steps  have  been  filled  in  and  compare  them  across 
different  innovation  types  (see  Figure  8).  The  figure  represents  the  percentage  of 
innovations per innovation type for  which none of the innovation steps have been filled 
in by the reviewers. Following patterns are observed: 

  Organisational/marketing  methods  and  service 

innovations  have  highest 

percentages of complete lack of information on the innovation steps; 

  Percentages  for  product  and  process  innovations  for  which  no  information  is 

available is relatively low.  

16 

 
 
 
 
 
 
 
Figure 8: Missing data on all innovation steps across innovation types  

Calculations: European Commission  JRC 
Data: European Commission DG Connect 
Note:  The  different  innovation  types  are  defined  in  the  following  way:  1)  Marketing/organisational  method 
includes both new and significantly improved methods, 2) Service innovation and others: new and significantly 
improved  services,  consulting  services  and  others,  3)  Product  innovations:  new  and  significantly  improved 
products,  4)  Process  innovations:  new  and  significantly  improved  process  innovations.  Percentages  are 
calculated per innovation type, i.e. relative to the total number of innovations in each innovation type. 

Recommendations 

We have the following recommendations for each dimension of the Innovation potential 
index: 

Market potential 

  Consider exclusion of market dynamics. 

Innovation management 

  Consider exclusion of "other" innovation steps. 

Innovation readiness 

  Consider hands-on support or training of reviewers. 
  Based  on  the  low  data  coverage  for  all  innovation  steps:  different  types  of 
innovations may require different types of innovation trajectories that are actually 
not  included  in  the  conceptual  framework  of  the  Innovation  Radar.  The 
questionnaire  and  conceptual  framework  could  be  adjusted  to  account  for  these 
differences. 

4.1.2  Innovator capacity index 

Assessment outcome 

The  indicators  of  the  Innovator’s  ability  have  no  missing  values.  The  innovator’s 
environment  has  only  a  few  missing  values  for  the  indicators  of  end-user  engagement 
and  commitment  to  innovate  as  indicated  in  previous  section.  Hence,  we  have  no 
particular  recommendations  concerning  the  data  coverage  of  the  Innovator  capacity 
index. 

17 

 
 
 
 
 
 
Recommendations 

Given  the  excellent  data  coverage  we  do  not  have  particular  recommendations  for  the 
innovator capacity index. 

4.2  Choice of the aggregation method 

Concept 

Every  ranking  score  in  composite  indicators  depends  on  subjective  modelling  choices. 
One  of  them  is  the  choice  to  use  arithmetic  averages  when  aggregating  data  into  the 
overall  index.  In  this  paragraph,  we  evaluate  how  rankings  differ  if  we  use  another 
aggregation method such as the geometric average. 

4.2.1  Innovation potential index 

Assessment outcome 

We  evaluate  another  aggregation  method  because  we  observe  a  low  diversity  in  the 
ranking scores when using arithmetic averages: 

  Only 60 out of the 1777 innovations (only 3 percent!) have a unique value for the 

Innovation potential Index; 

  A  large  majority  of  innovations  have  Innovation  potential  indices  that  appear 

twice and up to 23 times in the database. 

So far arithmetic averages have been used to aggregate indicators into dimensions and 
indices. It is used in wide range of well-known indices as it has the virtue of being simple 
and easy to interpret (Saisana and Saltelli, 2014).  

However, arithmetic averages provide low diversity in ranking scores caused by 
the following problems related to this method: 

  Perfect  substitutability:  i.e.  a  poor  performance  in  one  indicator  can  be  fully 

compensated by a good performance in another; 
It does not reward balanced achievement in all indicators; 

 
  No  impact  of  poor  performance:  it  does  not  consider  that  the  lower  the 
performance  in  a  particular  indicator,  the  more  urgent  it  becomes  to  improve 
achievements in that indicator. 

To  overcome  these  shortcomings  other  aggregation  methods  such  as  the  geometric 
mean  have  been  advanced  by  practitioners  (Munda,  2008).  This  average  method  is  a 
partially  compensatory  approach  that  rewards  projects  with  balanced  profiles  and 
motivates them to improve in the dimensions in which they perform poorly, and not just 
in any dimension. 

to 

In  addition 
these  advantages,  geometric  averages  accentuate  project 
differences  and  provide  more  diversity  in  the  rankings  scores.  This  is  exactly 
what  we  should  aim  at  with  the  Innovation  potential  index  as  ideally  an  index  should 
only have unique rankings scores that fully capture the project differences. 

This  is  well  illustrated  in  Figure  9  that  presents  the  distribution  of  similarity  in  ranking 
scores of innovations across the two types of aggregation: arithmetic and geometric. The 
figure  present  for  both  aggregation  methods  how  often  an  identical  ranking  score 
appears in the database.  The  fact that the  Innovation potential index has a majority of 
identical  ranking  scores  is  not  only  caused  by  the  restricted  scoring  system  of  the 
Innovation  Radar,  but  is  further  accentuated  by  the  use  of  arithmetic  averages.  The 
figures should be read  as a pyramid where the base is the ideal situation, representing 
the  number  of  ranking  scores  that  appear  only  once  in  the  database.  Hence,  these  are 

18 

 
 
 
 
 
 
the rankings that allow to differentiate projects in their innovation potential. Each layer 
above  represents  the  number  of  occurrences  that  a  same  ranking  score  appears.  For 
instance,  the  second  layer  represents  the  number  of  rankings  that  appear  twice  in  the 
database,  while  the  third  layer  represents  the  number  of  rankings  that  appear  three 
times in the database, etc. 

Based on both figures we can make the following conclusions: 

  The  number  of  unique  ranking  scores  when  using  geometric  averages  is 

significantly higher than for arithmetic averages; 

  The number of ranking scores that appear twice, three times, etc. in the database 
is gradually decreasing for the geometric average, while increases for  arithmetic 
averages; 

  The  number  of  occurrences  that  a  ranking  score  appears  in  the  database  is 

significantly lower for geometric averages. 

19 

 
 
 
  
Figure 9: Arithmetic versus geometric aggregation for the Innovation potential index 

                                               Arithmetic average                                                                Geometric average 

Calculations: European Commission JRC 
Data: European Commission DG Connect 
Note: The figures present the distribution of ranking scores along their number of occurrences in the database. Ranking score distributions are calculated when using 
arithmetic  and  geometric  aggregation.  The  base  of  the  pyramids  represents  the  number  of  unique  ranking  scores,  while  the  second  layer  are  ranking  scores  that 
appear twice, etc. The labels at the bars represent the number of ranking scores that appear in each layer of the pyramid. 

20 

 
 
 
 
 
 
 
Figure 10: Arithmetic versus geometric aggregation for the Innovator capacity index 

                                       Arithmetic average                                                                      Geometric average 

Calculations: European Commission JRC 
Data: European Commission DG Connect 
Note:  The  figures  present  the  distribution  of  ranking  scores  along  their  number  of  occurrences  in  the  database.  Ranking  score  distributions  are  calculated  when  using 
arithmetic and geometric aggregation. The base of the pyramids represents the number of unique ranking scores, while the second layer are ranking scores that appear 
twice, etc. The labels at the bars represent the number of ranking scores that appear in each layer of the pyramid. 

21 

 
 
 
 
 
4.2.2  Innovator capacity index 

Assessment outcome 
Using  a  similar  pyramid  comparison  as  for  the  Innovation  potential  index,  Figure  10 
presents  a  comparison  of  the  rankings  for  arithmetic  and  geometric  averages  for  the 
Innovator capacity index. Aggregation using geometric averages still accentuates project 
differences, though results are less pronounced than for the Innovation potential index.  

Recommendations for both indices 

Based on the assessment outcome of the choice of aggregation method for both indices 
it  is  recommended  to  use  geometric  averages  rather  than  arithmetic  ones.  In 
the  particular  case  of  the  Innovation  Radar,  ranking  scores  lack  diversity  due  to  the 
restricted  scoring  system.  However,  this  lack  of  diversity  is  further  accentuated  by  the 
use of arithmetic averages to aggregate the dimensions of market potential, innovation 
readiness and innovation management into the Innovation potential index. In Table 3 we 
recall the characteristics of the different aggregation methods for the Innovation Radar. 

Table 3: Comparison of aggregation method 

Note: Based on Munda (2008) and the assessment outcome of the Innovation Radar. 

22 

 
 
 
 
 
 
 
 
4.3  Choice of weighting method 

Concept 

The  results  of  principal  component  analyses  (see  section  3.4)  are  often  used  to 
determine  appropriate  weights  when  aggregating  indicators  into  dimensions.  Important 
to  notice  is  that  these  weights  are  then  used  to  correct  for  overlapping  information 
between  two  or  more  correlated  indicators  and  are  not  a  measure  of  the  theoretical 
importance  of  the  associated  indicator.  If  no  correlation  between  indicators  is  found  – 
which is the case for the Innovation Radar – then weights cannot be estimated with this 
method. 

Given  the  difficulty  of  obtaining  appropriate  weights  from  the  principal  component 
analyses, aggregation of the dimensions in the current version of the Innovation 
Radar is done using equal weights. 

4.3.1  Innovation potential index 

Assessment outcome 

We have analysed the scientific literature that investigates which factors are important in 
innovation  processes.  Many  of  these  papers  classify  important  indicators  in  relatively 
similar dimensions that have been used in the Innovation Radar.  Balachandra and Friar 
(1997)  proposes  four  major  categories  on  market,  technology,  environment,  and 
organisational related characteristics. These categories have been widely recognised and 
adopted  by  many  scholars  in  the  field  of  technology  commercialisation  of  R&D  projects 
(Astebro,  2004;  Linton  et  al.,  2002).  Alternatively,  Heslop  et  al.  (2001)  use  factor 
analyses to group more than fifty variables related to the technology commercialisation 
process  into  four  dimensions  of  market  readiness,  technology  readiness,  commercial 
readiness,  and  management  readiness.  However,  there  is  no  clear  evidence  of  which 
dimension is more important in the innovation process.  

Due  to  a  lack  of  convergence  in  the  scientific  literature  to  determine  which  factors  are 
most  important,  it  is  justified  to  follow  a  conservative  approach  and  to  opt  for 
equally  weighting  the  three  dimensions  of  market  potential,  innovation 
readiness  and  innovation  management.  With  this  approach  we  follow  the 
perspective of scholars claiming that successful development and commercialisation of a 
new technology is a matter of competence in all factors and of balance and coordination 
between  them  and  not  doing  one  or  two  things  brilliantly  well  (Conceição  et  al.,  2012; 
Cooper and Kleinschmidt, 1988; Rothwell, 1992). 

4.3.2  Innovator capacity index 

Assessment outcome 

A  similar  argumentation  applies  as  for  the  Innovation  potential index:  there  is  no  clear 
convergence  in  the  scientific  literature  of  which  indicators  are  important  to  determine 
innovator capacity. 

Recommendations for both indices 

At  the  moment,  the  Innovation  Radar  can  continue  using  equal  weighting  in  both  the 
Innovation potential index and the Innovator capacity index. 

23 

 
 
 
 
 
 
4.4  Multivariate analyses 

In order to assess the statistical and conceptual coherence of the structure of the data in 
the  indices  of  the  Innovation  Radar,  we  conduct  a  series  of  multivariate  analyses  that 
are  commonly  used  in  the  scientific  literature  of  composite  indicators  (OECD  and  JRC, 
2008). In particular, we conducted following analyses: 

  Correlation analyses: it provides insights about the statistical dimensionality and 

the grouping of indicators into the three dimensions; 

  Principal  Component  Analysis  (PCA):  it  is  used  to  assess  to  what  extent  the 
conceptual framework behind the indices of the Innovation Radar is confirmed by 
statistical approaches and to identify eventual pitfalls. 

  Cronbach  Alpha  Coefficient:  it  estimates  the  internal  consistency  in  each 

dimension of the innovation potential index. 

All  the  analyses  in  this  section  complement  each  other  and  aim  to  evaluate  to  what 
extent indicators that are fitting well in their respective dimensions.  

4.4.1  Correlation analyses 

Concept 

Correlation  analyses  allow  investigating  the  linear  statistical  relationships  across 
indicators  in  each  dimension  and  their  respective  relationship  with  the  final  index. 
Overall,  indicators  need  to  be  significantly  and  positively  correlated  but  not 
excessively  (above  0.95)  to  have  a  statistical  justification  to  aggregate  them 
together. 

4.4.1.1  Innovation potential index 

Assessment outcome 

Correlations  within  and  across  dimensions  are  presented  in  Table  4.  We  discuss  the 
correlation  matrices  for  each  dimension  of  market  potential,  innovation  readiness  and 
innovation management and conclude with the analyses of the correlations across these 
two dimensions and the Innovation potential index.  

Market potential 

We observe relatively low levels of correlations across indicators in the market potential 
dimension.  Indicators  in  the  market  potential  dimension  do  not  correlate  with 
each  other  because  they  measure  a  wide  range  of  different  phenomena.  The 
conceptual framework that provides a theoretical justification for the various items that 
are measured in the market potential dimension (see De Prato et al., 2015) highlights a 
large variety of phenomena that determine the market potential of an innovation. Market 
potential relates in essence to a wide range of technical and market characteristics that 
aim  to  capture  the  type,  level,  exploitation  stage  and  patentability  of  an  innovation  as 
well  as  market  conditions  with  respect  to  the  level  of  maturity,  competition  and 
dynamics. 

Although  these  items  are  relevant  to  determine  the  market  potential  of  an  innovation 
they  do  not  easily  fit  well  in  an  aggregated  measure  as  they  measure  different 
phenomena. This is difficulty is reflected in the correlation results. 

Concretely,  the  correlation  matrix  of  the  market  potential  dimension  reveals  the 
following: 

24 

 
 
 
 
 
  Correlations across indicators are close to zero and in the best case elevate up to 

0.40. 

  Skewed  contribution  of  the  indicators  to  the  aggregated  measure  of  market 

potential. 

The  market  potential  measure  seems  to  be  mainly  explained  by  the  indicators 
on  innovation  exploitation  stage  and  market  maturity.  The  relevance  and 
contribution  of  the  other  indicators  is  significantly  lower.  In  particular,  the  indicator  on 
market  competition  does  not  seem  to  contribute  to  the  aggregated  measure  of  market 
potential. This is not surprising as market competition seems to negatively correlate with 
other indicators in this pillar.   

Innovation readiness 

The  correlation  matrix  of  the  innovation  readiness  dimension  provide  a  more 
balanced  picture.  Correlations  across  indicators  are  higher  and  almost  all  positive 
significant,  suggesting  that  many  indicators  in  this  aggregated  measure  are  capturing 
related  phenomena.  By  consequence,  the  contribution  of  the  indicators  to  the 
aggregated  measure  is  more  balanced  as  well.  The  indicator  on  the  innovation 
development  stage  as  well  as  the  indicators  measuring  the  various  innovation  steps  in 
the development process (i.e. prototyping, pilot, demonstration, technology transfer and 
feasibility study) are capturing between 37 and 59 percent of the aggregated measure of 
innovation  readiness.  The  only  two  innovation  steps  that  contribute  less  to  the 
innovation readiness dimension are the launch of a startup or spinoff and other 
innovation steps. These are also the two indicators with the lowest data coverage. 

Innovation management 

Also  the  correlation  matrix  of  the  innovation  management  dimension  is 
relatively well balanced. All correlations are positive and significant. The indicator of a 
clear  owner  of  the  innovation  and  the  indicator  revealing  no  problems  of  IPR  issues 
within  the  project  consortium  are  the  only  two  exceptions.  Both  indicators  do  not 
correlate  with  other  indicators,  while  they  correlate  positively  with  each  other.  The  fact 
that  both  indicators  do  not  statistically  fit  with  the  other  indicators  is  also  reflected  in 
their  contribution  to  the  aggregated  measure  of  innovation  readiness.  Both  indicators 
only  explain  10  to  13  percent  of  the  variance  of  the  innovation  readiness  dimension, 
while  other  indicators  have  a  significantly  higher  contribution  (between  25  and  60 
percent).  

Correlations across the Innovation potential index and its three dimensions 

Analysing  the  correlations  across  the  Innovation  potential  index  and  its  three 
dimensions,  we  find  considerably  strong  linear  relationships  between  the  three 
dimensions  and  the  index.  This  suggests  that  the  three  dimensions  provide 
meaningful  information  on  the  variation  of  the  index  score.  The  contributions  of 
innovation  readiness  and  innovation  management  are  strongly  balanced  and  capture 
each 65 percent of the index variance. The contribution of the market potential is slightly 
lower, elevating at 41 percent. 

25 

 
 
 
 
 
 
Table 4: Correlations within and across dimension and the Innovation potential index  

Calculations: European Commission JRC 
Data: European Commission DG Connect 
Note: Correlations between indicators and pillars or  pillars and the innovation potential index are indicated in 
bold.  All  correlations  are  significant,  except  for  those  indicated  in  red  that  represent  correlations  with  a 
significance level below 5 percent. 

Recommendations 

We have the following recommendations for each dimension of the Innovation potential 
index: 

Market potential 

  Consider exclusion of market competition and number of patents. 

26 

12345671Market potential1.002Type of innovation0.291.003Innovation exploitation stage0.690.191.004Market maturity0.79-0.050.151.005Market dynamics0.370.060.020.391.006Level of Innovation0.350.120.130.120.281.007Market competition0.050.14-0.13-0.310.04-0.071.008Number of patents0.27-0.010.020.000.090.15-0.091234567891Innovation readiness1.002Innovation development stage0.671.003Technology transfer0.630.311.004Prototyping - real world0.760.410.411.005Pilot, demonstration0.770.450.440.731.006Feasibility study0.610.220.280.430.421.007Launch a startup or spin-off0.530.300.320.330.300.261.008Other0.330.130.070.170.24-0.040.501.009Time to market0.520.540.330.420.380.240.300.071.0010No workforce skills issues0.390.050.110.120.120.23-0.05-0.090.04123456789101Innovation management1.002Clear owner0.371.003Research engagement0.68-0.091.004Business plan0.78-0.040.581.005Market study0.77-0.130.570.851.006Application funding0.61-0.090.310.360.391.007Secure priv. Investment0.68-0.090.390.430.400.771.008Secure pub. Investment0.59-0.120.230.400.340.810.851.009No IPR issues0.320.22-0.03-0.090.01-0.02-0.01-0.081.0010End-user engagement0.500.040.290.380.350.050.050.01-0.161.0011Commitment to innovate0.570.050.330.360.370.170.300.180.100.29Dimension and indicatorsDimension and indicatorsDimension and indicatorsCorrelations within dimensions1231Innovation potential index1.002Market potential0.641.003Innovation readiness0.810.231.004Innovation management0.810.300.55Index and dimensionsCorrelations across dimensions and index 
 
 
 
 
 
Innovation management 

  Consider exclusion of Other innovation steps and No workforce skills issues.  

Innovation readiness 

  Consider exclusion of Clear owner and No IPR issues. 

4.4.1.2  Innovator capacity index 

Assessment outcome 

Correlations  within  and  across  dimensions  are  presented  in  Table  5.  We  discuss  the 
correlation  matrices  for  each  dimension  of  innovator  ability  and  innovator  environment 
and conclude with the analyses of the correlations across these two dimensions and the 
Innovator capacity index. 

Innovator ability 

We observe relatively low levels of correlations across indicators in the  innovator ability 
dimension.  Correlations  across  indicators  are  below  0.2,  which  leads  to  a  very  skewed 
contribution  of  indicators  to  the  aggregated  dimension.  Only  the  indicators  of  Most 
impressive  partner  and  Owner  of  the  innovation  contribute  significantly  to  the 
innovator ability dimension, while the impact of the other indicators is relatively low. 

Innovator environment 

The  indicators  of  Project  performance  and  Commitment  to  innovate  are  relatively 
strongly  correlated,  while  correlation  with  the  End-user  engagement  is  lower  to 
inexistent.  However,  all  indicators  seem  to  contribute  to  the  aggregated  measure  of 
innovator environment. The correlation between the dimension of innovator environment 
and the indicator Commitment to innovate is so high that only using that single indicator 
as measure for the innovator environment would yield a similar result. 

Correlations across the Innovator capacity index and its two dimensions 

Analysing  the  correlations  across  the  Innovator  capacity  index  and  its  two  dimensions, 
we  find  considerably  strong  linear  relationships  between  the  two  dimensions 
and  the  index.  This  suggests  that  the  two  dimensions  provide  meaningful information 
on  the  variation  of  the  index  score.  The  contributions  of  the  innovator  ability  and 
innovator environment are relatively balanced, with a slightly higher contribution of the 
latter dimension. The higher  contribution of innovator  environment is  mainly caused by 
the very imbalanced structure  of the innovator ability dimension that seems to regroup 
indicators that do not statistically relate to each other. 

27 

 
 
 
 
 
 
 
 
Table 5: Correlations within and across dimension and the Innovator capacity index 

Calculations: European Commission JRC 
Data: European Commission DG Connect 
Note: Correlations between indicators and pillars or  pillars and the innovation potential index are indicated in 
bold.  All  correlations  are  significant,  except  for  those  indicated  in  red  that  represent  correlations  with  a 
significance level below 5 percent. 

Recommendations 

We  have  the  following  recommendations  for  each  dimension  of  the  Innovator  capacity 
index: 

Innovator ability 

  Consider collection of other indicators of innovator's ability that fit better together 

from a statistical point of view. 

Innovator environment 

  Consider  collection  of  other  indicators  of  innovator's  environment  that  fit  better 

together from a statistical point of view. 

28 

123451Innovator ability1.002Number of times key organisation0.381.003Innovation potential index0.250.121.004Most impressive partner0.800.200.051.005Owner of innovation0.79-0.030.150.251.006Needs of organisation0.33-0.02-0.08-0.14-0.011231Innovator environment1.002End-user engagement0.751.003Project performance0.79-0.031.004Commitment to innovate0.910.260.58Dimension and indicatorsDimension and indicatorsCorrelations within dimensions121Innovator capacity index1.002Innovator ability0.651.003Innovator environment0.860.14Index and dimensionsCorrelations across pillars and index 
 
 
 
 
 
 
 
4.4.2  Principal component analyses 

Concept 

Principal  component  analysis  (PCA)  is  a  statistical  procedure  to  reveal  the  internal 
structure of the data in a way that best explains the variance in the data. PCA performs 
an  orthogonal  transformation  to  convert  the  different  sets  of  correlated  indicators  into 
linearly  uncorrelated  indicators.  In  layman's  words,  principal  component  analysis 
provides insights about the underlying structure of the data in each  dimension 
and  identify  which  indicators  statistically  belong  to  each  other.2  Ideally,  all 
indicators that have been categorised in one dimension based on theoretical/conceptual 
arguments, should show a similar structure from a statistical point of view. In this ideal 
case,  PCA  would  find  only  one  statistical  structure  per  dimension,  which  would  suggest 
that all the indicators included in one dimension are relatively highly correlated with each 
other and have similar statistical patterns. In more general terms, this would mean that 
the conceptual framework constructed on theoretical groundings would coincide with the 
statistical  structure  of  the  underlying  data.  This  is  needed  to  have  a  statistical 
justification to aggregate the data as outlined in the conceptual framework.  

Given  the  relatively  low  correlations  found  in  previous  section,  it  is  expected  that  the 
PCA will reveal more than one structure per dimension. In a sense, this is not surprising 
given  the  complex  nature  of  the  innovation  process  that  contains  many  different  steps 
that do not necessarily relate to each other. Nevertheless, it is important to analyse the 
data structure found by the PCA to see if it makes sense from a theoretical perspective 
as it can then be used to further improve the conceptual framework of the inidces of the 
Innovation Radar. 

To summarise, conducting a PCA is relevant for two reasons: 

  To provide statistical confirmation of the conceptual framework; 
  To  provide  new  insights  on  data  structures  that  can  be  used  to  revise  the 

conceptual framework. 

In  the  following  sections  we  present  that  the  results  of  the  PCA  for  the  Innovation 
potential index and the Innovator capacity index. 

4.4.2.1  Innovation potential index 

Assessment outcome 

Table  6  presents  the  different  structures  obtained  after  PCA  on  each  dimension.  The 
different  structures  are  presented  in  the  columns  and  the  red  values  indicate  which 
indicators  belong  to  the  respective  structures.3  Below  we  discuss  in  more  detail  the 
different structures that have been found for each dimension of the Innovation potential 
index.  

Market potential 

For  the  market  potential,  PCA  identifies  four  statistical  structures  that  respectively 
contain the following indicators: 

  Market maturity and market dynamics; 
  Market competition; 

2 In this report we only highlight the intuition behind PCA without going into detail concerning the 
mathematical  calculations  of  principal  component  analyses.  For  more  detailed  discussions  about 
this particular method, we refer to studies of OECD-JRC (2008) and Jolliffe (1986). 

3 A threshold value of 0.45 (absolute value) on the principal component loadings has been used to 
allocate indicators to their specific structure. These values are highlighted in red in the tables. 

29 

 
 
 
 
                                           
  Type of innovation and Innovation exploitation stage; 
  Number of patents. 

This result highlights that the indicators of the market potential capture a wide range of 
distinct phenomena.  

The  first  structure  identifies  indicators  that  relate  to  market  conditions.  Market 
maturity  and  market  dynamics  are  market-related  characteristics  that  are  important  to 
determine the market orientation and market potential of an innovation, but they seem 
not to relate the other indicators in this dimension. 

The  second  structure  identifies  market  competition  as  a  single  indicator.  This  is  not 
surprising  given  the  very  low  –  and  even  negative  –  correlation  that  this  indicator  has 
with  all  the  other  indicators  in  this  dimension.  Market  competition  acts  as  a  silent 
indicator,  meaning  that  its  inclusion  can  be  important  from  a  conceptual  point  of  view, 
but statistically it does not contribute to the market potential dimension.  

The third structure identifies indicators that relate to the  technology of the innovation. 
It  includes  indicators  on  the  type  of  innovation  and  its  exploitation  stage.  The  PCA 
outcome  is  however  not  clear-cut  about  the  level  of  innovation,  which  is  theoretically 
also  a  technological-related  aspect.  Statistically  that  indicator  does  not  seem  to  be 
categorised  in  any  particular  structure,  but  according  to  the  PCA  it  tend  to  fit  better  in 
the fourth structure. 

The  fourth  structure  contains  the  indicator  on  number  of  patents  which  provides  a 
measure of the patentability of the innovation. Hence, it is not surprising that the level 
of innovation seems to fit best in this structure as both aspects are undeniably related. A 
more  innovative  invention  that  satisfies  a  well-known  market  need  is  probably  more 
patented.  

To summarise, the PCA of the market potential highlights both market and technology-
related  aspects  of  innovations  and  reveals  that  indicators  in  each  of  these  dimensions 
relate to each other but that both aspects are distinct phenomena. This is in line with the 
scientific  literature  that  identifies  market  and  technology  as  two  of  the  most  relevant 
factors in the innovation process (Balachandra and Friar, 1997; Astebro, 2004). 

The finding of the PCA for the market potential has two important implications: 

 

It  provides  reliability  for  the  indicators  that  are  included  in  the  dimension  of 
market  potential  as  indicators  that  are  theoretically  related  seem  also  to  be 
statistically related; 

  The  distinction  between  market  and  technology  related  characteristics  in  the 
market  potential  dimension  should  be  further  emphasized  in  the  conceptual 
framework. 

Innovation readiness 

For the innovation readiness, PCA identifies three  statistical structures that respectively 
contain the following indicators: 

Innovation development stage and time to market; 

 
  Feasibility study and No workforce skills issues; 
  Launch a startup or spin-off and Other. 

The  first  structure  identifies  indicators  that  relate  to  commercialisation.  It  relates  to 
the  overall  development  stage  of  an  innovation  and  the  timing  to  market.  This  reveals 
consistency  in  the  underlying  data  as  an  innovation  that  is  more  advanced  in  its 
development stage should generally exhibit a shorter time to commercialisation.  

The second structure identifies indicators that relate to the feasibility of an innovation. 
It  identifies  both  Feasibility  study  and  No  workforce  skills  issues  in  the  same  latent 

30 

 
 
 
structure.  This  is  justified  from  the  fact  that  the  feasibility  of  an  innovation  is  directly 
affected by the lack of appropriate workforce skills in the project consortium. Hence, also 
this structure provides evidence for the consistency of responses to the questionnaires. 

The third structure is less comprehensible and hence we label it as other. PCA regroups 
the  indicators  of  Launch  a  startup  or  spin-off  and  Other  innovation  step  in  one  latent 
structure. This may mean that both indicator share a statistical pattern. However, so far, 
the  answers  of  the  “Other  innovation  steps”  indicator  have  not  been  explored  in  detail. 
Text-mining analyses on this indicator could shed more light on the type of answers that 
it  contains  and  could  potentially  unravel  correlation  patterns  with  the  indicator  of 
startup/spinoff launch. 

Other  innovation  steps  inserted  in  this  dimension  –  i.e.  technology  transfer,  pilot, 
demonstration  and  prototyping  –  are  not  allocated  to  any  particular  structure.  This 
means that statistically all these innovation steps appear as being distinct aspects of the 
innovation process that do not relate to each other. To better assess the reliability of the 
data  for  all  these  innovation  steps  the  PCA  of  this  dimension  should  be  complemented 
with an analysis of the internal consistency (which is done in next section). 

Innovation management 

For the innovation readiness, PCA identifies three  statistical structures that respectively 
contain the following indicators: 

  Business plan, Market study and End-user engagement; 
  Application funding, Secure private investment, Secure public investment; 
  Clear owner, No IPR issues. 

The first structure identifies indicators that relate to the business proposal. It contains 
the indicators of market study, business plan and user-engagement.  

The  second  structure  identifies  indicators  that  relate  to  the  financial  funding  of 
innovations. It regroups all the indicators that measure applications and actual attraction 
of  financial  investments  from  public  or  private  sources  that  are  needed  to  develop  an 
innovation.  

The final structure identifies indicators that relate to aspects concerning ownership. The 
fact that the indicator of clear ownership and no apparent IPR issues in the consortium is 
identified  to  be  in  one  structure  is  not  surprising  but  at  the  same  time  identifies  a 
weakness  of  the  conceptual  framework.  Even  if  both  indicators  are  measured  at  a 
different level (innovation versus project), innovations with a clear ownership may be in 
projects  where  there  are  no  IPR  issues  in  the  research  consortium.  The  scoring  system 
may penalise projects with only multiple owners. 

31 

 
 
 
 
Table 6: Statistical structure within the dimensions of the Innovation potential index  

Calculations: European Commission JRC 
Data: European Commission DG Connect 
Note: This table presents component loadings of a polychoric principal component analysis conducted on each 
pillar. Loadings greater than 0.45 (absolute values) are highlighted in red. Varimax rotation has been applied. 

Recommendations 

In  general  the  results  of  the  principal  component  analysis  confirm  the  findings  of  the 
correlation  analysis.  Hence,  similar  recommendations  apply  for  the  PCA.  Based  on  the 
PCA  results,  following  additional  recommendations  can  be  made  for  the  following 
dimensions of the Innovation potential index: 

32 

Market conditionsMarket competitionTechnologyPatentabilityType of innovation0.040.330.66-0.03Innovation exploitation stage-0.09-0.300.71-0.04Market maturity0.52-0.440.00-0.21Market dynamics0.750.13-0.060.00Level of Innovation0.380.080.240.41Market competition0.080.76-0.03-0.11Number of patents-0.04-0.06-0.050.88Explained variance1.461.321.241.08Cumulative0.210.400.580.73CommercialisationFeasibilityOtherInnovation development stage0.56-0.13-0.06Technology transfer0.270.210.07Prototyping - real world0.320.290.12Pilot, demonstration0.310.290.15Feasibility study0.080.540.00Launch a startup or spin-off0.100.020.60Other-0.11-0.080.75Time to market0.58-0.14-0.12No workforce skills issues-0.210.67-0.12Explained variance2.521.641.61Cumulative0.280.460.64Business proposalFundingOwnershipClear owner0.02-0.080.62Research engagement0.430.04-0.02Business plan0.490.07-0.04Market study0.480.07-0.02Application funding0.010.550.00Secure priv. Investment0.050.540.04Secure pub. Investment-0.030.58-0.05No IPR issues-0.050.050.72End-user engagement0.46-0.22-0.10Commitment to innovate0.360.010.29Explained variance2.812.721.26Cumulative0.280.550.68Market potentialInnovation readinessInnovation management 
 
 
 
Market potential 

Based  on  the  statistical  structure  found  in  the  PCA,  consider  creating  three  sub-
dimensions of market potential, including: 

  Market conditions (market maturity, market dynamics); 
  Technology (type of innovation and innovation exploitation stage); 
  Market orientation (level of innovation). 

As previous recommendations on the market potential suggested exclusion of a couple of 
indicators,  this  dimension  may  benefit  from  the  inclusion  of  indicators  related  to 
bottlenecks  of  innovation.  As  such,  this  dimension  would  not  only  account  for  positive 
indicators  towards  commercialisation  but  would  also  account  for  phenomena  that 
hamper the innovation process. Inclusion of the following indicators could be considered: 

  Bottlenecks to innovation such as standardisation, trade and regulation. 

Innovation readiness 

Given the fact that Other innovation steps and Launch a startup/spin-off are statistically 
grouped together by the PCA, text-mining analyses on Other innovation steps could shed 
more  light  on  the  type  of  answers  that  it  contains  and  could  potentially  unravel 
correlation patterns with the indicator of startup/spinoff launch. 

Innovation management 

There are no additional recommendations for innovation management. 

4.4.2.2  Innovator capacity index 

Assessment outcome 

Table  7  presents  the  different  structures  obtained  after  PCA  on  each  dimension.  The 
different  structures  are  presented  in  the  columns  and  the  red  values  indicate  which 
indicators  belong  to  the  respective  structures.4  Below  we  discuss  in  more  detail  the 
different structures that have been found for each dimension of the Innovation potential 
index.  

Innovator ability 

For  the  innovator  ability,  PCA  identifies  three  statistical  structures  that  respectively 
contain the following indicators: 

  Number of times key organisation; 
  Owner of innovation; 
  Needs of organisation. 

All the structures of this dimension contain only one indicator. In addition, two indicators 
on Most impressive partner and the Innovation potential index do not fit in any of these 
structures. The indicator of Most impressive partner is at the threshold of being included 
together  with  the  Owner  of  innovation,  which is  in line  with  the  correlation  analyses  as 
both  indicator  recorded  the  highest  correlation  in  this  dimension.  However,  overall  the 
PCA  reveals  that  none  of  the  indicators  in  the  innovator  ability  dimension  are 
related to each other from a statistical perspective.   

4 A threshold value of 0.45 (absolute value) on the principal component loadings has been used to 
allocate indicators to their specific structure. These values are highlighted in red in the tables. 

33 

 
 
 
 
 
                                           
Innovator environment 

For the innovator environment, PCA identifies two statistical structures that respectively 
contain the following indicators: 

  End-user engagement; 
  Project performance and Commitment to innovate.  

In  line  with  the  correlation  analysis  Project  performance  and  Commitment  to  innovate 
are grouped together in one structure. These are also the two most influential indicators 
in the dimension and explain a large part of the variance of the aggregated measure of 
innovator environment. 

Recommendations 

The  results  of  the  principal  component  analysis  confirm  the  findings  of  the  correlation 
analysis. Hence, following recommendations apply for the Innovator capacity index: 

Innovator ability 

  Consider collection of other indicators of innovator's ability that fit better together 

from a statistical point of view. 

Innovator environment 

  Consider  collection  of  other  indicators  of  innovator's  environment  that  fit  better 

together from a statistical point of view. 

Table 7: Statistical structure within the dimensions of the Innovator capacity index 

Calculations: European Commission JRC 
Data: European Commission DG Connect 
Note: This table presents component loadings of a polychoric principal component analysis conducted on each 
pillar. Loadings greater than 0.45 (absolute values) are highlighted in red. Varimax rotation has been applied. 

34 

OwnershipKey organisationInnovation needsNumber of times key organisation-0.120.870.03Innovation potential index0.340.380.12Most impressive partner0.440.27-0.35Owner of innovation0.82-0.170.09Needs of organisation0.050.040.93Explained variance1.241.141.05Cumulative0.240.470.68CommitmentEnd-user engagementEnd-user engagement-0.010.95Project performance0.74-0.20Commitment to innovate0.670.23Explained variance1.571.07Cumulative0.520.88Innovator abilityInnovator environment 
 
 
 
 
 
 
 
4.4.3  Internal consistency 

Concept 

In this section we measure the internal consistency of the various indicators included in 
each dimension. This is typically measured with the Cronbach Alpha Coefficient which is 
a  measure  of  reliability  that  indicators  that  propose  to  measure  a  similar 
concept  also  provide  similar  scores.5  A  high  Cronbach  Alpha  Coefficient  indicates 
that  the  indicators  of  a  dimension  are  measuring  the  same  underlying  construct. 
Important  to  keep  in  mind  is  that  the  Cronbach  Alpha  Coefficient  should  not  be  strictly 
interpreted  as  a  measure  of  uni-dimensionality.  In  this  respect,  the  Handbook  to 
construct composite indicators mentions that "(…) a set of individual indicators can have 
a  high  alpha  and  still  be  multi-dimensional.  This  happens  when  there  are  separate 
clusters  of  individual  indicators  (separate  dimensions)  which  intercorrelate  highly,  even 
though the clusters themselves are not highly correlated (…)" (OECD and JRC, 2008). 

Many  scholars  have  debated  on  how  large  the  Cronbach  Alpha  Coefficient  should  be  to 
be  acceptable.  According  to  Nunnally  (1978)  and  Hair  et  al.  (1998),  the  generally 
accepted  lower  limit  for  Cronbach’s  alpha  is  0.7,  although  this  may  decrease  to  0.6  in 
exploratory research. Below, we evaluate the internal consistency in both indices of the 
Innovation Radar. 

4.4.3.1  Innovation potential index 

Assessment outcome 

Market potential 

The Cronbach Alpha Coefficient for the market potential dimension is 0.08, which is very  
poor.  This  reflects  the  results  of  the  principal  component  analysis  and  the  correlation 
matrix  of  this  dimension.  Most  indicators  in  this  pillar  capture  different  phenomena, 
including  technological  and  market  related  characteristics  that  are  important  for  the 
development and commercialisation of innovations. 

Innovation readiness 

In contrast to market potential, the Cronbach Alpha Coefficient for innovation readiness 
is close to the acceptable reliability threshold, elevating at 0.66. When looking how the 
value of the Cronbach Alpha Coefficient changes after deleting one individual indicator at 
a time, we observe that the coefficient would decrease in most of the cases. This means 
that  almost  all  indicators  contribute  to  enhance  the  internal  consistency  of 
innovation  readiness. The only exception is the indicator capturing no workforce  skill 
issues,  where  deletion  of  this  indicator  would  increase  the  internal  consistency  of  the 
dimension.  Based  on  this  observation  and  in  line  with  the  recommendations  from  the 
correlation analysis, exclusion of this indicator could be considered.  

Even if the results of PCA in previous section may suggest that there is limited internal 
consistency  in  this  dimension  as  many  indicators  of  the  innovation  steps  are  not 
categorised  in  a  particular  structure,  a  more  detailed  investigation  is  needed.  To  gain 
further insights on the internal consistency of the innovation readiness, we analyse the 
number  of  innovation  steps  that  have  been  undertaken  and  compare  them 
across  the  different  development  stages  of  an  innovation.  Hence,  we  combine 
information  from  the  first  indicator  of  this  dimension  with  all  the  indicators  measuring 
innovation steps towards innovation readiness. We do this to measure the consistency in 
respondents'  replies  and  to  ensure  that  the  conceptual  framework  is  in  line  with  the 
underlying data.  

5  We  refer  to  studies  of  Cronbach  (1951)  and  Streiner  (2003)    for  more  details  about  the  mathematical 
construction of this coefficient. 

35 

 
 
 
 
                                           
We  proceed  in  the  following  way.  First,  we  count  the  number  of  innovation  steps  that 
have been undertaken per innovation and regroup them in three ordinal categories (low, 
medium  and high).  Then  we  calculate  the  frequencies  of  these  three  groups  across  the 
different development stages. Figure 11 presents the percentages of the three categories 
of  innovations  steps  per  development  stage.  In  line  with  the  expectations,  we  observe 
that  the  majority  of  innovations  that  are  still  under  development  have 
undertaken a limited number of innovation steps, as the share of the lowest group 
of  innovation  steps  is  the  highest.  Analysing  the  innovations  that  have  been  developed 
and are being exploited, we observe that the highest percentages gradually shift towards 
groups  with  more  innovation  steps.  These  results  provide  important  evidence  for  the 
consistency of reviewers’ replies to the questionnaire with respect to indicators related to 
the innovation readiness. 

Figure 11: Number of innovation steps across innovation development stages  

Calculations: European Commission JRC 
Data: European Commission DG Connect 
Note: The number of innovation steps in this figure is based on the following indicators: technology transfer, 
prototyping  –  real  world,  pilot,  demonstration,  feasibility  study,  launch  a  startup  or  spinoff  and  other.  The 
scores  of  the  indicators  have  been  summed  up  and  grouped  in  three  categories:  low  (score  0-1.5),  medium 
(score  2-3.5)  and  high  (score  4-6).  The  figure  presents  percentages  of  these  categories  across  different 
innovation development stages. 

Similarly, we analyse the time needed to bring an innovation on the market and 
compare  it  across  the  different  development  stages  of  an  innovation.  As  both 
indicators  (i.e.  innovation  development  stage  and  time  to  market)  aim  to  capture  a 
similar  latent  construct  –  namely  innovation  readiness  –  we  expect  them  to  follow  a 
similar  pattern.  In  particular,  innovations  that  are  exploited  should  be  close  to 
commercialisation  and  hence  report  a  shorter  time  to  market,  while  the  opposite  is 
expected  for  innovation  that  are  still  in  the  development  stage.  Figure  12  presents  the 
frequency distribution of the time to market across the different development stages of 
innovations and confirms our expectations.   

36 

 
 
 
 
 
 
Figure 12: Time to market across innovation development stages  

Calculations: European Commission JRC 
Data: European Commission DG Connect 
Note:  The  figure  presents  frequency  distributions  of  time  to  market  across  different  innovation  development 
stages. Time to market is grouped in three categories that represent the time needed to bring an innovation on 
the market: 3 or more years, between 1 and 2 years and less than 1 year. 

Innovation management 

The Cronbach Alpha Coefficient of the innovation management is also relatively close to 
the acceptable threshold and elevates at 0.63. Similar to the previous pillar, exclusion of 
individual  indicators  would  yield  the  coefficient  to  decrease,  which  means  that  almost 
all indicators contribute to the internal consistency of innovation management. 
Only two indicators have a positive impact on the Cronbach Alpha when being excluded: 
clear  owner  and  no  IPR  issues.  This  result  is  in  line  with  the  observations  from  the 
correlation  matrix  and  the  principal  component  analysis.  Both  indicator  seem  not  to 
belong to this dimension and could be considered to be excluded. Exclusion of the clear 
owner indicator would for instance increase the Cronbach Alpha Coefficient up to 0.68.  

To  gain  further  insights  on  the  internal  consistency  of  the  innovation  management,  we 
analyse  the  number  of  innovation  steps  that  have  been  undertaken  and 
compare them across various levels of commitment of the relevant partners to 
exploit  their  innovation.  Hence,  we  combine  information  from  the  indicator 
'commitment  to  innovate'  with  all  the  pillar  indicators  measuring  innovation  steps  that 
rely  on  an  effective  innovation  management.  Similar  to  the  previous  dimension,  we  do 
this  to  measure  consistency  in  respondents'  replies  and  to  ensure  that  the  conceptual 
framework of this dimension is in line with the underlying data. We calculate the number 
of  innovation  steps  in  this  pillar  that  have  been  undertaken  and  regroup  them  in  three 
categories  (low,  medium  and  high).  Figure  13  presents  the  percentages  of  each  group 
for  different  levels  of  partner  commitment  to  exploit  an  innovation.  The  figure  shows 
that  innovations  with  a  research  consortium  that  is  more  committed  to  exploit  an 
innovation  has  been  undertaking  more  innovation  steps  in  terms  of  business 
propositions, fund raising and research engagement.  

37 

 
 
 
 
 
Figure 13: Number of innovation steps across partner commitment 

Calculations: European Commission JRC 
Data: European Commission DG Connect 
Note: The number of innovation steps in this figure is based on the following indicators: research engagement, 
business  plan,  market  study,  application  funding,  secure  private  and  public  investment.  The  scores  of  the 
indicators  have  been  summed  up  and  grouped  in  three  categories:  low  (score  0-1.5),  medium  (score  2-3.5) 
and  high  (score  4-6).  The  figure  presents  percentages  of  these  categories  across  different  levels  of  partner 
commitment to exploit an innovation. 

Recommendations 

We have the following recommendations for each dimension of the Innovation potential 
index: 

Market potential 

  Similar  recommendations  apply  as  in  the  correlation  and  principal  component 

analysis. 

Innovation management 

 

Internal consistency analysis provides evidence of the reliability of the answers of 
the questionnaire, which enhances the validity of the Innovation potential index. 

Innovation readiness 

 

Internal consistency analysis provides evidence of the reliability of the answers of 
the questionnaire, which enhances the validity of the Innovation potential index. 

38 

 
 
 
 
 
 
4.4.3.2  Innovator capacity index 

Assessment outcome 

Innovator ability 

The Cronbach Alpha Coefficient for the innovator ability dimension is 0.24, which is very   
poor.  This  reflects  the  results  of  the  principal  component  analysis  and  the  correlation 
matrix of this dimension.  

Innovator environment 

The Cronbach Alpha Coefficient for the innovator ability dimension is 0.42, which is also 
relatively poor. This result reflects the fact that this dimension contains three indicators 
from which only two fit well together from a statistical perspective. Exclusion of the end-
user engagement indicator would increase the internal consistency of this dimension. 

Recommendations 

The  results  of  the  internal  consistency  analysis  confirm  the  findings  of  the  correlation 
and  principal  component  analysis.  Hence,  following  recommendations  apply  for  the 
Innovator capacity index: 

Innovator ability 

  Consider collection of other indicators of innovator's ability that fit better together 

from a statistical point of view. 

Innovator environment 

  Consider  collection  of  other  indicators  of  innovator's  environment  that  fit  better 

together from a statistical point of view. 

39 

 
 
 
 
 
 
5  Output: assessment of the final indices 

In  this  section  we  assess  the  output  of  the  current  version  of  the  Innovation  potential 
index and Innovator  capacity index.  In particular, we assess to  what extent the indices 
show biases towards certain types of innovations or types of research collaborations.  

5.1  Innovation potential index across innovation types 

Concept 

Innovation  is  a  complex  and  uncertain  process  that  involves  a  wide  range  of 
stakeholders. Most innovations are messy and the innovation process is characterised by 
feed-back  loops,  dead-ends  and  dynamic  interactions.  Simple  and  linear  innovation 
models have the advantages to be conceptually easy to understand but lack the capacity 
to  draw  attention  on  the  complex  ways  in  which  innovations  actually  evolve  over  time. 
The  Innovation  Radar  methodology  aims  to  approach  the  innovation  process  from  a 
holistic  point  of  view  and  attempts  to  synthesize  the  technological,  organisational  and 
commercial  aspects  of  the  innovation  process.  We  refer  to  Tidd  et  al.  (2005)  for  a 
detailed overview of the characteristics of innovation models and their evolvement over 
time. 

One  of  the  problems  of  holistic  approaches  of  innovation  models  as  the 
Innovation  Radar  is  that  they  may  not  be  suitable  of  all  types  of  innovations. 
The  various  innovation  steps  as  included  in  the  innovation  readiness  and  innovation 
management  pillars  of  the  Innovation  Radar  may  be  more  relevant  for  product 
innovations  than  for  other  types  of  innovations,  such  as  new  services  or  processes  and 
organisational/marketing methods. 

Assessment outcome 

To control for a potential bias of the innovation potential across innovation types, 
Figure  14  presents  the  distribution  of  the  innovation  potential  index  across  different 
types of innovations. The figure shows that: 

  The  distribution  and  mean  values  of  the  innovation  potential  for  product  and 

service innovations are similar; 

  The 

average 

innovation 

potential 

of 

process 

innovations 

and 

marketing/organisational methods is systematically lower. 

Recommendation 

The  actual  version  of  the  Innovation  potential  index  is  strongly  based  on  innovation 
models  for  product  development.  It  may  not  be  optimal  to  evaluate  the  innovation 
process  of  other  innovation  types  such  as  process  and  marketing/organisational 
methods. 

  Revision of the conceptual framework and adjustment of the questionnaire could 
be considered to account for differences in innovation processes across innovation 
types. 

40 

 
 
 
 
 
 
 
Figure 14: Distribution of the Innovation potential index across innovation types 

Calculations: European Commission JRC 
Data: European Commission DG Connect 
Note:  The  figure  presents  the  distribution  of  the  innovation  potential  index  across  different  types  of 
innovations.  The  different  innovation  types  are  defined  in  the  following  way:  1)  Marketing/organisational 
method  includes  both  new  and  significantly  improved  methods,  2)  Service  innovation  and  others:  new  and 
significantly  improved  services,  consulting  services  and  others,  3)  Product  innovations:  new  and  significantly 
improved products, 4) Process innovations: new and significantly improved process innovations. The box plots 
present the quartiles of the distribution (25% - 50% and 75%) while the reference lines represents the mean. 

5.2  Innovation potential index across research partners 

Concept 

The  Innovation  Radar  aims  to  capture  those  innovations  that  have  the  potential  to  be 
brought  on  the  market  in  the  near  future.  As  the  various  partners  of  a  research 
consortium  may  follow  different  trajectories  towards  commercialisation,  this  may  be 
reflected on their innovation potential. The innovation potential of innovations from 
consortia  with  private  partners  (firms)  may  be  higher  than  those  with  only 
public partners (universities/research centers) for the following reasons.  

Firms may have a strong strategic alignment with FP projects and explicit goals related 
to  innovation  outputs  such  as  developing  a  prototype,  a  patentable  technology,  or  a 
complementary  technology  that  will  directly  enhance  their  competitiveness.  They  focus 
on projects with an applied orientation and engage only in cooperative agreements that 
are likely to yield tangible benefits and guarantee their immediate survival and growth. 
In  this  sense,  the  innovation  process  as  measured  by  the  Innovation  potential  index 
follows well the various steps that private partners would undertake in the development 
of an innovation. 

Universities and public research centres, on the other hand, may primarily participate to 
FP  projects  to  advance  their  research  and  may  follow  a  different  development  path 
towards  innovation  that  is  not  accounted  for  by  the  actual  version  of  the  Innovation 
potential index. 

41 

Product innovationProcess innovationService innovation and othersMarketing/organisational methodInnovation type20406080100Innovation potential index 
 
 
 
Related to this issue, the role of organisational diversity on the innovation potential has 
been analysed by Nepelski and Piroli  (2017) and Nepelski et al. (2018) in other studies 
related to the Innovation Radar. 

Assessment outcome 

Figure 15 presents the distribution of the Innovation potential index across collaboration 
types,  accounting  for  collaborations  that  include  only  private,  only  public  or  public  and 
private partners. The figure shows that: 

 

Innovations  with  only  public  research  partners  score  systematically  less  on 
innovation potential than innovations from consortia including private partners. 

This  result  may  be  caused  by  the  fact  that  projects  including  only  public  key 
organisations are penalised by the actual scoring system in case they following different 
paths study to develop an innovation. 

Figure 15: Distribution of the Innovation potential index across collaboration types 

Calculations: European Commission JRC 
Data: European Commission DG Connect 
Note: The figure presents the distribution of the innovation potential index across different collaboration types. 
The different collaboration types are defined in the following way: 1) private only: innovations with only firms 
as  key  organisations,  2)  public  only:  innovations  with  only  universities,  research  centres,  governmental 
institutions or other types as key organisations, 3) public and private: innovations with a combination of public 
and  private key organisations.  The  box  plots  present  the  quartiles  of  the distribution  (25%  - 50%  and 75%) 
while the reference lines represents the mean. 

Recommendation 

The  conceptual  framework  to  measure  the  innovation  potential  of  FP  projects  could  be 
adjusted  to  account  for  different  innovation  development  paths  of  public  organisations 
such as universities, research centers or governmental institutions. 

42 

Private onlyPublic onlyPublic and privateCollaboration types20406080100Innovation potential index 
 
 
 
 
 
5.3  Innovator capacity index across organisation types 

Concept 

Similarly  as  for  the  Innovation  potential  index,  we  test  to  what  extent  the  Innovator 
capacity index varies across organisation types. The innovator capacity of SMEs may 
be higher than the one of other organisation types and large firms in particular 
for the following reason.  

It  may  be  due  that  SMEs  benefit  from  the  advantage  of  being  more  selected  as  most 
impressive partner as this question excludes large firms. 

Assessment outcome 

Figure  15  presents  the  distribution  of  the  Innovator  capacity  index  across  organisation 
types,  accounting  for  universities,  SMEs,  large  firms  and  other  organisations  (i.e. 
governmental institutions, research centers and others). The figure shows that: 

  On  average,  SMEs  have  the  highest  innovator  capacity,  while  large  firms  are 

lagging behind 

Figure 16: Distribution of the Innovator capacity index across organisation types 

Calculations: European Commission JRC 
Data: European Commission DG Connect 
Note: The figure presents the distribution of the innovator capacity index across different collaboration types. 
The  box  plots  present  the  quartiles  of  the  distribution  (25%  -  50%  and  75%)  while  the  reference  lines 
represents the mean. 

This  result  may  be  caused  by  the  fact  that  SMEs  scores  systematically  higher  as  most 
impressive partner compared to large firms. However, when observing the means of all 
the  indicators  included  in  the  Innovator  capacity  index,  it  seems  that  SMEs  are  on 
average  scoring  higher  on  all  the indicators  compared  to  large  firms.  This  may  suggest 

43 

UniversitiesLarge firmsSMEsResearch centra/governmentOrganisation types020406080100Innovator capacity index 
 
 
 
 
 
that  the  difference  of  the  Innovator  capacity  index  between  SMEs  and  large  firms  may 
remain even when the question about most impressive partner is not taken into account. 

To  evaluate  this  proposition,  we  calculated  a  revised  version  of  the  Innovator  capacity 
index  without  the  question  about  the  most  impressive  partner  and  plotted  the 
distribution  of  this  revised  index  across  organisation  types  in  Figure  17.  The  difference 
between SMEs and large firms remain, but is however less pronounced. 

Figure 17: Distribution of the revised Innovator capacity index across organisation types 

Calculations: European Commission JRC 
Data: European Commission DG Connect 
Note: The figure presents the distribution of the innovator capacity index across different collaboration types. 
The  box  plots  present  the  quartiles  of  the  distribution  (25%  -  50%  and  75%)  while  the  reference  lines 
represents the mean. 

Recommendation 

The  exclusion  of  large  firms  as  most  impressive  partners  in  that  particular  question  of 
the questionnaire seems to accentuate difference of the Innovator capacity index across 
SMEs and large firms. However, even after exclusion of that particular indicator from the 
Innovator  capacity  index,  a  difference  between  large  firms  and  SMEs  –  although  less 
pronounced – seems to remain. This seems to reveal that SMEs are the innovators with 
the strongest innovators' capacity. 

It  is  recommended  to  leave  the  question  open  to  all  organisation  types  in  order  to  see 
whether  SMEs  would  really  be  pointed  as  Most  impressive  partner.  At  least  it  would 
lower the probability of a biased answer and would yield a stronger result if more SMEs 
are chosen as Most impressive. 

44 

UniversitiesLarge firmsSMEsResearch centra/governmentOrganisation types20406080100Revised Innovator capacity index 
 
 
 
 
 
 
 
6  Synthesis of the assessment 

In this section we provide some tables that summarise the results of the assessment of 
the Innovation Radar presented in this report.  

The summary tables follow the structure of the report and are grouped in the following 
order: 

  Input: relates to the questionnaire and the scoring system that provide the input 

data that feeds the indices of the Innovation Radar (Table 8); 

  Process:  relates  to  the  statistical  process  to  construct  the  indices  of  the 

Innovation Radar (Table 9 to Table 11); 

  Output: relates to the statistical soundness of the final indices of the Innovation 

Radar (Table 12). 

Overall, the main findings of the current report on the validation of the Innovation Radar 
assessment framework can be summarised in the following way: 

Input 

  Questionnaire:  slight  adjustments  could  be  considered  as  to  maximise  a  clear 

alignment of reviewers on how to interpret questions; 

  Scoring system: slight adjustments could be considered as to accentuate project 

differences. 

Process 

 

Innovation potential index: statistically sound; 

o  The  innovation  management  and  innovation  readiness  dimensions  are 

statistically well-balanced and show a good internal consistency; 

o  More  room  for  improvement  is  observed  for  the  market  potential 

dimension. 

 

Innovator capacity index: conceptually sound but can be improved statistically; 
o  The index would benefit from a more balanced contribution of indicators; 
o  Hence, the collection of indicators that fit better together from a statistical 

perspective could be considered. 

Output 

  Adjustments to the conceptual framework of both indices could be considered as 
to account for differences in the innovation process across innovation types and 
research partners. 

45 

 
 
 
 
Table 8: Synthesis table of the input: questionnaire and scoring system 

Note: The table provides a synthesis of the findings when analyzing the statistical coherence of the questionnaire and scoring system behind the Innovation Radar. Data 
used in this assessment is owned by European Commission DG Connect. 

46 

 
 
 
 
 
Table 9: Synthesis table of the process: construction of the composite indicators 

Note: The table provides a synthesis of the findings when analyzing the statistical coherence of the construction method to produce the indices of the Innovation Radar. 
Data used in this assessment is owned by European Commission DG Connect. 

47 

 
 
 
 
 
Table 10: Synthesis table of the process: construction of the composite indicators (cont.) 

Note: The  table provides a synthesis of the findings when analyzing the statistical coherence of  the construction method to produce the indices of the Innovation Radar. 
Data used in this assessment is owned by European Commission DG Connect. 

48 

 
 
 
 
 
Table 11: Synthesis table of the process: construction of the composite indicators (cont.) 

Note: The  table provides a synthesis of the findings when analyzing the statistical coherence of the construction method to produce the indices of the Innovation Radar. 
Data used in this assessment is owned by European Commission DG Connect. 

49 

 
 
 
 
 
Table 12: Synthesis table of the output: assessment of the final indices 

Note:  The  table  provides  a  synthesis  of  the  findings  of  the  quality  assessment  of  the  final  indices  of  the  Innovation  Radar.  Data  used  in  this  assessment  is  owned  by 
European Commission DG Connect. 

50 

 
 
 
 
 
References 

Astebro,  T.,  2004.  Key  success  factors  for  technological  entrepreneurs  R&D  projects. 

IEEE Transactions on Engineering management 51, 381-399. 

Balachandra, R., Friar, J.H., 1997. Factors for success in R&D projects and new product 
IEEE  Transactions  on  Engineering 

framework. 

innovation:  a  contextual 
management 44, 276-287. 

Conceição,  O.,  Fontes,  M.,  Calapez,  T.,  2012.  The  commercialisation  decisions  of 
research-based spin-off: Targeting the market for technologies. Technovation 32, 
43-56. 

Cooper,  R.G.,  2007.  Managing  technology  development  projects.  Research-Technology 

Management 49, 23-31. 

Cooper, R.G., Kleinschmidt, E.J., 1988.  Resource allocation in the new product process. 

Industrial Marketing Management 17, 249-262. 

Cronbach, L.J., 1951. Coefficient alpha and the internal structure of tests. psychometrika 

16, 297-334. 

De  Prato,  G.,  Nepelski,  D.,  Piroli,  G.,  2015.  Innovation  Radar:  Identifying  Innovations 
and  Innovators  with  High  Potential  in  ICT  FP7,  CIP  &  H2020  Projects.  .  JRC 
Scientific and Policy Reports – EUR 27314 EN. Seville: JRC. 

EC,  2014.  Horizon  2020  Work  Programme  2014  -  2015:  Leadership  in  enabling  and 

industrial technologies - Information and communication technologies. 

Hair, J.F., Black, W.C., Babin, B.J., Anderson, R.E., Tatham, R.L., 1998. Multivariate data 

analysis. Prentice hall Upper Saddle River, NJ. 

Heslop,  L.A.,  McGregor,  E.,  Griffith,  M.,  2001.  Development  of  a  technology  readiness 
assessment measure: The cloverleaf model of technology transfer. The Journal of 
Technology Transfer 26, 369-384. 

Jolliffe,  I.T.,  1986.  Principal  Component  Analysis  and  Factor  Analysis,  Principal 

component analysis. Springer, pp. 115-128. 

Linton,  J.D.,  Walsh,  S.T.,  Morabito,  J.,  2002.  Analysis,  ranking  and  selection  of  R&D 

projects in a portfolio. r&D Management 32, 139-148. 

Munda, G., 2008. Social multi-criteria evaluation for a sustainable economy. Springer. 

Nepelski,  D.,  Piroli,  G.,  2017.  Organizational  diversity  and  innovation  potential  of  EU-

funded research projects. Journal of Technology Transfer forthcoming. 

Nepelski, D., Van Roy, V., Pesole, A., 2018. Organisational and geographic diversity and 
innovation  potential  of  EU-funded  research  projects.  Journal  of  Technology 
Transfer Under review. 

Nunnally, J.C., 1978. Psychometric theory. NY: McGraw-Hill. 

OECD,  JRC,  2008.  Handbook  on  Constructing  Composite  Indicators:  Methodology  and 

user guide. OECD Publishing, Paris. 

Pesole,  A.,  Nepelski,  D.,  2016.  Universities  and  collaborative  innovation  in  EC-funded 

research projects: An analysis based on Innovation Radar data. 

Rothwell,  R.,  1992.  Successful  industrial  innovation:  critical  factors  for  the  1990s.  r&D 

Management 22, 221-240. 

Saisana, M., Domínguez-Torreiro, M., Vertesy, D., 2017. Global Innovation Index 2017: 

Innovation Feeding the World - Appendix IV: Technical notes. INSEAD report. 

51 

 
 
 
Saisana, M., Saltelli, A., 2014. Joint Research Centre statistical audit of the 2014 Global 
Innovation  Index,  in:  Cornell  University,  INSEAD,  WIPO  (Eds),  The  Global 
Innovation  Index  2014:  The  human  factor  in  innovation.  Fontainebleau,  Ithaca, 
and Geneva, pp. 55-69. 

Streiner,  D.L.,  2003.  Starting  at  the  beginning:  an introduction  to  coefficient  alpha  and 

internal consistency. Journal of personality assessment 80, 99-103. 

Tidd,  J.,  Bessant,  J.,  Pavitt,  K.,  2005.  Managing  innovation  integrating  technological, 

market and organizational change. John Wiley and Sons Ltd. 

52 

 
 
 
 
 
 
List of figures 

Figure 1: The Innovation potential index and Innovator capacity index ........................ 4 

Figure 2: Methodological steps for the construction of the Innovation Radar ................. 7 

Figure 3: Number of most impressive partner per project ......................................... 10 

Figure 4: Overview of missing data for the dimension of market potential .................. 14 

Figure 5: Overview of missing data for the dimension of innovation readiness ............ 14 

Figure 6: Overview of missing data for the dimension of innovation management ....... 15 

Figure 7: Distribution of the number of missing innovation steps ............................... 16 

Figure 8: Missing data on all innovation steps across innovation types ....................... 17 

Figure 9: Arithmetic versus geometric aggregation for the Innovation potential index .. 20 

Figure 10: Arithmetic versus geometric aggregation for the Innovator capacity index .. 21 

Figure 11: Number of innovation steps across innovation development stages ............ 36 

Figure 12: Time to market across innovation development stages ............................. 37 

Figure 13: Number of innovation steps across partner commitment ........................... 38 

Figure 14: Distribution of the Innovation potential index across innovation types ........ 41 

Figure 15: Distribution of the Innovation potential index across collaboration types ..... 42 

Figure 16: Distribution of the Innovator capacity index across organisation types ........ 43 

Figure 17: Distribution of the revised Innovator capacity index across organisation types
......................................................................................................................... 44 

List of tables 

Table 1: Overview of innovation projects and organisation types ................................. 8 

Table 2: Change of scoring system for partners' commitment ................................... 11 

Table 3: Comparison of aggregation method ........................................................... 22 

Table 4: Correlations within and across dimension and the Innovation potential index . 26 

Table 5: Correlations within and across dimension and the Innovator capacity index ... 28 

Table 6: Statistical structure within the dimensions of the Innovation potential index .. 32 

Table 7: Statistical structure within the dimensions of the Innovator capacity index .... 34 

Table 8: Synthesis table of the input: questionnaire and scoring system .................... 46 

Table 9: Synthesis table of the process: construction of the composite indicators ........ 47 

Table 10: Synthesis table of the process: construction of the composite indicators (cont.)
......................................................................................................................... 48 

Table 11: Synthesis table of the process: construction of the composite indicators (cont.)
......................................................................................................................... 49 

Table 12: Synthesis table of the output: assessment of the final indices ..................... 50 

53 

 
 
 
  
 
 
Appendix 

1. Innovation Radar Questionnaire 

Innovation Radar Questionnaire by EC DG CONNECT 
Note: the first 19 questions below are to be answered for each innovation the project 
develops (up to a maximum of 3 innovations). 

1)  Title of the innovation 

2)  Describe the innovation (in less than 500 characters, spaces included): 

3)  Is the innovation developed within the project…: 

a)  Under development 
b)  Already developed but not yet being exploited 
c)  being exploited  

4)  Characterise the type of innovation  

a)  Significantly improved product  
b)  New product  
c)  Significantly improved service (except consulting ones)  
d)  New service (except consulting ones)  
e)  Significantly improved process  
f)  New process  
g)  Significantly improved marketing method  
h)  New marketing method  
i)  Significantly improved organisational method  
j)  New organisational method  
k)  Consulting services  
l)  Other  

5)  If other, please specify:  

6)  Will the innovation be introduced to the market or deployed within a 

partner: 
a)  Introduced new to the market (commercial exploitation) 
b)  Deployed within a partner (internal exploitation: Changes in organisation, new 

internal processes implemented, etc.)  

c)  No exploitation planned  

7)  If no exploitation planned, please explain why no exploitation is planned 

(answer only if 6(c) is selected) 

8)  Is there a clear owner of the innovation in the consortium or multiple 

owners? 
a)  A clear owner 
b)  Multiple owners 

9)  Indicate who is the "owner" of the innovation. Please use the exact name of 

the project partner as listed on the CORDIS project profile. 

10) 

Indicate the step(s) already done (or are foreseen) in the project in 
order to bring the innovation to (or closer to) the market (answer only if 
6(a) is selected) 

Done 

Planned in 
project 

Not 
Planned 

Desirable 

54 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
1. Technology transfer 
2. Engagement of both research team 
and partner's business units in project 
activities 
3. Business plan 
4. Market study 
5. Prototyping 
6. Pilot, Demonstration or Testing 
activities 
7. Feasibility study 
8. Launch a start-up or spin-off 
9. Standardisation 
10. Application for private or public 
investment 
11. Securing private investment 
12. Securing public investment 
13. Other 

11) 

If other, please specify  

12) 

Indicate which participant(s) (up to a maximum of 3) is/are the key 
organisation(s) in the project delivering this innovation. For each of these 
identify under the next question their needs to fulfil their market potential. 
Please use the exact name(s) of the project partner(s) as listed on the 
CORDIS project profile. 
Filed 1: Organization1:  
Filed 2: Organization 2:  
Filed 3: Organization 3:  

13) 

Indicate their needs to fulfil their market potential 

Investor 
readines
s 
training 

Investor 
introduct
ions 

Biz plan 
develop
ment 

Expandin
g to 
more 
markets 

Legal 
advice 
(IPR or 
other) 

Mentorin
g 

Partners
hip with 
other 
company 
(technolo
gy or 
other) 

Incubatio
n 

Startup 
accelerat
or 

Organi
zation 
1 
Organi
zation 
2 
Organi
zation 
3 

14)  Market size: What is the market size for this innovation 

a)  < €25M 
b)  €25M - €100M 
c)  €100M - €250M 
d)  €250M - €500M 
e)  > €500M 
f)  Not known 

15)  Market maturity: The market for this innovation is… 
a)  Nonexistent: customers are not yet buying such products 
b)  Emerging: There is a growing demand and few offerings are available 

55 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
c)  Mature: The market is already supplied with many products of the type proposed 

16)  Market dynamics: is the market…  

a)  In decline 
b)  Holding steady 
c)  Growing 

17) 

Level of innovation: What is the level of innovation 

a)  No innovation—other factors contribute to viability 
b)  Some distinct, probably minor, improvements over existing products 
c)  Innovative but could be difficult to convert customers 
d)  Obviously innovative and easily appreciated advantages to customer 
e)  Very innovative satisfies a well-known market need 

18)  Market competition: How strong is competition in the target market? 

a)  Patchy, no major players 
b)  Established competition but none with a proposition like the one under 

investigation 

c)  Several major players with strong competencies, infrastructure and offerings 

19)  When do you expect that such innovation could be commercialised? 

(answer only if 6(a) is selected) 

a)  Less than 1 year 
b)  Between 1 and 3 years 
c)  Between 3 and 5 years 
d)  More than 5 years 

General Questions 

(questions below are to be answered once in the project review, not for each innovation) 

1)  How does the consortium engage end-users?  

-  End user organisation in the consortium 
-  An end user organisation outside of the consortium is consulted 
-  No end user organisation in the consortium or consulted 

2)  Are there in the consortium internal IPR issues that could compromise the 
ability of a project partner to exploit new products/solutions/services, 
internally or in the market place?  
yes 
- 
-  no 

3)  Please provide specifics of the IPR issues: 

4)  Which are the external bottlenecks that compromise the ability of project 

IPR  

partners to exploit new products, solutions or services, internally or in the 
market place?   
- 
-  Standards  
-  Regulation  
- 
-  Workforce's skills  
-  Trade issues (between MS, globally)  
-  Others  

Financing  

56 

 
 
 
 
 
 
 
 
 
 
 
5)  If others, please specify:  

6)  Indicate how many patents have been applied for by the project: _________ 
7)  Does the review panel consider the project performance in terms of 

innovation? 
-  Exceeding expectations 
-  Meeting expectations  
- 

Performing below expectations 

8)  General observations of innovation expert on this project's innovation 

performance: 

9)  How would you rate the level of commitment of relevant partners to exploit 

the innovation? 
-  Very low  
- 
Low  
-  Average  
-  High  
-  Very High  
-  None 

10)  Please indicate the 1 partner (excluding large enterprises) that the panel 

considers to be the most impressive in terms of innovation potential: 
11)  Please enter some tag words (comma separated) to represent what 

"innovation elements" are strong in the project:  

12)  Please enter some tag words (comma separated) to represent what 
"innovation elements" can be improved (or are absent) in the project:  

57 

 
 
 
 
 
 
 
 
 
 
 
2.  Scoring  system:  matching  survey  questions  with  assessment 
criteria 

2.1 Innovation potential assessment framework 

Table 1: Innovation potential assessment framework: Market potential 

Question 
code* 
Q4 
b OR d OR f 
a OR c OR e 
h OR j 
g OR i 

Scoring 
Max: 
10 

1 
0.75 
0.5 
0.25 

Criteria & questions 

Market potential 

Type of innovation: 

New product, process or service 
Significantly improved product, process or service 
New marketing or organizational method 
Significantly  improved  marketing  or  organizational 
method 
Consulting services, other 

Innovation exploitation: 

Commercial exploitation 
Internal exploitation 
No exploitation 

Market maturity: The market for this innovation is… 

Nonexistent:  customers  are  not  yet  buying  such 
products 
Emerging:  There  is  a  growing  demand  and  few 
offerings are available 
Mature:  The  market  is  already  supplied  with  many 
products of the type proposed 

Market dynamics: is the market…  

In decline 
Holding steady 

          Growing 
Level of innovation: What is the level of innovation 

No innovation—other factors contribute to viability 
Some  distinct,  probably  minor,  improvements  over 
existing products. 

          Innovative but could be difficult to convert customers. 

Obviously 
innovative 
advantages to customer 
Very innovative satisfies a well-known market need. 

easily 

and 

appreciated    

Market  competition:  How  strong  is  competition  in  the  target 
market? 

Patchy, no major players 
Established  competition  but  none  with  a  proposition 
like the one under investigation 
Several  major  players  with  strong  competencies  and 
infrastructure 

k OR l 
Q6 
a 
b 
c 
Q15 
a 

b 

c 

Q16 
a 
b 
c 
Q17 
a 
b 

c 
d 

e 
Q18 

a 
b 

c 

Number of patents have been applied for by the project 

GQ6 

<2 
≥2 

58 

0 

2 
1 
0 

0 

1 

0.5 

0 
0.5 
1 

0 
0.25 

0.5 
0.75 

1 

1 
0.5 

0 

0.5 
1 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
Innovation potential assessment framework: Innovation readiness 

Criteria & questions 

Innovation Readiness 

Development phase 

Under development 
Developed but not exploited 
Being exploited 
Technology transfer** 

Done 
Planned 

Prototyping** 

Done 
Planned 

Pilot, Demonstration or Testing activities**  

Done 
Planned 

Feasibility study**  
Done 
Planned 

Launch a start-up or spin-off**  

Done 
Planned 

Other**  

Done 
Planned 
Time to market 

Less than 1 year 
Between 1 and 2 years 
Between 3 and 5 years 
More than 5 years 

No workforce's skills issues that could compromise the ability 
of a project partner to exploit the innovation 

Question 
code* 
Q3 
a 
b 
c 
Q10.1 

Q10.5 

Q10.6 

Q10.7 

Q10.8 

Q10.13 

Q19 
a 
b 
c 
d 
GQ4e 

Scoring 
Max: 
10 

0 
1 
2 

1 
0.5 

1 
0.5 

1 
0.5 

1 
0.5 

1 
0.5 

1 
0.5 

1 
0.75 
0.5 
0.25 

1 

59 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Innovation potential assessment framework: Innovation Management 

Innovation Management 

Criteria & questions 

There is a clear owner of the innovation 
Engagement  of  both  research  team  and  partner's  business 
units in project activities** 

Scoring 
Max: 
10 
1 

Question 
code* 
Q8 
Q10.2 

Done 
Planned 

Business plan**  

Done 
Planned 

Market study**  

Done 
Planned 

Q10.3 

Q10.4 

Application for private or public investment**  

Q10.10 

Done 
Planned 

Securing private investment**  

Done 
Planned 

Securing public investment **  

Done 
Planned 

Q10.11 

Q10.12 

No consortium internal IPR issues that could compromise the 
ability of a project partner to exploit the innovation  
End-user engagement  

GQ2 

GQ1 

End-user in the consortium  
End-user consulted 
No end-user in the consortium or consulted 

Commitment of relevant partners to exploit innovation 

GQ9 

Above average 
Average 
Below average 

1 
0.5 

1 
0.5 

1 
0.5 

1 
0.5 

1 
0.5 

1 
0.5 

1 

1 
0.5 
0 

1 
0.5 
0 

*GQ – general questions 
**  -  Steps  DONE  or  PLANNED  in  the  project  in  order  to  bring  the  innovation  to  the 
market. 

60 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2.1 Innovator capacity assessment framework  

Table 2: Innovator capacity assessment framework 

Innovator's ability 

Criteria & questions 

Number of innovations in the project for which an 
organization is identified as a key organisation(s) in the 
project delivering this innovation 

1 
2 
3 

Score of innovation for which an organization is identified as 
a key organisation(s) in the project delivering this innovation 

Scoring 
Max: 5 

Question 
code* 
Q12 

0.5 
0.75 
1 

Output of the 
innovation 
assessment 
framework 

Score 
between 
0-1 

Organization is considered as the most impressive in terms of 
innovation potential 
Organization is the owner of the innovation 
Total  number  of  needs  to  fulfil  the  market  potential  of  an 
innovation 

GQ10 

Q9 

Q13 

No needs 
Between 1 and 2 
Between 3 and 4 
Between 5 and 6 
More than 6 
Innovator's environment 

The engagement of end-users in the consortium 

End user organisation in the consortium 
An end user organisation outside of the consortium is 
consulted 
No  end  user  organisation  in  the  consortium  or 
consulted 

The project performance in terms of innovation 

Exceeding expectations 
Meeting expectations  
Performing below expectations 

The level of commitment of relevant partners to exploit the 
innovation 

Very High or high  
Average 
Below average 

*GQ – general questions 

Question 
code* 
GQ1 

GQ7 

GQ9 

1 

1 

1 
0.75 
0.5 
0.25 
0 
Max: 3  

1 

0.5 

0 

1 
0.5 
0 

1 
0.5 
0 

61 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3. Construction of the indices

3.1 Innovation Potential 

In  order to observe and measure the relevant criteria,  each  of them  was matched with 
relevant questions of the Innovation Radar Questionnaire. In this way, a composite sub-
indicator for each assessment criteria was created: 





Innovation Readiness Dimension (IR) is an arithmetic aggregate of all
relevant information in the domain of innovation readiness (see Table).

Innovation Management Dimension (IM) is an arithmetic aggregate of all
relevant information in the domain of innovation management (see Table).

 Market Potential Dimension (MP) is an arithmetic aggregate of all relevant

information in the domain of innovation market potential (see Table).

In  the  second  step,  the  Innovation  Potential  index  (IPI)  is  constructed.  IPI  is  an 
arithmetic composite indicator which aggregates the values of the three dimensions, i.e. 
MP, IR and IM. Equal weighting is applied. Figure visualizes this procedure.  

Figure 1: Construction of the Innovation Potential index 

Source: European Commission JRC 

3.2 Innovator Capacity 

In  order  to  create  a  measure  of  innovator  capacity,  we  proceed  in  two  steps.  In  a  first 
step,  composite  sub-indicators  are  created,  one  for  each  of  the  above  defined  criteria: 
Innovator's  Ability  and  Innovator's  Environment.  This  way,  two  intermediate  sub-
indicators are used in order to assess each innovation dimension, i.e.:  



Innovator's Ability Dimension (IA) is an arithmetic aggregate of all relevant
information in the domain of innovator's ability (see Table).



Innovator's Environment Indicator (IE) is an arithmetic aggregate of all
relevant information in the domain of innovator's environment (see Table).
In the second step, the Innovator Capacity Indicator (ICI) is constructed. The ICI is 
an  arithmetic  composite  indicator  aggregating  the  values  of  the  two  earlier  sub-
indicators,  i.e.  IA  and  IE.  Like  in  the  case  of  innovation  ranking,  equal  weighting  is 
applied. Figure 2 visualizes this procedure. 

62 

Figure 2: Construction of the Innovation Capacity index 

Source: European Commission JRC 

63 

Europe Direct is a service to help you find answers to your questions about the European Union 
Free phone number (*): 00 800 6 7 8 9 10 11 
(*) Certain mobile telephone operators do not allow access to 00 800 numbers or these calls may be billed. 

A great deal of additional information on the European Union is available on the Internet. 
It can be accessed through the Europa server http://europa.eu 

How to obtain EU publications 

Our publications are available from EU Bookshop (http://bookshop.europa.eu), 
where you can place an order with the sales agent of your choice. 

The Publications Office has a worldwide network of sales agents. 
You can obtain their contact details by sending a fax to (352) 29 29-42758. 

K
J
-
N
A
-
2
9
1
3
7
-
E
N
-
N

doi:10.2760/196017 

ISBN 978-92-79-80362-8