Analysis of patents, publications, innovations and new organisations from FP7-ICT and CIP-ICT projects.
Date: 2019-09-05
Institutions: Directorate-General for Communications Networks, Content and Technology
Subjects: Framework Programme for Research and Development, information technology, innovation, patent, report

Analysis of patents, 
publications and new 
organisations from FP7-ICT 
and CIP-ICT projects  

FINAL STUDY REPORT – D4 

A study prepared for the European Commission 
DG Communications Networks, Content & Technology 
by: 

Digital 
Single
Market

Final Study Report- D4 - SMART 2015/0078  2 

Final Study Report- D4 - SMART 2015/0078  3 

This  study  was  carried  out  for  the  European  Commission  by  the  UNU-MERIT,  Maastricht 
University, project team consisting of: 

• Dr. Jojo Jacob, Associate Professor at Grenoble Ecole de Management, Department of
Management  &  Technology,  and  Affiliate  Researcher  at  UNU-MERIT,  Maastricht
University.

• Dr.  Bulat  Sanditov,  Assistant  Professor  at  Institut  Mines-Télécom  Business  School,
Knowledge, Innovation & Network Dynamics research group, and Affiliate Researcher
at UNU-MERIT, Maastricht University.

• Alexandru Surpatean, MSc, Consultant Data Management at UNU-MERIT, Maastricht

University.

• Ad  Notten,  MSc,  Researcher  and  Information  Specialist  at  UNU-MERIT,  Maastricht

University.

Internal identification 
Contract number: 30-CE-0811804/00-47 
SMART number: SMART 2015/0078 

DISCLAIMER 

By the European Commission, Directorate-General of Communications Networks, Content & 
Technology. 

The information and views set out in this publication are those of the author(s) and do not necessarily 
reflect the official opinion of the Commission. The Commission does not guarantee the accuracy of the 
data  included  in  this  study.  Neither  the  Commission  nor  any  person  acting  on  the  Commission’s 
behalf may be held responsible for the use which may be made of the information contained therein. 

ISBN 978-92-76-00595-7 

doi: 10.2759/644641 

Luxembourg: Publications Office of the European Union, 2019 

©European Union, 2019. All rights reserved. Certain parts are licensed under conditions to the EU. 

Reproduction  is  authorized  provided  the  source  is  acknowledged.  The  reuse  policy  of 
European Commission documents is regulated  by Decision 2011/833/EU (OJ L 330, 14.12.2011, p. 
39).  For  any  use  or  reproduction  of  photos  or  other  material  that  is   not  under  the  EU  copyright, 
permission must be sought directly from the copyright holders. 

Final Study Report- D4 - SMART 2015/0078  4 

Final Study Report- D4 - SMART 2015/0078  5 

Contents 
Abstract .................................................................................................................... 9 

Abstrait ..................................................................................................................... 9 

Executive Summary .................................................................................................. 11 

Data management................................................................................................. 11 

Assessment Goals ................................................................................................. 12 

Productivity Analysis ........................................................................................... 13 

Impact Analysis ................................................................................................. 13 

Analysis of the cooperation behaviour ................................................................... 14 

Econometric Analysis .......................................................................................... 15 

Newly Created Organisations ............................................................................... 15 

Keyword and Key-phrase Analysis ........................................................................ 15 

Limitations of this study ...................................................................................... 16 

Résumé .................................................................................................................. 17 

Gestion de données ............................................................................................... 17 

Objectifs d'évaluation ............................................................................................ 18 

Analyse de la productivité ................................................................................... 19 

Analyse d'impact ................................................................................................ 20 

Analyse du comportement de coopération ............................................................. 20 

Analyse économétrique ....................................................................................... 21 

Organisations nouvellement créées ...................................................................... 21 

Analyse des mots-clés et des phrases-clés ............................................................ 21 

Limites de cette étude ........................................................................................ 22 

Research Highlights .................................................................................................. 23 

Recommendations .................................................................................................... 29 

1 

Introduction ...................................................................................................... 31 

1.1 

Literature overview ...................................................................................... 32 

1.2  Scientometric analysis .................................................................................. 32 

2  Data Management .............................................................................................. 33 

2.1 

Provided data .............................................................................................. 33 

2.2  Data Cleaning and Validation: Publications ...................................................... 33 

2.3  Data Cleaning and Validation: Patents ............................................................ 34 

Final Study Report- D4 - SMART 2015/0078  6 

2.4 

Publication data: Focal set ............................................................................ 36 

2.5 

Publication data: Control set.......................................................................... 37 

2.6 

Patent data: Focal set ................................................................................... 38 

2.7 

Patent data: Control set ................................................................................ 40 

3 

Publication Analysis ............................................................................................ 41 

3.1 

Publication data enhancement ....................................................................... 41 

3.2 

Journals and proceedings .............................................................................. 41 

3.3 

Publication productivity of FP7-ICT ................................................................. 44 

3.3.1 

Productivity by project ............................................................................ 44 

3.3.2 

Productivity by funding instrument ........................................................... 45 

3.3.3 

Productivity by strategic objective ............................................................ 47 

3.3.4 

Productivity by EU region ........................................................................ 48 

3.3.5 

Productivity by country ........................................................................... 49 

3.3.6 

Productivity by organization type ............................................................. 55 

3.3.7 

Private-Public ratio ................................................................................. 55 

3.4 

Publication impact of FP7-ICT ........................................................................ 56 

3.4.1 

Cited-Un-cited ratio ................................................................................ 56 

3.4.2 

Impact per journal and proceedings ......................................................... 57 

3.4.3  MOCR, MECR, RCR ................................................................................. 58 

3.5  Cooperation behaviour in FP7-ICT .................................................................. 60 

3.5.1 

Patterns of international cooperation (intra-EU) ......................................... 61 

3.5.2 

Patterns of international cooperation (extra-EU) ........................................ 65 

3.5.3  Diversity of collaborators in terms of nationality ........................................ 67 

3.5.4  Diversity of collaborators, in terms of nationality, per funding scheme .......... 71 

4 

Patent Analysis .................................................................................................. 73 

4.1 

Patenting productivity of FP7-ICT ................................................................... 73 

4.1.1 

Productivity by project ............................................................................ 73 

4.1.2 

Productivity by strategic objective ............................................................ 73 

4.1.3 

Productivity by funding instrument ........................................................... 75 

4.1.4 

Productivity by country ........................................................................... 75 

4.1.5 

Productivity by organization .................................................................... 78 

4.2 

Impact indicators ......................................................................................... 81 

 
 
 
Final Study Report- D4 - SMART 2015/0078  7 

4.3 

International cooperation behaviour in FP7-ICT................................................ 86 

4.3.1 
inventions ......................................................................................................... 87 

Internationalization of technology through cross-border ownership of 

5  Econometric Analysis .......................................................................................... 91 

5.1  Results of the publication model .................................................................... 92 

5.2  Results of the patent model ........................................................................... 95 

6  Newly Created Organizations ............................................................................... 97 

6.1 

Introduction to the New Questions Added to the Survey ................................... 97 

6.2 

Types and Geography of Newly Created Organizations ...................................... 97 

6.3  Geographical Division of Newly Created Organizations ...................................... 99 

6.4  Division of Start-ups and Spin-offs across Funding Schemes ........................... 100 

6.5  Division of Start-ups and Spin-offs across Strategic Objectives ........................ 101 

6.6  Conclusion of the Start-ups and Spin-offs Analysis ......................................... 103 

7  Keyword and Key-phrase Analysis ...................................................................... 105 

7.1  Keyword-based association strength ............................................................. 105 

7.2  Association strength versus control group ..................................................... 105 

7.3  Association strength over time ..................................................................... 106 

7.4  Association strength within the focal set ....................................................... 107 

Conclusions ........................................................................................................... 109 

Publication Analysis ............................................................................................. 109 

Patent Analysis ................................................................................................... 110 

Econometric Analysis ........................................................................................... 111 

Newly Created Organizations ................................................................................ 112 

Keyword and key-phrase Analysis ......................................................................... 112 

Horizon 2020 ...................................................................................................... 113 

References ............................................................................................................ 115 

Abbreviations and listing of funding instruments ........................................................ 117 

Abbreviations ..................................................................................................... 117 

Funding instruments ............................................................................................ 117 

Annexes ................................................................................................................ 119 

Annex 1: Full listing of the distribution of publications across strategic objectives, by 
number of projects. ............................................................................................. 119 

 
 
 
Final Study Report- D4 - SMART 2015/0078  8 

Annex 2: Distribution of projects across European regions, NUTS 3, top 5 countries in 
terms of productivity. .......................................................................................... 125 

Annex 3:  Summary statistics of the publication productivity model. .......................... 127 

Annex 4: Summary statistics of the patent productivity model. ................................. 128 

 
 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  9 

Abstract 

This final study report provides an in-depth examination of the publications and patents 
reported through the yearly survey waves undertaken by DG CNECT in the years 2008-
2017. It is also the culmination of two consecutive ex-post research evaluations, of which 
the first exercise starting in 2012 and the second in 2016. The results of both these 
research assessments, SMART 2011/0039 and SMART 2015/0078, are presented in this 
report. The assessments consist of the analyses of the productivity and quality of the 
research output of projects funded under FP7 ICT. Using results from these analyses we 
carry out an econometric analysis, in order to gain more insights on the factors contributing 
to the scientific and technological performance of the funded projects. Furthermore we 
conduct an analysis of the cooperation behavior of the project participants and, in a novel 
way, we measure the degree of overlapping national and regional research interests by 
analysing keywords that describe the publication output. Finally we conduct a preliminary 
analysis into the data collected on newly created organisations for the years 2014 to 2016. 
This report covers the entirety of the FP7 ICT projects launched during the years 2007 to 
2013, for their productive years from 2007 to 2016. 

Abstrait 

Ce rapport d'étude final propose une analyse approfondie des publications et des brevets 
rapportés au travers des vagues d’enquêtes annuelles menées par la DG CNECT au cours 
des années 2008-2017. Il est également l'aboutissement de deux évaluations de recherche 
ex post consécutives, la première ayant débuté en 2012 et la seconde en 2016. Les 
résultats de ces deux évaluations de recherche, SMART 2011/0039 et SMART 2015/0078, 
sont présentés dans ce rapport. Les évaluations consistent en des analyses de la 
productivité et de la qualité des résultats de recherche des projets financés par le FP7 ICT. 
Les résultats de ces analyses nous ont permis de réaliser une analyse économétrique qui 
nous a aidé à mieux comprendre les facteurs qui contribuent à la performance scientifique 
et technologique des projets financés. En outre, nous avons pu analyser le comportement 
de coopération des participants aux projets et mesurer de manière innovante le degré de 
chevauchement des intérêts de recherche nationaux et régionaux en analysant certains 
mots-clés décrivant la production des publications. Enfin, nous avons effectué une analyse 
préliminaire des données collectées parmi des organisations nouvellement créées au cours 
des années 2014 à 2016. Le présent rapport couvre l’ensemble des projets du FP7 ICT 
lancés au cours des années 2007 à 2013 pour les années productives allant de 2007 à 
2016. 

 
 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  10 

Final Study Report- D4 - SMART 2015/0078  11 

Executive Summary 

This study report provides an in-depth examination of the publications and patents reported 
through  the  yearly  survey  waves  undertaken  by  DG  CNECT  in  the  years  2008-2017.  It  is 
also  the  culmination  of  two  consecutive  ex-post  research  evaluations,  of  which  the  first 
exercise  starting  in  2012  and  the  second  in  2016.  The  results  of  both  these  research 
assessments,  SMART  2011/0039  and  SMART  2015/0078,  are  created  in  this  report. 
Additional analysis of the productivity and quality of the research output of projects funded 
under FP7 ICT using a longitudinal dataset is foremost concentrated on two data sets; one 
of  patents,  the  other  of  publications.  Next  to  these  two  datasets,  we  also  conduct  a 
preliminary  analysis  into  the  data  collected  on  newly  created  organisations  for  the  years 
2014 to 2016. This report covers the entirety of the FP7 projects launched during the years 
2007 to 2013, for their productive years from 2007 to 2016. 

Data management 
The raw data for the analyses in this report as well as for the yearly reports (D2 & D3) have 
been  collected  by  DG  CNECT  through  yearly  surveys  of  the  project  coordinators  of  the 
funded  projects.  These  were  subsequently  aggregated  and  sent  on  to  the  MERIT  team  at 
Maastricht  University.  The  team  has  undertaken,  for  all  of  the  data  delivered  to  them,  an 
intensive  process  of  cleaning,  validation  and  matching,  using  automated  as  well  as  semi-
manual  techniques.  The  patents  received  were  validated  using  a  combination  of  the 
European Patent Office’s (EPO) web portal Espacenet, as well as using the more expansive 
EPO  PATSTAT  database  as  well  as  an  OECD  database  focusing  on  patent  quality  (which  is 
derived  from  the  aforementioned  PATSTAT  data).  The  publication  data  were  validated  in 
Elsevier’s  Scopus  database,  using  an  API,  while  further  analysis  was  enabled  through  the 
use of a Scopus Custom Data set.  

The  validated  dataset  is  essentially  a  sub-part  of  a  larger  universe  of  publications  and 
patents  in  the  field  of  ICT.  We  use  this  larger  universe  to  construct  both  a  focal  sample 
representing FP7 ICT projects, and a control sample by which we would like to benchmark 
the  output  reported  to  DG  CNECT,  using  a  stratified  random  sampling  design  with 
proportional allocation. We focus our attention on the publication output as published in the 
same journals and proceedings as used by FP7 ICT researchers. For patents we match our 
data  on  the  patenting  authority,  priority  year  and  IPC  class  as  patented  in  by  FP7  ICT 
inventors.  The  samples  are  used  for  all  the  comparative  analyses,  while  the  full  validated 
dataset is used for the descriptive statistics and related analyses. 

The analyses performed in this study depend greatly on the quality of the data provided to 
us by DG CNECT. It is however important to acknowledge the fact that this data is compiled 
from  the  survey  answers  of  project  leaders  in  a  voluntary  reporting  process,  which  takes 
place each year. As such, data is not available for all funded projects,  and for the projects 
for which it is available, it is not necessarily available for all the funding years. 

Final Study Report- D4 - SMART 2015/0078  12 

Assessment Goals 
The  various  analyses  in  this  report  correspond  to  four  main  assessment  goals.  These  four 
assessment  goals  endeavour  to  shed  light  not  only  on  the  productivity  and  impact  of  the 
research  conducted,  but  also  on  the  knowledge  generated  within  the  relevant  knowledge 
domains and diffused through the research networks of the participants.  

The first goal is to analyse and evaluate the productivity of the  ICT FP7 research projects. 
This analysis encompasses evaluations of the productivity of projects, types of projects and 
strategic  objectives.  A  comparative  analysis  is  added,  focussing  on  the  geographical  and 
organizational distribution of productivity within the EU-28 and selected third countries. 

The  second  assessment  goal  concerns  itself  with  examining  the  impact  of  the  funded 
research, based on a citation analysis of both patents and publications. Here, the evaluation 
focuses on the quality of the research funded by FP7 in the field of ICT, which is compared 
with similar research worldwide.  

The third assessment goal explores and evaluates the cooperation behaviour of the funded 
project  participants  at  an  aggregated  geographical  level.  Tools  and  methods  used  in  this 
evaluation  consist  mainly  of  a  social  network  analytical  approach  as  well  as  descriptive 
statistics. 

The  fourth  assessment  goal  ventures  into  the  contextual  nature  of  the  knowledge  diffused 
through  the  social  networks  mentioned  above.  It  primarily  focuses  on  the  geographical 
concentration  and  topical  overlap  of  the  funded  ICT  research.  Using  co-word  analysis 
techniques, trends in ICT research are compared at the national level, specifically looking at 
the topical overlap between countries and regions.  

Finally,  an  econometric  analysis  endeavours  to  correlate  a  number  of  the  findings  of  the 
different evaluations in order to assess their significance and policy relevance. 

Final Study Report- D4 - SMART 2015/0078  13 

Productivity Analysis 

•  Both  the  publication  and  patent  outputs  per  project  are  skewed;  a  few  projects 
account for the highest share of both types of output. We find that on average there 
are  9.12  publications  per  project  funded,  within  the  sample  of  2706  projects.  The 
division of the proceedings papers and journal articles, funded by FP7 in the field of 
ICT, within the validated publication set is roughly 65% versus 35% respectively.  
•  Although highly skewed in terms of distribution as well, we find that for the absolute 
number  of  publications  generated,  the  "Collaborative  Projects"  (CP-FP-INFSO)  
performs  best,  with  “Networks  of  Excellence”  (NoEs)  and  "  Large  scale  integrating 
collaborative projects" (CP-IPs) coming in second and third place. This is almost the 
same  for  the  patents,  with  CP-FP-INFSO  first  and  CP-IPs  second  in  terms  of 
productivity. NoEs come third in terms of generating patents (in absolute numbers). 

•  When  relating  the  patent  and  publications  output  to  the  number  of  projects  funded 
through  each  of  the  funding  schemes,  we  find  that  for  both  patents  and  for 
publications the NoEs rank first.  

•  For  the  strategic  objectives,  both  the  patents  and  publications  productivity  trends 
show a long-tailed distribution. While the strategic objective “Networks of the Future” 
generates  the  most  publications  and  patents  in  absolute  terms,  the  strategic 
objective  “Distributed  Computing  Infrastructure”  is  in  relative  terms  the  most 
productive  in  generating  publications.  For  patents  “Networks  of  the  Future” 
generates the most output overall. 

•  When  looking  at  the  comparison  of  the  geographic  distribution,  we  see  naturally  a 
strong  bias  towards  the  EU-28  countries  in  the  focal  sample,  while  in  the  control 
sample several strong and competitive non-EU countries take the lead. This is more 
so  for  patents  than  for  publications,  where  the  EU  is  performing  well  overall. 
However,  the  general  picture  emerging  is  that  the  trends  suggest  a  substantially 
higher  involvement  of  EU  organizations  in  ICT  research  due  to  the  FP7  programme 
than  what  one  observes  in  academia  and  the  industry  as  a  whole,  which  is  exactly 
what  the  funding  programmes  were  meant  for.  Germany  leads,  in  terms  of 
productivity,  the  group  of  EU-28  countries  for  both  publications  and  patents.  The 
United States takes this role worldwide.  

•  For the comparison of organizational productivity, we find that the traditional Public-
Private division still holds. Corporations lead the field within patenting activity, while 
Academic  organizations  lead  the  field  for  the  organizational  productivity  in 
publications.  

Impact Analysis 

•  The  impact  analysis,  or  quality  assessment,  in  this  report  is  based  on  a  citation 
analysis  of  the  citations  received  by  the  scientific  publications  reported  in  the  FP7 
survey waves to DG CNECT and re-validated in 2018.  

 
 
 
Final Study Report- D4 - SMART 2015/0078  14 

•  We  find  that  only  5%  of  the  articles  produced  within  DG  CNECT  funded  research 

projects remain un-cited. For proceeding papers the un-cited rate is 15%. 

•  For the total set of validated publications, including proceedings papers, the un-cited 

rate is only slightly higher than the worldwide (control) un-cited rate. 

•  The  mean  observed  citation  rate  (MOCR1) is  well  above  the  mean  expected  citation 
rate  (MECR2)  for  the  total  set  of  validated  publications,  as  well  as  for  articles  and 
proceedings  papers  separately.  Particularly,  for  research  published  in  conference 
proceedings  the  relative  citation  rate  (MOCR/MECR)  is  much  higher  than  the 
average. This means that overall ICT FP7 research published in the popular and high 
impact journals and conference proceedings receive on average more citations than 
non-DG  CNECT  funded  research.  Specifically,  in  front-line  research  published  in 
conference  proceedings,  the  EU  supported  research  outperforms  the  world-wide 
average. 

Analysis of the cooperation behaviour 

•  For  the  assessment  of  the  cooperation  behaviour  in  FP7  ICT  research,  we  employ 
network  analytical  methods  alongside  standard  statistics.  Using  network  measures, 
we  seek  to  identify  the  most  important  knowledge  brokers  at  the  country  level. 
Furthermore, we benchmark the cooperation characteristics with a control sample. 

•  We  find  that  there  seem  to  be  several  “cliques”  of  countries  with  overall  high  tie 
strength 3.  The  Czech  Republic,  Romania  and  Hungary  are  on  such  clique,  while 
Portugal,  Spain  and  Italy  seem  to  make  up  another.  Austria  has  similarly  high  tie 
strength and could be seen as the connecting gatekeeper between these cliques.  

•  We further notice that most patents that are owned by European organizations in our 

sample are also invented by residents of their own countries.  

•  For most European organizations involved in patenting, the most important source of 
knowledge outside Europe is the US. Interestingly, the only two European countries 
that  use  inventors  from  BRICS  countries  are  France  (inventors  from  China)  and 
Spain (inventors from India). 

•  On a whole, we can state that the FP7 funded research favours intra-EU knowledge 

flows, and displays a high degree of internationalisation between EU countries. 

1  The  mean  observed  citation  rate  or  MOCR,  represents  the  average  citation  rate  per  publication;  (number  of 
citations)  /(number  of  publications).  In  other  words  this  metric  represents  the  average  amount  of  citations  each 
publication receives (Schubert and Braun, 1986).  
2  The  mean  expected  citation  rate  or  MECR,  is  the  average  expected  citation  rate  per  publication;  (expected 
number  of  citations)/(number  of  publications).  Here  the  expected  number  of  citations  is  based  on  the  average 
citation rates of the publishing journal (Schubert and Braun, 1986). 
3 Tie strength is defined as the level of intensity of a (social) relationship between two network nodes (Granovetter, 
1973). Ties can be weighted using several metrics. In this case we use the number of collaborations. 

 
 
 
                                                           
Final Study Report- D4 - SMART 2015/0078  15 

Econometric Analysis 

•  We carry out separate econometric analyses for publications and patents. In addition 
to  using  project-related  variables  such  as  funding,  number  of  participants  involved, 
the  type  of  financing  instrument,  etc.,  we  introduced  a  novel  measure  in  order  to 
capture  the  diversity  of  scientific  capabilities  at  the  national  level  of  organizations 
involved in each project. 

•  The  analysis  of  the  publication  productivity  reveals  that  scientific  diversity  of 
research  groups,  in  terms  of  different  national  knowledge  bases,  plays  a  significant 
role  in  enhancing  their  scientific  productivity.  We  also  find  evidence  on  the  positive 
contribution  of  funding  and  the  number  of  participants  involved  in  a  project  for 
publication productivity.  

•  We also find a reinforcing effect of patents on publications and vice versa, and that a 
larger share of companies and universities in the projects has a significant effect on 
the generation of patents and publications respectively. 

•  The  analysis  of  the  publication  impact  shows  up  quite  different  results,  where  the 
positive  relation  of  patenting,  funding  and  project  size  is  not  visible  anymore,  or 
even  negative.  A  probable  explanation  for  this  is  that  larger  projects  increase  the 
pressure on participants to publish thereby impinging on quality. 

Newly Created Organisations 

•  The  analysis  of  newly  created  organisations  resulting  from  FP7  ICT  research  shows 
that  the  CP-IP  and  CP-FP-INFSO  funding  schemes  are  the  main  schemes  producing 
university spin-offs, start-ups and corporate spin-offs. 

•  The  use-case  oriented  strategic  objectives  such  as  “Expansion  of  Use  Cases”  and 
“Use Case scenarios and early trials” are, as might be expected, the ones leading in 
number  of  organisations  newly  created.  This  might  be  due  to  the  practical  “Follow 
Through” nature of these strategic objectives. 

Keyword and Key-phrase Analysis 

•  The keyword and key-phrase analysis presented at the end of the study is provided 
as  a  novel  way  of  analysing  FP7  ICT  project  output  and  providing  extra  context  to 
the  analysis.  We  measure  the  degree  of  overlapping  national  and  regional  research 
interests  by  analysing  the  keywords  that  describe  the  publication  output.  This  is 
presented  as  a  measure  of  overlapping  research  interests,  but  not  computed  based 
on explicit cooperation ties, but through indirect labels of interest. 

•  We note that FP7 ICT research has a positive affinity with all the considered control 
sample groups. This shows that FP7 ICT research is following the global trends in ICT 
research.  

 
 
 
Final Study Report- D4 - SMART 2015/0078  16 

•  The  FP7  ICT  output  has  greatest  affinity  with  the  EU-28  control  group  of  non-DG 
CNECT-funded projects, which is to be expected. The focal sample also has a positive 
affinity with the US and BRICS control sample groups, and a positive affinity, albeit 
much smaller, with the Japan and South Korea control samples. This shows that the 
FP7  ICT  research  is  closely  following  the  research  trends  in  the  EU-28,  as  well  as 
those  in  the  US  and  BRICS  countries.  The  least  overlap  in  research  themes  can  be 
found  with  South  Korea  and  Japan.  Over  time  we  notice  a  decrease  in  affinity  with 
US  ICT  research  which is  not on line  with  the  affinity  for  the  control  which  remains 
largely the same for the US. Although less pronounced this also seems to be the case 
for South Korean and Japanese ICT research. Conversely, the affinity with the BRICS 
countries increases over time. 

Limitations of this study 

•  As  the  data  used  in  this  study  is  reported  directly  to  DG  CNECT  by  the  project 
coordinators,  we  cannot  independently  verify  the  data.  Possible  instances  of 
reporting  bias  are  imaginable,  as  exemplified  by  the  double-count  and  non-FP7 
output reporting. 

•  As  the  report  only  assesses  one  scientific  area,  i.e.  ICT,  the  results  cannot  be 

generalised to other scientific areas. 

•  As  the  report's  results  are  based  on  three  types  of  output;  patents,  articles,  and 
proceedings, we should be aware of possible other media types that are not captured 
in  this  report  but  that  could  be  of  importance  to  knowledge  diffusion  within  the 
scientific  area  of  ICT.  An  example  could  be  online  media  types  not  captured 
elsewhere. 

•  The  keyword  analysis  is  provided  as  a  novel  way  of  analysing  FP7  ICT  output. 
Although we have chosen a robust method, it is a simple one, which can be further 
improved upon if found to be valuable. For instance instead of aggregating all topics 
into  one  overarching  field,  topic  models  of  specific  interest  areas  could  be  analysed 
separately.   

•  As  a  rule,  quantitative  evaluations,  no  matter  how  thorough  and  robust,  should  be 
used alongside other, qualitative, evaluation schemes in order to gain the necessary 
depth before drawing conclusions on possible policy implications. 

 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  17 

Résumé 

Ce rapport d'étude fournit un examen approfondi des publications et des brevets rapportés 
au  cours  des  vagues  d'enquêtes  annuelles  menées  par  la  DG  CNECT  au  cours  des  années 
2008-2017.  Il  est  également  l'aboutissement  de  deux  évaluations  de  recherche  ex  post 
consécutives, la première ayant débuté en 2012 et la seconde en 2016. Ce rapport présente 
les  résultats  des  deux  évaluations  de  recherche  SMART  2011/0039  et  SMART  2015/0078. 
Une  analyse  supplémentaire  de  la  productivité  et  de  la  qualité  des  résultats  de  recherche 
issus de projets financés dans le cadre du programme TIC du FP7  à partir d’un ensemble de 
données  longitudinales  a  principalement  été  axée  sur  deux  ensembles  de  données;  le 
premier  recouvrant  les  brevets,  l'autre  les  publications.  Outre  ces  deux  ensembles  de 
données,  nous  avons  également  effectué  une  analyse  préliminaire  des  données  collectées 
parmi les organisations nouvellement créées au cours des années 2014 à 2016. Ce rapport 
couvre  l'intégralité  des  projets  FP7  lancés  de  2007  à  2013  pour  les  années  productives  de 
2007 à 2016. 

Gestion de données 
Les données brutes qui ont servi aux analyses présentées dans ce rapport ainsi que pour les 
rapports  annuels  (D2  et  D3)  ont  été  collectées  par  la  DG  CNECT  à  travers  des  enquêtes 
annuelles  menées  par  les  coordinateurs  des  projets  financés.  Ceux-ci  ont  ensuite  été 
regroupés  et  envoyés  à  l'équipe  MERIT  de  l'Université  de  Maastricht.  Cette  équipe  s’est 
livrée, pour toutes les données qui lui ont été livrées, à un processus intensif de nettoyage, 
de validation et d’appariement, à l’aide de techniques automatisées et semi-manuelles. Les 
brevets reçus ont été validés à l'aide d'une combinaison du portail Web Espacenet de l'Office 
Européen  des  Brevets  (OEB/EPO),  de  la  base  de  données  plus  étendue  EPO  PATSTAT  et 
d'une  base  de  données  de  l'OCDE  axée  sur  la  qualité  des  brevets  (dérivée  des  données 
PATSTAT mentionnées ci-dessus). Les données de publication ont été validées dans la base 
de données Scopus d’Elsevier à l’aide d’une API. Par ailleurs, des analyses plus poussées ont 
été activées grâce à l’utilisation d’un ensemble de données personnalisées Scopus (Scopus 
Custom Data). 

L'ensemble  de  données  ainsi  validé  fait  essentiellement  partie  d'un  vaste  univers  de 
publications  et  de  brevets  dans  le  domaine  des  TIC.  Nous  utilisons  cet  univers  plus  large 
pour  construire  à  la  fois  un  échantillon  focal  représentant  les  projets  TIC  du  FP7  et  un 
échantillon  de  contrôle  permettant  de  comparer  le  résultat  communiqué  à  la  DG  CNECT,  à 
l'aide  d'un  plan  de  sondage  aléatoire  stratifié  à  répartition  proportionnelle.  Nous 
concentrons  notre  attention  sur  le  nombre  de  publications  publiées  dans  les  mêmes 
journaux et actes que ceux utilisés par les chercheurs en TIC du FP7. Pour les brevets, nous 
comparons  nos  données  par  rapport  à  l’autorité  de  délivrance  des  brevets,  l’année  de 
priorité  et  la  classe  IPC  telles  que  brevetées  par  les  inventeurs  des  TIC  du  FP7.  Les 
échantillons  sont  utilisés  pour  toutes  les  analyses  comparatives,  tandis  que  l'ensemble  de 
données  validé  complet  est  utilisé  pour  les  statistiques  descriptives  et  les  analyses 
connexes. 

 
 
 
Final Study Report- D4 - SMART 2015/0078  18 

Les  analyses  effectuées  dans  cette  étude  dépendent  en  grande  partie  de  la  qualité  des 
données  qui  nous  ont  été  fournies  par  la  DG  CNECT.  Il  est  toutefois  important  de 
reconnaître  le  fait  que  ces  données  ont  été  compilées  à  partir  des  réponses  d'enquête  des 
coordinateurs de projet dans le cadre d'un processus de participation volontaire, qui a lieu 
chaque  année.  Par  conséquent,  les  données  ne  sont  pas  disponibles  pour  tous  les  projets 
financés.  Et,  pour  les  projets  pour  lesquels  elles  sont  disponibles,  elles  ne  le  sont  pas 
nécessairement pour toutes les années de financement. 

Objectifs d'évaluation 
Les  différentes  analyses  présentées  dans  ce  rapport  correspondent  à  quatre  objectifs 
principaux  d’évaluation.  Ces  quatre  objectifs  d’évaluation  visent  à  mettre  en  lumière  non 
seulement  la  productivité  et  l’impact  de  la  recherche  effectuée,  mais  également  les 
connaissances  générées  dans  les  domaines  de  connaissance  pertinents  et  diffusées  à 
travers les réseaux de recherche des participants. 

Le  premier  objectif  consiste  à  analyser  et  évaluer  la  productivité  des  projets  de  recherche 
du  FP7  relatifs  aux  TIC.  Cette  analyse  comprend  des  évaluations  de  la  productivité  des 
projets,  des  types  de  projets  et  des  objectifs  stratégiques.  Une  analyse  comparative  est 
ajoutée,  mettant  l'accent  sur  la  répartition  géographique  et  organisationnelle  de  la 
productivité dans les 28 pays de l'UE et dans certains pays tiers choisis. 

Le deuxième objectif de l'évaluation consiste à examiner l'impact de la recherche financée, 
sur la base d'une analyse des citations de brevets et de publications. Ici, l'évaluation porte 
sur  la  qualité  des  recherches  financées  par  le  FP7  dans  le  domaine  des  TIC,  qui  est 
comparée à celle des recherches similaires menées dans le monde entier. 

Le  troisième  objectif  de  l'évaluation  explore  et  évalue  au  niveau  géographique  global  le 
comportement de coopération des participants aux projets financés. Les outils et méthodes 
utilisés  dans  cette  évaluation  consistent  principalement  en  une  approche  analytique  des 
réseaux sociaux et en statistiques descriptives. 

Le  quatrième  objectif  de  l'évaluation  explore  la  nature  contextuelle  des  connaissances 
diffusées à travers les réseaux sociaux mentionnés ci-dessus. Il se concentre principalement 
sur  la  concentration  géographique  et  le  chevauchement  thématique  des  recherches  TIC 
financées. À l’aide de techniques d’analyse par mots associés, les tendances de la recherche 
sur  les  TIC  sont  comparées  au  niveau  national,  notamment  en  ce  qui  concerne  le 
chevauchement des thèmes entre pays et régions. 

Enfin, nous cherchons à corréler au moyen d’une analyse économétrique un certain nombre 
des conclusions des différentes évaluations afin d’évaluer leur importance et leur pertinence 
pour les politiques scientifique, technologique et d'innovation de l'UE. 

 
 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  19 

Analyse de la productivité 

•

La  publication  et  les  résultats  de  brevets  par  projet  sont  biaisés;  quelques  projets
représentent la part la plus élevée des deux types de production. Nous constatons en
moyenne 9,12 publications par projet financé, sur un échantillon de 2706 projets. La
répartition des comptes rendus et des articles de revues scientifiques, financés par le
FP7  dans  le  domaine  des  TIC,  au  sein  du  groupe  de  publications  validées  est
d'environ 65% contre 35%, respectivement.

•

•

• Bien que très inégalement distribués, nous constatons que pour le nombre absolu de
publications  générées,  ce  sont  les  “projets  collaboratifs”  (CP-FP-INFSO)  qui
fonctionnent  le  mieux,  avec  les  “réseaux  d’excellence”  (NoE)  et  les  “projets
collaboratifs  d'intégration  à  grande  échelle"(CP-IP)  en  deuxième  et  troisième
positions.  Il  en  est  presque  de  même  pour  les  brevets,  avec  le  CP-FP-INFSO  en
première  position  et  le  CP-IP  en  deuxième  position  en  termes  de  productivité.  Les
NoE  viennent  en  troisième  position  pour  ce  qui  est  de  la  génération  de  brevets  (en
chiffres absolus).
Lorsque nous établissons un lien entre la production de brevets et de publications et
le  nombre  de  projets  financés  par  chacun  des  mécanismes  de  financement,  nous
constatons  que,  pour  les  brevets  comme  pour  les  publications,  les  NoE  sont  au
premier rang.
En  ce  qui  concerne  les  objectifs  stratégiques,  les  tendances  de  la  productivité  des
brevets  et  des  publications  montrent  une  distribution  à  long  terme.  Alors  que
l'objectif stratégique “Réseaux du futur” génère le plus grand nombre de publications
et  de  brevets  en  valeur  absolue,  l'objectif  stratégique  “Infrastructure  de  calcul
distribué” est en termes relatifs le plus productif pour ce qui concerne la production
de  publications.  Pour  ce  qui  est  des  brevets,  les  “Networks  of  the  Future”  génèrent
globalement le plus grand nombre de résultats.
Lorsque nous comparons la répartition géographique, nous constatons naturellement
un fort biais en faveur des pays de l'UE-28 dans l'échantillon central, tandis que dans
l'échantillon  de  contrôle,  plusieurs  pays  forts  et  compétitifs  non-membres  de  l'UE
viennent  en  tête.  Ceci  est  observé  davantage  pour  les  brevets  que  pour  les
publications,  où  l'UE  se  comporte  globalement  bien.  De  manière  générale,  les
tendances suggèrent une implication plus importante des organisations de l'UE dans
la  recherche  sur  les  TIC  grâce  au  programme  du  FP7  par  rapport  à  celle  observée
dans  le  monde  universitaire  et  de  l'industrie  dans  son  ensemble,  ce  qui  correspond
exactement à l'objectif initial des programmes de financement. L'Allemagne vient en
tête  en  termes  de  productivité  au  sein  du  groupe  des  pays  de  l'UE-28  en  ce  qui
concerne les publications et les brevets. Les États-Unis jouent ce rôle dominant dans
le monde entier.
Pour  ce  qui  est  de  la  comparaison  de  la  productivité  organisationnelle,  nous
constatons que la division traditionnelle public-privé reste en vigueur. Les entreprises
mènent  le  jeu  dans  le  domaine  des  brevets,  tandis  que  les  organisations
académiques  se  distinguent  par  leur  productivité  organisationnelle  au  niveau  des
publications.

•

•

Final Study Report- D4 - SMART 2015/0078  20 

Analyse d'impact 

•  L'analyse  d'impact,  ou  l’évaluation  de  qualité,  présentée  dans  ce  rapport  est  basée 
sur  une  analyse  des  citations  reçues  par  les  publications  scientifiques  rapportées 
dans les vagues d'enquêtes du FP7 à la DG CNECT et revalidées en 2018. 

•  Nous constatons que seulement 5% des articles produits dans le cadre de projets de 
recherche financés par  la DG CNECT ne sont pas cités. Pour les comptes  rendus, le 
taux des textes non-cités est de 15%. 

•  Pour l'ensemble des publications validées, y compris les comptes rendus, le taux des 
textes  non-cités  n'est  que  légèrement  supérieur  au  taux  mondial  (de  contrôle)  de 
textes non-cités. 

•  Le taux de citations moyen  observé (MOCR) est bien supérieur  au taux de citations 
moyen  prévu  (MECR)  pour  l'ensemble  des  publications  validées,  ainsi  que  pour  les 
articles  et  les  comptes  rendus  pris  séparément.  En  particulier,  pour  les  recherches 
publiées dans les actes de conférence, le taux de citations relatif (MOCR / MECR) est 
beaucoup  plus  élevé  que  la  moyenne.  Cela  signifie  que  l'ensemble  des  recherches 
dans  le  domaine  des  TIC  du  FP7  publiées  dans  les  journaux  populaires  et  à  fort 
impact  et  les  actes  de  conférence  reçoivent  en  moyenne  plus  de  citations  que  les 
recherches  non  financées  par  la  DG  CNECT.  En  particulier,  si  l’on  regarde  les 
recherches de première ligne publiées dans les actes de conférences, les recherches 
soutenues par l’UE dépassent la moyenne mondiale. 

Analyse du comportement de coopération 

•  Pour évaluer le comportement de coopération de recherche dans le domaine des TIC 
du  FP7,  nous  utilisons  des  méthodes  analytiques  de  réseau  parallèlement  aux 
statistiques standard. En utilisant des mesures de réseau, nous cherchons à identifier 
les principaux intermédiaires de connaissances au niveau national. Par ailleurs, nous 
comparons les caractéristiques de coopération à l’aide d’un échantillon de contrôle. 
•  Nous  constatons  qu’il  semble  exister  plusieurs  “cliques”  de  pays  ayant  une  force 
d’association  généralement  élevée.  La  République  tchèque,  la  Roumanie  et  la 
Hongrie  font  partie  de  cette  clique,  alors  que  le  Portugal,  l’Espagne  et  l’Italie 
semblent  en  constituer  une  autre.  L’Autriche  a  une  force  d’égalité  similaire  et 
pourrait être considérée comme la gardienne de la connexion entre ces cliques. 

•  Nous  notons  en  outre  que  la  plupart  des  brevets  détenus  par  des  organisations 
européennes dans notre échantillon ont également été inventés par des résidents de 
leurs propres pays. 

•  Pour  la  plupart  des  organisations  européennes  impliquées  dans  le  domaine  des 
brevets, les États-Unis sont la source de connaissances la plus importante en dehors 
de  l'Europe.  Fait  intéressant,  les  deux  seuls  pays  européens  qui  utilisent  des 
inventeurs des pays des BRICS sont la France (qui utilise des inventeurs de Chine) et 
l’Espagne (qui utilise des inventeurs de l’Inde). 

•  Globalement, nous pouvons affirmer que la recherche financée par le FP7 favorise les 
flux de connaissances au sein de l'UE et affiche un degré élevé d'internationalisation 
entre les pays de l'UE. 

 
 
 
Final Study Report- D4 - SMART 2015/0078  21 

Analyse économétrique 

•  Nous  réalisons  des  analyses  économétriques  séparées  pour  les  publications  et  les 
brevets.  En  plus  d'utiliser  des  variables  liées  au  projet  telles  que  le  financement,  le 
nombre  de  participants  impliqués,  le  type  d'instrument  de  financement,  etc.,  nous 
avons  introduit  une  nouvelle  mesure  qui  nous  permet  de  rendre  compte  de  la 
diversité des capacités scientifiques au niveau national des organisations impliquées 
dans chaque projet.  

•  L'analyse  de  la  productivité  des  publications  révèle  que  la  diversité  scientifique  des 
groupes  de  recherche,  en  termes  de  bases  de  connaissances  nationales  différentes, 
joue  un  rôle  important  dans  l'amélioration  de  leur  productivité  scientifique.  Nous 
trouvons  également  des  preuves  de  la  contribution  positive  dans  la  productivité  de 
publication,  liée  au  financement  ainsi  qu’au  nombre  de  participants  impliqués  dans 
un projet. 

•  Nous  constatons  également  un  effet  de  renforcement  mutuel  des  brevets  sur  les 
publications, et qu'une part plus importante d'entreprises et d'universités impliquées 
dans les projets a un effet significatif sur la génération de brevets et de publications, 
respectivement. 

•  L’analyse  de  l’impact  des  publications  donne  des  résultats  assez  différents,  dans 
lesquels la relation positive entre brevet, le financement et la taille du projet n’est ni 
visible,  ni  même  négative.  Une  explication  probable  pourrait  être  que  les  grands 
projets augmentent la pression de publier exercée sur les participants, ce qui nuit à 
la qualité. 

Organisations nouvellement créées 

•  L'analyse  des  organisations  nouvellement  créées  à  la  suite  des  recherches  menées 
dans le domaine des TIC du FP7 montre que les régimes de financement CP-IP et CP-
FP-INFSO sont les principaux mécanismes produisant des spin-offs, des start-ups et 
des spin-off d'entreprise dans les universités. 

•  Les  objectifs  stratégiques  axés  sur  les  cas  d'utilisation,  tels  que  “Extension  des  cas 
d'utilisation”  et  “Scénarios  de  cas  d'utilisation  et  essais  préliminaires”  viennent  en 
premier,  comme  l’on  pouvait  s'y  attendre,  en  ce  qui  concerne  le  nombre 
d'organisations  nouvellement  créées.  Cela  pourrait  être  dû  à  la  nature  pratique  du 
“suivi” de ces objectifs stratégiques. 

Analyse des mots-clés et des phrases-clés 

•  L'analyse des mots-clés et des phrases-clés présentée à la fin de cette 'étude est une 
nouvelle  façon  d'analyser  les  résultats  des  projets  TIC  du  FP7  et  de  fournir  un 
contexte supplémentaire à l'analyse. Nous mesurons le degré de chevauchement des 
intérêts de recherche nationaux et régionaux en analysant les mots-clés décrivant le 
résultat  de  la  publication.  Il  s’agit  d’une  mesure  d’intérêts  de  recherche  qui  se 
chevauchent,  non  pas  calculé  sur  la  base  de  liens  de  coopération  explicites,  mais  à 
travers des étiquettes d'intérêts indirectes. 

 
 
 
Final Study Report- D4 - SMART 2015/0078  22 

•  Nous notons que la recherche sur les TIC du FP7 a une affinité positive avec tous les 
groupes d'échantillons de contrôle considérés. Cela montre que la recherche  sur les 
TIC du FP7 suit les tendances mondiales en matière de recherche sur les TIC. 

•  Les résultats en matière de TIC du FP7 ont la plus grande affinité avec le groupe de 
contrôle  de  l'UE-28  composé  de  projets  non  financés  par  la  CNECT,  ce  qui  était  à 
prévoir.  L'échantillon  focal  a  également  une  affinité  positive  avec  les  groupes 
d'échantillons de contrôle des États-Unis et des pays des BRICS, ainsi qu'une affinité 
positive, bien que beaucoup plus petite, avec les échantillons de contrôle du Japon et 
de la Corée du Sud. Cela montre que la recherche sur les TIC du FP7 suit de près les 
tendances de la recherche dans l'UE-28, ainsi que celles des  États-Unis et des pays 
des  BRICS.  Nous  constatons  le  chevauchement  le  plus  réduit  dans  les  thèmes  de 
recherche avec la Corée du Sud et le Japon. Au fil du temps, nous avons observé une 
diminution  de  l'affinité  avec  la  recherche  américaine  sur  les  TIC  dans  l'échantillon 
focal,  qui  ne  correspond  pas  à  l'affinité  pour  le  groupe  de  contrôle,  qui  reste 
largement  identique  pour  les  États-Unis.  Bien  que  moins  prononcé,  cela  semble 
également  être  le  cas  pour  la  recherche  en  TIC  en  Corée  du  Sud  et  au  Japon. 
Inversement, les affinités avec les pays des BRICS augmentent avec le temps. 

Limites de cette étude 

•  Les  coordinateurs  de  projet  ayant  transmis  directement  les  données  utilisées  dans 
cette  étude  à  la  DG  CNECT,  nous  ne  pouvons  pas  les  vérifier  de  manière 
indépendante.  Des  exemples  possibles  de  biais  de  notification  sont  envisageables, 
comme  l'illustrent  les  déclarations  de  résultats  à  double  comptabilisation  et  les 
déclarations de résultats autres que celles du FP7. 

•  Le rapport n’évaluant qu’un seul domaine scientifique, à savoir les TIC, les résultats 

ne peuvent être généralisés à d’autres domaines scientifiques. 

•  Les résultats du rapport étant basés sur trois types de résultats; brevets, articles et 
comptes  rendus,  il  nous  convient  d’être  conscients  de  l’existence  d’autres  types  de 
supports  qui  ne  sont  pas  décrits  dans  le  présent  rapport  mais  qui  pourraient  être 
importants pour la diffusion des connaissances dans le domaine scientifique des TIC. 
Un exemple pourrait être un type de support en ligne non capturé ailleurs. 

•  L’analyse des mots-clés est une nouvelle méthode d’analyse des résultats des TIC du 
FP7.  Bien  que  nous  ayons  choisi  une  méthode  robuste,  elle  est  simple  et  peut  être 
encore  améliorée  si  elle  est  jugée  utile.  Par  exemple,  au  lieu  d'agréger  tous  les 
sujets  dans  un  seul  domaine  global,  les  modèles  de  sujet  de  domaines  d'intérêt 
spécifiques pourraient être analysés séparément. 

•  En  règle  générale,  les  évaluations  quantitatives,  aussi  minutieuses  et  robustes 
soient-elles, devraient être utilisées en parallèle avec d’autres systèmes d’évaluation 
qualitatifs afin d’obtenir une profondeur d’analyse nécessaire permettant de tirer des 
conclusions sur les implications possibles en matière de politiques. 

 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  23 

Research Highlights 

Highlights from the published and patented research, funded by FP7 in the area of ICT are 
presented  below.  We  have  taken  the  top  5  publications  that  have  gathered,  and  are  still 
gathering, an above average number of citations. We also show the top 3 patents in terms 
of  forward  citations.  As  part  of  the  highlights  we  add  an  abstract  of  the  patent  or 
publication,  the  number  of  citations  the  research  has  gathered,  and  under  which  type  of 
project and strategic objective they were produced.  

•  Authors:  Alegre, T.  et  al. (2011) “Laser  cooling of a nanomechanical oscillator into 

its quantum ground state”, Nature, Vol. 478(7367): pp. 89-92 

The simple mechanical oscillator, canonically consisting of a coupled mass-spring system, is 
used  in  a  wide  variety  of  sensitive  measurements,  including  the  detection  of  weak  forces 
and  small  masses.  On  the  one  hand,  a  classical  oscillator  has  a  well-defined  amplitude  of 
motion;  a  quantum  oscillator,  on  the  other  hand,  has  a  lowest-energy  state,  or  ground 
state,  with  a  finite-amplitude  uncertainty  corresponding  to  zero-point  motion.  On  the 
macroscopic  scale  of  our  everyday  experience,  owing  to  interactions  with  its  highly 
fluctuating  thermal  environment  a  mechanical  oscillator  is  filled  with  many  energy  quanta 
and  its  quantum  nature  is  all  but  hidden.  Recently,  in  experiments  performed  at 
temperatures  of  a  few  hundredths  of  a  kelvin,  engineered  nanomechanical  resonators 
coupled to electrical circuits have been measured to be oscillating in their quantum ground 
state  3,4.  These  experiments,  in  addition  to  providing  a  glimpse  into  the  underlying 
quantum  behaviour  of  mesoscopic  systems  consisting  of  billions  of  atoms,  represent  the 
initial steps towards the use of mechanical devices as tools for quantum metrology 5,6 or as 
a  means  of  coupling  hybrid  quantum  systems  7-9.  Here  we  report  the  development  of  a 
coupled,  nanoscale  optical  and  mechanical  resonator  10  formed  in  a  silicon  microchip,  in 
which  radiation  pressure  from  a  laser  is  used  to  cool  the  mechanical  motion  down  to  its 
quantum ground state (reaching an average phonon occupancy number of0:85±0:08). This 
cooling  is  realized  at  an  environmental  temperature  of  20  K,  roughly  one  thousand  times 
larger  than  in  previous  experiments  and  paves  the  way  for  optical  control  of  mesoscale 
mechanical oscillators in the quantum regime.  

1008  citations  gathered  as  of  September  2018.  This  article  was  produced  by  the  HiPEAC 
project. This 2011 article is from the DG CNECT project 248095 (call FP7-ICT-2009-4) titled 
from  the  “FET  proactive  2:  Quantum  Information  Foundations  and 
Q-ESSENCE, 
Technologies” strategic objective (CP-IP-INFSO-FET project). 

•  Authors:  Baldauf,  M.,  et  al.  (2007)  “A  survey  on  context-aware  systems”, 
International  Journal  of  Ad  Hoc  and  Ubiquitous  Computing,  Volume  2(4):  pp.  263-
277 

 
 
 
  
  
Final Study Report- D4 - SMART 2015/0078  24 

Context-aware  systems  offer  entirely  new  opportunities  for  application  developers  and  for 
end users by gathering context data and adapting systems behaviour accordingly. Especially 
in  combination  with  mobile  devices  these  mechanisms  are  of  high  value  and  are  used  to 
increase  usability  tremendously.  In  this  paper,  we  present  common  architecture  principles 
of context-aware systems and derive a layered conceptual design framework to explain the 
different  elements  common  to  most  context-aware  architectures.  Based  on  these  design 
principles, we introduce various existing context-aware  systems  focusing on context-aware 
middleware  and  frameworks,  which  ease  the  development  of  context-aware  applications. 
We discuss various approaches and analyse important aspects in context-aware computing 
on the basis of the presented systems. 

1154 citations gathered as of September 2018. This article was an output of the DG CNECT 
project 216462 (call FP7-ICT-2007-1) titled C-CAST, and part of the strategic objective “The 
Network of the Future” (CP-IP-INFSO project).  

•  Authors:  Novotny,  L.,  and  N.  van  Hulst  (2011)  “Antennas  for  light”,  Nature 

Photonics, Vol. 5: pp. 83–90 

Optical antennas are devices that convert freely propagating optical radiation into localized 
energy,  and  vice  versa.  They  enable  the  control  and  manipulation  of  optical  fields  at  the 
nanometre scale, and hold promise for enhancing the performance and efficiency of photo-
detection,  light  emission  and  sensing.  Although  many  of  the  properties  and  parameters  of 
optical  antennas  are  similar  to  their  radio-wave  and  micro-wave  counterparts,  they  have 
important  differences  resulting  from  their  small  size  and  the  resonant  properties  of  metal 
nanostructures.  This  Review  summarizes  the  physical  properties  of  optical  antennas, 
provides  a  summary  of  some  of  the  most  important  recent  developments  in  the  field, 
discusses the potential applications and identifies the future challenges and opportunities. 

1442 citations gathered as of September 2018. This article was produced by the DG CNECT 
project 224014 (call FP7 ICT-2007-2), named PHOTONICS4LIFE. This project is part of the 
strategic objective “Photonic components and subsystems” (NoE project).  

•  Authors:  Edfors,  O,  et  al.  (2014)  “Massive  MIMO  for  next  generation  wireless 

systems”, IEEE Communications Magazine, Vol. 52(2): pp. 186-198 

Multi-user MIMO offers big advantages over conventional point-to-point MIMO: it works with 
cheap single-antenna terminals, a rich scattering environment is not required, and resource 
allocation is simplified because every active terminal utilizes all of the time-frequency bins. 
However, multi-user MIMO, as originally envisioned, with roughly equal numbers of service 
antennas  and  terminals  and  frequency-division  duplex  operation,  is  not  a  scalable 
technology.  Massive  MIMO  (also  known  as  large-scale  antenna  systems,  very  large  MIMO, 
hyper MIMO, full-dimension MIMO, and ARGOS) makes a clean break  with current practice 
through  the  use  of  a  large  excess  of  service  antennas  over  active  terminals  and  time-

 
 
 
 
Final Study Report- D4 - SMART 2015/0078  25 

division duplex operation. Extra antennas help by focusing energy into ever smaller regions 
of  space  to  bring  huge  improvements  in  throughput  and  radiated  energy  efficiency.  Other 
benefits  of  massive  MIMO  include  extensive  use  of  inexpensive  low-power  components, 
reduced  latency,  simplification  of  the  MAC  layer,  and  robustness  against  intentional 
jamming.  The  anticipated  throughput  depends  on  the  propagation  environment  providing 
asymptotically  orthogonal  channels  to  the  terminals,  but  so  far  experiments  have  not 
disclosed  any  limitations  in  this  regard.  While  massive  MIMO  renders  many  traditional 
research  problems  irrelevant,  it  uncovers  entirely  new  problems  that  urgently  need 
attention:  the  challenge  of  making  many  low-cost  low-precision  components  that  work 
effectively  together,  acquisition  and  synchronization  for  newly  joined  terminals,  the 
exploitation  of  extra  degrees  of  freedom  provided  by  the  excess  of  service  antennas, 
reducing  internal  power  consumption  to  achieve  total  energy  efficiency  reductions,  and 
finding  new  deployment  scenarios.  This  article  presents  an  overview  of  the  massive  MIMO 
concept and contemporary research on the topic. 

1779 citations gathered as of September 2018. This article was produced by project 619086 
(call  FP7-ICT-2013-11)  which  is  named  MAMMOET.  This  project  is  part  of  the  strategic 
objective “Future Networks”  (CP-IP-INFSO project).  

•  Authors:  Pronk,  S.,  et  al.  (2013)  “GROMACS  4.5:  A  high-throughput  and  highly 
parallel  open  source  molecular  simulation  toolkit”,  Bioinformatics,  Vol.  29(7):  pp. 
845-854 

Molecular simulation has historically been a low-throughput technique, but faster computers 
and increasing amounts of genomic and structural data are changing this by enabling large-
scale  automated  simulation  of,  for  instance,  many  conformers  or  mutants  of  biomolecules 
with or without a range of ligands. At the same time, advances in performance and scaling 
now make it possible to model complex bio-molecular interaction and function in a manner 
directly  testable  by  experiment.  These  applications  share  a  need  for  fast  and  efficient 
software  that  can  be  deployed  on  massive  scale  in  clusters,  web  servers,  distributed 
computing  or  cloud  resources.  We  present  a  range  of  new  simulation  algorithms  and 
features  developed  during  the  past  4  years,  leading  up  to  the  GROMACS  4.5  software 
package.  The  software  now  automatically  handles  wide  classes  of  biomolecules,  such  as 
proteins,  nucleic  acids  and  lipids,  and  comes  with  all  commonly  used  force  fields  for  these 
molecules built-in. GROMACS supports several implicit solvent models, as well as new free-
energy  algorithms,  and  the  software  now  uses  multithreading  for  efficient  parallelization 
even on low-end systems, including windows-based workstations. Together with hand-tuned 
assembly  kernels  and  state-of-the-art  parallelization,  this  provides  extremely  high 
performance  and  cost  efficiency  for  high-throughput  as  well  as  massively  parallel 
simulations. 

 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  26 

2397  Citations  as  of  September  2018.  This  article  is  output  from  the  DG  CNECT  261523 
project  (call  FP7-INFRASTRUCTURES-2010-2).  This  project  is  named  ScalaLife.  The  RI 
project is part of the “Simulation software and services”  strategic objective (CP-CSA-Infra-
PP project). 

•  Inventors:  Tukaram  K.  Hatwar,  et  al.  (Priority  date:  2009-05-13)  “Internal 
CNECTor  for  organic  electronic  devices”.  EPO  patent:  EP2430677.  CPC  Class: 
H01L51/5044  Stack  of  electroluminescent  layers  with  spacer  layers  between  the 
emissive layers (+12 other classifications). 

GmbH, 
Original 
https://patents.google.com/patent/EP2430677 

Assignee: 

NovaLED 

Global 

OLED 

Technology 

LLC. 

The  invention  provides  an  electronic  device  including  an  anode  and  a  cathode,  between 
which there are at least two  organic phototransducing units where the units are  separated 
by an intermediate CNECTing region which comprises in sequence: an organic p-type layer, 
an  intermediate  layer  in  direct  contact  with  the  organic  p-type  layer  and  including  a 
compound  that  has  a  LUMO  more  negative  than  -3.0  eV  and  is  different  from  the  organic 
compound in the organic p-type layer, and an n-type doped  organic layer in direct contact 
with  the  intermediate  layer  and  including  an  electron  transport  material  as  a  host  and  an 
organic  n-dopant  with  a  HOMO  less  negative  than  -4.5  eV.  In  one  embodiment,  the 
electronic device is a tandem OLED. 

32  forward  citations  according  to  OECD  patent  quality  indicators  (Google  patents:  115 
citations.) This patent is output from the DG CNECT project number 216815 (call FP7-ICT-
2007-1). This project is named AMAzOLED. This project was part of the “Organic and large-
area  electronics,  visualisation  and  display  systems”  strategic  objective  (CP-FP-INFSO 
project). 

•  Inventors: Nicole L.W. Eikelenberg, et al. (Priority date: 2010-06-21) “Method and 
system  for  determining  a  route  for  efficient  energy  consumption”.  USPTO  patent: 
US2011309926.  CPC  Class:  G01C21/3469  Fuel  consumption;  Energy  use;  Emission 
aspects (+1 other classification). 

Original 
https://patents.google.com/patent/US20110309926 

Assignee: 

Ford 

Global 

Technologies 

LLC. 

Various  embodiments  relate  to  determining  a  route  for  efficient  energy  consumption.  Data 
defining  one  or  more  routes  and  one  or  more  consumption  influencing  parameters  which 
influence  a  consumption  of  an  energy  source  of  the  vehicle  along  the  one  or  more  routes 
may  be  received  at  a  vehicle  computer.  Based  on  the  route  data  and  the  consumption 
influencing parameters, at least  one  route which minimizes the  consumption of the energy 

 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  27 

source for the vehicle may be determined. For the at least one route, one or more braking 
strategies  may  be  determined  for  implementation  by  a  vehicle  operator  in  the  vehicle  in 
accordance  with  reducing  the  consumption  of  the  vehicle  energy  source.  The  one  or  more 
braking strategies may be output in the vehicle. 

 30  forward  citations  according  to  OECD  patent  quality  indicators  (Google  patents:  43 
citations).  This  patent is  output  from  the  DG  CNECT  project  246587  (call  FP7-ICT-2009-4) 
and  titled  "interactIVe".  This  project  was  part  of  strategic  objective  “ICT  for  Safety  and 
Energy Efficiency in Mobility” (CP-IP project). 

•  Inventors:  Alexander  Glikson,  Assaf  Israel  and  Dean  Har'el  Lorenz  (Priority  date: 
2009-12-16)  “Live  multi-hop  vm  remote-migration  over  long  distance”.  USPTO 
patent:  US2011145380.  CPC  Class:  G06F9/5077  Logical  partitioning  of  resources; 
Management or configuration of virtualized resources (+3 other classifications). 

Original 
https://patents.google.com/patent/US20110145380 

International 

Assignee: 

Business 

Machines 

Corp. 

Methods and systems of computing, involve identifying a target location for live migration of 
virtual  machine  services,  delivering  the  virtual  machine  services  to  an  operating  system 
from  a  source  location,  migrating  the  virtual  machine  services  a  first  time  via  a  data 
network from the source location to an intermediate location, delivering the virtual machine 
services  to  the  operating  system  from  the  intermediate  location,  and  migrating  the  virtual 
machine services a second time from the intermediate location to the target location via the 
data  network.  Breaks  in  the  virtual  machine  services  to  the  operating  system  while 
migrating the virtual machine are minimal. 

28  forward  citations  according  to  OECD  patent  quality  indicators  (Google  patents:  43 
citations). This patent is output from DG CNECT project 215605 (call FP7-ICT-2007-1) titled 
"RESERVOIR",  and  part  of  the  strategic  objective  “Service  and  Software  Architectures, 
Infrastructures and Engineering”, (CP-IP project). 

 
 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  28 

 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  29 

Recommendations 

Scientometric  analysis  has  proven  itself  as  an  effective  tool  in  research  evaluations. 
However, accessing the necessary data is not straightforward. Monopolistic behaviour by the 
large aggregators, e.g. citation indexes, is often a show-stopper in this respect. Therefore, if 
the  EC  would  initiate  efforts  in  creating  access  to  free  meta-data  (from  large  bibliographic 
sources),  it  would  be  beneficial  to  evaluations  and  beyond.  Scale  effects  would  be  created 
by studies, outside of any contractual work, on the basis of this data. 

In terms of surveying, we could envisage improvements through the use of an online survey 
tool  with  a  more  compulsory  nature,  not  just  in  terms  of  compulsory  reporting  but  also  in 
terms of compulsory, and controlled, fields within the survey tool. This would ideally not be 
a  tool  based  on  Excel  spreadsheets,  but  one  in  which  the  survey  tools'  fields  would  feed 
directly  into  a  type  of  relational  database,  with  predefined  fields  and  character  encodings, 
and ideally through a web-based portal. 

Following  from  the  above  we  note  that  implementing  target  thresholds  for  the  surveying 
would  be  an  additional  tool  to  improve  the  availability  of  data  for  evaluation  purposes. 
Basing these thresholds on the number of active projects for each of the productive years of 
the  framework  would  be  advisable.  Although  FP7  projects  were  launched  from  2007  to 
2013,  the  projects  duration  was  from  2007  to  2018,  with  “high  volume”  years  running  to 
2016 after which productive projects significantly reduce towards 2018. Adjusting surveying 
thresholds accordingly would be as such beneficial. 

In  the  context  of  the  present  study,  only  a  number  of  patents  is  considered,  due  to  the 
“recentness”  of  the  patents  reported.  Citation  lag  plays  an  important  influence  in  this 
regard. Although understandable in terms of an ex-post evaluation, which needs to produce 
facts  upon  which  to  act  relatively  quickly,  a  thorough  impact  assessment  on  the  basis  of 
patent  citations  should  be  taken  into  account  in  future  evaluations  by  using  a  longer 
evaluation horizon. 

Although  the  funded  projects  are  goal-oriented,  these  goals  are  not  directly  measured  by 
the  type  and  number  of  outputs  but  are  rather  seen  as  contributing  to  a  larger  objective. 
However  we  do  see  some  possibility  for  adjustment  of  the  (funding) policy  on  the  basis  of 
the  outcomes  of  the  present  study.  For  instance  when  looking  at  the  outcomes  of  the 
econometric analysis we can hypothesise that if the larger strategic objective of a call would 
be  in  supporting  or  increasing  applied  research  (i.e.  patenting),  the  focus  should  be  on 
setting  up  smaller  teams  (so  with  fewer  participants),  with  a  high  diversity  in  terms  of 
organizational types, and large per capita funding. Conversely, when the interest would be 
in    supporting  basic  science  and  knowledge  creation  (i.e.  publications),  larger  teams  (with 
multiple  participating  organisations),  with  high  diversity  in  terms  of  nationality,  but  a  low 
diversity  in  organization  type  (so  just  academic/public  research  institutes),  would  make  a 
bigger impact even though the per capita funding would not be that important a factor. Care 

 
 
 
Final Study Report- D4 - SMART 2015/0078  30 

has  to  be  taken  however  with  respect  to  over-emphasizing  diversity  and  team  size  as  this 
seems to have adverse effects on publication quality. 

From  our  keyword  analysis  we  can  also  derive  a  possibility  for  a  policy  shift.  Instead  of 
country specific calls, “knowledge" specific calls could be of interest. These calls could look 
at third countries with which topical affinity is low, in order to gain knowledge which is less 
well researched in the EU-28 and gain a competitive advantage in the process.  

Finally,  and  also  with  respect  to  calls  that  should  lead  to  citations,  an  effectiveness  and 
efficiency  gain  could  be  made  by  incorporating  regulations  on  where  to  publish,  while 
making  it  easier  for  scientists  (with  a  proven  track-record  in  highly  cited  publications  and 
publishing in high-impact journals) to gain access to funding. New academic entrants should 
follow  university  standards  in  publishing:  e.g.  targeting  “A”  journals.  These  sorts  of 
requirements are already put into practice by other governmental funding agencies (i.e. NIH 
and NSF in the US4). 

4 National Institutes of Health, US Department of Health, and US National Science Foundation. 

 
 
 
 
 
                                                           
Final Study Report- D4 - SMART 2015/0078  31 

1  Introduction 

The document presented here is the final study report on the “Analysis of publications and 
patents of ICT research in FP7 and CIP”, and the final deliverable D4 of SMART 2015/0078. 
This document presents an additional analysis to the earlier Annual Reports, D2 and D3, as 
well as re-validation and re-evaluation of the data analysed for SMART 2011/00395,6.  

This study report mainly deals with a broad scientometric7 and econometric analysis of the 
publications and patents reported through the  surveys to DG CNECT.  It is divided into five 
main chapters. After the Introduction, Chapter 2 deals with the management of the data as 
received  by  the  project  team  from  DG  CNECT,  as  well  as  the  Scopus  Custom  data  bought 
from Elsevier, and the patent data as received from the European Patent Office (EPO). The 
main items discussed in this chapter concern the validation and matching of the publications 
and  patents,  and  the  construction  of  the  data  sets  that  are  the  basis  of  the  analysis 
presented in subsequent chapters.  

In  Chapter  3,  a  bibliometric  analysis  of  the  publications  as  reported  to  DG  CNECT  is 
provided.  Here,  in  addition  to  the  yearly  information  provided  in  the  annual  reports,  we 
undertake  an  analysis  of  the  aggregated  and  re-validated  data  for  the  FP7  years  of  2007-
2013.  

In  Chapter  4,  an  in-depth  analysis  of  the  patents  reported  is  undertaken.  This  analysis  is 
likewise  based  on  the  aggregated  and  re-validated  data  for  the  years  2007-2013.  Both 
Chapter 3 and 4 seek to address four main assessment focal points: 

•  The productivity of the projects, funding schemes and strategic objectives under FP7. 

•  The impact of the ICT research done in FP7. 

•  The collaborative behaviour of the participants in the FP7 projects. 

Chapter 5 presents an econometric analysis drawing a set of variables taken from Chapters 
3 and 4. This analysis consists of an econometric model which gives a longitudinal picture of 
the productivity of the projects sponsored by DG CNECT during FP7. 

Chapter  6  gives  an  overview  of  the  newly  created  organisations  resulting  from  the  FP7 
funded ICT projects. 

Finally, chapter 7 presents a novel method in the context of scientometrics. A keyword and 
key-phrase  analysis  of  the  textual  information,  as  present  in  the  publication  records, 
attempts  to  capture  the  position  of  the  DG  CNECT  sponsored  research  on  the  basis  of  its 
content. 

5 https://ec.europa.eu/digital-single-market/en/news/inventory-patents-and-publications-fp7-projects-field-ict 
6 http://ec.europa.eu/newsroom/dae/document.cfm?action=display&doc_id=14504 
7 Scientometrics is concerned with the quantitative features and characteristics of science and scientific research. It 
consists mostly of research in which the development and mechanism of science are studied. 

 
 
 
                                                           
Final Study Report- D4 - SMART 2015/0078  32 

1.1  Literature overview 
For the preparation of this report, we have based ourselves on prior research in the area of 
scientometrics  and  on  this  particular  topic.  Foremost,  the  work  by  professor  Breschi  on 
“Using performance indicators in monitoring the implementation of ICT research in FP6 and 
FP7” (Breschi, 2010), and by professors Breschi and Malerba (2011) on the same topic and 
methodology  were  influential.  Furthermore,  the  literature  on  bibliometrics  and  patent 
analysis presented the authors of this report with a number of interesting and inspirational 
methods.  For  the  bibliometrics  used  in  this  report,  we  have  based  the  construction  of  our 
data  set  on  the  work  by  professor  Leydesdorff  and  co-authors.  Specifically,  the  use  of 
journals  sets  as  a  delineation  of  an  interdisciplinary  field  (Leydesdorff,  L.  and  C.  Probst, 
2009), as well as the use of citations as a comparative performance indicator (Zhou, 2009, 
Braun et al., 1985), are methods we have implemented. For the patent analysis, the works 
of  Francis  Narin  (1995)  and  Hariolff  Grupp  (1998)  have  been  the  methodological  basis, 
while also more recent work by the authors has been used. 

A  separate  methodological  literature  has  been  the  one  on  network  analysis.  Network 
analytical  methods  are  increasingly  used  and  created  into  other  strands  of  scientific 
research,  and  this  is  certainly  the  case  with  scientometrics.  The  main  authors  and  works 
used  here  are  Granovetter  (1973),  Wasserman  and  Faust  (1994),  and  Knoke  and  Yang 
(2008). 

1.2  Scientometric analysis 
In  our  understanding,  scientometric  analysis  can  be  defined  as  the  combination  of 
bibliometrics  and  patent  analysis,  which  are  two  closely  related  approaches  to  measuring 
scientific research and development, and science in general. Apart from productivity indices 
and  impact  analyses  involving  various  types  of  citation  analysis,  the  data  used  in  this 
research can inform us about networks of scientists and scientific communication. 

As part of scientometrics, bibliometrics is the most important way of measuring the impact 
of  scientific  publications.  If  an  article  is  published  in  a  journal  with  a  high  impact  factor, 
which  in  turn  is  determined  by  the  number  of  citations  towards  the  universe  of  articles 
within  that  particular  journal, it has  an impact on  the  profile  of  the  respective  author.  The 
numbers  of  citations  to  articles  over  time  are  as  such  an  important  and  widely  used 
measure of the productivity and the impact of science and R&D. 

Patent analysis uses principally the same measures as used in bibliometrics, but for applied 
research and its outputs. Techniques in this field are very well developed and are not  only 
used in public research but also for commercial purposes. 

 
 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  33 

2  Data Management 

2.1  Provided data 
The  data  provided  to  us  by  DG  CNECT  consists  of  a  series  of  Excel  files,  on  the  reporting 
output for the 2014-2016 years of EU-funded projects, under FP7-ICT. The Excel files are a 
product  of  the  annual  surveys  conducted  by  DG  CNECT  in  which  project  coordinators  are 
requested  to  report  the  outputs  that  their  projects  produced  in  the  year  preceding  the 
survey.  These  reports  are  themselves  Excel  files.  We  also  had  access  to  the  previous 
reporting output for the years 2007-2013, as  well as the analysis previously performed by 
us on the earlier years. 

The  reporting  procedure  has  evolved  throughout  the  years.  In  the  early  years  of  FP7 
reporting initiatives, Excel files have been sent out to project leaders for filling in the output 
information and these files were later collected and provided to us either as a collection of 
files, or first aggregated manually at the DG and provided to us as a consolidated Excel file. 
In recent years, the project coordinators have been invited to download, fill in, and upload 
Excel files, which were  collected using automated software. Special care has been put into 
adapting our practices and algorithms throughout the years to the changing data structure 
and reporting style. 

For  the  recent  years,  the  individual  project  files  contain  four  main  sheets,  providing 
information about patents (e.g. patent office, application number, etc.), articles and papers 
(e.g.  title,  DOI  code,  journal  name,  authors,  etc.),  new  organization  (e.g.  legal  name, 
country of registration, etc.), and innovations (e.g. type, description, etc.) corresponding to 
the  projects.  Files  containing  metadata  about  the  projects  are  also  at  team's  perusal, 
contain  information  such  as  the  programme,  framework,  funding  scheme,  and  strategic 
objective under which each project is categorized.  

The analysis performed in this study depends greatly on the quality of the data provided to 
us by DG CNECT. It is however important to acknowledge the fact that this data is compiled 
from  the  survey  answers  of  project  leaders  in  a  voluntary  reporting  process,  which  takes 
place each year. As such, data is not available for all funded projects,  and for the projects 
for which it is available, it is not necessarily available for all the funding years. 

2.2  Data Cleaning and Validation: Publications 
In  accordance  with  our  methodology,  the  first  step  in  validating  the  provided  data  was 
performed  using  Scopus.  For  this  purpose,  the  Scopus  API  (Application  Programming 
Interface)  has  been  used  to  automatically  match  the  publications  with  Scopus  data,  and 
automatic  validation  has  been  attempted.  As  expected,  through  our  experience  in  the 
previous  SMART  2011/0039  project,  this  approach  provides  the  best  return  on  invested 
time. 

Two  methods  have  been  used  for  automatic  validation.  Firstly,  the  API  has  been 
automatically  queried  with  the  provided  DOI  number  (if  available)  of  each  publication  in 

 
 
 
Final Study Report- D4 - SMART 2015/0078  34 

turn,  and  in  the  case  of  exactly  one  returned  match,  the  match  has  been  automatically 
validated  by  title:  on  same  title  this  match  is  recorded  as  valid,  and  on  different  title  the 
publication is returned to the pending pool. 

Secondly, the API has been automatically queried with the provided title of each publication 
in  turn,  and  in  the  case  of  exactly  one  returned  match,  the  match  has  been  automatically 
validated  by  first  author:  on  same  first  author  this  match  is  recorded  as  valid,  and  on 
different first author the publication is returned to the pending pool.  

Manual  validation  of  publications  on  the  other  hand  raises  the  problem  of  diminishing 
returns  on  time  invested.  As  we  noted  in  our  previous  work,  the  process  of  manual 
validation  is  time  consuming.  In  order  to  utilize  this  relatively  short  time  window  most 
efficiently,  emphasis  is  put  on  using  manual  validation  as  a  way  to  identify  potential 
problem areas and increase the validation range of the automated process. 

The  results  of  the  publication  validation  and  cleaning  efforts  for  the  survey  years  2014-
2016,  combined  with  the  validation  and  cleaning  efforts  for  the  survey  years  2007-2013, 
are presented in Table 1 below by publication year. Although low or relatively low numbers 
in the early years are easily explained as a clear start-up issue, the low figures for the final 
productive years are disappointing. Specifically, due to the low response rate, the reported 
output for 2016 does not correspond with the projects still running in this specific year. 

Table 1. Results of the publication validation, survey waves 2007-2016. 

Years 

2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  Total 

Papers 

27 

1014  1143  2661  2526  1434  2839  2014  1808  175 

33 

Articles  5 

247 

458 

1190  1354  899 

1749  1224  974 

183 

62 

Other 

2 

18 

39 

93 

110 

70 

113 

80 

85 

18 

8 

Total 

34 

1279  1640  3944  3990  2403  4701  3318  2867  376 

103 

2 

6 

1 

9 

15676 

8351 

637 

24664 

2.3  Data Cleaning and Validation: Patents 
Similarly to the publications dataset the focal dataset of FP7 patents has been built on two 
sources:  

•  New  data  from  the  DG  CNECT  surveys  of  2016  and  2017  year,  covering  patenting 

activity of FP7 participants in 2014-15 and 2016 respectively, and  

•  The  set  of  FP7  patents  validated  in  the  previous  study  based  on  the  DG  CNECT 

surveys from 2009 to 2013.   

 
 
 
 
Final Study Report- D4 - SMART 2015/0078  35 

The data from the new surveys was, first, cleaned and then validated by matching reported 
patents  to  existing  online  patent  databases  –  Google  Patents,  EPO’s  Espacenet,  USPTO 
PatFT, and WIPO’s Patentoscope. In a few cases, online databases of national patent offices 
(UK  Ipsum  and  German  DPMA)  had  to  be  utilized.  Matching  was  fairly  straightforward  in 
cases  where  the  respondents  to  the  surveys  provided  the  correct  link in  the  filed  “Patent's 
URL at Espacenet” (32 patents out of 136 in survey 2016, and 4 out of 8 in 2017 surveys), 
however  in  many  questionnaires  this  part  of  information  on  patents  was  missing  (104  in 
2016  survey  and  4  in  2017  survey).  In  those  cases  we  resorted  to  searching  the 
abovementioned  online  patent  databases  using  the  other  information  provided  by  the 
survey respondents; patent’s application number, patent title, names of the inventor(s) and 
applicant(s).  In such searches Google Patents, based on fuzzy words matching, generated 
most satisfactory results (wider than Espacenet’s “Smart search”). However Google Patents 
covers  only  patent  applications  filed  at  the  major  patent  offices:  EPO,  USPTO,  WIPO,  and 
only  a  few  national  patent  offices  such  as  DPMA.  Therefore  we  first  tried  finding  reported 
patents or their equivalents (patents of the same patent family) on Google Patents. For the 
patents which we could not find there, but for which we knew that they were applied at the 
national patent offices, we reverted to searching Espacenet. 

Not all reported patents, however,  could be  found in the available online patent databases 
(43 out of 136 for survey 2016 and 4 out of 8 in the survey 2017); we attribute such cases 
to the following reasons: 

• 

Information  provided  by  the  respondents  of  the  survey  was  not  sufficient  to 
(unambiguously) identify the patent, 

•  Corresponding  patent  application  has  not  yet  been  filed  by  the  organization 
(respondents  provide  only  a  tentative  title  and/or  internal  to  the  organization 
reference number)  

•  Patent  application  has  been  filed  but  not  yet  published  by  the  patent  office  by  the 
time we have Collected data (e.g. EPO publishes patent application 18 months after 
filing or 18 months after priority date),  

•  Patent  application  has  been  filed  but  withdrawn  by  the  applicant  before  it  was 

published,  

•  Patent application has been filed with national patent office where online search were 

unavailable or difficult to perform. 

After  we  have  found  a  patent  in  one  of  the  online  patent  databases,  we  recorded  its 
publication reference number and used this number to match it to the most recent version 
of EPO PATSTAT database (April 2018) in order to obtain patent information in a structural 
form. 

 
 
 
Final Study Report- D4 - SMART 2015/0078  36 

In  some  cases,  when  the  same  invention  has  been  covered  by  patent  applications  in 
multiple  patent  offices,  those  multiple  equivalent  patents  have  been  reported  as  separate 
patents. To mitigate double counting of such patents, we have adopted a procedure, which 
has been employed in the earlier studies (the KITES-Bocconi pilot study and SMART project 
2011/0039). If the reported patent has an equivalent at EPO, we count it as an EPO patent, 
when  the  reported  patent  does  not  have  an  EPO  equivalent  but  has  an  equivalent  at  the 
USPTO  we  count  it  as  a  USPTO  patent.  If  it  has  neither  an  EPO  nor  a  USPTO  but  a  PCT 
application at WIPO, we count it as a WIPO patent, and in the remaining cases where patent 
applications have been filed only in a national patent office we assign it to the corresponding 
national office. 

We  have  repeated  such  procedures  also  for  the  data  produced  in  the  previous  SMART 
project  (SMART  2011/0039),  to  provide  uniform  treatment  for  the  FP7  patents  earlier 
recorded  as  WIPO  or  ‘national’  patents.  Even  though  some  of  those  patents  had  been 
already  applied  at  EPO  and/or  USPTO,  these  were  not  yet  published  at  the  time  we  were 
conducting  the  previous  study.  Therefore  their  EPO  or  USPTO  equivalents  were  not  yet  in 
PATSTAT,  but  have  since  been  published  (during  the  time  elapsed  between  the  previous 
SMART 2011/0039 and the current SMART 2015/0078 study). 

As  has  been  noted  already  in  the  previous  study  and  the  Annual  Reports,  some  projects 
reported patents with priority dates preceding the start dates of those projects. In such case 
the invention which the patent is referring to cannot possibly stem from the research funded 
by the project. We have identified another 3 of these patents among the patents reported in 
the  2016  survey,  in  addition  to  25  such  patents  observed  in  the  previous  study.  Those 
patents have been excluded from our analysis. 

2.4  Publication data: Focal set 
The validated dataset is essentially a sub-part of a larger universe of publications in the field 
of  ICT.  We  use  this  larger  universe  to  construct  both  a  focal  sample  representing  FP7  ICT 
projects, and a control sample by which we would like to benchmark the output reported to 
DG  CNECT.  We  primarily  focus  our  attention  on  the  publication  output  as  published  in  the 
most popular journals and proceedings.  

The  total  sampling  population  is  based  on  a  selection  of  source  titles  in  which  FP7  funded 
authors have published 4 or more articles or papers in any of the target analysis years. As 
such,  we  have  a  target  journal  and  proceedings  set  of  1176  individual  source  titles, 
containing 19273 DG-sponsored publications. This can be seen in Table 2. 

 
 
 
 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  37 

Table 2. Total focal sampling population, distribution across publication types and resulting target set. 

Years 

2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  Total 

Papers 

17 

708 

835 

2063  1926  1040  2329  1622  1445  116 

25 

Articles  4 

208 

376 

996 

1083  706 

1428  1018  809 

111 

40 

Other 

0 

8 

19 

48 

56 

33 

71 

56 

55 

11 

6 

Total 

21 

924 

1230  3107  3065  1779  3828  2696  2309  238 

71 

2 

2 

1 

5 

12128 

6781 

364 

19273 

We apply a proportional sampling methodology for the extraction of both a focal sample and 
a control sample. In both samples, the proportional allocation by year and document type is 
fixed to the proportions of DG-sponsored output present in each year and by each document 
type.  For  the  focal  set,  given  a  population  of  19273  publications  and  using  a  confidence 
level  of  99%  and  a  confidence  interval  of  2%,  we  extract  a  focal  sample  of  3422 
publications. Shown below, in Table 3, the year and document type distribution for this focal 
sample. 

Table 3. Focal set, stratified on years and publication types. 

Years 

2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  Total 

Papers 

Articles 

Other 

Total 

4 

1 

0 

5 

141 

157 

366 

347 

196 

386 

280 

251 

25 

35 

63 

163 

186 

123 

238 

170 

136 

26 

3 

8 

17 

17 

12 

18 

12 

16 

5 

5 

9 

3 

179 

228 

546 

550 

331 

642 

462 

403 

56 

17 

1 

1 

1 

3 

2159 

1151 

112 

3422 

2.5  Publication data: Control set 
We apply a proportional sampling methodology for the extraction of both a focal sample and 
a control sample. In both samples, the proportional allocation by year and document type is 
fixed to the proportions of DG-sponsored output present in each year and by each document 
type. The control population against which we would like to benchmark the output reported 
to DG CNECT, is the set of publications present in the above defined 1176 individual source 
titles.  

 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  38 

As such, for the control set, given a control population of 6846926 publications and using a 
confidence  level  of  99%  and  a  confidence  interval  of  2%,  we  extract  a  control  sample  of 
4158  publications  with  the  year  and  document  type  distribution  matching  the  proportional 
distribution in the focal set and focal sample. 

Table 4. Control set, stratified on years and publication types.  

Years 

2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  Total 

Papers 

Articles 

Other 

Total 

5 

1 

0 

6 

169 

190 

448 

423 

241 

479 

340 

305 

30 

6 

42 

78 

200 

229 

152 

292 

207 

165 

31 

11 

5 

8 

12 

11 

15 

18 

17 

18 

5 

3 

216 

276 

660 

663 

408 

789 

564 

488 

66 

20 

0 

2 

0 

2 

2636 

1410 

112 

4158 

2.6  Patent data: Focal set 
Following  the  data  cleaning  and  validation  procedure  outlined  in  section  2.2  we  have 
validated  353  patent  applications  reported  across  181  FP7  projects.  Table  5  reports  the 
number of validated patents in each wave of the DG CNECT survey. 

Table 5. Focal sample of validated FP7 patents by survey year. 

Years 

2009 

2010 

2011 

2012 

2013 

2016 

2017 

Patents 

61 

41 

91 

74 

29 

90 

3 

The number of distinct patents (single patent per patent family) is, however, lower than the 
sum  of  the  patent  counts  in  the  table,  because  some  patents  were  reported  across  survey 
years (mostly because respondents consider patents from the same patent family as distinct 
patents).  In  total,  the  focal  dataset  contains  353  distinct  patents  validated  and  found  in 
PATSTAT. 

 
 
 
 
 
 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  39 

Figure 1. Distribution of patents by patent office. 

Most  FP7  patents  have  equivalent  patents  at  EPO  (198  out  of  353  patents,  >50%)  and/or 
USPTO  (96,  >25%)  and/or  WIPO  (38,  >10%),  only  few  national  patents  have  no 
equivalents in the international patent offices: 16 German, 2 Italian, and 1 patent in French, 
Greek and Polish patent (see Figure 1). 

Figure 2 reports the priority years of the FP7 patent. The peak of patenting activity was in 
2010-11. 

Figure 2. Patent counts by priority year. 

 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  40 

2.7  Patent data: Control set 
We have produced two control samples which mimic specific features of the focal sample of 
FP7 patents. To generate control sample 1 for each patent in the focal sample we randomly 
draw one control patent from the population of patents which match the focal patent by:  

•  Patent office at which the focal patent was applied. 

•  Priority year of the patent application. 

•  Same technology as reflected by matching at least one 4-digit IPC class of the focal 

patent. 

We made sure that the matching population contains neither members of the patent family 
of the focal patent, nor patents of the same applicant8. This has produced a sample of 346 
focal-control patent pairs. The 7 patents missing cannot be matched. 

For  control  sample  2  we  impose  an  additional  restriction  to  the  three  criteria  above:  the 
country of the applicant in the control patent (at least one) must match the country of the 
applicant in the focal patent (at least one). This has produced a control sample of 339 focal-
control pairs (14 focal patents could not be matched) with 339 focal and 338 control patents 
(one control patent is matched to two different focal patents). 

8 Here we rely on disambiguation of patent applicants provided in PATSTAT. 

 
 
 
 
 
                                                           
Final Study Report- D4 - SMART 2015/0078  41 

3  Publication Analysis 

3.1  Publication data enhancement 
The validation and matching process  of the DG CNECT survey data,  which was established 
in  the  previous  ex-post  evaluation  project  SMART  2011/0039,  has  been  fine-tuned  during 
the  preparation  of  the  Annual  Reports  2014-2015,  and  2016,  that  have  been  delivered  to 
DG CNECT (D2 and D3). Using automated validation, backed up by manual validation where 
necessary, a matching rate of more than 60% has been achieved. After the validation stage 
the UNU-MERIT team further  enhanced the survey data by combining the validated survey 
data with data from other sources, mainly Elsevier’s Scopus citation index (using a Scopus 
custom  data  set)  and  project  participant  data  from  the  EC  and  DG  CNECT.  Several  key 
fields, such as the media type data, author data, organizational affiliation data, EC contract 
and sub-contract, as well as topic fields, are of specific interest. 

The  data  filtering  was  done  using  VantagePoint9  text-mining  software  which  enables  the 
user  to  parse  the  original  data-fields  of  a  record  into  sub-fields  which  can  be  used  more 
efficiently.  That  said,  one  must  not  under-estimate  the  amount  of  time  invested  in  the 
filtering as well as further (sub) field clean-ups using fuzzy algorithms and manual cleaning 
efforts.  For  instance,  the  affiliation  field  for  the  4148  record  control  sample  consisted  of 
8680  different  instances  of  affiliations,  while  the  larger  validated  file  of  24664  publication 
records had a staggering 42529 instances of affiliations that were subsequently reduced to 
15819.  A  further  enhancement  was  provided  through  the  use  of  thesauri  with  which  we 
classified the organization types (STREAM classification) and the regional blocks (EU-28, FP7 
Associated, BRICS, etc.). 

3.2  Journals and proceedings 
As  explained  earlier,  our  publication  data  consists  of  the  validated  publication  output  as 
reported to DG CNECT in their surveys for the years 2007 to 2016. This validated dataset is 
essentially a sub-part of a larger universe of publications in the field of ICT. We can further 
see this universe as split up into three categories: 

•  Articles originating from journals, 

•  Proceeding papers originating from conference proceedings. 

•  Literature contained in books, book chapters, reviews etc.  

Peer reviewed articles have been the backbone of bibliometric studies and therefore are the 
main part (though not the largest) of this study. However, in the previous project (SMART 
2011/0039)  as  well  as  in  the  Pilot  Study  by  KITes  Bocconi,  the  importance  of  conference 
proceedings has been underlined, specifically within the fields of mathematics and computer 

9  VantagePoint  is  a  powerful  text-mining  tool  for  discovering  knowledge  in  search  results  from  patent  and 
literature databases. See: https://www.thevantagepoint.com/ 

 
 
 
                                                           
Final Study Report- D4 - SMART 2015/0078  42 

science,  as  well  as  the  ICT  research  field  at  large  (Montesi  and  MackenzieOwen,  2008). 
Therefore  we continue the inclusion of this, as we now know, important media type in our 
study.  As  our  dataset  has  grown  so  has  the  relative  size  of  the  “other”  category.  This 
category also consists of peer-reviewed literature and should as such not be ignored, hence 
the inclusion of this category in our study. 

Our validation process results with respect to  the division of media types are shown below 
in  Table  6.  Here  we  see  that  the  Scopus  API  finds  matches  for  the  reported  data  in  more 
than 10 categories. On closer inspection we find that several categories can be combined as 
they  are  from  the  same  type  of  source.  For  instance  Conference  Papers  and  Conference 
Reviews are in essence both proceedings from a conference. In the right three columns we 
have  aggregated  these  categories  in  the  earlier  mentioned  three  broad  categories  of 
articles, proceedings and “other”. 

Table 6. Distribution of validated output across Scopus media types. 

Percentage 

Media type 

Percentage*  Media type 

Number of 
publications 
8717 
15681 
253 

Number of 
publications* 
8351 
6 
8 
124 
15676 
5 
121 
6 
21 
14 
319 
13 
24664 

35.4 
63.6 
1.0 

Articles 
Proceedings 
Other 

Article 
Article in Press 
Book 
Book Chapter 
Conference Paper 
Conference Review 
Editorial 
Erratum 
Letter 
Note 
Review 
Short Survey 
Total 
Note:  Second  and  third  column  represent  the  division  as  classified  by  Scopus.  The  fifth  and  sixth 
columns are aggregated by the authors as described in the text. 

33.86 
0.02 
0.03 
0.50 
63.56 
0.02 
0.49 
0.02 
0.09 
0.06 
1.29 
0.05 
100% 

24664 

100% 

If  we  aggregate  the  proceedings,  we  see  a  clearer  division  between  the  articles  and 
proceedings  of  around  approximately  60/40.  This  corresponds  with  the  results  from  the 
earlier  studies  (SMART  2011/0039  and  the  KITes  Bocconi  Pilot  Study),  and  are  correct  in 
assuming  that  the  scientists  in  the  field  of  computers  science  and  ICT  have  a  large 
propensity for publishing in conference proceedings.  

The assumption that the proceedings would not have as significant an impact when looking 
at citation rates, as opposed to productivity, is also still valid but we find that a subtle shift 
in  favour  of  the  proceedings  has  taken  place.  In  Table  7  below,  we  show  the  percentage 
distribution  of  the  citations  gathered  over  the  media  types,  as  classified  by  Elsevier's 
Scopus, as well as aggregated into the three broad categories. 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  43 

Table 7. Distribution of citations across media types. 

Articles 
Conference 
Other 

Percentage 
63.0 
36.8 
0.2 

Times Cited 
258967 
151062 
906 

Media type 
Article 
Article in Press 
Book 
Book Chapter 
Conference Paper 
Conference Review 
Editorial 
Erratum 
Letter 
Note 
Review 
Short Survey 
Total 
Note: Column five and six represent the aggregated distribution across journal articles and proceeding 
papers. 

Percentage*  Media type 
57.5 
0.0 
0.1 
0.1 
36.8 
0.0 
0.1 
0.0 
0.1 
0.1 
5.2 
0.1 
100% 

Times cited* 
236489 
8 
207 
490 
151062 
0 
209 
1 
496 
279 
21407 
287 
410935 

410935 

100% 

We find that over two thirds of all the citations are gathered by the journal articles but that 
well  over  one  third is  gathered  by  the  proceedings.  Previously, in  2014,  we  calculated  this 
difference as ¾ of all citations where gathered by the articles while only ¼ was gathered by 
the  proceedings.  Each  proceeding  paper  now  gathers  9.63  citations  on  average,  as  can  be 
seen  in  Table  8,  while  each  reported  article  gathers  29.66  citations  on  average.  The  other 
category has a negligible amount of citations, but gathers on average only 3.58 citations per 
publication. 

Table 8. Distribution of citations across media types, and per publication. 

Media type 
Article 
Conference Paper 
Other 
Total 

# of pub 
8730 
15681 
253 
24664 

Percentage 
35.4 
63.6 
1 
100% 

# of cit 
258967 
151062 
906 
410935 

Percentage 
63 
36.8 
0.2 
100% 

Cit per pub 
29.66 
9.63 
3.58 
16.66 

These  facts  underscore  our  choice  for  the  use  of  the  journal  set  methodology  versus  a 
disciplinary  model  in  which  a  universe  would  be  constructed  of  all  output  in  a  number  of 
relevant  disciplines.  The  theoretical  rationale  behind  our  use  of  a  journal  set,  or  perhaps 
more aptly termed “journals and proceedings set”, to benchmark and compare the research 
reported  to  DG  CNECT  rests  on  the  assumption  that  interdisciplinary  and  frontier  research 
takes  place  at  the  interface  between  two  or  more  disciplines.  Therefore,  it  is  difficult  to 
capture  the  disciplinary  classifications,  using  “ex  ante”  classification  schemes.  One  can 
expect citation flows between disciplines to be less dense than within the disciplinary cores. 
This  will  diminish  the  numbers  of  citations  that  are  visible  and  consequently  also  any 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  44 

indicator that is derived from these citation statistics. Research by Laudel and Origgi (2006) 
further  suggests  that  interdisciplinary  research  systematically  receives  lower  grades  than 
core disciplinary research efforts in research assessment exercises (RAEs) (Leydesdorff and 
Probst, 2009). We believe that in the field of ICT there is a large overlap in research topics. 
This  is  due  to  the  high  level  of  interdisciplinary  research  which  incorporates  ICT.  Like 
biotechnology  and  nanotechnology,  ICT  is  not  a  discipline  in  itself  but  an  enabling 
technology. If we were to select journals or proceedings which represent distinct disciplinary 
fields (physics, computer science etc.) we would have to make choices or we would need to 
leave out a large number of titles which show high overlap. Alternatively we could "double-
count"  journal/proceeding  titles  in  several  fields.  Either  way,  this would  most  probably  not 
cover all the ICT research sponsored by DG CNECT, or alternatively over-represent it. This 
is  not  desirable  and  further  supports  our  choice  for  the  one-on-one  "journal  set"  approach 
where we use the exact source titles reported in the survey to construct our journal set. 

3.3  Publication productivity of FP7-ICT 
Under this heading we undertake the assessment of the productivity of each project, as well 
as  the  productivity  for  each  of  the  funding  instrument  and  the  strategic  objectives  of  DG 
CNECT, as defined under FP7. Moreover, in an effort to test national performance, we chart 
the  distribution  of  publication  output  across  countries,  country  blocks  and  the  NUTS  3 
regions of the top 5 countries. Further to this, we perform a comparison of the control and 
focal sample to detect any deviations in productivity from the norm. And finally, we look at 
the  productivity  per  organizational  type.  In  general  we  can  say  that,  for  each  of  the 
productivity analyses, a Pareto  or power law probability distribution can be discerned. This 
means  that  high  productivity  performance  is  concentrated  in  only  a  few  of  the  researched 
variables. 

For  the  productivity  analysis,  we  use  the  complete  validated  dataset  of  publication  output 
reported  to  DG  CNECT.  Out  of  the  2706  projects  that  were  funded,  we  find  that  1286 
projects  have  reported  24664  validated  publication  outputs  over  the  FP7  time  period 
surveyed. This includes the publications that were validated as being published in the period 
of  2007  to  201810,  but  excludes  the  duplicates11  and  the  144  publications  that  were 
validated as being published before 2007 and as such outside of FP7. 

3.3.1  Productivity by project 
In  Table  9  we  report  the  distribution  of  publications  across  the  projects.  The  distribution 
reveals a Pareto distribution (also called power law distribution) which, contrary to a normal 
distribution does not show a “peak” at the average. We can see that the majority of projects 

10  2018  is  included due  to  the publishing  process; publications  that  have  been finished  in  the  previous years,  for 
instance in 2016 in this case, can be published in the follow years; and we assume these outputs to still be valid 
2016 outputs. 
11 Duplication of reported output has been a serious problem in this and in the previous SMART 2011/0039 project. 
There are several ways of dealing with duplicates, as described in the Annual Reports, where we can use “soft” de-
duplication  in  the  case  of  multiple  duplicates  across  multiple  projects,  or  “hard”  de-duplication  in  the  case  of 
duplicates in a single project, or “strict” de-duplication where the publication is assigned to the earliest project. For 
this latter one we decided for the final study report. 

 
 
 
                                                           
Final Study Report- D4 - SMART 2015/0078  45 

produce  only  few  publications,  which is the  so-called  “tail”  of  the  curve,  while  a  select  few 
generate  a  large  amount  of  publications.  To  generate  Table  9  we  divided  the  curve  into 
several  intervals.  We  also  separate  the  small  percentage  of  projects  generating  high 
numbers.  We  find  that  although  79.2%  of  the  projects  produce  between  0  and  10 
publications, a respectable amount of projects produces between 11 and 50 publications. As 
expected, only a small percentage (2.7%) produces more than 50 publications. The average 
amount of publications per project is 9.115. 

Table 9. Distribution of Publications per Project. 

Number of Projects 
1420 
346 
376 
263 
228 
56 
17 

Number of Publications 
0 
1 to 3 
4 to 10 
11 to 20 
21 to 50 
51 to 100 
100+ 

Percentage 
52.5 
12.8 
13.9 
9.7 
8.4 
2.1 
0.6 

Mean 
9.115 

StdDev 
74.048 

Min 
0 

Max 
3714 

3.3.2  Productivity by funding instrument 
Our next analysis concerns the distribution of publications across the funding schemes. Here 
we  are  interested  in  discovering  whether  there  are  any  differences  in  productivity  with 
respect to the way the projects are funded. What we find is that the small or medium-scale 
focused research projects (STREPs), under the heading of CP-FP-INFSO, are responsible for 
more than 50% of all output generated in absolute terms. This is reported in Table 10. The 
networks  of  excellence  (NoEs)  and  the  integrated  projects  (IPs)  take  a  second  and  third 
place respectively. 

Table 10. Distribution of publications across funding schemes, sub-contract level. 

Funding instrument 
CP-FP-INFSO 
NoE 
CP-IP 
CP-CSA-Infra-PP 
CP-IP-INFSO-FET 
CP-FP-INFSO-FET 
CSA-CA 
CSA-SA 
CSA-CA-INFSO-FET 
CP-CSA-PCP 
CIP-ICT-PSP-PB 
CIP-ICT-PSP-BPN 

Publications 
10580 
5646 
5490 
915 
863 
564 
251 
150 
60 
41 
40 
26 

Percentage 
42.9 
22.9 
22.3 
3.7 
3.5 
2.3 
1.0 
0.6 
0.2 
0.2 
0.2 
0.1 

 
 
 
 
 
 
 
 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  46 

CP-SICA-INFSO 
CP 
CP-CSA-Infra-IP 
CSA-SA-INFSO-FET 
CIP-ICT-PSP-TN 
CIP-ICT-PSP-PA 
CIP-ICT-PSP-TN(EC) 
CSA-ERA-Plus 
Total 

11 
8 
8 
6 
5 
0 
0 
0 
24664 

0.0 
0.0 
0.0 
0.0 
0.0 
0.0 
0.0 
0.0 
100 

However,  if  we  control  for  the  number  of  projects  that  make  up  each  of  the  funding 
schemes, as shown in Table 11, we see quite a different picture. 

Table  11.  Distribution  of  publications  across  funding  schemes  (sub-contract  level),  by  number  of 
projects (total projects FP7 ICT). 

Funding scheme 
NoE 
CP-IP-INFSO-FET 
CP-IP 
CP-CSA-Infra-PP 
CP-FP-INFSO-FET 
CP-FP-INFSO 
CP-CSA-Infra-IP 
CP-CSA-PCP 
CP 
CSA-CA-INFSO-FET 
CSA-CA 
CP-SICA-INFSO 
CIP-ICT-PSP-BPN 
CSA-SA 
CSA-SA-INFSO-FET 
CIP-ICT-PSP-PB 
CIP-ICT-PSP-TN 
CIP-ICT-PSP-PA 
CIP-ICT-PSP-TN(EC) 
CSA-ERA-Plus 
Total 

Number of projects 
39 
19 
362 
71 
62 
1424 
2 
13 
3 
23 
146 
8 
24 
261 
12 
172 
39 
15 
8 
3 
2706 

Number of publications 
5646 
863 
5490 
915 
564 
10580 
8 
41 
8 
60 
251 
11 
26 
150 
6 
40 
5 
0 
0 
0 
24664 

Average 
144.8 
45.4 
15.2 
12.9 
9.1 
7.4 
4 
3.2 
2.7 
2.6 
1.7 
1.4 
1.1 
0.6 
0.5 
0.2 
0.1 
0 
0 
0 
9.12 

We now find that NoEs generate on average almost 145 publications per project, while the 
integrated  projects  take  second  and  third  place  with  an  aggregated  average  of  more  than 
60 publications per project. The infrastructure projects take third place. The STREPs, which 

 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  47 

accounted for the largest percentage of publications overall, now are fourth with on average 
13  publications  per  project.  This  outcome  is  supported  by  the  Annual  Reports,  and  by  the 
results  from  the  SMART  2011/0039  project,  which  also  showed  that  the  networks  of 
excellence are on average the most productive funding scheme. 

3.3.3  Productivity by strategic objective 
If we now look at the distribution of publications per strategic objective of the funded calls 
for proposal, we see similar characteristics as for the funding schemes. Strategic objectives 
which performed well in absolute terms, as shown for the top 20 objectives in Table 12, can 
be seen to perform less well once controlled for by the number of projects. 

Table 12. Top 20 Strategic objectives in terms of absolute publication numbers. 

Strategic objective 
Computing Systems 
The Network of the Future 
FET Open 
Cognitive Systems and Robotics 
Cognitive Systems, Interaction, Robotics 
Trustworthy ICT 
Future Networks 
Service and Software Architectures, Infrastructures and Engineering 
Embedded Systems Design 
Secure, dependable and trusted Infrastructures 
Photonic components and subsystems 
Networked Media 
Distributed computing infrastructure (DCI) 
Internet of Services, Software & virtualisation 
Networked Embedded and Control Systems 
Next-Generation Nano-electronics Components and Electronics 
Integration 
Virtual Physiological Human 
Intelligent Information Management 
Technology-Enhanced Learning 
ICT forever yours 
Note: For the full listing of the absolute distribution of publications across strategic objectives, please 
see Annex 1. 

Publications 
4382 
1680 
1617 
1098 
723 
679 
651 
595 
557 
464 
461 
415 
408 
379 
329 
308 

Percentage 
17.8 
6.8 
6.6 
4.5 
2.9 
2.8 
2.6 
2.4 
2.3 
1.9 
1.9 
1.7 
1.7 
1.5 
1.3 
1.2 

293 
288 
279 
274 

1.2 
1.2 
1.1 
1.1 

In  Table  13  below,  we  see  that  the  strategic  objective  titled  "Distributed  computing 
infrastructure" features as the best performing objective with 196 publications per project, 
while  the  same  strategic  objective  occupies  the  twelfth  place  in  absolute  publication 
numbers generated. Networks of the future  occupied the top spot in terms of the absolute 

 
 
 
Final Study Report- D4 - SMART 2015/0078  48 

number of publications generated, whereas it occupies the 17th spot in average number of 
publications per project. 

Table 13. Top 20 Distribution of publications across strategic objectives, by number of projects. 

Strategic objective 

Distributed computing infrastructure (DCI) 
Computing Systems 
FET proactive 2: Quantum Information Foundations and 
Technologies 
ICT forever yours 
Human-Computer Confluence 
Brain-inspired ICT 
FET Proactive: Quantum ICT (QICT) 
Self-Awareness in Autonomic Systems 
Deployment of e-Infrastructures for scientific communities 

Number of 
projects 
2 
28 
2 

Number of 
publications 
408 
4382 
238 

Average 

204.0 
156.5 
119.0 

3 
2 
4 
4 
2 
5 

274 
101 
201 
193 
88 
214 

3 
2 

118 
71 

Science of Complex Systems for socially intelligent ICT 
FET Proactive 3: Bio-chemistry-based Information Technology 
(CHEM-IT) 
Strengthening cooperation in ICT R&D in an enlarged  Europe 
Content analytics and language technologies 
Future Networks 
Coordinating Communities, Plans and Actions in FET Proactive 
Initiatives 
Service and Software Architectures, Infrastructures and 
Engineering 
The Network of the Future 
FET Open 
FET Proactive: Dynamics of Multi-Level Complex Systems (DyM-
CS) 
Embedded Systems Design 
Note: For the full listing of the performance of strategic objectives, please see Annex 1. 

33 
62 
651 
30 

1680 
1617 
141 

1 
2 
21 
1 

59 
57 
5 

557 

595 

20 

20 

3.3.4  Productivity by EU region 
For  the  analysis  under  this  sub-chapter  we  have  used  the  participants’  data  that  was 
available  from  DG  CNECT  and  combined  it  with  project  and  NUTS  3  geo-coding  data.  This 
was  achieved  by  matching  the  postal  codes  or  city  names  with  the  NUTS  3  information 
available  from  Eurostat12  and  from  national  sources.  Not  surprisingly  we  find  that  again 
there  is  a  Pareto  distribution,  where  a  small  number  of  regions  account  for  the  highest 
number of projects. These high scoring NUTS codes encompass the large metropolitan areas 

12 See: http://epp.eurostat.ec.europa.eu/portal/page/portal/nuts_nomenclature/introduction 

91.3 
50.5 
50.3 
48.3 
44.0 
42.8 

39.3 
35.5 

33.0 
31.0 
31.0 
30.0 

29.8 

28.5 
28.4 
28.2 

27.9 

 
 
 
                                                           
Final Study Report- D4 - SMART 2015/0078  49 

of each of the countries, with Munich, Berlin, Milan, Rome, London, Barcelona, Madrid, and 
Paris taking the top spots. The listing of projects by NUTS 3 region can be found in Annex 2.  

3.3.5  Productivity by country 
In  order  to  transform  the  European  R&D  landscape,  which  has  long  been  plagued  by 
duplication of efforts on a transnational level, the European Research Area was introduced. 
One of the mainstays of this policy framework is the promotion of cross-border collaborative 
partnerships in order to remove the barriers that obstruct transnational research activities. 

Here we show productivity per country. We have separated the EU-28 countries, as well as 
the  FP7  Associated  countries,  the  BRICS  countries  and  the  USA,  Japan,  and  South  Korea. 
The  data  used  for  constructing  Tables  14  to  18  has  been  taken  from  the  validated  data 
reported  to  DG  CNECT  and  as  such  represent  all  output  in  which  at  least  one  DG  CNECT 
sponsored  author  is  found.  It  should  be  noted  that,  due  to  co-authorships,  a  double-
counting of the articles is unavoidable. 

Table 14. Distribution of publications per EU-28 country. 

Country 
Germany 
Italy 
United Kingdom 
Spain 
France 
The Netherlands 
Greece 
Sweden 
Austria 
Belgium 
Portugal 
Denmark 
Finland 
Ireland 
Poland 
Czech Republic 
Romania 
Hungary 
Slovenia 
Cyprus 
Slovakia 
Croatia 
Estonia 
Bulgaria 
Lithuania 

Publications 
5520 
5067 
4250 
3692 
3015 
2052 
1789 
1423 
1307 
1223 
805 
602 
569 
497 
470 
351 
336 
275 
184 
161 
130 
126 
94 
85 
79 

 
 
 
Final Study Report- D4 - SMART 2015/0078  50 

Luxembourg 
Latvia 
Malta 

40 
16 
7 

In the above  Table 14, we see the division of publication numbers per EU-28 country.  The 
figures  presented  are  absolute  numbers.  The  use  of  these  absolute  numbers  is  alas 
unavoidable  at  this  stage  as,  due  to  insufficient  data,  we  cannot  say  without  a  doubt  how 
many  author-organization  pairs  have  contributed  to  each  article.  The  affiliation  data  is 
arranged  in  such  a  way  that  for  instance  a  publication  with  two  authors  from  the  same 
affiliation and a third author from a different affiliation is ascribed to two affiliations while it 
should  in  theory  be  three.  Nevertheless,  the  tables  showing  the  distributions  across 
countries  do  give  a  good  general  impression  of  the  national  productivity.  In  Table  14,  we 
also  see  that  the  top  spots  are  taken  by  the  five  largest  countries  in  the  European  Union. 
These  are  also  the  leading  countries  in  terms  of  number  of  institutions  taking  part  in  DG 
CNECT  projects.  Interestingly,  the  Netherlands  and  Greece  take  a  6th  and  7th  spot 
respectively which, when compared to the number of organizations undertaking R&D within 
these countries, shows us that the productivity in these countries is comparatively high.  

In  Table  15  below,  we  report  on  a  similar  distribution,  but  then  for  the  FP7  Associated 
Countries13.  Here  we  see  that  Switzerland  is  leading  the  other  Associated  Countries  by  a 
comfortable margin. When compared to the EU-28, we can put Switzerland on par with the 
Netherlands which in itself can be explained by their similarities in educational system and 
infrastructure  as  well  as  by  the  relatively  high  incidence  of  multinational  corporations 
working within their borders. 

Table 15. Distribution of co-publications per FP7 Associated Country. 

FP7 Associated country 
Switzerland 
Israel 
Norway 
Turkey 
Serbia 
Macedonia 
Albania 
Iceland 
Moldova 
Montenegro 
Bosnia and Herzegovina 
Liechtenstein 

Publications 
1930 
576 
398 
253 
241 
44 
10 
8 
8 
4 
2 
1 

13 See: http://ec.europa.eu/research/participants/portal//desktop/en/funding/reference_docs.html#fp7 

 
 
 
 
 
                                                           
Final Study Report- D4 - SMART 2015/0078  51 

A  further  indication  of  the  interest  of  the  EU  in  strengthening  collaborative  R&D  ties is  the 
publishing  of  several  specifically  targeted  calls,  calling  for  EU  cooperation  with  a  specific 
third country. There were two calls published in 2011, for Brazil and Russia, as well as two 
in  2013  calling  for  R&D  cooperation  with  Brazil  and  Japan.  These  specific  calls  were 
additional to the CP-SICA-INFSO calls which are also focussed on third countries. 

Table 16. Productivity of the four targeted calls identified from the survey and participant data. 

Targeted Call  
FP7-ICT-2011-EU-Brazil 
FP7-ICT-2011-EU-Russia 
FP7-ICT-2013-EU-Brazil 
FP7-ICT-2013-EU-Japan 
Total 
Note: The above publications fall outside of the CP-SICA-INFSO funding instrument. 

Number of projects 
4 
2 
4 
6 
16 

Number of publications 
6 
12 
0 
2 
20 

The publications in Table 16 above do not amount to a large number as these targeted calls 
have  also  been  shown  to  be  less  productive  than  others.  However  the  shortfall,  as  yet,  of 
these calls is made up by an ample amount of publications originating from other projects. 
Below in Table 17 we show the publication productivity of the BRICS countries. Here we see 
China  leading  Brazil  and  Russia  while  India  and  South  Africa  trail  behind.  We  can 
hypothesize  that  future  country  specific  targeted  calls  might  remedy  this  by  further 
targeting  the  BRICS  countries,  in  particular  India  and  South  Africa,  that  are  shown  to  be 
lagging at the moment. 

Table 17. Distribution of co-publications per BRICS country. 

BRICS country 
China 
Brazil 
Russian Federation 
India 
South Africa 

Publications 
667 
355 
301 
165 
75 

Traditionally, many European countries have sought close R&D ties with the United States in 
order  to  benefit  from  the  research  infrastructure  and  world leading institutions  which have 
their  home  base  there.  Table  18  shows  that,  for  publications  with  at  least  one  DG  CNECT 
sponsored author, this is not necessarily the case. Although from the countries listed in this 
table  the  US  has  the  highest  amount  of  publications,  EU-28  countries  seem  to  prefer 
cooperation  with  Switzerland  over  the  US  as  the  former  have  a  number  of  publications 
almost on par with the US. 

 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  52 

Table 18. Distribution of co-publications per USA, Japan, and South Korea. 

Country 
United States 
Japan 
South Korea 

Publications 
2234 
294 
136 

Next  to  the  above  regional  blocks  and  known  technology  giants  we  also  find  a  number  of 
third countries that are preferred collaboration partners, chief among which are Canada and 
Australia. 

Table 19. Distribution of co-publications per other third country, Top 10. 

Country 
Canada 
Australia 
Singapore 
Taiwan 
Mexico 
Ukraine 
Colombia 
New Zealand 
Armenia 
Iran 

Publications 
338 
316 
145 
110 
108 
107 
100 
91 
87 
81 

In  order  to  provide  a  fuller  assessment  of  the  relative  productivity  of  the  EU-28  countries 
benefiting from the funding available within FP7 ICT, we juxtapose in Table 20 the country 
ranking  in  terms  of  publications  produced.  Here  we  take  the  data  on  publication  numbers 
from  our  focal  and  control  sample.  A  percentage  distribution  is  added  for  further 
clarification.  What  we  find is  that  there  are  obvious  differences  between  the  sample  of  DG 
CNECT  publications  and  the  benchmarking  sample  (control  sample).  The  control  sample 
shows  a  big  lead  for  the  US  in  terms  of  publications  in  which  an  American  author  has 
participated.  The  other  large  countries  follow  at  a  distance.  For  the  focal  sample  we  see  a 
distribution  similar  to  the  one  shown  in  our  first  table  on  national  productivity.  The  US 
features  just  above  Switzerland,  while  China  occupies  a  spot  quite  a  way  further  down. 
Furthermore,  we  notice  that  countries  such  as  Japan,  India,  South  Korea,  Russia  and 
Canada feature much further down the focal sample list than expected. This is confirmed by 
the comparison with the control sample listing. 

 
 
 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  53 

Table 20. Distribution of publication output over the affiliation countries, focal vs. control* 

Percentage 
12.3 
11.7 
9.7 
8.8 
6.8 
4.8 
4.4 
4.3 
4.2 
3.1 
3.1 
2.9 
1.7 
1.7 
1.3 
1.2 
1.2 
1.2 
1.1 
0.9 
0.8 
0.8 
0.7 
0.7 
0.6 
0.6 
0.6 
0.6 
0.5 
0.5 
0.5 
0.4 
0.4 
0.4 
0.4 
0.3 
0.3 
0.3 
0.3 

Focal 
Publications  Country 
Germany 
763 
Italy 
723 
United Kingdom 
604 
Spain 
548 
France 
423 
United States 
298 
Switzerland 
275 
Netherlands 
268 
Greece 
260 
Sweden 
195 
Austria 
191 
Belgium 
180 
China 
104 
Portugal 
104 
Finland 
80 
Ireland 
77 
Israel 
77 
Poland 
73 
Denmark 
69 
Norway 
55 
Canada 
49 
Brazil 
48 
Russian Federation 
43 
Czech Republic 
41 
Japan 
40 
Australia 
39 
Serbia 
38 
Romania 
37 
Hungary 
34 
Cyprus 
32 
South Korea 
28 
Singapore 
25 
India 
24 
Slovenia 
24 
Turkey 
23 
Mexico 
20 
Ukraine 
18 
Croatia 
16 
Iran 
16 

Country 
United States 
China 
Germany 
United Kingdom 
France 
Japan 
Italy 
Canada 
Spain 
Australia 
India 
South Korea 
Switzerland 
Netherlands 
Taiwan 
Russian Federation 
Sweden 
Poland 
Austria 
Brazil 
Singapore 
Belgium 
Portugal 
Finland 
Greece 
Israel 
Denmark 
Czech Republic 
Mexico 
Hong Kong 
Malaysia 
Ireland 
Norway 
Iran 
Turkey 
Romania 
New Zealand 
Hungary 
Saudi Arabia 

Control 

Publication 
1064 
591 
412 
358 
307 
293 
214 
193 
156 
144 
132 
107 
95 
92 
86 
82 
76 
67 
66 
66 
61 
57 
56 
48 
48 
48 
46 
44 
44 
41 
32 
30 
30 
29 
28 
23 
21 
20 
20 

Percentage 
19.0 
10.5 
7.4 
6.4 
5.5 
5.2 
3.8 
3.4 
2.8 
2.6 
2.4 
1.9 
1.7 
1.6 
1.5 
1.5 
1.4 
1.2 
1.2 
1.2 
1.1 
1.0 
1.0 
0.9 
0.9 
0.9 
0.8 
0.8 
0.8 
0.7 
0.6 
0.5 
0.5 
0.5 
0.5 
0.4 
0.4 
0.4 
0.4 

 
 
 
Final Study Report- D4 - SMART 2015/0078  54 

15 
14 
14 
14 
13 
11 
11 
10 
10 
9 
9 
9 
8 
8 
7 
7 
7 
6 
6 
47 
6197 

0.2 
0.2 
0.2 
0.2 
0.2 
0.2 
0.2 
0.2 
0.2 
0.1 
0.1 
0.1 
0.1 
0.1 
0.1 
0.1 
0.1 
0.1 
0.1 
0.8% 
100% 
*Note:  Double  counting  is  an  obvious  part  of  the  distribution  using  co-author  affiliations  due  to 
multiple affiliations per paper and even per author. 

Slovakia 
Estonia 
New Zealand 
Taiwan 
Lithuania 
Armenia 
Colombia 
Belarus 
Bulgaria 
Argentina 
Georgia 
Pakistan 
Egypt 
Viet Nam 
Chile 
Hong Kong 
Thailand 
Luxembourg 
Morocco 
Other 
Total 

Tunisia 
Chile 
Egypt 
Ukraine 
Argentina 
Thailand 
Pakistan 
Slovakia 
Algeria 
Qatar 
UAE 
Slovenia 
Cuba 
Lithuania 
South Africa 
Bulgaria 
Colombia 
Indonesia 
Luxembourg 
Other 
Total 

0.3 
0.2 
0.2 
0.2 
0.2 
0.2 
0.2 
0.2 
0.2 
0.2 
0.2 
0.1 
0.1 
0.1 
0.1 
0.1 
0.1 
0.1 
0.1 
1.7% 
100% 

17 
14 
14 
14 
13 
12 
10 
10 
9 
9 
9 
8 
7 
7 
7 
6 
6 
6 
6 
94 
5605 

 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  55 

3.3.6  Productivity by organization type 
Another  barrier  that  the  ERA  has  sought  to  overcome  was  a  perceived  low  level  of  public-
sector support, as well as inadequate conditions for private investment in R&D. In order to 
promote  public-private  collaborations,  specifically  with  SMEs,  a  number  of  financial  and 
funding  schemes  have  been  put  in  place  to  support  research  and  innovation  projects  and 
capacity building. The fruits of this endeavour should become visible when we compare the 
share of publications distributed across four organizational types. 

Table 21. Share of publications by organizations type, as taken from the focal and control samples. 

Focal 

Number of 
affiliations 

Perc. 

Control 

Number of 
affiliations 

Academic 
Government/NGO 
Corporate Total 
Of which Corporate 
Large is 
And SMEs is 

6520 
708 
629 
447** 

182** 
7857 

83% 
9% 
8% 
71% 

29% 
100% 

Academic 
Government/NGO 
Corporate Total 
Of which Corporate 
Large is 
And SMEs is 

6597 
1389 
694 
520** 

174** 
8680 

Perc. 

76% 
16% 
8% 
75% 

25% 
100% 

Note: * percentage of total, part of Academic, ** as part of Corporate Total, *** percentage of total, 
part of Academic and Government. Data as taken from the stratified random sample. 

In  Table  21  above,  we  see  that  this  is  indeed  the  case.  Although  total  corporate  research 
activity in terms of co-publications/affiliations is on par with the benchmark, the data shows 
a slightly higher amount of productivity on the part of SMEs. SMEs make up 99% of all EU 
enterprises, and provide around 75 million jobs14. They are therefore vital to delivering the 
growth and higher employment objectives connected to the ERA policies. We can state that 
the data suggests that the ERA has fulfilled this part of its policy goals for FP7 ICT. 

Academic,  Government  and  NGO  involvement  are  quite  dissimilar  with  the  percentages  as 
observed  in  the  (stratified)  control  sample,  with  a  higher  influence  of  Government/NGO  in 
the control.  

3.3.7  Private-Public ratio 
From  the  analysis,  we  can  observe  a  number  of  interesting  outcomes.  If  we  now  want  to 
look at the ratio of private versus public organizational output, we can see that there is an 
overwhelming  percentage  which  can  be  attributed  to  the  public  sector.  From  our  focal 
sample we can see that 92% of the output is generated by public research. This number is 
an aggregation of the Academic and Government/NGO output. Private organizational output 
accounts  for  only  8%  in  the  focal  sample.    In  terms  of  the  division  of  large  corporations 
versus  SMEs  within  our  samples,  we  find  that  the  DG  CNECT  sponsored  projects  have  a 
more or less similar level of SME productivity than the control sample, and as such conform 
to the “real-world” conditions in this field of research. 

14 http://ec.europa.eu/enterprise/policies/sme/files/sme_definition/sme_user_guide_en.pdf 

 
 
 
 
 
                                                           
Final Study Report- D4 - SMART 2015/0078  56 

On  a  whole,  we  can  say  that  the  Private-Public  ratio  of  ICT  research  sponsored  by  DG 
CNECT is 8%-92%, which corresponds with the same ratio for  ICT R&D performed outside 
of  the  FP7  framework.  This  shows  that  the  sponsored  research  projects  have  a  good 
inclusion in terms of corporations, which is in line with the policy goals of the Lisbon Agenda 
on the creation of a more inclusive European Research Area. 

3.4  Publication impact of FP7-ICT 
Measuring  the  impact  of  a  science  field  through  a  citation  analysis  is  traditionally  the 
mainstay  of  any  bibliometric  analysis.  Although  the  question  of  whether  citations  actually 
represent impact of research through its perceived quality can be debated, we can say that 
“ceteris paribus” the citations gathered by a publication are a measure of the exposure and 
uptake  of  the  knowledge  contained  within  this  publication  by  the  scientific  community  at 
large. One important fact that we need to point out is that traditions governing referencing 
and  citing  can  differ  significantly  across  scientific  disciplines.  This  we  have  tried  to 
circumvent however, by using the journal set methodology explained earlier. 

3.4.1  Cited-Un-cited ratio 
Under  this  heading  we  report  on  the  ratio  of  cited  to  un-cited  publications.  As  the  citation 
rates  of  both  media  types  are  known  to  be  different  we  separate  in  Table  22  the 
publications we perceived as being articles from the proceedings papers and from the other 
publication  types.  As  expected,  we  notice  a  marked  difference,  with  95%  of  the  FP7  ICT 
articles  being  cited  at  least  once,  whereas  for  the  proceedings  this  is  85%  of  the  FP7 
proceedings  are  being  cited  at  least  once,  while  the  remaining  publications  have  77% 
chance of getting cited at least once. Although it is remarked that citation lag, e.g. the time 
it takes to get cited, can be several years, we do not see this in our sample, as exemplified 
by a 2016 article which has received 178 citations already. 

Table  22.  Ratio  of  cited  to  un-cited  for  the  complete  validated  set  of  publications  reported  to  DG 
CNECT, divided into articles and proceedings. 

Uncited 
Cited 
Total 

Articles  
388 
7969 
8357 

Percentage  
4.64 
95.36 
100%  

Proceedings  
2413 
13268 
15681 

Percentage  
15.39 
84.61 
100%  

Other 
145 
481 
626 

Percentage 
23.16 
76.84 
100% 

If  we  now  carry  this  analysis  through  to  the  comparative  level,  comparing  the  cited  –  un-
cited  ratios  for  the  control,  focal  sample,  and  for  the  complete  validated  set,  we  find  that 
the DG CNECT sponsored research has a lower percentage of un-cited work as well as lower 
percentage of highly cited works. Although at first sight this might seem unfavourable, we 
need  to  offset  these  descriptive  statistics  with  the  more  advanced  mean  observed  citation 
rate  (MOCR),  mean  expected  citation  rate  (MECR),  and  the  resulting  relative  citation  rate 
(RCR), as performed in the next section. 

 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  57 

Table 23. Comparison of the percentage distribution of citations, cited and un-cited, for the focal and 
control sample, with average citation rates in italics. 

Publications 

932 
1114 
836 
1276 
4158 

Control 
Times 
Cited 
0 
1 to 3 
4 to 10 
10+ 
14.499 

Percent 

Publications 

Times Cited  Percent 

Publications 

Focal 

22.4% 
26.8% 
20.2% 
30.6% 
100% 

385 
798 
848 
1391 
3422 

0 
1 to 3 
4 to 10 
10+ 
19.612 

11.3% 
23.3% 
24.8% 
40.6% 
100% 

2946 
6093 
6355 
9270 
24664 

Complete 
Times 
Cited 
0 
1 to 3 
4 to 10 
10+ 
16.661 

Percent 

11.9% 
24.8% 
25.7% 
37.6% 
100% 

3.4.2  Impact per journal and proceedings 
A further descriptive statistic shows the share of citations per journal or proceedings title as 
part  of  the  total  number  of  citations  gathered.  Here  we  see  that  for  the  top  15  titles,  the 
popular proceedings series “Lecture Notes in Computer Science” takes top place in terms of 
the total share gathered. However, on a relative scale, it is outperformed by the high impact 
journals that take up the 3th and 4th spots. 

Table 24. Distribution of the number of citations per Journal and Proceedings title, top fifteen titles. 

Journal or Proceedings title 

Lecture Notes in Computer Science 
ACM International Conference Proceeding Series 
Physical Review Letters 
Physical Review A - Atomic, Molecular, and Optical Physics 
Proceedings of SPIE 
CEUR Workshop Proceedings 
Physical Review B - Condensed Matter and Materials Physics 
Optics Express 
IEEE International Conference on Intelligent Robots and Systems 
Proceedings - IEEE International Conference on Robotics and Aut. 
Proceedings -Design, Automation and Test in Europe 
New Journal of Physics 
IEEE International Conference on Communications 
Applied Physics Letters 
European Conference on Optical Communication 

Number of 
citations 
2544 
372 
291 
245 
224 
216 
210 
187 
186 
185 
152 
145 
142 
119 
117 

Percentage 

10.31% 
1.508% 
1.179% 
0.993% 
0.908% 
0.875% 
0.851% 
0.758% 
0.754% 
0.750% 
0.616% 
0.587% 
0.575% 
0.482% 
0.474% 

 
 
 
 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  58 

3.4.3  MOCR, MECR, RCR 
A perhaps more insightful impact assessment can be done using the mean citation rates of 
both the focal and control samples. For this, we employ the earlier mentioned MOCR, MECR 
and RCR indicators. These can be defined as: 

Mean  Observed  Citation  Rate  (MOCR):  MOCR  is  defined  as  the  ratio  of  citation  count  to 
publication count (Braun, Glanzel and Schubert, 1985). 

Mean  Expected Citation Rate (MECR): is a journal-based citation measure which expresses 
the expected citation rate of a publication set. The expected citation rate of a single paper is 
defined  as  the  mean  citation  rate  of  all  papers  published  in  the  same  journal  in  the  same 
year. MECR is then defined as the average of the individual expected citation rates over the 
given publication set (Braun, Glanzel and Schubert, 1985). 

Relative Citation Rate (RCR): RCR is defined as the ratio of the two previous measures, that 
is, RCR = MOCR/MECR. RCR = 0 would correspond to complete “un-citedness”, while RCR < 
1 (RCR > 1) would mean lower (higher)-than-average citation rates.  RCR = 1 if the set of 
papers in question attracts just the number of citations expected on the basis of the citation 
impact of the journals where the papers have been published (Braun, Glanzel and Schubert, 
1985, and Zhou, 2009). 

In the table below we perform the above described analysis for our complete set of journals 
and proceedings. When we compare the focal (MOCR) and control (MECR) samples, we find 
that an average RCR of just below 1.4 is reached, which would mean that the focal sample 
has a citation rate above the observed world average. 

Table  25.  MECR,  MOCR  and  RCR  calculations  for  the  DG  CNECT  reported  and  control  sample 
publications sets (articles and proceedings papers). 

Year 

Publ 
Focal 

Min  Max  MOCR  Publ 
Ctrl 

Min  Max  MECR  RCR 

t test 

p 
value 

184 
007/008 
774 
009/010 
011/012 
881 
013/014  1104 
459 
015/016 

930  27.500 
0 
679  23.362 
0 
0 
906  22.575 
0  2623  17.620 
9.959 
0 

510 

222 
936 
1071 
1353 
554 

574  13.995  1.965 
0 
0  1515  20.135  1.160 
785  17.411  1.297 
0 
659  11.469  1.536 
0 
7.455  1.336 
261 
0 

2.776 

0.035 

P corr 

0.757 

1.459  Average 

Note: The data as presented above is taken from the stratified samples 

Although  this  in  itself  is  of  course  a  good  score  we  would  like  to  explore  which  of  the 
variables,  or  perhaps  all,  is  responsible  for  the  above  average  RCR  score.  First  we  can 
imagine that the media type would be of influence. As seen earlier, in Table 7, we were able 
to confirm Lisee, Lariviere and Archambault's (2008) finding that proceedings generate less 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  59 

citations.  However,  would  the  DG  CNECT  sponsored  output  still  score  above  the  world 
average  when  looking  at  quality  when  measured  by  RCR?  Below  in  Tables  26  and  27,  we 
compare the MOCR and MECR for articles and proceedings respectively. 

Table  26.  MECR,  MOCR  and  RCR  calculations  for  the  DG  CNECT  reported  and  control  sample 
publications sets (articles only). 

Year 

007/008 
009/010 
011/012 
013/014 
015/016 

Publ 
Focal 
36 
226 
309 
408 
162 

Min  Max  MOCR 

930 
0 
679 
0 
0 
420 
0  2623 
510 
0 

75.278 
45.823 
35.084 
33.049 
16.704 

Publ 
Ctrl 

43 
278 
381 
499 
196 

Min  Max  MECR 

RCR 

t test 

p value 

0 
574 
0  1515 
785 
0 
659 
0 
261 
0 

47.628 
51.795 
38.651 
21.689 
14.020 

0.350 

P corr 

0.779 

2.776 

1.581 
0.885 
0.908 
1.524 
1.191 
1.218  Average 

Note: The data as presented above is taken from the stratified samples. 

Table  27.  MECR,  MOCR  and  RCR  calculations  for  the  DG  CNECT  reported  and  control  sample 
publications sets (proceedings only). 

Year 

007/008 
009/010 
011/012 
013/014 
015/016 

Publ 
Focal 
145 
523 
543 
665 
276 

Min  Max  MOCR  Publ 
Ctrl 

Min  Max  MECR  RCR 

t test 

p value 

0 
0 
0 
0 
0 

230  15.841 
265  12.493 
906  12.700 
7.756 
4.612 

93 
49 

174 
638 
664 
817 
335 

0 
0 
0 
0 
0 

109 
132 
87 
146 
34 

5.753 
6.288 
5.078 
4.689 
2.752 

0.018 

P corr 

0.866 

2.776 

2.754 
1.987 
2.501 
1.654 
1.676 
2.114  Average 

Note: The data as presented above is taken from the stratified samples. 

The results, as presented above, show a surprising and interesting difference in the average 
ratios.  We  find  that,  for  articles,  the  average  RCR  over  the  years  examined  is  just  above 
1.2, and as such above the expected citation rate. When we look at Table 27, we see that 
the  proceedings  actually  have  a  considerably  better  rate  as  compared  to  the  articles,  with 
an average RCR of 2.1. We can deduct from both calculations that, for the complete set, the 
ratio  is  higher  than  expected;  this  is  true  for  both  articles  and  proceedings,  although  the 
inclusion  of  the  proceedings  papers  in  our  examined  universe  has  been  beneficial  to  an 
overall  higher  quality  of  the  research  output,  as  examined  in  our  focal  sample.  DG  CNECT 
sponsored  research  published  in  the  popular  and  high  impact  journals,  as  well  as  in  the 
popular  conference  proceedings  examined,  performs  well  and  above  the  expected  world-
wide ratio. 

The  results  of  our  citation  rate  calculations  are  very  positive  showing  that  on  average  the 
DG  CNECT  sponsored  output  performs  well  above  the  world-wide  average.  Specifically 
research  published  in  conference  proceedings  performs  strongly,  at  almost  double  the 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  60 

expected  rate.  The  higher  impact  of  the  DG  CNECT  sponsored  research  published  in 
conference proceedings is a positive sign, especially as the bulk of the research is published 
in proceedings. These proceeding papers normally get a relatively low share of citations (as 
seen in Table 7) and it is therefore a very positive sign that DG CNECT sponsored research 
scores well in this popular medium, in number of publications and citations. 

3.5  Cooperation behaviour in FP7-ICT 
In  order  to  transform  the  European  R&D  landscape,  which  has  long  been  plagued  by 
duplication  of  efforts  on  a  transnational  level,  the  European  Research  Area  and 
accompanying policy framework was established. One of the mainstays of this framework is 
the  promotion  of  cross-border  collaborations.  As  such,  cooperative  research  is  also  a  key 
characteristic of the research sponsored under FP7. This cooperation is not restricted by the 
ERA  however,  and  in  this  sub-chapter  we  analyse  the  collaborative  ties  within  the  EU-28, 
with FP7 Associated countries (the ERA area), as well as with the BRICS countries and the 
United States, Japan and South Korea.  

A  first  indication  of  the  interest  by  the  EU  to  strengthen  collaborative  R&D  ties  with  the 
above  mentioned  countries  outside  of  the  ERA,  is  the  publishing  of  several  specifically 
targeted calls, calling for EU cooperation with a specific third country. These were two calls 
published in 2011, for Brazil and Russia, and one in 2013 calling for  R&D cooperation with 
Japan. 

Table 28. Productivity of the targeted calls identified from the survey data. 

Call  
CP-SICA-INFSO 
FP7-ICT-2011-EU-Brazil 
FP7-ICT-2011-EU-Russia 
FP7-ICT-2013-EU-Brazil 
FP7-ICT-2013-EU-Japan 
Total 

Number of projects 
8 
4 
2 
4 
6 
24 

Number of publications 
11 
6 
12 
0 
2 
31 

Citations 
165 
79 
169 
0 
48 
461 

Cit/Pub 
15.0 
13.2 
14.1 
0 
24.0 
14.9 

With  1.29  publications  per  project  on  average  for  these  calls  specifically  targeted  at  third 
country  cooperation  the  productivity  falls  significantly  below  the  overall  average  of  9.115 
publications per project. The citation rate is also below the average, although the difference 
is  not  as  large  as  with  the  number  of  publications.  Here  we  see  an  average  of  14.870 
citations per publication while the overall average was 16.661. However, as we know from 
the  literature  (Narin,  Stevens  and  Whitlow,  1991;  Glanzel,  2001)  and  as  we  will  see  from 
the  econometric  analysis, 
improve  both 
productivity  and  citation  rates.  This  is  specifically  the  case  in  collaborations  with  those 
countries that have the potential to add new knowledge to the EU knowledge stock. This last 
point is being shown by us in the chapter on keyword and key-phrase analysis. So, for the 

international  cooperation  should  actually 

 
 
 
 
Final Study Report- D4 - SMART 2015/0078  61 

time  being  it  remains  a  conundrum  as  to  why  these  specific  calls  do  not  generate  the 
expected number of publications and, to a lesser extent, citations. 

3.5.1  Patterns of international cooperation (intra-EU) 
A set of publications (as we have in our journal set) and its associated set of co-authors can 
be  represented  as  a  network,  where  the  nodes  of  the  network  represent  the  authors  or  in 
this case the author’s country of affiliation. Each author is associated with a finite number of 
other  authors  for  a  single  publication.  Similarly,  summing  over  the  entire  set  of  journal 
articles  and  proceedings  papers,  each  author  can  be  linked  to  a  finite  set  of  other  authors 
through  its  particular  set  of  publications.  Again,  considering  all  authors,  author  affiliations, 
or  affiliation  countries,  a  network  can  be  generated  to  represent  the  links  between  all 
authors associated with a set of collective publications in the universe of all publications. 

The above described network(s), resulting from co-authorships, are governed by a number 
of characteristics which can be measured and visualized in order to give an indication of the 
position and characteristics of this position in the network(s). We  employ two measures in 
our network analysis of the patterns of cooperation: 

Betweenness  Centrality  of  an  actor  (author,  organization,  or  country):  This  centrality 
concept gives an indication of the extent to which an actor, or country in this case, controls 
the  flow  of  information  and  knowledge  between  two  other  actors  which  are  not  directly 
connected.  A  country  with  a  high  Betweenness  Centrality  score  can  be  considered  as  a 
knowledge  broker  or  gate  keeper  of  knowledge  flows  in  the  network15.  One  of  the 
advantages  of  the  type  of  network  data16  we  are  working  with  is  that  we  can  more  easily 
apply the Betweenness Centrality measure. The Betweenness Centrality of a certain node is 
equal  to  the  number  of  geodesic  (shortest)  paths  from  all  nodes  to  all  others  that  pass 
through  that  certain  node,  and  as  such  measures  the  control  this  node  has  over  the 
knowledge transferred along the connecting links. The geodesic path is the shortest distance 
from  one  node  to  the  other  travelling  along  the  least  amount  of  "edges"  or  links.  To 
illustrate:  if  organization  A  has to  communicate  “through”  organization  B in  order  to  reach 
the information or knowledge owned by organization C, we can see that organization B has 
the power to influence and control various attributes of the information or knowledge owned 
by organization C and desired by  organization A. Then  organization B might be deemed to 
have  a  powerful  position  in  this  network.  However,  this  will  be  so  only  if  there  are  no 
alternative geodesic paths of similar distance for organization A to choose from in order to 
reach organization C. 

Density  of  collaboration,  also  known  as  tie  strength  (Granovetter,  1973),  between  two 
countries: This is simply the number of publications, which they have “authored” together. 

15  Betweenness  Centrality  is  in  this  case  a  more  logical  metric  to  use  instead  of  Degree Centrality  as  we  are  not 
interested  in  the  popularity  of  a  certain  node,  e.g.  the  number  of  edges  or  links  leading  to  or  from  the 
aforementioned node, but in the impact of a certain node’s position on the network. 
16 The data we have collected is undirected in nature. This means that we cannot discern in- or out-going links, e.g. 
we  do  not  know  which  of  the  countries  is  providing  the  knowledge  and which  country  is  using  the  knowledge. It 
might even be both; each country is receiving from, and giving knowledge to, the other partner country. 

 
 
 
                                                           
Final Study Report- D4 - SMART 2015/0078  62 

In  this  paper,  densities  were  identified  using  UCINET  6  social  network  analysis  software. 
This  software  builds  ‘adjacency  matrices’  or  a  square  matrix  between  all  co-authors 
indicating the number of publications that any two countries have in common. In a network, 
a  higher  density  can  be  visualized  by  a  thicker  arc  representing  the  stronger  dyadic 
relationship  between,  or  a  greater  number  of  joint  publications  by,  the  two  associated  co-
authors. 

Figure 3. Network representation of the Intra-EU-28 cooperative ties. 

Note:  Node  size  based  on  Betweenness  Centrality,  arc  size  based  on  tie  strength  in  terms  of  the 
number cooperations. 

Figure 3 represents the cooperation between the EU-28 countries as based on the affiliation 
data  resulting  from  our  focal  sample.  We  see  that,  contrary  to  expectations  resulting  from 
the productivity indicators, certain countries take a prominent position within the knowledge 
network. Particularly Italy, the United Kingdom and, to a somewhat lesser degree, Portugal 
have both: a high Betweenness Centrality measure supported by a large number of high tie 
strength  cooperative  links.  Austria  and  Sweden  follow  closely,  while  Poland,  the  Czech 
Republic,  Hungary  and  Romania  seem  to  be  equally  important  knowledge  brokers  in  their 
region.  Overall,  we  see  a  very  dense  network  graph  which  suggests  that  there  are  strong 
ties  between  almost  all  EU-28  countries.  This  finding,  through  a  different  type  of  analysis, 
backs  up  our  earlier  findings,  which  also  found  that  there  was  a  higher  propensity  to 
undertake Intra-EU collaboration. 

 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  63 

Table 29. Degree, Power and Betweenness Centrality weights within the EU-28 cooperation network. 

Bonacich Power** 

Germany 
United Kingdom 
Italy 
Spain 
France 
Netherlands 
Greece 
Austria 
Poland 
Sweden 
Belgium 
Portugal 
Romania 
Czech Republic 
Hungary 
Denmark 
Ireland 
Finland 
Slovakia 
Slovenia 
Croatia 
Cyprus 
Bulgaria 
Estonia 
Lithuania 
Luxembourg 
Malta 
Latvia 
Note: *Degree Centrality and **Bonacich Power concepts explained below. ***Betweenness Centrality 
concept explained above. 

Betweenness centrality *** 
(Freeman)  
1.431 
2.576 
2.576 
1.451 
1.451 
1.451 
0.431 
2.576 
1.431 
2.576 
1.451 
2.576 
2.576 
2.576 
2.576 
0.288 
1.431 
0.431 
0.862 
0.045 
1.431 
0.143 
0.093 
1.431 
0.141 
0 
0 
0 

Degree*  
(Freeman) 
27.285 
24.011 
23.957 
21.061 
20.648 
13.923 
12.438 
10.407 
10.34 
9.322 
8.64 
8.076 
7.52 
7.219 
6.869 
6.277 
5.479 
5.399 
3.897 
3.56 
3.515 
3.071 
2.802 
2.713 
2.524 
0.377 
0.157 
0.103 

1.102 
1.26 
1.054 
1.272 
1.527 
1.512 
1.55 
1.774 
4.896 
1.46 
1.574 
2.234 
4.976 
4.574 
5.551 
2.322 
2.454 
2.112 
6.634 
4.292 
6.173 
4.228 
7.267 
6.368 
7.037 
2.049 
4.375 
1.353 

Further  to  the  concept  of  centrality,  there  turn  out  to  be  different  kinds  of  centrality 
indicating different positional advantages. Above we have described Betweenness Centrality 
but one  of the more well-known and simple ones is degree centrality. Here (in-) degree is 
regarded  as  the  basic  indicator  of  popularity,  whereas  power  centrality  (for  instance 
Bonacich  Power)  is  more  complex  and  dependent  on  the  degree  centrality  of  the  alters  in 
the network 

The degree centrality measure is based  on the star network,  which is the most  centralized 
network,  and  connected,  for  any  number  of  actors.  In  the  star  network,  all  the  actors  but 

 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  64 

one  have  a  degree  centrality  score  of  one,  and  the  "star"  has  a  degree  centrality  score 
equivalent  to  the  number  of  actors  in  the  network,  less  one.  Freeman  (1978)  felt  that 
network  degree  centrality  measure  can  be  understood  as  the  degree  of  inequality  in  a 
network as a proportion to that of a perfect “star” network of the same size. 

Now  we  can  wonder  whether  a  popular  position  in  a  network  is  the  same  as  a  powerful 
position.  Bonacich  (1987)  questioned  this  idea.  He  argued  that  being  connected  to  well-
connected  alters  might  make  an  actor  central  in  terms  of  popularity,  but  not  necessarily 
powerful.  This  is  somewhat  ironic;  by  being  connected  to  other  actors  that  are  not  well 
connected  you  are  in  a  powerful  position  in  the  network,  because  these  other  actors 
dependent  on  you  for  connecting  them  to  the  wider  network.  Conversely,  well  connected 
actors  are  not  dependent  on  you  because  they  have  many  more  “friends”  to  turn  to.  So, 
Bonacich proposed that both centrality and power were a function of the connections of the 
actors  in  one's  neighbourhood.  The  more  connections  the  actors  in  your  neighbourhood 
have, the more central you are. The fewer the connections the actors in your neighbourhood 
have to the others, the more powerful you are as a gatekeeper (Cook, et al., 1983). 

This we can clearly see in Table 29 and in our graph, Figure 3, above. Germany here is the 
case  in  point.  Although  Germany  has  the  highest  degree  centrality  score,  being  the  most 
popular country to  co-author with, its well-connected-ness also makes it less “powerful” in 
terms  of  its  gatekeeping  position  (Freeman  Betweenness  Centrality)  and  in  its  overall 
“Bonacich” power. Estonia and Croatia, in this respect, seem to be the most promising and 
potentially powerful gatekeepers in their respective “neighbourhoods”. This is a sign of the 
knowledge  diversity  within  the  EU-28  network  of  organizations  undertaking  research 
sponsored by DG CNECT. Further proof if this high diversity can be seen under section 3. 

 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  65 

3.5.2  Patterns of international cooperation (extra-EU) 
The same measures as applied above are of course also applicable to the country blocks and 
separate  countries  we  are  interested  in  exploring.  Below  in  Figures  4,  5  and  6,  we  show 
respectively  the  networks  linking  the  EU-28  countries  to  the  FP7  Associated  Countries,  the 
BRICS, and to the United States, Japan and South Korea. 

Figure 4. EU-28 - FP7 Associated Countries cooperative ties. 

Note:  Node  size  based  on  Betweenness  Centrality,  arc  size  based  on  tie  strength  in  terms  of  the 
number cooperations. Blue squares denoted the FP7 Associated Countries. 

In  line  with  expectations,  we  find  that  of  the  FP7  Associated  Countries,  Switzerland  has  a 
clear  leading  position. Of  the  EU-28  countries  in  this network,  Germany,  Spain  and  France 
take  important  positions  both  in  terms  of  knowledge  brokerage  as  well  as  in  terms  of 
strength of ties. 

 
 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  66 

Figure 5. EU-28 - BRICS cooperative ties. 

Note:  Node  size  based  on  Betweenness  Centrality,  arc  size  based  on  tie  strength  in  terms  of  the 
number cooperations. Blue squares denoted the BRICS countries. 

Again, in line with expectations, we  find in our network depicting the cooperative ties with 
the BRICS countries that the Russian Federation has both a prominent position in terms of 
knowledge  brokerage  as  well  as  in  terms  of  ties  with  a  number  of  EU-28  countries  (Italy, 
UK, Franc, Spain, Poland and Germany). 

Final Study Report- D4 - SMART 2015/0078  67 

Figure 6. EU-28 cooperative ties with the Unites States, Japan and South Korea. 

Note:  Node  size  based  on  betweenness  centrality,  arc  size  based  on  tie  strength  in  terms  of  the 
number cooperations. Blue squares denoted the USA, Japan and South Korea. 

Lastly,  in  Figure  6,  we  explore  the  network  of  the  other,  and  traditionally  important 
innovative countries, United States, Japan and South Korea. We find that the United States 
is  the  most  prominent  node  in  the  network,  both  in  terms  of  tie  strength  as  well  as  for 
Betweenness  Centrality.  Japan  and  South  Korea  share  similar  characteristics  for  both 
measures. 

3.5.3  Diversity of collaborators in terms of nationality 
Under this heading we would like to further assess the diversity of EU-28 collaborations for 
the five most cooperative  countries within the EU, as well as for the  five most  cooperative 
countries from outside of the EU. We will again use the focal and control samples generated 
and used earlier. 

In  Table  30  below,  we  can  see  from  our  focal  sample  that  from  all  of  the  DG  CNECT 
sponsored  co-authorships,  Germany  is  leading  in  terms  of  co-authorships  with  763  co-
authorships  and  47%  if  its  total  co-authorships  coming  from  within  the  country.  However, 
Italy  seems  to  be  slightly  more  focused  on  national  collaborations,  while  France  has  the 
least nationally focused number of co-authorships. 

 
 
 
 
Final Study Report- D4 - SMART 2015/0078  68 

Table 30. Diversity of Intra-EU collaborations, focal sample. 

Germany 

Italy 

Germany 
Italy 
United Kingdom 
Spain 
France 
Netherlands 
Greece 
Sweden 
Austria 
Belgium 
Portugal 
Finland 
Ireland 
Poland 
Denmark 
Czech Republic 
Romania 
Hungary 
Cyprus 
Slovenia 
Croatia 
Slovakia 
Estonia 
Lithuania 
Bulgaria 
Luxembourg 
Malta 

763 
113 
122 
81 
100 
55 
57 
30 
52 
43 
19 
19 
24 
34 
16 
17 
17 
17 
6 
6 
8 
5 
6 
6 
7 
0 
0 

47.0 % 
7.0% 
7.5% 
5.0% 
6.2% 
3.4% 
3.5% 
1.8% 
3.2% 
2.6% 
1.2% 
1.2% 
1.5% 
2.1% 
1.0% 
1.0% 
1.0% 
1.0% 
0.4% 
0.4% 
0.5% 
0.3% 
0.4% 
0.4% 
0.4% 
0.0% 
0.0% 

113 
723 
93 
83 
115 
44 
36 
28 
28 
28 
22 
20 
19 
27 
12 
16 
20 
17 
6 
7 
8 
5 
7 
6 
7 
0 
0 

United 
Kingdom 
122 
93 
604 
72 
64 
37 
45 
23 
34 
27 
18 
16 
26 
28 
9 
14 
21 
17 
19 
5 
9 
6 
6 
6 
7 
3 
1 

9.2% 
7.0% 
45.3% 
5.4% 
4.8% 
2.8% 
3.4% 
1.7% 
2.6% 
2.0% 
1.4% 
1.2% 
2.0% 
2.1% 
0.7% 
1.1% 
1.6% 
1.3% 
1.4% 
0.4% 
0.7% 
0.5% 
0.5% 
0.5% 
0.5% 
0.2% 
0.1% 

7.6 % 
48.5% 
6.2% 
5.6% 
7.7% 
3.0% 
2.4% 
1.9% 
1.9% 
1.9% 
1.5% 
1.3% 
1.3% 
1.8% 
0.8% 
1.1% 
1.3% 
1.1% 
0.4% 
0.5% 
0.5% 
0.3% 
0.5% 
0.4% 
0.5% 
0.0% 
0.0% 

Spain 

France 

81 
83 
72 
548 
71 
40 
33 
26 
28 
36 
18 
13 
21 
29 
11 
16 
16 
15 
9 
8 
8 
4 
6 
7 
6 
1 
0 

6.7% 
6.9% 
6.0% 
45.4% 
5.9% 
3.3% 
2.7% 
2.2% 
2.3% 
3.0% 
1.5% 
1.1% 
1.7% 
2.4% 
0.9% 
1.3% 
1.3% 
1.2% 
0.7% 
0.7% 
0.7% 
0.3% 
0.5% 
0.6% 
0.5% 
0.1% 
0.0% 

100 
115 
64 
71 
423 
39 
30 
28 
21 
37 
15 
12 
18 
27 
18 
17 
22 
14 
6 
3 
7 
5 
6 
6 
7 
0 
0 

9.0% 
10.4% 
5.8% 
6.4% 
38.1% 
3.5% 
2.7% 
2.5% 
1.9% 
3.3% 
1.4% 
1.1% 
1.6% 
2.4% 
1.6% 
1.5% 
2.0% 
1.3% 
0.5% 
0.3% 
0.6% 
0.5% 
0.5% 
0.5% 
0.6% 
0.0% 
0.0% 

If we now look at the same resulting measures from our control sample, we see a markedly 
different picture. From  the co-authorships reported here,  which are not sponsored through 
DG CNECT, we can see a clear national focus for all five of the most cooperative countries. 
Only Spain has a relatively low percentage of  national cooperation, but still well above the 
ones  calculated  in  the  previous  table.  These  findings  again  support  our  earlier  findings  on 
patterns  of  co-authorship.  The  DG  CNECT  sponsored  projects  have  a  much  more  even 
distribution of collaboration as well as a much lower national focus. 

 
 
 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  69 

Table 31. Diversity of Intra EU collaborations, control sample. 

Germany 

Germany 
United Kingdom 
France 
Italy 
Spain 
Netherlands 
Sweden 
Poland 
Austria 
Belgium 
Portugal 
Finland 
Greece 
Denmark 
Czech Republic 
Ireland 
Romania 
Hungary 
Slovakia 
Slovenia 
Lithuania 
Bulgaria 
Luxembourg 
Croatia 
Estonia 
Latvia 
Cyprus 

412 
36 
29 
23 
18 
12 
11 
11 
9 
11 
6 
4 
4 
6 
4 
4 
2 
3 
1 
2 
1 
2 
1 
3 
1 
1 
0 

66.8% 
5.8% 
4.7% 
3.7% 
2.9% 
1.9% 
1.8% 
1.8% 
1.5% 
1.8% 
1.0% 
0.6% 
0.6% 
1.0% 
0.6% 
0.6% 
0.3% 
0.5% 
0.2% 
0.3% 
0.2% 
0.3% 
0.2% 
0.5% 
0.2% 
0.2% 
0.0% 

United 
Kingdom 
36 
358 
31 
20 
18 
16 
11 
5 
5 
4 
12 
5 
6 
3 
2 
4 
5 
1 
2 
1 
1 
3 
1 
2 
1 
1 
0 

6.5% 
64.6% 
5.6% 
3.6% 
3.2% 
2.9% 
2.0% 
0.9% 
0.9% 
0.7% 
2.2% 
0.9% 
1.1% 
0.5% 
0.4% 
0.7% 
0.9% 
0.2% 
0.4% 
0.2% 
0.2% 
0.5% 
0.2% 
0.4% 
0.2% 
0.2% 
0.0% 

France 

Italy 

Spain 

29 
31 
307 
16 
14 
10 
5 
5 
4 
10 
8 
3 
2 
7 
3 
3 
4 
5 
1 
2 
2 
2 
1 
2 
1 
1 
0 

6.1% 
6.5% 
64.2% 
3.3% 
2.9% 
2.1% 
1.0% 
1.0% 
0.8% 
2.1% 
1.7% 
0.6% 
0.4% 
1.5% 
0.6% 
0.6% 
0.8% 
1.0% 
0.2% 
0.4% 
0.4% 
0.4% 
0.2% 
0.4% 
0.2% 
0.2% 
0.0% 

23 
20 
16 
214 
9 
4 
3 
4 
5 
4 
3 
2 
5 
2 
2 
1 
2 
4 
1 
4 
1 
2 
1 
2 
2 
1 
0 

6.8% 
5.9% 
4.7% 
63.5% 
2.7% 
1.2% 
0.9% 
1.2% 
1.5% 
1.2% 
0.9% 
0.6% 
1.5% 
0.6% 
0.6% 
0.3% 
0.6% 
1.2% 
0.3% 
1.2% 
0.3% 
0.6% 
0.3% 
0.6% 
0.6% 
0.3% 
0.0% 

18 
18 
14 
9 
156 
6 
3 
3 
4 
1 
8 
4 
2 
2 
2 
3 
4 
3 
2 
1 
1 
2 
1 
3 
1 
1 
1 

6.6% 
6.6% 
5.1% 
3.3% 
57.1% 
2.2% 
1.1% 
1.1% 
1.5% 
0.4% 
2.9% 
1.5% 
0.7% 
0.7% 
0.7% 
1.1% 
1.5% 
1.1% 
0.7% 
0.4% 
0.4% 
0.7% 
0.4% 
1.1% 
0.4% 
0.4% 
0.4% 

For the Extra-EU collaborations, a different picture emerges. When comparing Table 32 and 
Table 33, we can see that the EU-28 countries have a higher propensity to cooperate with 
the five most cooperative Extra-EU countries outside of the DG CNECT sponsored projects. 
Specifically we notice that EU-28 countries like to cooperate with Switzerland outside of the 
DG CNECT sponsored projects. 

 
 
 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  70 

Table 32. Diversity of Extra-EU collaborations, focal sample. 

Germany 
Italy 
United Kingdom 
Spain 
France 
Netherlands 
Greece 
Sweden 
Austria 
Belgium 
Portugal 
Finland 
Ireland 
Poland 
Denmark 
Czech Republic 
Romania 
Hungary 
Cyprus 
Slovenia 
Croatia 
Slovakia 
Estonia 
Lithuania 
Bulgaria 
Luxembourg 

United States 
11.4% 
72 
14.0% 
88 
11.3% 
71 
9.8% 
62 
8.4% 
53 
4.9% 
31 
5.4% 
34 
2.5% 
16 
4.0% 
25 
3.0% 
19 
1.9% 
12 
2.2% 
14 
2.2% 
14 
3.8% 
24 
1.3% 
8 
1.7% 
11 
2.5% 
16 
1.7% 
11 
2.2% 
14 
0.5% 
3 
1.1% 
7 
0.6% 
4 
1.0% 
6 
1.1% 
7 
1.0% 
6 
0.3% 
2 

Switzerland 
70 
82 
43 
39 
42 
27 
18 
7 
19 
19 
17 
8 
13 
25 
5 
12 
16 
12 
7 
4 
7 
5 
6 
6 
7 
2 

13.5% 
15.8% 
8.3% 
7.5% 
8.1% 
5.2% 
3.5% 
1.4% 
3.7% 
3.7% 
3.3% 
1.5% 
2.5% 
4.8% 
1.0% 
2.3% 
3.1% 
2.3% 
1.4% 
0.8% 
1.4% 
1.0% 
1.2% 
1.2% 
1.4% 
0.4% 

China 
31 
26 
34 
24 
21 
20 
12 
13 
12 
8 
9 
11 
11 
19 
5 
10 
15 
10 
6 
3 
8 
4 
6 
6 
6 
0 

9.4% 
7.9% 
10.3% 
7.3% 
6.4% 
6.1% 
3.6% 
3.9% 
3.6% 
2.4% 
2.7% 
3.3% 
3.3% 
5.8% 
1.5% 
3.0% 
4.5% 
3.0% 
1.8% 
0.9% 
2.4% 
1.2% 
1.8% 
1.8% 
1.8% 
0.0% 

Russian Fed 
8.4% 
25 
7.4% 
22 
7.7% 
23 
7.0% 
21 
7.4% 
22 
4.7% 
14 
3.7% 
11 
1.7% 
5 
4.3% 
13 
2.7% 
8 
3.3% 
10 
2.3% 
7 
4.0% 
12 
7.4% 
22 
1.3% 
4 
4.0% 
12 
5.4% 
16 
3.3% 
10 
2.0% 
6 
1.0% 
3 
2.3% 
7 
2.0% 
6 
2.0% 
6 
2.3% 
7 
2.3% 
7 
0.0% 
0 

Table 33. Diversity of Extra-EU collaborations, control sample. 

United States 

China 

Switzerland 

Japan 

Germany 
United Kingdom 
France 
Italy 
Spain 
Netherlands 
Sweden 
Belgium 
Poland 

40 
36 
32 
12 
8 
9 
11 
2 
2 

23% 
20% 
18% 
7% 
5% 
5% 
6% 
1% 
1% 

4 
9 
6 
0 
2 
1 
6 
1 
0 

11% 
26% 
17% 
0% 
6% 
3% 
17% 
3% 
0% 

13 
6 
5 
6 
4 
2 
1 
1 
2 

28% 
13% 
11% 
13% 
9% 
4% 
2% 
2% 
4% 

8 
10 
9 
5 
1 
0 
1 
1 
1 

20% 
24% 
22% 
12% 
2% 
0% 
2% 
2% 
2% 

Japan 
18 
11 
16 
14 
12 
7 
5 
7 
5 
3 
3 
3 
1 
6 
4 
4 
4 
5 
0 
3 
1 
4 
1 
0 
0 
0 

13.1% 
8.0% 
11.7% 
10.2% 
8.8% 
5.1% 
3.6% 
5.1% 
3.6% 
2.2% 
2.2% 
2.2% 
0.7% 
4.4% 
2.9% 
2.9% 
2.9% 
3.6% 
0.0% 
2.2% 
0.7% 
2.9% 
0.7% 
0.0% 
0.0% 
0.0% 

Russian 
Federation 
7 
4 
10 
1 
2 
1 
4 
1 
2 

18% 
11% 
26% 
3% 
5% 
3% 
11% 
3% 
5% 

 
 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  71 

Czech Republic 
Finland 
Greece 
Austria 
Denmark 
Portugal 
Ireland 
Hungary 
Romania 
Lithuania 
Slovakia 
Bulgaria 
Croatia 
Estonia 
Slovenia 

2 
3 
1 
3 
3 
2 
3 
3 
3 
0 
0 
1 
1 
0 
0 

1% 
2% 
1% 
2% 
2% 
1% 
2% 
2% 
2% 
0% 
0% 
1% 
1% 
0% 
0% 

0 
2 
0 
0 
1 
0 
1 
0 
1 
1 
0 
0 
0 
0 
0 

0% 
6% 
0% 
0% 
3% 
0% 
3% 
0% 
3% 
3% 
0% 
0% 
0% 
0% 
0% 

1 
1 
1 
1 
1 
1 
0 
0 
0 
0 
1 
0 
0 
0 
0 

2% 
2% 
2% 
2% 
2% 
2% 
0% 
0% 
0% 
0% 
2% 
0% 
0% 
0% 
0% 

1 
0 
0 
0 
0 
1 
2 
1 
0 
0 
0 
0 
0 
0 
0 

2% 
0% 
0% 
0% 
0% 
2% 
5% 
2% 
0% 
0% 
0% 
0% 
0% 
0% 
0% 

1 
1 
0 
0 
0 
1 
1 
2 
0 
0 
0 
0 
0 
0 
0 

3% 
3% 
0% 
0% 
0% 
3% 
3% 
5% 
0% 
0% 
0% 
0% 
0% 
0% 
0% 

3.5.4  Diversity of collaborators, in terms of nationality, per funding scheme 
In  this  last  section  of  this  sub-chapter,  we  look  at  the  diversity  of  nationalities  involved  in 
each  of  the  funding  schemes  under  FP7.  Due  to  the  sheer  number  of  different  countries 
collaborating  within  each  of  the  funding  schemes,  we  have  chosen  to  sum  the  different 
countries and asses the diversity as a share of the total number of United Nations Member 
States (193).  

In  Table  34  below  we  report  the  share  and  number  of  different  nationalities  per  funding 
scheme. By far the most diverse funding scheme in terms of countries involved is the CSA-
SA  sub-contract  type,  incorporating  59%  of  all  the  UN  Member  States,  followed  at  a 
distance  by  the  CP-CSA-Infra-PP  with  39%.  Overall,  we  can  conclude  that  the  DG  CNECT 
sponsored research is diverse in terms of nationality and outward looking when concerning 
cooperation with other EU-28 and non-EU countries. 

Table 34. Diversity of collaborators in terms of nationality, per strategic objective, top 10 listed. 

Funding scheme 
CSA-SA 
CP-CSA-Infra-PP 
CP-FP-INFSO 
CSA-CA 
CP-IP 
CIP-ICT-PSP-PB 
CIP-ICT-PSP-BPN 
CP-SICA-INFSO 
CIP-ICT-PSP-TN 
CIP-ICT-PSP-PA 

Number of different countries involved 
113 
75 
60 
59 
57 
40 
36 
36 
35 
32 

Percentage 
59 
39 
31 
31 
30 
21 
19 
19 
18 
17 

 
 
 
 
Final Study Report- D4 - SMART 2015/0078  72 

CIP-ICT-PSP-TN(EC) 
CP-FP-INFSO-FET 
CP-IP-INFSO-FET 
CP-CSA-PCP 
CSA-CA-INFSO-FET 
CP-CSA-Infra-IP 
CSA-SA-INFSO-FET 
CSA-ERA-Plus 
NoE 
CP 

30 
28 
27 
26 
26 
21 
11 
10 
6 
4 

16 
15 
14 
13 
13 
11 
6 
5 
3 
2 

 
 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  73 

4  Patent Analysis 

4.1  Patenting productivity of FP7-ICT 
Under this heading we undertake the assessment of the patent productivity of each project, 
as well as the productivity for each of the funding instrument and the strategic objectives of 
DG CNECT, as defined under FP7. 

4.1.1  Productivity by project 
Patenting results of projects supported by framework programmes is relatively rare and, by 
far,  less  frequent  than  publishing:  Out  of  2706  projects  in  FP7,  only  181  account  for  any 
patent(s).    The  distribution  of  patents  across  FP7  projects  is  shown  in  Figure  7.  This 
distribution  is  highly  skewed  with  more  than  half  of  the  181  projects  resulting  in  a  single 
patent, while only 5 projects claimed 6 or more patents. 

Figure 7. Distribution of patent applications by projects. 

4.1.2  Productivity by strategic objective 
Figure 8 reports the distribution of patents across  strategic objectives. Again, the research 
output  measured  by  the  number  of  patents  per  objective  is  distributed  unequally:  of  185 
different  strategic  objectives,  only  74  have  a  project  with  patent(s),  and  half  of  those 
objectives account for less than 3 patents. 

 
 
 
 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  74 

Figure 8. Distribution of patents across strategic objectives. 

The  output  measured  by  the  number  of  patents  per  objective  is  distributed  unequally:  of 
185  different  strategic  objectives,  only  74  have  a  project  with  patents,  and  half  of  those 
objectives account for less than 3 patents. At the same time the top 5 strategic objectives 
as listed in Table 35 below account for 30% of all patents. 

Table 35. Strategic objectives by patent counts, top 5 listed. 

Strategic objective 

The Network of the Future 

Next-Generation Nano-electronics Components and Electronics Integration 

Photonic components and subsystems 

Micro/nanosystems 

Secure, dependable and trusted Infrastructures 

Patents 

48 

19 

17 

16 

14 

The  ranking  of  strategic  objectives  however  changes  if  we  take  into  account  that  the 
number  of  projects  funded  under  different  strategic  objectives  varies  significantly.  Thus, 
when we compare objectives by the average number of patents per project, the top 5 looks 
quite different (Table 36). The only objective that scores high in terms of both patent counts 
as  well  as  patents  per  project  is  “Next-Generation  Nanoelectronics  Components  and 
Electronics Integration” which is present in top 3 of Tables 35 and 36.   

 
 
 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  75 

Table 36. Top 5 Strategic objectives, by patents per projects. 

Strategic objective 

Manufacturing solutions for new ICT products 
FET Proactive: Quantum ICT (QICT) 
Technology foundation: Future Internet Core Platform 
Next-Generation Nanoelectronics Components and 
Electronics Integration 
Embodied Intelligence 

Number of 
projects 
2 
6 
1 
20 

Number of 
patents 
6 
6 
1 
19 

6 

4 

Average 

3 
1 
1 
0.95 

0.67 

4.1.3  Productivity by funding instrument 
The  distribution  of  patents  by  funding  instrument  is  shown  in  Table  37.  The  two  main 
funding  schemes  claim  60%  and  30%  of  all  patents  are  CP-FP-INFSO  and  CP-IP 
respectively.    However,  similarly  to  the  distribution  of  patents  by  strategic  objective,  the 
ranking changes if we consider patents per project instead of simple patent counts. In terms 
of  patents  per  project  the  top  funding  instrument  is  “Networks  of  Excellence”  with  39 
patents per 15 projects (an average of 0.38 patents per project). 

Table 37. Distribution of patent applications by funding instrument, by patents per projects. 

Strategic objective 

CP-CSA-Infra-PP 
CP-FP-INFSO 
CP-FP-INFSO-FET 
CP-IP 
CP-IP-INFSO-FET 
CP-SICA-INFSO 
CSA-CA 
NoE 

Number of 
projects 
71 
1424 
62 
362 
19 
8 
146 
39 

Number of 
patents 
2 
212 
5 
109 
5 
2 
1 
15 

Average 

0.03 
0.15 
0.08 
0.30 
0.26 
0.25 
0.01 
0.38 

4.1.4  Productivity by country 
There  are  two  ways  in  which  we  can  attribute  a  patent  to  a  country:  by  its  inventor(s), 
whom we may consider as the source of the patented invention, and by applicant(s), who as 
we may expect, would be user(s) of the invention. Tables 38 and 39 report the number of 
patents by the country of the applicant(s) and inventor(s) of the FP7 patents. 

First, we notice that, by and large, the distributions of patent counts by country of inventor 
and by country  of applicant exhibit similar pattern: both are dominated by the two largest 
European  countries:  Germany  and  France  (with  Germany  having  almost  twice  as  many 
patents  as  France),  followed  by  Spain,  Italy,  Belgium  and  the  Netherlands.  In  what  the 
ranking differs, however, is the place of the United States: in terms of inventors’ country it 

 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  76 

follows top 5 taking place between Belgium and the Netherlands, while in terms of the origin 
of the applicant it is in the third place following Germany and France. Such pattern is what 
one  may  expect  given  importance  of  the  US  in  the  ICT  sector  in  general  and,  more 
specifically, because  of  active participation of large US  companies in the FP7 projects (two 
large US companies are among top 10 owners of the FP7 patents).  

To  mitigate  potential  effects  of  factors  specific to  country  and  reference  year,  we  compare 
our sample of FP7 patents against a control sample of patents where each patent is drawn 
randomly  from  a  population  of  patents  similar  to  the  given  FP7  patent:  patents  applied  at 
the same patent office having the same (priority) year and the same technological class. We 
draw one control patent per each focal patent. 

Since  framework  programme  funding  favours  European  individuals  and  organizations,  the 
FP7  patents  naturally  feature  Europeans.  For  example,  we  find  three  times  more  French 
patents in the focal sample of FP7 patents than among random control patents, for German 
patents  this  ratio  is  about  4,  while  for  Austria  and  Italy  it  is  15.  The  reverse  is  also  true; 
some countries with strong competencies in ICT are well represented in the control sample, 
however  much  less  so  in  the  sample  of  FP7  patents:  the  top  3  countries  in  the  control 
sample are the USA, Japan and South Korea and only the USA has any significant number of 
patents  in  the  focal  sample  (46  patents  vs.  117  in  the  random  sample),  while  Japanese 
patents  are  only  marginally  present  among  in  the  FP7  patents,  and  there  are  no  Korean 
patents. 

Table 38. Distribution of patents by applicant country, focal vs. control. 

Focal 

Control 

Percentage  Patents 
27 
13 
11 
7 
7 
5 
5 
4 
3 
2 
2 
2 
2 
2 
1 
1 
1 

112 
56 
46 
30 
29 
22 
22 
15 
13 
10 
8 
8 
7 
7 
5 
5 
4 

Country 
Germany 
France 
USA 
Italy 
Spain 
Belgium 
Netherlands 
Austria 
Finland 
UK 
Switzerland 
Sweden 
Israel 
Japan 
Czech 
Greece 
Poland 

Country 
USA 
Japan 
Germany 
South Korea 
France 
China 
Sweden 
Canada 
Switzerland 
Taiwan 
Netherlands 
Spain 
Italy 
Finland 
India 
Belgium 
Austria 

Patents 
117 
62 
33 
26 
19 
15 
9 
8 
7 
7 
6 
3 
2 
2 
2 
1 
1 

Percentage 
35 
19 
10 
8 
6 
4 
3 
2 
2 
2 
2 
1 
1 
1 
1 
0 
0 

 
 
 
Final Study Report- D4 - SMART 2015/0078  77 

1 
1 
0 
0 
0 
0 
0 
0 
0 
0 
0 
0 
0 
0 
100% 

3 
3 
2 
2 
2 
2 
2 
1 
1 
0 
0 
0 
0 
0 
417 

Canada 
Ireland 
Denmark 
India 
Norway 
Portugal 
Slovenia 
Armenia 
Singapore 
Australia 
Barbados 
Bermuda 
Brazil 
China 
Total 

Israel 
Czech 
Greece 
Poland 
Ireland 
Denmark 
Singapore 
Australia 
Barbados 
Bermuda 
Brazil 
Caymans 
Malta 
Philippines 
Total 

1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
334 

Table 39. Distribution of patents by inventor’s country. 

Focal 

Control 

Percentage  Patents 
25 
16 
9 
8 
6 
5 
5 
3 
3 
3 
3 
2 
2 
1 
1 
1 
1 
1 
1 
0 
0 
0 
0 

109 
67 
38 
34 
27 
23 
21 
15 
14 
14 
11 
10 
7 
6 
5 
5 
4 
3 
3 
2 
2 
2 
2 

Country 
Germany 
France 
Spain 
Italy 
Belgium 
USA 
Netherlands 
Switzerland 
Austria 
Israel 
Finland 
UK 
Poland 
Sweden 
Czech 
Greece 
Japan 
India 
Portugal 
Canada 
Denmark 
Norway 
Slovenia 

Country 
USA 
Japan 
Germany 
South Korea 
China 
France 
Sweden 
Canada 
Taiwan 
Italy 
Switzerland 
India 
Belgium 
Netherlands 
UK 
Russia 
Spain 
Israel 
Finland 
Poland 
Austria 
Czech 
Portugal 

Patents 
117 
61 
37 
26 
17 
15 
11 
10 
8 
5 
5 
5 
4 
4 
4 
4 
3 
3 
2 
2 
1 
1 
1 

0 
0 
0 
0 
0 
0 
0 
0 
0 
0 
0 
0 
0 
0 
100% 

Percentage 
33 
17 
10 
7 
5 
4 
3 
3 
2 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
0 
0 
0 

 
 
 
 
Final Study Report- D4 - SMART 2015/0078  78 

0 
0 
0 
0 
0 
0 
0 
0 
0 
100% 

1 
1 
1 
1 
1 
1 
1 
1 
0 
432 

Armenia 
Brazil 
China 
South Korea 
Singapore 
Turkey 
Ukraine 
Vietnam 
Australia 
Total 

Denmark 
Norway 
Brazil 
Singapore 
Australia 
Columbia 
Hungary 
Ireland 
Malta 
Total 

1 
1 
1 
1 
1 
1 
1 
1 
1 
355 

0 
0 
0 
0 
0 
0 
0 
0 
0 
100% 

We  observe  similar  pattern  when  we  turn  to  the  distribution  of  patents  by  country  of 
inventor:  European  inventors  dominate  the  focal  sample,  while  inventors  from  the  US, 
Japan and South Korea are underrepresented there. 

4.1.5  Productivity by organization 
Next in our analysis is the distribution of patents by the applicant organizations, their types 
and  funding  schemes  associated  with  their  projects.    Table  40  shows  the  distribution  of 
patents  by  applicants.  While  most  of  FP7  applicants  have  only  one  patent,  there  are 
applicants owning as many as 22 patents. Among 6 applicants who own more than 10 FP7 
patents are 1 university, 3 public research organizations, and 2 large corporations. 

Table 40. Distribution of patents by applicants. 

Patents by 
applicant 
Frequency 

1 

2 

3 

309 

48 

15 

4 

7 

5 

11 

6 

2 

7 

1 

8 

5 

12 

14 

15 

18 

22 

2 

1 

1 

1 

1 

Table 41 shows the distribution of the patents across the types of patent applicants. About 
50%  of  FP7  patents  belong  to  public  sector:  higher  education  organizations  (ACAD)  and 
public  research  organizations  (PRO),  industry:  large  companies  (CORP)  and  SMEs  (SME) 
also  own  about  50%  of  all  patents.  Significant  fraction  of  patents  is  (co)owned  by 
individuals  (INDUM).17  However  participation  of  different  types  of  organizations  varies  by 
country:  if  in  Belgium,  Italy,  Spain  and  France  most  FP7  patents  is  owned  by  universities 
and public research centres, in Austria, Netherlands and Sweden majority of FP7 patents is 
owned  by  companies.  Shares  of  public  and  private  organizations  among  the  applicants  of 
German  patents  are  roughly  similar.  As  for  the  key  non-European  countries,  the  US  and 
Japan are mostly represented by private sector, while the applicants of all 6 Israeli patents 
are individuals.  

17 Since the table reports whole patent counts, when a patent is co-owned by organizations of different types we 
count  this  patent  to  both  types  of  the  organizations.  The  pattern  of  co-ownership  of  patents  by  organizations  of 
different types is reported below.   

 
 
 
 
 
                                                           
Final Study Report- D4 - SMART 2015/0078  79 

Table 41. Distribution of patents by applicant’s type of organization and country. 

ACAD 

Applicant’s 
Country 
0 
Armenia 
2 
Austria 
12 
Belgium 
1 
Canada 
3 
Switzerland 
0 
Czech 
23 
Germany 
1 
Denmark 
9 
Spain 
2 
Finland 
10 
France 
3 
UK 
2 
Greece 
1 
Ireland 
1 
Israel 
0 
India 
15 
Italy 
Japan 
0 
Netherlands  4 
0 
Norway 
2 
Poland 
0 
Portugal 
1 
Sweden 
0 
Singapore 
2 
Slovenia 
3 
USA 
Total per 
90 
type 

PRO 

CORP 

SME 

INDUM 

0 
0 
8 
0 
0 
1 
24 
0 
6 
4 
24 
0 
4 
0 
0 
0 
5 
1 
3 
2 
0 
0 
0 
0 
0 
1 
80 

0 
7 
5 
1 
1 
3 
52 
1 
8 
8 
21 
0 
0 
2 
0 
0 
6 
5 
11 
0 
0 
0 
6 
0 
0 
37 
171 

0 
3 
2 
1 
2 
0 
9 
0 
3 
0 
4 
7 
0 
0 
0 
0 
1 
0 
2 
0 
1 
1 
0 
0 
0 
3 
40 

1 
7 
5 
2 
2 
1 
37 
0 
11 
0 
10 
0 
1 
0 
6 
2 
9 
1 
7 
1 
1 
2 
1 
1 
2 
9 
92 

Total per 
country 
1 
15 
23 
3 
8 
5 
113 
2 
31 
13 
57 
10 
5 
3 
7 
2 
30 
7 
22 
2 
4 
2 
8 
1 
2 
48 

Concerning the funding instruments, and as has been shown in Table 37, most patents are 
attributed  to  two  instruments:  CP-FP-INFSO  and  CP-IP.  Among  CP-FP-INFSO  sponsored 
patents the share of private patents is fairly close to the share of public patents, but in CP-
IP projects most patents are from industry (large corporations). We also notice that most of 
the SME patents resulted from projects funded by CP-FP-INFSO projects (Table 42). 

 
 
 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  80 

Table 42. Distribution by applicants’ type of organization and funding instrument. 

Funding instrument 
CP-CSA-Infra-PP 
CP-FP-INFSO 
CP-FP-INFSO-FET 
CP-IP 
CP-IP-INFSO-FET 
CP-SICA-INFSO 
CSA-CA 
NoE 
CIP-ICT-PSP-PB 
CP-CSA-Infra-PP 
Patents per 
applicants’ type 

ACAD 
1 
57 
5 
17 
3 
1 
1 
9 
0 
0 
90 

PRO 
0 
52 
1 
16 
1 
1 
1 
8 
0 
1 
80 

CORP 
0 
89 
0 
74 
0 
0 
0 
2 
2 
0 
171 

SME 
0 
33 
0 
4 
2 
1 
0 
0 
0 
0 
40 

HOSP 
0 
1 
0 
0 
0 
0 
0 
0 
0 
0 
1 

INDUM 
0 
59 
0 
27 
0 
0 
0 
3 
2 
0 
91 

Table 43 reports on the public-private collaboration in FP7 as measured by co-ownership of 
patents  by  applicants  of  different  types  (excluding  patents  owned  by  individuals)  across 
funding  instruments.  First,  we  notice  that  most  patents  are  owned  by  single  applicant  of 
either type (some of them co-owned by individuals/inventors).  

Second, although, in general, public partners (universities of public research centres) patent 
less, they more likely co-own an FP7 patent than do their industrial counterparts. Most co-
owned patents resulted from collaborative research involving public organizations (43 out of 
47  co-owned  patents),  and  large  majority  of  such  patents  (34  patents)  are  the  result  of 
collaboration  among  public  organizations.  While  collaboration  among  public  organizations 
resulting in co-ownership of patents is characteristic for almost all funding instruments, we 
observe collaboration with industrial partners only in CP-FP-INFSO, CP-IP and NoE projects. 

Table  43.  Public  -  Private  collaboration  by  funding  instrument  (Public:  ACAD,  PRO,  Industry:  CORP, 
SME). 

Funding instrument 

Single 
Industry 
applicant* 

Single 
Public 
applicant* 

CIP-ICT-PSP-PB 
CP-CSA-Infra-PP 
CP-FP-INFSO 
CP-FP-INFSO-FET 
CP-IP 
CP-IP-INFSO-FET 

2 
0 
111 
0 
76 
2 

0 
2 
62 
4 
27 
2 

Public - 
Industry 
collaborati
on 
0 
0 
6 
0 
2 
0 

Industry - 
Industry 
collaborati
on 
0 
0 
4 
0 
0 
0 

Public - 
Public 
collaborati
on 
0 
0 
24 
1 
2 
1 

Total per 
funding 
instrument 

2 
2 
212 
5 
109 
5 

 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  81 

0 
CP-SICA-INFSO 
0 
CSA-CA 
0 
NoE 
Total per coll. type  
4 
Note: * Including patents owned by organizations and individuals. 

0 
0 
8 
102 

1 
0 
1 
197 

0 
0 
1 
9 

1 
1 
5 
34 

2 
1 
15 

Narrowing our focus onto collaboration with SMEs, we notice that in our sample SMEs do not 
co-own  patents  neither  with  large  companies;  nor  with  other  SMEs.  The  only  type  of 
research  collaboration  involving  SMEs  which  we  find  in  our  patent  data  is  with  public 
institutions and only within CP-FP-INFSO projects (Table 44). 

Table 44. SME patterns of collaboration by funding instrument. 

Funding instrument 

Single SME 
applicant* 

SME - Public 
collaboration 

Patents per funding 
instrument 

CIP-ICT-PSP-PB 
CP-CSA-Infra-PP 
CP-FP-INFSO 
CP-FP-INFSO-FET 
CP-IP 
CP-IP-INFSO-FET 
CP-SICA-INFSO 
CSA-CA 
NoE 

0 
0 
29 
0 
4 
2 
1 
0 
0 

0 
0 
4 
0 
0 
0 
0 
0 
0 

2 
2 
212 
5 
109 
5 
2 
1 
15 

Patent per Type of 
collaboration 
Note: *Including patents owned by organizations and individuals. 

36 

4 

4.2  Impact indicators 
One  of  the  most  robust  findings  on  patent-based  statistics  as  indicators  of  knowledge 
production  is  the  skewness  of  patent  value  distribution  –  only  few  patents  are  valuable, 
while  most  others  worth  little,  if  anything  at  all.  Because  of  this  skewness,  a  thorough 
assessment of an R&D programme should not only analyse the raw patent counts resulting 
from the supported research activities, but also attempt to examine quality of the produced 
patents.  In  this  study  we  employ  several  well-known  and  widely  used  indicators  of  patent 
quality.  The  patent  quality  metrics  are  listed  and  shortly  described  in  Table  45.  Those 
indicators have been reviewed and systematically studied in Squicciarini et al. (2013). 

 
 
 
 
 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  82 

Table 45. Definition of patent quality indicators used in this study. 

Indicator 

Definition 

Meaning 

References 

Patent 
scope 

The number of distinct 
4-digit IPC subclasses 
listed in the patent 
application. 

Measure of patent’s 
technological breadth. Broad 
patents tend to be more 
valuable. 

Lerner(1994) 

1 

2 

Patent 
family size 

There exist several 
alternative definitions of 
patent families. We used 
INPADOC patent families 
frequently used in the 
patent related literature. 

3 

Grant lag 

The time elapsed 
between the date when 
the application was filed 
and the granting date. 

4 

Backward 
citations 

The number of citations 
to patent documents 
made by the patent. 

Lanjouw et al. 
(1998), Harhoff et 
al. (2003) 

Harhoff and 
Wagner (2009) 

 (Popp et al. 
2004) 

Harhoff et al. 
(2003), 

Lanjouw & 
Schankerman 
(2001) 

A patent family is a set of 
patent applications (filed in the 
same or multiple countries) 
covering the same invention. 
Invention covered by multiple 
patents filed in different 
jurisdictions is more likely to 
have higher value than 
invention covered by a single 
patent.  

If applicants put more effort 
into writing the application, 
avoid controversial claims and 
work closely with the patent 
examiners they can reduce the 
grant lag. However, it requires 
time and effort of experienced 
patent attorneys that is not 
cheap, so it pays only for 
valuable patents. 

On the other hand, patents in 
newer more complex 
technologies take significantly 
longer 

On the one hand, broader and 
therefore more valuable 
patents tend to cite more prior 
art, than narrow patents (e.g. 
Harhoff et al., 2003).  

On the other hand, many 
backward citations may be a 
sign of lack of novelty and 
incremental innovation 
(Lanjouw & Schankerman, 
2001). 

 
 
 
 
Final Study Report- D4 - SMART 2015/0078  83 

5 

Citations to 
non-patent 
literature 
(NPL) 

Count citations made to 
scientific papers, 
proceedings of 
conferences etc. 

Measure of proximity to 
scientific (state-of-art) 
research (Narin et al. 1997). 
High number of references to 
the scientific literature point to 
patents of relatively high value 
(Harhoff et al., 2003). 

6 

Forward 
citations 

The number of citations 
received by the patent 

Measure of the technological 
impact of the patent. The 
greater is the number of 
forward citation the more 
valuable if the invention.  

7 

Patent 
originality 

1 - (s1

2+s2

2+..+sn

2), 

where si = share of 
backward citations to 
patents with ith 4-digit 
IPC subclass 

Originality index measures the 
breadth of the technological 
basis of the patent. Patents 
citing diverse technological 
fields have higher originality 
scores. 

8 

Patent 
generality 

1 - (s1

2+s2

2+..+sn

2), 

where si = share of 
forward citations to 
patents with ith 4-digit 
IPC subclass 

Generality index measures 
impact of the patented 
inventions across different 
technologies. Patents cited by 
later generation of patents 
from diverse technological 
fields have higher generality 
scores. 

Narin et al. 
(1997), 

Carpenter et al. 
(1980), 

Harhoff et al. 
(2003), 
Branstetter 
(2005) 

Trajtenberg 
(1990), Harhoff et 
al. (2003) 

Trajtenberg et al. 
(1997) 

Hall (2002) 
version of the 
index corrects for 
statistical bias. 

Tranjtenberg et 
al. (1997) 

Hall (2002) 
version of the 
index corrects for 
statistical bias.  

Squicciarini et al. 
(2013) proposes 
an alternative 
(more advanced) 
version of the 
index. 

9 

Patent 
claims 

Number of claims in the 
patent application. 

Broader, and therefore more 
valuable, patents have more 
claims. 

Lanjouw & 
Schankerman 
(2001) 

In  order  to  avoid  potential  idiosyncrasies  of  specific  time  period,  technology  and  also 
country  of  the  applicant  we  compare  our  sample  of  FP7  patents  against  a  set  of  randomly 
chosen control patents which mimic certain features of the focal patent set. The population 
of  patents  from  which  we  draw  our  control  sample  matches  the  focal  patents  by  patent 
office, priority year and technology (4-digit IPC code) and the country of the applicants (if 
there  are  multiple  applicants,  then  at  least  one  of  them).  The  details  of  the  matching 

 
 
 
 
Final Study Report- D4 - SMART 2015/0078  84 

procedure and the characteristics of the control sample are described in section 2.6, Table 5 
and Figures 1 and 2.  

The  quality  indicators  for  the  focal  and  control  patents  were  extracted  from  the  latest 
available  edition  of  OECD  database  on  patent  quality  (March  2018  edition  based  on 
PATSTAT  Fall  2017).  Since  the  OECD  database  covers  only  EPO  and  USPTO  patents,  thus 
patents of WIPO and national patent offices could not be matched, yet we end up with the 
sample of 274 focal-control pairs of patents.  

Table  46  reports  on  the  results  of  the  Welch  unequal  variances  t-test.  We  run  a  paired  t-
test, where “focal-control” pairing is the output of the random matching procedure outlined 
in section 2.6. This test tells us whether the focal and control samples systematically differ 
with  respect  to  the  quality  metrics  listed  in  Table  45,  and  whether  the  differences,  if  they 
exist, are statistically significant. 

Table 46. Patent quality: FP7 patents vs. control patents. 

Indicator 
Number of claims 
Patent scope 
Family size 
Grant lag 
Backward citations 
NPL citations 
Forward citations 
Generality 
Originality 

FP7 
15.3 
1.5 
2.9 
1282.3 
6.7 
5.7 
2.3 
0.37 
0.67 

control 
14.1 
2.0 
3.5 
1298.2 
9.0 
2.6 
3.9 
0.40 
0.74 

t statistics 
2.17 
-5.94 
-3.25 
0.69 
-2.53 
4.16 
-0.99 
-1.12 
-4.13 

p-value 
0.03 
0.00 
0.00 
0.49 
0.01 
0.00 
0.32 
0.27 
0.00 

DF 
135 
268 
268 
67 
268 
268 
268 
67 
222 

The  FP7  patents  tend  to  have  higher  number  of  claims  than  do  patents  from  the  control 
sample, which may suggest that the FP7 patents are technologically broader. On the other 
hand,  they  also  tend  to  be  narrower  in  the  scope  and  have  smaller  families.  Although,  in 
general, a larger patent family might be an indicator of higher patent value (e.g. Lanjouw et 
al.  1998),18  one  should  bear  in  mind  that  unlike  applicants  of  control  patents  many 
applicants  of  the  FP7  patents  are  universities  or  academic  research  institutions,  and 
academics  usually  have  less  resources  and  fewer  incentives  for  vigorous  patenting 
compared  to  companies.  We  find  no  significant  difference  between  the  focal  and  control 
samples with respect to grant lag.  

The  FP7  patents  tend  to  cite  other  patents  (backward  citations)  less  frequently  and,  as  a 
result,  the  technological  diversity  of  their  citations  (originality)  is  lower.  This,  however,  is 
not very surprising because when comparing backward citations we should take into account 
that FP7 patents are closer to fundamental research than patents from control sample and, 

18 Also see Sapsalis, van Pottelsberghe de la Potterie and Navon (2006) who examined Belgian biotech patents and 
find that family size has no significant effect on patent values. 

 
 
 
 
                                                           
Final Study Report- D4 - SMART 2015/0078  85 

as  a  result,  they  tend  to  cite  scientific  papers  rather  than  patent  documents.  This  pattern 
becomes  clear  when  we  turn  to  the  difference  in  the  number  of  citations  made  to  non-
patent literature (NPL citations). Furthermore, the fact that patent examiners also have not 
added  many  patent  citations  may  also  suggest  that  FP7  patents  have  fewer  predecessors 
and probably explore new fields for research rather than exploit existing ones. 

What  is  interesting,  however,  is  that  from  the  first  glance  our  FP7  patents  tend  to  receive 
fewer  citations  than  patents  in  the  control  sample  even  though  this  difference  is  not 
statistically  significant.  Although  forward  citations  are  not  a  very  reliable  indicator  for 
recently  published  patents,  it  is  still  rather  surprising.  For  further  investigation  we  have 
divided our sample into the patents applied in EPO and at the USPTO. The results of the t-
test for the split sample are shown in Table 47. 

Table 47. Patent quality: FP7 patents vs. control patents (split sample). 

Indicator  
USPTO patents 
Number of claims 
Patent scope 
Family size 
Grant lag 
Backward citations 
NPL citations 
Forward citations 
Generality 
Originality 
Indicator 
EPO patents 
Number of claims 
Patent scope 
Family size 
Grant lag 
Backward citations 
NPL citations 
Forward citations 
Generality 
Originality 

FP7 

control 

t statistics 

p-value 

18.53 
1.51 
1.69 
1187.09 
11.91 
8.83 
5.11 
0.39 
0.75 
FP7 

13.76 
1.51 
3.55 
1371.96 
4.24 
4.20 
0.87 
0.34 
0.62 

17.02 
2.10 
2.47 
1126.35 
16.75 
3.18 
10.76 
0.51 
0.80 
control 

12.49 
1.98 
4.07 
1502.25 
5.25 
2.34 
0.56 
0.27 
0.72 

1.74 
-3.61 
-3.63 
1.43 
-1.87 
3.20 
-1.12 
-2.40 
-1.92 
t statistics 

1.33 
-4.72 
-2.02 
-0.64 
-2.45 
2.72 
1.41 
1.64 
-3.66 

0.09 
0.00 
0.00 
0.16 
0.06 
0.00 
0.26 
0.02 
0.06 
p-value 

0.19 
0.00 
0.04 
0.53 
0.02 
0.01 
0.16 
0.12 
0.00 

DF 

45 
87 
87 
45 
87 
87 
87 
50 
68 
DF 

89 
180 
180 
21 
180 
180 
180 
16 
153 

The general pattern is similar to the one we saw in Table 46, however in the EPO subsample 
FP7 patents are more  cited than the control patents, thus the result in the original sample 
largely owes to the USPTO patents. 

When we explore the control patents in the USPTO subsample, we notice that most patents 
there received fewer  citations than the corresponding FP7 patents. What drives the  results 

 
 
 
 
Final Study Report- D4 - SMART 2015/0078  86 

is the presence of few highly cited patents in the control sample19. Presence of such outliers 
in the control sample illustrates a fundamental nature of uncertainty surrounding innovation 
process  which  has  received  significant  attention  in  innovation  literature  (e.g.  Scherer, 
Harhoff 2000; Silverberg, Verspagen 2007). The distribution of values of inventions is often 
heavy-tailed: most inventions have low or no value, but there are few which have extremely 
high  values.  The  FP7  sample  is  probably  too  small  to  have  such  important  patents. 
Therefore for valid comparison we have decided to exclude highly cited patents.   

After  excluding  the  few  most  highly  patents  (control  and  FP7  patents  with  more  than  30 
forward citations and their matching pairs) we repeated our t-test. The results are shown in 
Table 48. 

Table 48. Patent quality: FP7 patents vs. control patents (outliers excluded). 

Indicator 
Number of claims 
Patent scope 
Family size 
Grant lag 
Backward citations 
NPL citations 
Forward citations 
Generality 
Originality 

FP7 
15.15 
1.51 
2.97 
1266.4 
6.57 
5.42 
2.11 
0.38 
0.66 

control 
13.91 
2.01 
3.54 
1297.4 
7.94 
2.47 
1.29 
0.40 
0.74 

t statistics 
2.02 
-5.89 
-3.05 
0.49 
-1.79 
4.25 
2.97 
-0.54 
-4.11 

p-value 
0.04 
0.00 
0.00 
0.63 
0.07 
0.00 
0.00 
0.59 
0.00 

DF 
132 
260 
260 
64 
260 
260 
260 
59 
215 

As one can see after excluding the few outliers the difference between citations received by 
FP7  patents  and  control  patents  becomes  positive  and  statistically  significant.  Thus,  on 
average,  the  patents  resulted  from  FP7-ICT  sponsored  research  have  higher  value  than 
similar  patents  as  can be  seen  by  most  patent  quality indicators.  It,  however,  lacks  highly 
valuable patents. 

4.3  International cooperation behaviour in FP7-ICT 
Support  for  transnational  cooperation  in  research  and  development  is  one  of  the  EU 
strategic priorities, and one of the FP7 objectives has been fostering research collaboration 
within  EU  and  beyond.  In  this  section  we  examine  the  pattern  of  international  research 
collaborations  in  our  focal  sample  of  FP7  patents  by  comparing  it  to  a  ‘typical’  pattern  of 
international  collaborations.  For  that  we  draw  two  random  control  samples  as  described  in 
section 2.6. Control sample 1 is drawn from population of patents matched by patent office, 
priority  year  and  technology,  while  in  Control  sample  2  we  also  request  that  the  applicant 
origin of control patents match the country of the applicant of the focal patent. The results 
of the comparison are shown below in Table 49. 

19  One  such  control  sample  patent  has  a  staggering  481  citations,  and  had  been  originally  patented  by  a  UK 
company and later acquired by Samsung Electronics.  

 
 
 
 
                                                           
Final Study Report- D4 - SMART 2015/0078  87 

Table 49. Patterns of international research collaborations: Focal vs. control sample. 

Focal sample 

Control sample 1 

Control sample 2 

Single applicant 
EU 
non - EU 
Multiple applicants, same country 
non-EU 
intra-EU 
International collaboration 
EU-EU 
EU-USJP 
EU-BRICS 
EU-EM 
EU-OTHER 

196 
26 

6 
64 

26 
23 
2 
1 
2 

57% 
8% 

2% 
18% 

8% 
7% 
1% 
0% 
1% 

82 
235 

11 
2 

2 
0 
0 
0 
0 

In  Table  49  we  first  notice  that  FP7  patents  show  stronger  tendency  to  collaborate.  In 
Control sample 1, patents resulting from collaboration between two and more applicants are 
very rare and constitute less than 5% of the sample. While part of that may be owed to the 
fact  that  collaboration  is  more  frequent  among  European  applicants  which  are  ‘over-
represented’  in  the  FP7  patent  sample,  we  still  find  that  even  after  we  control  for  the 
applicants’  origin, in  Control  sample  2,  the  share  of  patent  with  multiple  applicants  is  only 
slightly  above  10%.  This  is  in  drastic  contrast  with  the  40%  of  such  patents  in  the  FP7 
sample.  

Second, we also notice that when FP7 patentees collaborate it is relatively more likely to be 
a collaboration between applicants from the same country (intra-EU). In Control sample 2, 
the ratio of intra-EU to  EU-EU  collaboration is 12:14, while in the focal sample it is 64:26. 
Furthermore,  relative  frequencies  of  intra-EU  vs.  EU  -  US/Japan  collaborations  are  lower 
among  FP7  patents.  In  the  focal  sample  this  ratio  is  26:23,  while  in  random  sample  it  is 
14:8. 

4.3.1  Internationalization  of  technology  through  cross-border  ownership 

of inventions 

Table 50 reports two additional indicators, IdAf and AdIf (Guellec and Pottelsberghe 2001), 
measuring intensity of transnational research collaboration based on cross-border ownership 
of FP7 patents. The former, IdAf, is defined as the share of all patents signed by domestic 
inventors (column Id) assigned to foreign applicants (column Af), therefore it indicates the 
intensity of outgoing knowledge flows.  The  other indicator, AdIf, is defined as the share  of 
patents  owned  by  domestic  organizations  (or  individuals)  with  foreign  inventors,  thus  it 
measures the intensity of incoming knowledge flows. 

 
 
 
 
 
 
 
 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  88 

Table 50. Patterns of international research collaborations: Focal vs. control sample. 

Country 
Austria 
Belgium 
Brazil 
Canada 
Switzerland 
China 
Czech Republic 
Germany 
Denmark 
Spain 
Finland 
France 
United Kingdom 
Greece 
Ireland 
Israel 
India 
Italy 
Japan 
South Korea 
Netherlands 
Norway 
Poland 
Portugal 
Sweden 
Singapore 
Turkey 
Ukraine 
United States 
Vietnam 
Note: Patents with missing information either on the country of inventor(s) or country of applicant(s) 
were excluded from the analysis.   

IdAf 
0.21 
0.44 
1.00 
1.00 
0.67 
1.00 
0.20 
0.23 
0.50 
0.50 
0.09 
0.34 
0.20 
0.40 
0.00 
0.93 
1.00 
0.34 
0.75 
1.00 
0.42 
0.50 
0.71 
1.00 
0.33 
1.00 
1.00 
1.00 
0.81 
1.00 

AdIf 
0.14 
0.33 
0.00 
1.00 
0.75 
0.00 
0.20 
0.25 
1.00 
0.28 
0.31 
0.29 
0.13 
0.25 
1.00 
0.00 
1.00 
0.36 
0.86 
0.00 
0.41 
0.50 
0.50 
1.00 
0.43 
0.00 
0.00 
0.00 
0.96 
0.00 

Ad 
14 
21 
0 
3 
8 
0 
5 
110 
2 
29 
13 
56 
8 
4 
2 
0 
2 
28 
7 
0 
22 
2 
4 
2 
7 
0 
0 
0 
46 
0 

Id 
14 
25 
1 
2 
15 
1 
5 
108 
2 
36 
11 
67 
10 
5 
0 
14 
3 
32 
4 
1 
19 
2 
7 
3 
6 
1 
1 
1 
21 
1 

If 
2 
7 
0 
3 
6 
0 
1 
28 
2 
8 
4 
16 
1 
1 
2 
0 
2 
10 
6 
0 
9 
1 
2 
2 
3 
0 
0 
0 
44 
0 

Af 
3 
11 
1 
2 
10 
1 
1 
25 
1 
18 
1 
23 
2 
2 
0 
13 
3 
11 
3 
1 
8 
1 
5 
3 
2 
1 
1 
1 
17 
1 

Focusing  on  the  countries  with  more  than  10  patents  (for  countries  with  fewer  patents 
indicators are prone to large fluctuations) we notice that European countries can be roughly 
divided  into  three  groups.  First,  there  are  small  open  economies  (Switzerland,  Spain, 
Belgium,  the  Netherlands)  with  IdAf  ratio  above  0.40.  Inventors  of  FP7  patents  in  those 
countries  often  sign  patents  by  foreign  applicants  (often  co-owned  by  domestic 
organization). On the other extreme, there is a group of countries including Finland, the UK, 

 
 
 
 
Final Study Report- D4 - SMART 2015/0078  89 

Austria,  and  Germany  which  have  small  values  of  IdAf  indicator.  Inventors  in  those 
countries  are  less  likely  to  sign  FP7  patents  owned  by  foreign  organizations.  In  the  middle 
there are Italy and France both of which have IdAf about 0.3, meaning that about third of 
patented  inventions  of  Italian  and  French  inventors  are  (co)owned  by  foreign  applicants. 
Similar  pattern  is  observed  when  we  turn  to  the  AdIf  indicator,  again  Switzerland  and  the 
Netherlands are the top countries according to this indicator, and the UK is at the bottom of 
the list. Interestingly, Spain moves from top 3 to middle-bottom of the list, which suggests 
that  for  this  country  the  intensity  of  outgoing  knowledge  flows  exceeds  the  intensity  of 
incoming flows. 

 
 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  90 

Final Study Report- D4 - SMART 2015/0078  91 

5  Econometric Analysis 

In  this  section  we  examine  the  characteristics  of  the  projects  and  of  the  teams  that 
influence  the  scientific  and  technological  productivity  of  the  projects  over  the  period  of 
2008-2017. We capture the scientific and technological productivity of the projects using the 
number  of  publications  and  patents  per  project  per  year.  We  also  carry  out  additional 
analysis  using  citations  per  publications  and  patents  respectively  as  dependent  variables. 
Using  these  variables  has  the  advantage  of  capturing  the  quality  of  the  output.  The 
explanatory variables we employ in publication and patent models remain largely the same.  

Annex  3  and  Annex  4  provide  summary  statics  of  the  publication  sample  and  the  patent 
sample respectively. In what follows we discuss these variables. 

One  of  the  key  explanatory  variables  that  we  employ  in  this  paper  is  the  diversity  of  the 
national  science  base  of  project  participants.  Recent  research  has  emphasized  the 
importance  of  sourcing  knowledge  from  diverse  locations  endowed  with  location  specific 
knowledge.  Accessing  such  broad  knowledge  elements  would  increase  the  possibilities  for 
knowledge  recombination,  thereby  potentially  increasing  scientific  and  technological 
productivities  (Leiponen  and  Helfat,  2011,  Nieto  and  Rodriquez,  2011,  Lahiri,  2010, 
Freeman, 1991, and Ahuja, 2000).  

To  construct  this  variable,  we  first  assembled  the  number  of  ICT-related  publications  per 
year for each country to which one of the focal organizations belong. We then calculated the 
Herfindahl index of the distribution of country-level publications across each organization in 
a  project.  Next  we  derive  a  diversity  measure  of  national  science  base  by  subtracting  the 
Herfindahl index from 1. As Annexes 3 and 4 show, in both samples the diversity measure 
never takes the extreme value of 1. 

To account for the effect of funding, we employ a mix of variables. These are Total Funding 
per  project,  Number  of  Participating  organizations,  and  Funding  per  participants.  The  first 
and the last two are highly correlated to each other, so we use these two sets of variables in 
separate models.  

In  order  to  examine  whether  publication  output  of  an  organization  influences  its  patent 
output and vice versa, we use current publications as an explanatory variable in the patent 
model and current patents as an explanatory variable in the publication model.  

We  furthermore  use  several  dummy  variables  to  identify  the  effect  of  different  funding 
instruments.  Note  in  this  respect  the  correlation  between  funding  instruments  are  rather 
moderate.  However,  in  order  to  ensure  the  robustness  of  our  results  to  the  potential 
inclusion  or  exclusion  of  these  variables,  we  estimate  models  with  and  without  them.  To 
take into account the potential impact of 2008 economic crisis we used a dummy that takes 
a value of 1 for projects that started after 2008. 

 
 
 
Final Study Report- D4 - SMART 2015/0078  92 

Given that the dependent variables, publications and patents, are count variables, and they 
furthermore exhibit substantial dispersion we employ a negative binomial model to estimate 
the models on publication productivity and patent productivity, with robust standard errors. 
Below we discuss the results of the analysis in turn. 

5.1  Results of the publication model 
We  begin  by  discussing  the  results  of  the  publication  output  models  which  are  reported  in 
Table 51, where we carry out an econometrics analysis on a sample of 1285 observations at 
the level of project. All the variables are contemporaneous; they refer to the same year as 
that  of  the  dependent  variable.  As  noted  earlier,  some  of  the  variables  are  highly 
correlated20, and therefore we estimate these variables separately. Model 1 consists of the 
variables  Project  duration,  Total  funding,  and  Patents,  all  of  which  have  positive  and 
significant  coefficients.  These  results  are  consistent  with  our  expectations.  Naturally, 
projects  that  receive  more  funding  and  that  receive  sustained  financial  support  tend  to 
outperform  those  that  receive  lower  financial  support  and  sustained  assistance.  Projects 
with high patenting activity tend also to publish more, indicating the complementary nature 
of  the  two.  A  negative  coefficient  for  the  period  dummy  suggests  a  lower  degree  of 
publishing during the period following the financial crisis. In model 2 we add diversity of the 
national  science  base  of  participants  in  a  project.  This  variable  turns  out  to  have  a 
significantly  positive  coefficient.  This  result  too  is  in  line  with  our  expectations:  a  project 
team that is highly international, and as such possessing diverse endowments of knowledge 
seems  to  have  strong  effect  on  publication  productivity.    Model  3  replicates  model  2,  with 
funding  instrument  dummies  as  additional  variables.  The  significant  effect  of  all  variables 
remains intact, with total funding turning out to have a significant effect in this full model. 
The  results  on  the  funding  instrument  suggest  that  compared  to  the  baseline  categories 
such  as  CP,  CSA  and,  in  particular,  NoEs  seem  to  generate  a  significant  performance 
enhancing effect.  

Models 4 to 6 replicate those from 1 to 3, but with one critical difference. The latter set of 
models  replaces  project  funding  in  the  former  set  of  models  with  number  of  participating 
organizations and funding per participants. The results show that enhancing the number of 
participating  organizations  enhances  productivity;  however,  funding  per  person  does  not 
show positive effect on performance after controlling for funding instruments. 

The  results of the analysis based on  citations per publication are  reported in Table 52. We 
discuss  a  few  key  differences  here.  Patents  do  not  have  a  significant  effect  anymore. 
Funding  and  the  number  of  participants  have  a  negative  effect  on  the  quality  of 
publications. One interpretation of the latter results seems to be that projects that are large 
may have a focus on scale due to the possibility of a high pressure to publish in large scale 
projects.  Smaller  projects  on  the  other  hand  may  face  less  publication  pressure,  allowing 
them to focus more on quality. More research is needed to further explore these questions. 

20 See Annex 3 for summary statistics of the publication productivity model 

Final Study Report- D4 - SMART 2015/0078  93 

Table 51. Negative Binomial Regression Analysis of Projects’ Publication Productivity, 2008-2013. The 
Dependent Variable = Total Publications. 

(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

VARIABLES 

Patents 

Project duration 

Total Funding 

Diversity of national science base 

Nr of participants 

Funding per partner 

contr_BPN 

contr_CP 

contr_CP_CSA 

contr_CSA 

contr_NOE 

contr_PB 

0.276*** 
(0.084) 
2.402*** 
(0.180) 
0.213*** 
(0.057) 

0.321*** 
(0.081) 
2.207*** 
(0.181) 
-0.025 
(0.061) 
3.076*** 
(0.278) 

0.375*** 
(0.076) 
1.366*** 
(0.184) 
0.139** 
(0.061) 
1.527*** 
(0.309) 

0.338*** 
(0.083) 
2.478*** 
(0.181) 

0.336*** 
(0.081) 
2.265*** 
(0.182) 

0.376*** 
(0.076) 
1.371*** 
(0.185) 

2.565*** 
(0.330) 
0.138 
(0.086) 
-0.137* 
(0.075) 

0.516*** 
(0.074) 
-0.159** 
(0.076) 

1.450*** 
(0.331) 
0.185** 
(0.080) 
0.118 
(0.081) 
1.524 
(1.087) 
2.655*** 
(1.005) 
2.808*** 
(1.008) 
2.575** 
(1.002) 
4.838*** 
(1.013) 
1.326 
(1.042) 
-0.154** 
(0.064) 
-7.889*** 
(1.336) 
1,285 

1.484 
(1.083) 
2.578*** 
(0.991) 
2.770*** 
(1.003) 
2.518** 
(0.995) 
4.777*** 
(1.005) 
1.277 
(1.037) 
-0.157** 
(0.063) 
-7.982*** 
(1.261) 
1,285 

Constant 

Dummy ( >2008=1) 

-0.115* 
(0.068) 
-6.996*** 
(0.811) 
Observations 
1,285 
Note:  Robust  standard  errors  in  parentheses,  ***  p<0.01,  **  p<0.05,  *  p<0.1.  All  explanatory 
variables are log transformed (except diversity of national science base and dummy variables). 

-0.157** 
(0.070) 
-9.036*** 
(0.792) 
1,285 

-0.092 
(0.068) 
-5.836*** 
(0.946) 
1,285 

-0.085 
(0.069) 
-5.425*** 
(0.964) 
1,285 

 
 
 
  
  
  
  
  
  
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  94 

Table  52.  Negative  Binomial  Regression  Analysis  of  Projects’  Publication  Impact,  2008-2013.  The 
Dependent Variable = Citations per Publication. 

VARIABLES 

Patents 

Project duration 

Total Funding 

Diversity of national science base 

Nr of participatnts 

Funding per partner 

contr_BPN 

contr_CP 

contr_CP_CSA 

contr_CSA 

contr_NOE 

o.contr_PA 

contr_PB 

(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

0.100 
(0.080) 
1.275*** 
(0.164) 
-0.402*** 
(0.051) 

0.113 
(0.079) 
1.382*** 
(0.166) 
-0.529*** 
(0.057) 
1.663*** 
(0.331) 

0.128 
(0.078) 
1.370*** 
(0.181) 
-0.575*** 
(0.061) 
1.898*** 
(0.343) 

1.310 
(0.903) 
2.129*** 
(0.785) 
2.521*** 
(0.800) 
2.013** 
(0.791) 
1.642** 
(0.807) 
- 

0.089 
(0.080) 
1.178*** 
(0.173) 

0.090 
(0.079) 
1.146*** 
(0.172) 

0.109 
(0.078) 
1.246*** 
(0.183) 

2.342*** 
(0.355) 
-0.797*** 
(0.079) 
-0.342*** 
(0.067) 

-0.490*** 
(0.064) 
-0.323*** 
(0.068) 

2.296*** 
(0.358) 
-0.785*** 
(0.080) 
-0.377*** 
(0.080) 
0.910 
(0.906) 
1.499* 
(0.799) 
1.945** 
(0.812) 
1.606** 
(0.795) 
1.192 
(0.814) 
- 

Constant 

Dummy ( >2008=1) 

-0.072 
(0.066) 
4.599*** 
(0.779) 
1,285 
Observations 
Note:  Robust  standard  errors  in  parentheses,  ***  p<0.01,  **  p<0.05,  *  p<0.1.  All  explanatory 
variables are log transformed (except diversity of national science base and dummy variables). 

-0.092 
(0.066) 
4.275*** 
(0.770) 
1,285 

-0.107 
(0.067) 
3.887*** 
(0.820) 
1,285 

-0.107 
(0.066) 
3.331*** 
(0.815) 
1,285 

0.529 
(0.841) 
-0.071 
(0.066) 
3.055*** 
(1.162) 
1,285 

0.115 
(0.845) 
-0.101 
(0.066) 
1.904 
(1.220) 
1,285 

 
 
 
  
  
  
  
  
  
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  95 

5.2  Results of the patent model 
We replicate the analysis of publications above for patents (see Tables 53 and 54). It must 
be  noted  that  whereas  we  had  a  sizeable  number  of  observations  in  the  analysis  of 
publications with positive outcomes (over 1280), in the patent sample only a small number 
of  about  145  projects  have  positive  outcomes.  Among  the  variables  related  to  project  size 
and funding, only total funding shows a significant positive impact consistently in all models. 
Of  course,  patenting  is  costly  and  projects  that  are  well  funded  are  therefore  likely  to  be 
able to meet the  costs  associated with patent filing. We find a complementary relationship 
between  publishing  and  patenting,  although  not  consistently  across  models.  Finally,  the 
models  with  citations  per  patent  as  the  dependent  variable  yield  few  significant  outcomes. 
We  desist  from  making  any  definitive  conclusions  from  these  results,  but  also  from  the 
patent analysis in general due to the small number of observations. 

Table  53.  Negative  Binomial  Regression  Analysis  of  Projects’  Patent  productivity.  The  Dependent 
Variable = Total Patents. 

VARIABLES 

Publications 

Project duration 

Total Funding 

Diversity of national science base 

Nr of participatnts 

Funding per partner 

contr_CP 

contr_CP_CSA 

contr_CSA 

Dummy ( >2008=1) 

Constant 

(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

0.077 
(0.048) 
-0.592* 
(0.324) 
0.175** 
(0.079) 

0.082* 
(0.049) 
-0.597* 
(0.324) 
0.204** 
(0.096) 
-0.333 
(0.611) 

0.070 
(0.050) 
-0.699** 
(0.329) 
0.254** 
(0.102) 
-0.514 
(0.628) 

-0.487* 
(0.281) 
-1.521* 
(0.777) 
-0.907 
(1.042) 

0.080* 
(0.048) 
-0.605* 
(0.327) 

0.083* 
(0.049) 
-0.601* 
(0.328) 

0.072 
(0.050) 
-0.726** 
(0.334) 

-0.299 
(0.654) 
0.206 
(0.154) 
0.213 
(0.130) 

0.163 
(0.122) 
0.205 
(0.129) 

-0.410 
(0.658) 
0.217 
(0.156) 
0.306** 
(0.145) 
-0.543* 
(0.302) 
-1.568** 
(0.783) 
-0.886 
(1.043) 

-0.168 
(0.109) 
0.085 
(1.631) 
178 

-0.173 
(0.109) 
-0.109 
(1.669) 
178 

-0.191* 
(0.109) 
0.155 
(1.683) 
178 

-0.171 
(0.109) 
-0.256 
(1.905) 
178 

-0.174 
(0.109) 
-0.271 
(1.902) 
178 

-0.196* 
(0.109) 
-0.381 
(1.930) 
178 

Observations 
Note:  Robust  standard  errors  in  parentheses,  ***  p<0.01,  **  p<0.05,  *  p<0.1.  All  explanatory 
variables are log transformed (except diversity of national science base and dummy variables). 

 
 
 
  
  
  
  
  
  
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  96 

Table  54.  Negative  Binomial  Regression  Analysis  of  Projects’  Patent  productivity.  The  Dependent 
Variable = Citations per Patent. 

VARIABLES 

Publications 

Project duration 

Total Funding 

Diversity of national science 
base 

Nr of participatnts 

Funding per partner 

contr_CP 

contr_CP_CSA 

contr_CSA 

perioddum 

Constant 

(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

0.110 
(0.160) 
-1.273 
(1.157) 
0.229 
(0.226) 

0.146 
(0.163) 
-1.373 
(1.156) 
0.385 
(0.282) 

-1.760 
(1.952) 

0.152 
(0.164) 
-1.565 
(1.175) 
0.386 
(0.288) 

-1.388 
(1.942) 

0.654 
(1.048) 
-1.276 
(2.040) 
-14.014 
(1,391.904) 
-0.272 
(0.308) 
0.494 
(4.097) 

-0.253 
(0.305) 
1.573 
(3.897) 

-0.286 
(0.307) 
0.726 
(3.993) 

0.081 
(0.174) 
-1.311 
(1.165) 

0.109 
(0.173) 
-1.451 
(1.171) 

0.104 
(0.173) 
-1.681 
(1.184) 

-2.038 
(1.995) 
0.648 
(0.466) 
0.262 
(0.337) 

0.382 
(0.391) 
0.134 
(0.312) 

-0.286 
(0.313) 
2.621 
(4.705) 

-0.339 
(0.315) 
2.236 
(4.773) 

-1.761 
(1.980) 
0.734 
(0.472) 
0.221 
(0.342) 
0.671 
(1.049) 
-1.526 
(2.074) 
-20.656 
(35,700.573) 
-0.342 
(0.316) 
2.573 
(4.884) 

Observations 
Note:  Robust  standard  errors  in  parentheses,  ***  p<0.01,  **  p<0.05,  *  p<0.1.  All  explanatory 
variables are log transformed (except diversity of national science base and dummy variables). 

145 

145 

145 

145 

145 

145 

 
 
 
  
  
  
  
  
  
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  97 

6  Newly Created Organizations 

For  the  2016  survey  wave  surveying  the  years  2014  and  2015,  DG  CNECT  has  added  two 
new questions (with two sub-questions) which query the respondents on output other than 
publications  and  patents.  The  first  one  of  these  is  inquiring  into  any  start-up  or  spin-off 
companies resulting from the DG CNECT sponsored projects. 

6.1  Introduction to the New Questions Added to the Survey 
As  already  mentioned  in  the  Inception  Report  (D1)  the  addition  of  the  above  mentioned 
questions is interesting as it goes beyond the traditional line of enquiry, but it also possess 
some challenges.  

In order to apply the same weight to the new data as we do to the publication and patent 
data,  we  do  need  to  control  for  the  quality  (e.g.  accuracy  and  reliability)  of  the  data 
collected  for  the  new  organisations  and  for  the  innovations.  A  basic  level  of  validation  is 
called  for  in  this  respect  and  we  have  set  out  our  intentions  to  do  so  and  the  tools  we 
believe can be utilized for this task in the Inception Report.   

For the data generated as a response to questions Q3 and Q3a, on Start-ups and Spin-offs, 
we  have  used  the  databases  as  proposed.  Open  Corporates,  Crunchbase  and  StartXplore 
were the main source for validation here.  In all we managed to validate 41 newly created 
companies  while  59  remained  un-validated.  Below  a  first  analysis  of the  validated  data  for 
this question can be found. 

6.2  Types and Geography of Newly Created Organizations 
In  the  following  sections  we  will  discuss  the  analysis  of  the  data  submitted  by  the 
respondents for the questions Q3 and Q3a of the survey. Question Q3 asks whether the “a 
new  organisation  (start-up,  spin-off,  other)  been  created  during  the  year  2014,  2015  or 
2016?” in the course of the project, while question Q3a asks which type of organisation the 
reported start-up or spin-off (newly created organization) is. 

We can make a sub-division in the types of organizations newly created:  

•  Corporate  spin-offs; former parts (can be in the form of intellectual property)  of an 
existing  company  that  then  become  independent  companies  with  their  own  assets 
(intellectual property) taken from the parent company. The parent company (e.g. its 
shareholders) completely owns or has a main share in the new company. 

•  University spin-offs; a company founded on the findings of a member or by members 
of a research group at a university, but clearly still part of this university or with the 
university being the main shareholder. 

•  Start-ups;  which  are  defined  as  newly  formed  companies,  with  one  or  more 
shareholders but not one distinguishable entity from which it has spun off, or which 
is the main shareholder. 

 
 
 
Final Study Report- D4 - SMART 2015/0078  98 

Below in Table 55 we can see the 41 newly formed companies, the type of organization they 
are assigned to and which are the different parent organizations. 

Table 55. Type of organization with parent. 

Typological division 

Corporate spin-off 
Bristol Energy 
Bristol Pound CIC 
etventure GmbH 
European Fashion Association 
Insero E-Mobility A/S 
Insero Energy A/S 
Solintel 
Synelixis Solutions 
TEDESYS 
Trilogis Srl 
VTT Technical Research Centre of Finland Ltd. 
Grupo Inclam & Eurecat 

University spin-off 

CERN, Unitar, University of Geneve 
Fraunhofer FOKUS Institute 
Lulea Technical University 
RWTH Aachen 
MODUL University 
Scuola Sup di Studi Universitari e di Perfezionamento Sant'Anna, Pisa 
Universite Paris Descartes 
University of Bologna 
University of Bordeaux 
University of Nottingham 
University of Verona 

Start-up 
Total 

11 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
12 
1 
1 
1 
1 
1 
1 
1 
2 
1 
1 
1 
17 
41 

From  Table  55  we  can  see  that  spin-offs  make  up  the  largest  part  of  the  newly  formed 
companies,  more  or  less  split  even  between  corporate  and  university  spin-offs.  Start-ups 
however follow closely behind. 

 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  99 

6.3  Geographical Division of Newly Created Organizations 
From  Table  56  below  we  can  see  that  Spain,  the  United  Kingdom,  Germany,  France  and 
Italy  are  the  favoured  countries  for  the  incorporation  of  the  newly  former  companies. 
Switzerland  and  Hong  Kong  are  the  only  non-EU  countries  where  companies  have  been 
created. 

Table 56. Geographical division. 

Geographical division 

Corporate spin-off 
Austria 
Denmark 
Finland 
France 
Germany 
Hong Kong China 
Italy 
Spain 
UK 
United Kingdom 
Start-up 
Austria 
Germany 
Greece 
Italy 
Portugal 
Spain 
United Kingdom 
University spin-off 
UK 
France 
Germany 
Italy 
Sweden 
Switzerland 
Grand Total 

13 
1 
2 
1 
1 
1 
1 
1 
2 
1 
2 
17 
1 
3 
1 
1 
2 
7 
2 
11 
1 
2 
2 
4 
1 
1 
41 

 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  100 

6.4  Division of Start-ups and Spin-offs across Funding Schemes 
Under  this  heading  we  explore  the  division  of  the  newly  created  companies  across  the 
different funding schemes, but also the type of organization is taken into this division. 

Table 57. Division of newly created companies across funding schemes. 

Funding scheme 

CIP-ICT-PSP-BPN 
CIP-ICT-PSP-PB 
CP-CSA 
CP-FP-INFSO 
CP-FP-INFSO-FET 
CP-IP 
CSA-SA 

Companies 

1 
3 
4 
17 
1 
14 
1 

Table 57 clearly shows that the small or medium-scale focused research projects (STREPs), 
denoted  by  CP-FP-INFSO  and  CP-FP-INFSO-FET  are  by  far  generating  most  of  the  new 
companies. 

Table 58. Division across funding scheme according to type of organization. 

Funding scheme 
Corporate spin-off 
CIP-ICT-PSP-BPN 
CIP-ICT-PSP-PB 
CP-CSA 
CP-FP-INFSO 
CP-IP 
Start-up 

CP-CSA 
CP-FP-INFSO 
CP-FP-INFSO-FET 
CP-IP 
CSA-SA 

University spin-off 
CP-FP-INFSO 
CP-IP 

Total 

Companies 
12 
1 
3 
1 
4 
3 
16 
3 
6 
1 
6 
1 
11 
7 
5 
41 

 
 
 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  101 

However when we further divide according to type of organization (Table 58 above) we see 
that this shows a much more nuanced, and scattered, picture. CIP sponsored projects only 
seem  to  turn  out  corporate  spin-offs,  while  the  aforementioned  STREPs  are  more  focussed 
on start-ups and university spin-offs. CP-IP is the only funding scheme that turns out new 
companies in all types, and in almost equal parts. 

6.5  Division  of  Start-ups  and  Spin-offs  across  Strategic 

Objectives 

Next we focus on the division of the new companies across each of the Strategic Objectives. 
Here it is interesting to see not necessarily the number of new companies but the type and 
the division of these types across Strategic Objectives. From  

From Table 59 below we find that Use case focused strategic objectives (SOs) “Expansion of 
Use  Cases”  and  “Use  Case  scenarios  and  early  trials”  generate  the  highest  number  of  new 
companies,  namely  8  combined.  However,  the  typological  division  for  these  use  case 
focused  SOs  is  less  clear,  with  4  Start-ups,  3  Corporate  spin-offs  and  only  one  University 
spin-off. 

Table 59. Division across Strategic Objectives according to type. 

Strategic Objective and type of newly created organization 

Aggregating content in Europeana 

Corporate spin-off 

Application experiments for robotics and simulation 

Start-up 

Brain-inspired ICT 

Start-up 

Cognitive Systems and Robotics 

University spin-off 
Computing Systems 

Start-up 

Core and disruptive photonic technologies 

Start-up 

Equipment assessment for sensor and laser based applications 

Start-up 

Expansion of Use Cases 
Corporate spin-off 
Start-up 

Future Networks 

University spin-off 

ICT for Ageing and Wellbeing 

1 
1 
1 
1 
1 
1 
2 
2 
1 
1 
2 
2 
1 
1 
4 
1 
3 
1 
1 
1 

 
 
 
 
Final Study Report- D4 - SMART 2015/0078  102 

University spin-off 
ICT for energy efficiency 

Start-up 
Corporate spin-off 

ICT for Environmental Management and Energy Efficiency 

Corporate spin-off 

ICT for efficient water resources management 

Corporate spin-off 

Networked Media and Search Systems 

Corporate spin-off 
Start-up 

New paradigms for embedded systems, monitoring and control 

Corporate spin-off 

Open data 

Corporate spin-off 

Open objective for innovation 

Corporate spin-off 

Personal Health Systems (PHS) 

University spin-off 
Scalable data analytics 

Start-up 

Smart components and smart systems integration 

University spin-off 
Smart Energy Grids 

Corporate spin-off 

Software Engineering, Services and Cloud Computing 

Start-up 

Technology-enhanced learning 

University spin-off 

Use Case scenarios and early trials 

Corporate spin-off 
Start-up 
University spin-off 

Virtual Physiological Human 

Corporate spin-off 

Total 

1 
3 
2 
1 
1 
1 
1 
1 
3 
1 
2 
1 
1 
1 
1 
1 
1 
3 
3 
1 
1 
1 
1 
1 
1 
1 
2 
2 
2 
4 
2 
1 
1 
1 
1 

41 

 
 
 
 
Final Study Report- D4 - SMART 2015/0078  103 

6.6  Conclusion of the Start-ups and Spin-offs Analysis 
DG CNECT have, with this new category, extended their ex post evaluation beyond the basic 
R&D indicators. The commercialization of basic and applied research is the next step in the 
commercialization  process,  moving  past  the  concept  testing  and  on  to  the  actual  business 
analytics  and  product  development.  Although  some  interesting  characteristics  have  been 
discovered  with  respect  to  the  funding  scheme  and  strategic  objective  productivity,  there 
are  numerous  questions  unanswered.  For  this  more  data  would  be  needed  on  for  instance 
investment (and division thereof), shareholders, employment etc. 

Final Study Report- D4 - SMART 2015/0078  104 

 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  105 

7  Keyword and Key-phrase Analysis 

7.1  Keyword-based association strength 
In this final chapter we focus on a non-citation based analysis; co-word analysis. Specifically 
we  use  Knowledge  Discovery  in  Databases  techniques  in  order  to  analyse  the  publication 
and patent outputs from the perspective of their underlying keywords and key phrases. As a 
starting  point  for  our  analysis  of  keywords  and  key  phrases,  we  use  the  “index  terms” 
provided by Scopus (and available for approximately 80% of the parsed publications). These 
index  terms  are  assigned  by  professional  indexers,  ensuring  therefore  a  high  indexing 
quality. As the terms are extracted from controlled vocabularies, uniformity and consistency 
is much greater than that of the keywords provided by the authors themselves.  Moreover, 
the indexing is based on a diverse set of Elsevier thesauri, well suited to cover ICT research 
as well as broader scientific topics. The controlled vocabulary used by Scopus for the index 
terms contains thesauri such as: 

EI thesaurus (engineering, technology, physical sciences),
Emtree medical terms (life sciences & health sciences),

•
•
• MeSH (life sciences & health science),
• Geobase  Subject  Index  (geology,  geography,  earth  and  environmental

science),
FLX terms, WTA terms (fluid sciences & textile sciences),

•
• Regional Index (geology, geography, earth and environmental science),
• Species Index (biology, life sciences).

For  our  analysis,  we  measure  the  degree  of  overlapping  national  and  regional  research 
interests  by  analysing  the  keywords  that  describe  the  publication  output.  We  start  by 
identifying  the  keywords  describing  the  publication  output  of  each  region,  in  a  full  count 
manner, and we construct a binary occurrence matrix, tracking on one side regions and on 
the other keywords, as well as a co-occurrence matrix, which tracks the keyword overlap of 
each pair of regions. We compute the association strength between regions, also known as 
the probabilistic affinity index, which normalises the observed number of co-occurrences by 
the  expected  number  of  co-occurrences  given  each  region’s  share  of  occurrences.  We 
normalise using a probability similarity measure as opposed to a set-theoretic measure, as 
it corrects for the size effect (van Eck and Waltman, 2009) which is typical for our dataset. 
The  association  strength  is  then  normalized  to  values  between  -1  and  1,  with    positive 
values  marking  an  affinity  greater  than  the  baseline  expected  value  given  random 
independent keyword occurrence, and negative values marking an affinity smaller than the 
baseline expected value given random independent keyword occurrence. 

7.2  Association strength versus control group 
Table 60 below shows the research interest affinity between the EU-28 focal sample and the 
various control samples from our study, as computed from keyword co-occurrence. As such, 

Final Study Report- D4 - SMART 2015/0078  106 

this  is  a  measure  of  overlapping  research  interests,  but  not  computed  based  on  explicit 
cooperation ties as was done throughout the study, but through indirect labels of interest. 

Table 60. EU-28 focal and control sample comparisons on topical overlap with BRICS and Thirds 
Countries. 

EU28 

c_EU28 

EU28 

c_EU28 

c_US 

c_JP 

c_KR 

c_BRICS 

0.362 

0.192 

0.072 

0.01 

0.204 

0.362 

0.336 

0.1 

0.003 

0.227 

c_US 

0.192 

0.336 

0.108 

0.1 

0.251 

c_JP 

0.072 

0.1 

0.108 

0.231 

0.078 

c_KR 

0.01 

0.003 

0.1 

0.231 

-0.007 

c_BRICS 

0.204 

0.227 

0.251 

0.078 

-0.007 

We  notice  that  the  EU-funded  output  has  a  positive  affinity  with  all  the  considered  control 
samples,  affinity  greater  than  the  baseline  expected  value  given  random  independent 
occurrence.  The  DG  CNECT  funded  output  has  its  greatest  affinity  with  the  EU-28  control 
group of non-EU-funded projects, which is to be expected. The focal sample also has a large 
positive  affinity  with  the  US  and  BRICS  control  samples,  although  lower  than  the  EU-28 
control  sample  has  with  these  samples.  Finally,  the  focal  sample  has  a  positive  affinity, 
albeit small, with the Japan and South Korea control samples.  

DG-funded  projects  have  a  high  topical  overlap  with  wider  EU  research,  as  is  to  be 
expected.  And  compared  with  the  control  EU  research,  DG-funded  research  features  a 
similarly  high  affinity  with  BRICS  research  and  similarly  low  affinity  with  Japan  and  South 
Korean  research.  However,  we  notice  that  DG-funded  research  features  a  significantly 
decreased topic overlap with US when compared with wider EU research control. 

7.3  Association strength over time 
We  split  the  above  analysis  of  EU  funded  projects  versus  the  control  sample  groups  into 
three intervals, as shown in Table 61, in order to ascertain possible changes over time. We 
observe a consistently significant affinity of the DG CNECT funded output with the wider EU-
28 control group of non-EU-funded publications, which is to be expected. We also observe a 
gradual affinity increase with the BRICS publication control over time, and a gradual affinity 
decrease with the US publication control over time. 

 
 
 
 
 
  
  
  
  
  
  
 
 
 
Final Study Report- D4 - SMART 2015/0078  107 

Table  61.  EU-28  focal  and  control  interval  comparisons  on  topical  overlap  with  BRICS  and  Thirds 
Countries.  

2007-2010 

EU28 

EU28 

c_EU28 

c_US 

c_JP 

c_KR 

c_BRICS 

0.375 

0.211 

0.087 

-0.033 

0.159 

2011-2013 

EU28 

EU28 

c_EU28 

c_US 

c_JP 

c_KR 

c_BRICS 

0.344 

0.206 

0.096 

-0.004 

0.217 

2014-2016 

EU28 

EU28 

c_EU28 

c_US 

c_JP 

c_KR 

c_BRICS 

0.371 

0.125 

-0.142 

-0.049 

0.225 

c_EU28 

0.375 

0.337 

0.07 

0.092 

0.252 

c_EU28 

0.344 

0.359 

0.145 

-0.051 

0.256 

c_EU28 

0.371 

0.331 

0.16 

0.047 

0.196 

c_US 

0.211 

0.337 

0.133 

0.096 

0.251 

c_US 

0.206 

0.359 

0.038 

0.216 

0.178 

c_US 

0.125 

0.331 

0.207 

-0.054 

0.336 

c_JP 

0.087 

0.07 

0.133 

0.391 

-0.013 

c_JP 

0.096 

0.145 

0.038 

0.082 

0.056 

c_JP 

-0.142 

0.16 

0.207 

0.106 

0.041 

c_KR 

-0.033 

0.092 

0.096 

0.391 

-0.288 

c_KR 

-0.004 

-0.051 

0.216 

0.082 

0.031 

c_KR 

-0.049 

0.047 

-0.054 

0.106 

0.123 

c_BRICS 

0.159 

0.252 

0.251 

-0.013 

-0.288 

c_BRICS 

0.217 

0.256 

0.178 

0.056 

0.031 

c_BRICS 

0.225 

0.196 

0.336 

0.041 

0.123 

7.4  Association strength within the focal set 
We can also split the same analysis by country. In Table 62, on the next page, we provide 
the  same  type  of  analysis,  but  inside  the  EU-28  focal  set,  showing  the  research  interest 
affinity of DG CNECT funded projects within EU-28 countries. We must note however that at 
lower  granularity  the  results  can  be  less  reliable,  as  scores  of  small  countries  with  little 
publication output can be heavily over- or under-estimated. 

 
 
 
  
  
  
  
  
  
 
 
 
 
 
 
 
  
  
  
  
  
  
 
 
 
 
 
 
 
  
  
  
  
  
  
 
 
 
Final Study Report- D4 - SMART 2015/0078  108 

2
6
0

.

8
5
0

.

3
3
0

.

4
5
0

.

6
2
0

.

3
8
0

.

.

8
0
-

4
1
0
-

.

7
4
0

.

8
0
0

.

5
0

.

7
3
0

.

4
3
0

.

8
5
0

.

5
0
0

.

1
3
0

.

2
5
0

.

3
0
0

.

2
0
0
-

.

3
0
0
-

.

5
1
0
-

.

1
0
0
-

.

7
1
0

.

6
4
0

.

8
5
0

.

8
5
0

.

5
1
0
-

.

5
1
0
-

.

1
0

.

4
0
0
-

.

7
2
0

.

3
3
0

.

6
0
0

.

2
0
0
-

.

8
0
0

.

1
0
0

.

4
2
0

.

4
1
0

.

3
0
0

.

1
4
0

.

1
4
0

.

4
2
0

.

6
1
0
-

.

2
1
0
-

.

2
0
0
-

.

1
5
0

.

2
1
0
-

.

3
3
0

.

K
S

I
S

E
S

1
5
0

.

6
1
0
-

.

5
0

.

O
R

1
1
0

.

7
2
0

.

4
1
0

.

2
1
0

.

1
0
0

.

5
0
0

.

5
1
0

.

6
4
0

.

1
7
0

.

7
0
0

.

3
0
0

.

4
1
0

.

6
0
0

.

7
2
0

.

8
0
0

.

T
P

3
4
0

.

9
1
0
-

.

8
9
0

.

3
0

.

3
0
0
-

.

3
7
0
-

.

4
2
0

.

2
0
0
-

.

1
1
0
-

.

5
6
0

.

7
2
0

.

7
2
0

.

0

8
2
0
-

.

6
0
0
-

.

4
0
0

.

7
4
0

.

6
3
0

.

4
3
0

.

1
1
0
-

.

1
5
0

.

6
3
0

.

1
2
0

.

2
5
0
-

.

8
1
0
-

.

4
1
0
-

.

9
0
0
-

.

4
1
0

.

8
3
0
-

.

5
1
0
-

.

1
2
0
-

.

1
5
0

.

6
3
0

.

5
4
0

.

6
0
0

.

7
1
0
-

.

3
0
0

.

4
2
0

.

4
0
0
-

.

1
0

.

6
1
0
-

.

8
2
0
-

.

7
1
0
-

.

7
0
0

.

1
0
0

.

8
0
0

.

3
0
0

.

1
-

6
0
0

.

1
7
0

.

5
1
0
-

.

5
1
0
-

.

2
5
0

.

7
4
0

.

4
1
0
-

.

.

8
0
-

L
P

L
N

T
M

8
1
0
-

.

2
3
0
-

.

3
1
0
-

.

4
5
0
-

.

1
6
0

.

6
7
0

.

5
4
0

.

6
4
0

.

8
5
0

.

7
1
0

.

1
3
0

.

3
8
0

.

V
L

8
2
0
-

.

6
1
0
-

.

3
1
0
-

.

8
1
0
-

.

1
-

4
5
0
-

.

2
3
0
-

.

4
0
0

.

6
7
0

.

4
0
0

.

1
6
0

.

6
3
0

.

5
1
0

.

8
5
0

.

1
5
0

.

5
0
0

.

6
4
0

.

3
2
0

.

4
2
0

.

9
0
0

.

1
2
0
-

.

2
1
0
-

.

.

1
0
-

5
2
0
-

.

9
1
0
-

.

6
3
0
-

.

5
0
0

.

3
1
0

.

3
1
0
-

.

1
0
0

.

3
1
0
-

.

1
0
0
-

.

5
1
0
-

.

4
0
0

.

3
0
0
-

.

5
0
0

.

6
2
0

.

U
L

8
5
0

.

4
5
0

.

T
L

3
0
0
-

.

2
0
0
-

.

5
2
0
-

.

3
3
0

.

T
I

E
I

4
0
0

.

4
0
0

.

8
0
0

.

1
1
0

.

2
1
0
-

.

4
0
0
-

.

1
5
0

.

1
2
0

.

4
3
0

.

7
2
0

.

6
3
0

.

6
3
0

.

7
2
0

.

1
1
0
-

.

3
7
0
-

.

2
0
0
-

.

3
0
0
-

.

4
2
0

.

2
1
0

.

4
2
0

.

3
0

.

4
1
0

.

1
4
0

.

1
0
0

.

4
3
0

.

8
5
0

.

U
H

Table 62. Intra-EU-28 topical overlap. 

K
S

I
S

1
1
0

.

7
0
0

.

3
1
0
-

.

8
6
0

.

2
0

.

7
0

.

0

6
3
0

.

1
2
0

.

4
3
0

.

3
2
0
-

.

4
1
0
-

.

7
2
0

.

8
0
0
-

.

5
1
0

.

9
1
0

.

6
1
0
-

.

1
1
0
-

.

4
1
0

.

3
0
0

.

6
2
0
-

.

6
1
0
-

.

2
2
0
-

.

9
0
0
-

.

1
1
0
-

.

1
0
0
-

.

E
S

7
0
0

.

6
0
0

.

6
2
0
-

.

7
0
0
-

.

6
0
0

.

8
0
0

.

9
0
0

.

4
0
0
-

.

8
0
0

.

2
2
0

.

8
0
0

.

4
0
0

.

4
0
0

.

4
1
0

.

1
0
0

.

O
R

7
0
0

.

T
P

0

9
0
0
-

.

3
0
0
-

.

3
6
0

.

5
3
0

.

8
2
0

.

1
2
0

.

6
2
0

.

5
1
0

.

L
P

2
0
0

.

3
1
0

.

1
3
0

.

3
4
0

.

9
2
0

.

.

2
0
-

7
0
0
-

.

7
0
0
-

.

6
0
0

.

1
5
0

.

4
1
0
-

.

6
1
0

.

1
0
0

.

2
4
0

.

3
0
0

.

6
0
0

.

1
0

.

1
0

.

3
0
0

.

5
0
0

.

1
1
0
-

.

2
0
0
-

.

6
0
0
-

.

9
0
0
-

.

1
0
0
-

.

7
0
0
-

.

L
N

1
0
0

.

6
1
0

.

5
2
0
-

.

1
1
0
-

.

0

7
1
0

.

7
0
0

.

7
0
0

.

1
1
0

.

2
0
0

.

9
5
0

.

7
0
0

.

.

4
0
-

.

1
0
-

6
3
0
-

.

T
M

V
L

U
L

8
4
0

.

8
0
0

.

3
0
0

.

1
4
0
-

.

8
1
0
-

.

2
0
0
-

.

7
7
0

.

4
0

.

1
4
0

.

8
4
0

.

3
7
0

.

6
1
0

.

T
L

8
0
0

.

6
0
0

.

7
7
0

.

9
4
0

.

9
1
0

.

T
I

5
0
0

.

5
0
0

.

4
3
0
-

.

5
2
0
-

.

6
1
0
-

.

6
3
0

.

1
3
0

.

7
0
0

.

5
6
0

.

5
0
0

.

4
7
0

.

E
I

4
0
0

.

4
0
0

.

6
2
0

.

8
1
0

.

3
0
0

.

2
0
0

.

U
H

1
0
0

.

3
0
0

.

1
5
0

.

4
2
0

.

9
1
0

.

4
7

.

0

2
2

.

0

5
2

.

0
-

4

.

0
-

3
2

.

0

8
1

.

0
-

7
7

.

0

3
0

.

0
-

7
2

.

0
-

7
4

.

0

2
6

.

0

2
0

.

0

R
H

R
G

3
0

.

0

1
0

.

0

1
1

.

0
-

3
0

.

0

B
G

5
0

.

0

2
0

.

0

R
F

2
0

.

0

3
1

.

0

I
F

0

0

5
0

.

0

3
0

.

0
-

7
0

.

0

6
0

.

0

3
0

.

0
-

7
0

.

0

2
0

.

0

1
0

.

0

1
0

.

0
-

1
1

.

0

8
1

.

0

7
0

.

0

5
0

.

0
-

9
0

.

0

4
0

.

0

S
E

E
E

K
D

E
D

Z
C

Y
C

G
B

E
B

T
A

5

.

0

8
1

.

0

3
0

.

0
-

2
2

.

0
-

3
2

.

0

5
1

.

0
-

9
6

.

0

3
1

.

0

6
2

.

0
-

7
2

.

0

2
6

.

0

7
0

.

0

5
0

.

0
-

1
5

.

0

8
0

.

0
-

1

.

0
-

6
0

.

0
-

1
1

.

0

5
0

.

0
-

7
3

.

0

3
2

.

0

3
1

.

0
-

7
2

.

0

7
4

.

0

3
0

.

0
-

7
0

.

0

1
0
0
-

.

9
2
0
-

.

1
0

.

3
2
0

.

9
1
0

.

6
0
0
-

.

8
2
0

.

5
1
0

.

1
1

.

0

3
0

.

0

1
0

.

0
-

3
0

.

0
-

4
0

.

0
-

8
3

.

0

6
0

.

0

3
0

.

0
-

1
1

.

0

3
2

.

0

3
2

.

0

0

7
0
0
-

.

4
7
0
-

.

2
2
0
-

.

3
2
0

.

7
4
0

.

4
6

.

0

3
0

.

0
-

6
2

.

0
-

7
2

.

0
-

8
3

.

0

2
2

.

0
-

8
0

.

0

7
2

.

0
-

7
3

.

0

9
6

.

0

7
7

.

0

3
0

.

0
-

1
0

.

0
-

7
0
0

.

4
4
0
-

.

1
3
0
-

.

4
1
0
-

.

1
1
0
-

.

2
2
0

.

6
0
0
-

.

6
0
0
-

.

5
1

.

0
-

9
0

.

0

8
1

.

0

8
1

.

0

4
0

.

0
-

2
2

.

0
-

3
0

.

0
-

1
2

.

0

5
0

.

0
-

5
1

.

0
-

8
1

.

0
-

5
0

.

0

1
0

.

0

.

4
0
-

4
3
0
-

.

9
1
0
-

.

4
2
0

.

1
1
0
-

.

6
2

.

0
-

1
1

.

0

4
2

.

0

2

.

0

3
0

.

0
-

1
2

.

0

7
2

.

0
-

2
0

.

0
-

3
1

.

0
-

6
2

.

0
-

7
2

.

0
-

6
0

.

0

2
0
0
-

.

5
0
0
-

.

1
0

.

1
0

.

0

0

1
0

.

0
-

3
0

.

0

6
0

.

0

3
0

.

0
-

8
0

.

0

2
0

.

0
-

3
2

.

0

3
1

.

0

3
0

.

0
-

7
0

.

0

7
0
0
-

.

3
1
0
-

.

3
2

.

0
-

5
0

.

0

4
1

.

0

3
0

.

0
-

8
1

.

0

7
2

.

0
-

3
0

.

0

2

.

0

6
0

.

0
-

2
2

.

0
-

4

.

0
-

3
1

.

0

1
1
0

.

8
0

.

0

4
0

.

0

4
0

.

0

7
0

.

0
-

8
2

.

0

6
0

.

0
-

3
2

.

0

5
0

.

0
-

2
0

.

0

3
0

.

0

1
5

.

0

4
0

.

0
-

2
1

.

0
-

3
1

.

0
-

5
1

.

0

6
0

.

0
-

7
4

.

0

1

.

0

1
1

.

0
-

9
1

.

0

4
2

.

0

8
1

.

0

1
5

.

0

6
2

.

0

3
0

.

0

4
0

.

0

4
2

.

0
-

2
1

.

0

4
1

.

0

1
0

.

0
-

8
1

.

0

6
2

.

0
-

1
0

.

0
-

4
2

.

0

1

.

0
-

3
0

.

0
-

5
2

.

0
-

2
0

.

0

2
0

.

0

2
1

.

0

5
0

.

0

3
0

.

0

9
0

.

0

3
0

.

0
-

0

1
1

.

0

8
0

.

0
-

8
1

.

0

2
2

.

0

3
0

.

0

2
0

.

0

4
2

.

0
-

3
2

.

0
-

1
1

.

0

5
1

.

0
-

4
6

.

0

1
0

.

0

6
2

.

0
-

1
5

.

0

5

.

0

4
7

.

0

1
1

.

0
-

3
0

.

0

5
2
0
-

.

3
0
0
-

.

4
0
0

.

3
1
0
-

.

3
1
0
-

.

3
1
0

.

5
0
0

.

6
3
0
-

.

9
1
0
-

.

5
2
0
-

.

.

1
0
-

2
1
0
-

.

1
2

.

0
-

9
0

.

0

4
2

.

0

3
2

.

0

6
0

.

0
-

2
2

.

0

2
2

.

0
-

2
0

.

0
-

4
2

.

0

6
1

.

0
-

5
2

.

0
-

4
3

.

0
-

5
0

.

0

1
5

.

0

1
1

.

0
-

1
2

.

0
-

9
0

.

0
-

9
1

.

0

1
1

.

0
-

4
7

.

0

5
0

.

0

9
1

.

0
-

9
1

.

0

7
4

.

0

4
0

.

0

5
1

.

0
-

4
1

.

0
-

3
2

.

0

4
1

.

0
-

5
6

.

0

7
0

.

0

4
3

.

0
-

6
1

.

0

9
4

.

0

3
7

.

0

7
7

.

0

6
0

.

0

8
4

.

0

2
0

.

0
-

3
0

.

0

5
6

.

0

6
0

.

0
-

8
3

.

0
-

8
1

.

0
-

1

.

0

1
3

.

0
-

1
3

.

0

6
3

.

0

8
9

.

0

8
2

.

0
-

4
1

.

0

2
5

.

0
-

9
2

.

0
-

4
4

.

0
-

4
7

.

0
-

6
3

.

0
-

4

.

0
-

1

.

0
-

1
4

.

0

4

.

0

7
7

.

0

8
1

.

0
-

8
0

.

0

4

.

0
-

7
0

.

0

9
5

.

0

1
4

.

0
-

8
4

.

0

9
1

.

0
-

2
0

.

0

1
1

.

0

7
0

.

0

1
0

.

0
-

7
0

.

0

7
0

.

0
-

7
0

.

0

7
1

.

0

0

1
1

.

0
-

5
2

.

0
-

6
1

.

0

3
4

.

0

7
2

.

0

1
4

.

0

4
1

.

0

4
0

.

0

4
0

.

0

8
0

.

0

2
2

.

0

8
0

.

0

4
0

.

0
-

9
0

.

0

8
0

.

0

6
0

.

0

7
0

.

0
-

6
2

.

0
-

6
0

.

0

0

7
0

.

0
-

6
0

.

0
-

5
0

.

0

1
1

.

0

1
0

.

0
-

2
0

.

0
-

6
0

.

0

3
0

.

0

3
0

.

0

1

.

0

1

.

0

7
0

.

0
-

9
2

.

0

2
4

.

0

1
0

.

0

7
0

.

0
-

5
1

.

0

3
0

.

0

9
0

.

0
-

1
1

.

0
-

6
1

.

0

4
1

.

0
-

1
5

.

0

6
0

.

0

2

.

0
-

8
2

.

0

3
4

.

0

6
2

.

0

5
3

.

0

1
3

.

0

3
1

.

0

1
2

.

0

3
0

.

0
-

0

3
6

.

0

9
0

.

0
-

7
0

.

0

7
3

.

0

1
0

.

0
-

9
0

.

0
-

6
1

.

0
-

3
0

.

0

1
1

.

0
-

9
1

.

0

5
1

.

0

4
1

.

0
-

4
3

.

0

1
2

.

0

6
3

.

0

0

2
6

.

0

1
1

.

0
-

2
2

.

0
-

6
2

.

0
-

4
1

.

0

6
1

.

0
-

8
0

.

0
-

7
2

.

0

3
2

.

0
-

R
H

R
G

B
G

R
F

I
F

S
E

E
E

K
D

E
D

7

.

0

Z
C

2

.

0

Y
C

8
6

.

0

3
1

.

0
-

1
1

.

0

G
B

E
B

T
A

4
0

.

0

9
0

.

0

8
1

.

0

1
1

.

0

0

2
0

.

0

5
0

.

0

1
0

.

0

1
0

.

0

4
0

.

0

5
0

.

0

8
0

.

0

1
0

.

0

2
0

.

0

7
0

.

0

7
0

.

0

T
A

E
B

G
B

Y
C

Z
C

E
D

K
D

E
E

S
E

I
F

R
F

B
G

R
G

R
H

U
H

E
I

T
I

T
L

U
L

V
L

T
M

L
N

L
P

T
P

O
R

E
S

I
S

K
S

 
 
 
 
Final Study Report- D4 - SMART 2015/0078  109 

Conclusions 

In  this  report  we  have  carried  out  an  in  depth  examination  of  the  patent  and  publication 
data reported in the annual survey waves conducted by DG CNECT from 2008 to 2017, for 
the FP7 productive years 2007-2016. We began by validating the raw publication and patent 
data  by  first  matching  and  validating  them  with  data  in  independent  bibliographic 
databases. For publications, this was Elsevier's Scopus database and a Scopus Custom Data 
set,  while  for  patents,  Google  Patent  search,  the  EPO’s  PATSTAT  database  and  Espacenet 
were used. For patents, we  retained after validating only a single patent per patent family 
and only those patents with a priority year greater than or equal to 2007. For these patents, 
we  collected  additional  information  (from  the  PATSTAT  database  and  Espacenet),  such  as 
the  location  of  inventors,  applicants,  and  IPC  classes  which  were  used  in  the  subsequent 
analysis. For the reported publications we validated using the DOI in the first instance and 
the  author-title  pair  in  the  second  instance.  We  retained  all  publications  that  were  either 
articles (from journals), proceedings (from conference proceedings) or other peer-reviewed 
publications output (books, book chapters etc.) and which had a validated publication year 
greater  than  or  equal  to  2007.  Additional  bibliographic  information  was  generated  using 
text-mining  software.  For  both  the  patents  and  publications  the  data  was  further  enriched 
using  a  list  of  projects  and  participants  provided  by  DG  CNECT  and,  where  missing,  data 
from CORDIS. 

Publication Analysis 
A sizeable amount of publications were validated and cleaned, which provided a robust basis 
for  further  evaluation.  The  first  aspect  highlighted  in  the  bibliometric  analysis  was  the 
productivity related descriptive indicators. Here, we found that each project generates 9.12 
publications on average, although one project produced 3714 publications! We further found 
that,  of  the  funding  instruments,  NoEs  are  the  most  productive  with  an  average  of  144.8 
publications per project. The strategic objective “Computing Systems” generates the highest 
number of publications, with 4382 publications in total. Although there are only 2 projects in 
this strategic objective “Distributed Computing Infrastructure (DCI)” is the most productive 
in absolute terms, generating 204 publications per project. Second is “Computing Systems” 
(28  projects)  with  156.5  publications  per  project.  Looking  beyond  these  descriptive 
indicators to the geographical and organizational distribution, we found that for the full set 
of  validated  publications,  as  well  as  for  the  focal  sample,  Germany  leads  Italy,  the  United 
Kingdom,  Spain,  and  France  in  terms  of  publication  output  within  the  EU-28,  while  in  the 
control sample excluding non EU-28 countries Germany leads the UK and France, Italy and 
Spain.  For  both  the  focal  and  control  samples  we  find  that,  in  terms  of  participating 
organisations, the vast bulk of output is produced by academic institutions. 

 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  110 

Next,  we  undertook  a  citation  analysis  to  assess  the  impact  of  the  research  funded  under 
FP7-ICT. We find that, for the total stratified sample, the relative citation rate is well above 
the  world  average  for  the  years  examined.  From  the  citation  analyses  on  a  more 
disaggregate level, we furthermore find that the front line research published in conference 
proceedings is of even higher relevance and quality. 

The  last  aspect  to  be  examined  was  the  cooperation  behaviour  between  the  different 
countries engaged in (FP7-) ICT research, as well as the diversity of nationalities involved in 
FP7  funded  projects.  Within  this  aspect,  our  evaluation  discovered,  through  the  use  of 
network analytical methods, that Italy and the United Kingdom are the principal knowledge 
brokers within the EU-28 group of countries, while Germany is the most favoured country to 
cooperate  with.  Austria  and  Poland  seem  to  play  important  gatekeeping  roles  for  the 
terms  of 
diffusion  of  knowledge  between  Eastern  and  Western  Europe. 
internationalization  of  research,  we  find  that  the  cooperation  among  the  EU-28  countries 
engaged in ICT research under FP7 is more evenly distributed than for the control sample. 
We  can  infer  that  the  project  participants  display  a  significantly  higher  degree  of 
internationalization  than  their  control  sample  counterparts,  whom  are  rather  nationally 
focused.  Finally,  we  calculate  the  national  diversity  of  the  countries  receiving  funding 
through  the  different  funding  schemes.  Here  we  find  that  a  staggering  113  different 
nationalities have received funding through the CSA-SA funding instrument.  

In 

Patent Analysis 
In general, only a small proportion of projects (181) reported patenting activity, and most 
of  those  reported  applied  fewer  than  5  patents  for  the  whole  period.  We  began  by  a 
descriptive analysis of the focal patents. We noted that more than 90% of the patents were 
accounted  for  by  two  funding  instruments,  namely  CP-FP-INFSO  and  CP-IP.  We  further 
explored  the  distribution  of  patents  across  different  strategic  objectives  and  found  that 
about  one  fifth  of  the  patents  resulted  from  projects  with  the  strategic  objective  of  “The 
Network of the Future”. In relative terms “Manufacturing solutions for new ICT products” is 
the  best  performing,  while  “Next-Generation  Nano-electronics  Components  and  Electronics 
Integration”  is  the  most  consistent  in  absolute  and  relative  terms.  The  vast  number  of 
remaining strategic objectives accounted for only small shares of the total focal patents.  

With respect to impact and quality of the patenting activity of FP7 ICT projects, we find that 
from  the  focal  and  control  comparison  it  becomes  evident  that  on  the  whole  the  FP7 
sponsored patents are of higher than average quality.   

In order to evaluate the effectiveness of FP7 funding program in ICT, we compared the focal 
patents  with  two  randomly  selected  control  samples—one  representing  the  ICT  industry  in 
general, and the other the Europe-based ICT industry. We found that the focal patents were 
associated  with  very  high  inter-organizational  collaboration  —  much  more  than  what  is 
observed  in  both  the  industry  in  general  and  specifically  EU-based  industry.  While  most  of 
the control patents originated in a single organization, outside of EU in the general industry 
sample and within EU in the EU-based sample, inter-organizational collaborations within an 

 
 
 
Final Study Report- D4 - SMART 2015/0078  111 

EU country, between EU countries, and between EU, Japan or the US were important for the 
focal patents. 

We  further  found  that  the  focal  patents  display  a  unique  EU  bias  also  in  regard  to  the 
location  of  both  applicants  and  inventors.  Although  a  couple  of  EU  countries  dominate  in 
both of these dimensions, a broad spectrum of EU countries is represented among inventors 
and  applicants.  This  is  in  sharp  contrast  to  what  we  observe  in  the  control  sample,  where 
only  a  few  large  EU  economies  such  as  Germany  and  France  have  a  noticeable  presence, 
while small EU economies are barely visible. 

Econometric Analysis 
We then carried out an econometric analysis on a sample of 1285 observations at the level 
of  project  in  order  to  gain  more  fine  grained  insights  on  the  factors  contributing  to  the 
scientific  and  technological  performance  of  the  FP7  projects.  We  carried  out  separate 
analysis for publications and patents. In addition to using project-related variables such as 
project duration, funding, number of participants involved, the type of financing instrument, 
etc., we introduced a novel measure to capture the diversity in the scientific capabilities at 
the national level of organizations involved in each project. Recent studies have highlighted 
the importance of a geographically diverse knowledge sourcing strategy that would enhance 
the  possibilities  for  knowledge  recombination—a  vital  ingredient  to  innovation.  In  line  with 
this  view,  our  analysis  of  the  publication  productivity  revealed  that  scientific  diversity  of 
project  teams  played  a  significant  role in  enhancing  their  scientific  productivity.  Consistent 
with  our  expectations  we  found  evidence  on  the  positive  contribution  of  funding  duration, 
total  funding  and  number  of  patents,  as  well  as  the  number  of  participants  involved  in  a 
project  and  funding  per  participant,  for  publication  productivity.  We  further  found  that  the 
patent output of projects has a complementary effect on their publication output, and that 
projects  with  a  greater  share  of  university  partners  tend  to  perform  better.  Naturally, 
projects  that  receive  more  funding  and  that  receive  sustained  financial  support  tend  to 
outperform  those  that  receive  lower  financial  support  and  sustained  assistance.  Projects 
with high patenting activity tend also to publish more, indicating the complementary nature 
of  the  two.  Interestingly,  a  negative  coefficient  for  the  period  dummy  suggests  a  lower 
degree of publishing during the period following the financial crisis. 

For  the  econometric  model  with  funding  instrument  dummies  as  additional  variables  we 
observe that the significant effect of all variables remains intact, with total funding turning 
out to have a significant effect in this model. The results on the funding instrument suggest 
that compared to the baseline categories such as CP, CSA and, in particular, NoEs seem to 
generate a significant performance enhancing effect.  

When  looking  at  the  results  of  the  analysis  based  on  the  citations  per  publication,  we  find 
that patenting activity does not have a significant effect anymore.  Funding and the number 
of  participants  have  a  negative  effect  on  the  quality  of  publications.  One  interpretation  of 
the latter results seems to be that projects that are large may have a focus on scale due to 
the possibility of a higher pressure to publish in large scale projects. Smaller projects on the 

 
 
 
Final Study Report- D4 - SMART 2015/0078  112 

other hand may face less publication pressure, allowing them to focus more on quality. More 
research is needed to further explore the context behind these results. 

We replicate the analysis of publications above for patents.  It must be noted that whereas 
we  had  a  sizeable  number  of  observations  in  the  analysis  of  publications  with  positive 
outcomes  (over  1280),  in  the  patent  sample  we  only  have  a  small  number  of  about  145 
projects  with  positive  outcomes.  We  find  that  total  funding  shows  a  significant  positive 
impact consistently in all models which are further positively shaped by funding per person, 
publication output, and a high share of industry participants. Of course, patenting is costly 
and projects that are well funded are therefore likely to be able to meet the costs associated 
with patent filing. We find a complementary relationship between publishing and patenting, 
although not consistently across models.  

In  sum,  our  analyses  revealed  a  significantly  high  internationalization  of  the  inventive 
activities  in  FP7  projects  in  the  field  of  ICT.  This  is  in  sharp  contrast  to  the  general,  non-
collaborative, trend in the industry globally, and in the EU-based industry. Our econometric 
analysis  confirmed  that  the  diversity  of  the  project  team  in  terms  of  the  scientific 
capabilities  of  the  countries  in  which  they  are  embedded  plays  a  critical  role  in  enhancing 
the productivity of the projects. 

Newly Created Organizations 
For  the  survey  waves  for  2014-2016  DG  CNECT  have  extended  their  ex  post  evaluation 
beyond the basic R&D indicators. The commercialization of basic and applied research is the 
next step in the commercialization process, moving past the concept testing and patenting 
and  on  to  the  actual  business  and  product  development.  Start-ups,  Corporate  and 
University Spin-offs are the primary results of this activity and respondents are asked about 
the development of such activities emanating from the sponsored projects.  

For  the  three  survey  waves  we  positively  identified  and  validated  41  newly  created 
organizations. These are more or less evenly divided across the three different types (Start-
up, Corporate Spin-off and University Spin-off). We also find that very similar to patenting 
the main funding schemes used are CP-FP-INFSO and CP-IP.  

Keyword and key-phrase Analysis 
The keyword and key-phrase analysis presented at the end of the study was provided as a 
novel  way  of  analysing  DG  CNECT  funded  project  output.  We  measured  the  degree  of 
overlapping  national  and  regional  research  interests  by  analysing  the  keywords  that 
describe  the  publication  output.  This  was  presented  as  a  measure  of  overlapping  research 
interests, but not computed based on explicit cooperation ties, but through indirect labels of 
interest. 

We  noted  that  the  DG  CNECT  funded  output  has  a  positive  affinity  with  all  the  considered 
control sample groups, with affinity greater than the baseline expected value given random 
independent occurrence. The DG CNECT funded output has greatest affinity with the EU-28 

Final Study Report- D4 - SMART 2015/0078  113 

control group of non-DG CNECT-funded projects, which is to be expected. The focal sample 
also  has  a  large  positive  affinity  with  the  US  and  BRICS  control  sample  groups,  although 
lower than the EU-28 control has with these groups. Finally, the focal sample has a positive 
affinity, albeit small, with the Japan and South Korea control samples. 

Over time we notice a decrease in affinity with US ICT research which is not in line with the 
affinity  for  the  control  which  remains  largely  the  same  for  the  US.  Although  less 
pronounced,  this  also  seems  to  be  the  case  for  South  Korean  and  Japanese  ICT  research. 
Conversely, the affinity with the BRICS countries increases over time. 

Horizon 2020 
ICT, as an enabling technology, has an important impact across EU society and its economy. 
As we have  shown in this report the research  and development activities in ICT funded by 
FP7 support the goals of the ERA and are mutually reinforcing public and private efforts in 
the  broader  ICT  field.  These  efforts  should  be  further  supported  under  the  Horizon  2020 
(H2020)  framework  programme  for  research  and  innovation.  The  EU  has  successfully 
tackled  some  of  the  earlier  noticed  discrepancies  with  the  wider  ICT  R&D,  such  as  further 
inclusion  of  SMEs  in  funded  projects,  while  also  creating  a  framework  within  which  a  high 
diversity  of  organizations  and  countries  are  invited  to  cooperate.  Our  analysis  has  shown 
that these collaborations, resting on diversity, are successful and mutually reinforcing. 

The further inclusion of small and medium sized partners from industry into the sponsored 
projects  would  strengthen  the  “Industrial  Leadership”  goals  from  Horizon  2020,  while 
attracting  large  industrial  corporations  as  main  partners  would  both  enable  access  to 
additional funding and to the much desired patenting capacities.  

The EU has also been shown to own a large and high quality knowledge pool in the field of 
ICT which, stimulated by the high degree of internationalization, should grow further in the 
years  to  come.  Attracting  and  educating  more  knowledge  workers  will  solidify  the  EU 
leadership position in ICT research. A further step could be engaging in more, and perhaps 
less regulated, collaborations with new research partners outside of the EU. As shown in this 
report, there seems to be room for more cooperation with BRICS countries as well as with 
countries  such  as  Japan  and  South  Korea.  It  is  inevitable  that  knowledge  exchanges  with 
these countries will lead to further innovations and a strengthening of the EU’s “Excellence 
in Science”–an H2020 goal. 

As already mentioned before, ICT has had and still has a tremendous impact on our society. 
The topics tackled by DG CNECT, and visible in the Strategic Objectives, do not only satisfy 
the larger need for advances in core science but also for  ICT advances that can be readily 
applied  to  tackle  societal  problems.  In  this  respect,  health  and  aging,  as  well  as 
environment  and  mobility,  are  pertinent  issues  within  society  for  which  DG  CNECT  has 
successfully funded ICT R&D projects under FP7. However our research also shows that we 
should reach out to extra-EU partners facing similar issues, such as an aging population or 
rapid urbanization, and try to derive common solutions to these challenges. 

Final Study Report- D4 - SMART 2015/0078  114 

Final Study Report- D4 - SMART 2015/0078  115 

References 

Ahuja,  G.  (2000).  “Collaboration  networks,  structural  holes,  and  innovation:  A  longitudinal 
study”, Administrative Science Quarterly, Vol. 45(3): pp. 425-455 

Bonacich,  P.  (1987).  "Power  and  Centrality:  A  Family  of  Measures",  American  Journal  of 
Sociology, Vol. 92: pp. 1170-1182 

Braun,  T.,  W.  Glänzel,  and  A.  Schubert  (1985).  Scientometric  Indicators:  A  32  Country 
Comparison of Publication Productivity and Citation Impact. Singapore: World Scientific.  

Breschi, S. (2010).  “Using performance indicators in monitoring the implementation of  ICT 
research  in  FP6  and  FP7.”,  Milan: KITeS-CESPRI  –  Boccuni  University.  Report  prepared  for 
DG INFSO. 

Breschi, S., and F. Malerba (2011). “Assessing the scientific and technological output of EU 
Framework  Programmes:  evidence  from  the  FP6  projects  in  the  ICT  field”,  Scientometrics, 
Vol. 88: pp. 239–257 

Cook, K., R. Emerson, M. Gillmore, and T. Yamagishi (1983). "The Distribution of Power in 
Exchange Networks: Theory and Experimental Results", American Journal of Sociology, Vol. 
89(2): pp. 275 - 305 

Elsevier  B.V. 
http://info.sciencedirect.com/scopus/scopus-in-detail/content-coverage-guide/metadata 

“Coverage  of  metadata”.  Retrieved 

from  ScienceDirect: 

(2014). 

Freeman,  C.  (1991).  “Networks  of  innovators:  A  synthesis  of  research  issues”,  Research 
Policy, Vol. 21(5): pp. 499-514 

Freeman,  L.  (1979).  "Centrality  in  Social  Networks:  Conceptual  Clarification",  Social 
Networks, Vol. 1; pp. 215-239 

Glänzel, W., B. Schlemmer, A. Schubert, and B. Thijs (2006). “Proceedings literature as an 
additional data source for bibliometric analysis.” Scientometrics, Vol. 68: pp. 457–473 

Granovetter,  M.  (1973).  “The  Strength  of  Weak  Ties”.  American  Journal  of  Sociology,  Vol. 
78(6): pp. 1360-1380  

Grupp,  H.  (1998).  Foundations  of  the  Economics  of  Innovation:  Theory,  Measurement  and 
Practice, Houndmills: Edward Elgar. 

Guellec,  D.,  and  B.  van  Pottelsberghe  de  la  Potterie  (2001).  “The  internationalisation  of 
technology analysed with patent data”, Research Policy, Vol. 30(8): pp. 1253-1266 

Knoke, D. and S. Yang (2008). Social Network Analysis, 2nd ed., London: Sage 

 
 
 
Final Study Report- D4 - SMART 2015/0078  116 

Lahiri,  N.  (2010).  “Geographic  Distribution  of  R&D  Activity:  How  Does  It  Affect  Innovation 
Quality?” Academy of Management Journal, Vol.53(5): pp. 1194-1209 

Laudel,  G.,  and  G.  Origgi  (2006).  “Introduction  to  a  special  issue  on  the  assessment  of 
interdisciplinary research.” Research Evaluation, Vol. 15(1): pp. 2-4 

Leiponen  A.,  and  C.  Helfat  (2011).  “Location,  Decentralization,  and  Knowledge  Sources  for 
Innovation.”, Organization Science, Vol. 22: pp. 641-658 

Leydesdorff,  L.  and  C.  Probst  (2009).  “The  delineation  of  an  interdisciplinary  specialty  in 
terms of a journal set: The case of communication studies.” Journal of the American Society 
for Information Science and Technology, Vol. 60(8): pp. 1709-1718 

Lisee, C., V. Lariviere, and E. Archambault (2008). “Conference proceedings as a source of 
scientific  information:  A  bibliometric  analysis”.  Journal  of  the  American  Society  for 
Information Science and Technology, Vol. 59(11): pp. 1776-1784 

Montesi,  M.,  and  J.  MackenzieOwen  (2008).”From  conference  to  journal  publication:  How 
conference papers in software engineering are extended for publication in journals.” Journal 
of the American Society for Information Science and Technology, Vol. 59(5): pp. 816–829 

Narin,  F.  (1995).  “Patents  as  indicators  for  the  evaluation  of  industrial  research  output”, 
Scientometrics, Vol. 34(3), pp: 489-496 

Nieto,  M.,  and  A.  Rodriguez  (2011).  "Offshoring  of  R&D:  Looking  abroad  to  improve 
innovation performance." Journal of International Business Studies, Vol. 42(3): pp. 345-361 

Notten,  A.,  and  S.  Ramani  (2014).  “Co-patenting  networks  in  nanotechnology:  A 
comparison  of  South  Korea  and  Germany”.  In:  Ramani,  S.  (ed.)  Nanotechnology  and 
Development:  What’s  in  it  for  Emerging  Countries?,  Cambridge:  Cambridge  University 
Press. 

Schubert, A., and T. Braun (1986). “Relative indicators and relational charts for comparative 
assessment of publication output and citation impact”. Scientometrics, Vol. 9: pp. 281-291 

Van Eck, N. J., and L. Waltman (2009) “How to Normalize Co-Occurrence Data? An Analysis 
of  Some  Well-Known  Similarity  Measures”.  In:  ERIM  Report  Series:  Research  in 
Management, ERS-2009-001-LIS 

Wasserman,  S.,  and  K.  Faust  (1994).  Social  Network  Analysis:  Methods  and  Applications, 
Cambridge: Cambridge University Press.  

Zhou,  P.  (2009).  Mapping  knowledge  production  and  scholarly  communication  in  China. 
Thesis. Amsterdam: UvA. 

 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  117 

Abbreviations and listing of funding instruments 

Abbreviations 
•
•
•

CA: Coordination actions
CP: Collaborative projects
CP-CSA/CP-CSA-Infra:  Combination  of  Collaborative  Projects  and
Coordination and Support Actions
CP-CSA-Infra-IP:  Combination  of  Collaborative  Projects  and  Coordination
and Support Actions, Implementation phase
CP-CSA-Infra-PP:  Combination  of  Collaborative  Projects  and  Coordination
and Support Actions, Preparatory phase
CP-FP  (CP-FP-INFSO):  Small  or  medium-scale  focused  research  project
(STREP)
CP-IP: Collaborative project - Large scale integrating collaborative projects
CP-SICA-INFSO: Specific international cooperation actions
CSA: Coordination and support actions
CSA-CA: Coordination and support actions - Coordination actions
CSA-SA: Coordination and support actions - Specific support actions
FET - Future and Emerging Technologies
ICT - Information and communications technologies
INFRA: e-Infrastructures
INFSO: Information Society
IP: Large scale integrating collaborative projects
NoE: Networks of Excellence
SA: Specific support actions
STREP: Small or medium-scale focused research projects (CP-FP)

•

•

•

•
•
•
•
•
•
•
•
•
•
•
•
•

Funding instruments 

• Collaborative Projects (CP)

-

-

Small  or  medium-scale  focused  research  actions  (STREP);  abbreviations
used: CP-FP, CP-FP-INFSO
Large-scale integrating projects (IP); abbreviation used: CP-IP.

• Combination of Collaborative Projects and Coordination and Support Actions (CP-

CSA)

• Coordination and Support Actions (CSA)

-
-

Coordination Actions (CA), abbreviations used: CSA-CA
Specific Support Actions (SA), abbreviations used: CSA-SA.

• Networks of Excellence (NoE)
• Specific international cooperation actions. (CP-SICA-INFSO)

Final Study Report- D4 - SMART 2015/0078  118 

Final Study Report- D4 - SMART 2015/0078  119 

Annexes 

Annex 1: Full listing of the distribution of publications across strategic objectives, 
by number of projects. 
Strategic objective 

Average 

Number of 
projects 
408 
4382 
238 

Number of 
publications 
2 
28 
2 

Distributed computing infrastructure (DCI) 
Computing Systems 
FET proactive 2: Quantum Information Foundations and 
Technologies 
ICT forever yours 
Human-Computer Confluence 
Brain-inspired ICT 
FET Proactive: Quantum ICT (QICT) 
Self-Awareness in Autonomic Systems 
Deployment of e-Infrastructures for scientific 
communities 
Science of Complex Systems for socially intelligent ICT 
FET Proactive 3: Bio-chemistry-based Information 
Technology (CHEM-IT) 
The Network of the Future 
Strengthening cooperation in ICT RStrengthening 
cooperation in ICT R&D in an enlarged in an enlarged 
Europe 
Content analytics and language technologies 
Future Networks 
Coordinating Communities, Plans and Actions in FET 
Proactive Initiatives 
Service and Software Architectures, Infrastructures and 
Engineering 
FET Open 
FET Proactive: Dynamics of Multi-Level Complex 
Systems (DyM-CS) 
Embedded Systems Design 
Networked Media 
Networked Embedded and Control Systems 
Secure, dependable and trusted Infrastructures 
Coordinating communities, identifying new research 
topics for FET Proactive initiatives and fostering 
interdisciplinary dialogue 
FET Proactive: Unconventional Computation (UCOMP) 
Pervasive adaptation 
Virtual Research Communities 

274 
101 
201 
193 
88 
214 

118 
71 

1058 
33 

62 
651 
30 

595 

1617 
141 

557 
415 
329 
464 
27 

76 
124 
168 

3 
2 
4 
4 
2 
5 

3 
2 

30 
1 

2 
21 
1 

20 

57 
5 

20 
15 
12 
17 
1 

3 
5 
7 

204.0 
156.5 
119.0 

91.3 
50.5 
50.3 
48.3 
44.0 
42.8 

39.3 
35.5 

35.3 
33.0 

31.0 
31.0 
30.0 

29.8 

28.4 
28.2 

27.9 
27.7 
27.4 
27.3 
27.0 

25.3 
24.8 
24.0 

Final Study Report- D4 - SMART 2015/0078  120 

Technologies and scientific foundations in the field of 
creativity 
Photonic components and subsystems 
Trustworthy ICT 
Cognitive Systems and Robotics 
Network of the Future 
International Partnership building and support to 
dialogues 
Low carbon multi-modal mobility and freight transport 
A reliable, smart and secure Internet of Things for 
Smart Cities 
ICT ICT & Ageing 
Molecular Scale Devices and Systems 
Nano-scale ICT devices and systems 
Next-Generation Nanoelectronics Components and 
Electronics Integration 
Cognitive Systems, Interaction, Robotics 
FET proactive 1: Concurrent Tera-device Computing 
FET Young Explorers 
Digital libraries and technology-enhanced learning 
Internet of Services, Software & virtualisation 
Networked Media and 3D Internet 
Networked Media and Search Systems 
Design of semiconductor components and electronic 
based miniaturised systems 
ICT and Ageing 
ICT for Safety and Energy Efficiency in Mobility 
Robotics, Cognitive Systems and Smart Spaces, 
Symbiotic Interaction 
FET Proactive: Evolving Living Technologies (EVLIT) 
Nanoelectronics Technology 
ICT for Ageing and Wellbeing 
ICT in support of the networked enterprise 
Personal Health Systems (PHS) 
Internet-CNECTed objects 
Organic and large-area electronics, visualisation and 
display systems 
Technology-Enhanced Learning 
Advanced Computing, embedded and Control Systems 
Embodied Intelligence 
Software Engineering, Services and Cloud Computing 
Smart components and smart systems integration 
Joint Call for ICT Challenge 6 and Energy 7 

45 

461 
679 
1098 
622 
126 

83 
41 

100 
40 
116 
308 

723 
57 
73 
262 
379 
262 
209 
95 

95 
47 
62 

30 
74 
118 
88 
160 
86 
172 

279 
83 
55 
41 
68 
53 

2 

21 
31 
51 
29 
6 

4 
2 

5 
2 
6 
16 

38 
3 
4 
15 
22 
16 
13 
6 

6 
3 
4 

2 
5 
8 
6 
11 
6 
12 

20 
6 
4 
3 
5 
4 

22.5 

22.0 
21.9 
21.5 
21.4 
21.0 

20.8 
20.5 

20.0 
20.0 
19.3 
19.3 

19.0 
19.0 
18.3 
17.5 
17.2 
16.4 
16.1 
15.8 

15.8 
15.7 
15.5 

15.0 
14.8 
14.8 
14.7 
14.5 
14.3 
14.3 

14.0 
13.8 
13.8 
13.7 
13.6 
13.3 

 
 
 
Final Study Report- D4 - SMART 2015/0078  121 

Intelligent information management 
Intelligent Content and Semantics 
New Paradigms and Experimental Facilities 
FET Proactive: Minimising Energy Consumption of 
Computing to the Limit (MINECC) 
Organic photonics and other disruptive photonics 
technologies 
Language Technologies 
International Cooperation on Virtual Physiological 
Human 
ICT for access to cultural resources 
ICT for smart and personalised inclusion 
EU-Russia Research and Development cooperation 
Towards Zero-Power ICT 
New paradigms for embedded systems, monitoring and 
control towards complex systems engineering 
Challenging current Thinking 
Technology building blocks for creating, monitoring and 
managing secure, resilient and always available 
information infrastructures that link critical 
infrastructures 
Heterogeneous Integration and take-up of of Key 
Enabling Technologies for Components and Systems 
Virtual Physiological Human 
Cooperative Systems for energy efficient and 
sustainable mobility 
Accessible and Assistive ICT 
Studies, conferences and coordination actions 
supporting policy development in the context of 
international cooperation for e-Infrastructures 
Third implementation phase of the European High-
Performance Computing (HPC) service PRACE 
Accessible and Inclusive ICT 
Engineering of Networked Monitoring and Control 
Systems 
Nanoelectronics 
Simulation software and services 
Future Internet Experimental Facility and 
Experimentally-driven Research 
Smart Energy Grids 
Bio-ICT convergence 
Exa-scale computing, software and simulation 
Manufacturing solutions for new ICT products 
Patient Guidance Services (PGS), safety and healthcare 

288 
270 
115 
51 

102 

89 
62 

61 
133 
12 
24 
119 

164 
74 

21 

293 
31 

82 
20 

10 

109 
87 

19 
38 
111 

46 
63 
18 
18 
53 

22 
21 
9 
4 

8 

7 
5 

5 
11 
1 
2 
10 

15 
7 

2 

28 
3 

8 
2 

1 

11 
9 

2 
4 
12 

5 
7 
2 
2 
6 

13.1 
12.9 
12.8 
12.8 

12.8 

12.7 
12.4 

12.2 
12.1 
12.0 
12.0 
11.9 

10.9 
10.6 

10.5 

10.5 
10.3 

10.3 
10.0 

10.0 

9.9 
9.7 

9.5 
9.5 
9.3 

9.2 
9.0 
9.0 
9.0 
8.8 

 
 
 
Final Study Report- D4 - SMART 2015/0078  122 

record information reuse 
ICT for Patient Safety 
Smart Factories: Energy-aware, agile manufacturing and 
customisation 
Future Internet Research and Experimentation (FIRE) 
Flexible, Organic and Large Area Electronics and 
Photonics 
ICT for Governance and Policy Modelling 
Micro/nanosystems 
Co-operative mobility 
Equipment assessment for sensor and laser based 
applications 
ICT for Mobility of the Future 
ICT Solutions for Fall prevention, and ICT and Ageing 
network 
Personal Health Systems for Monitoring and Point-of-
Care diagnostics 
Technology foundation: Future Internet Core Platform 
Cloud Computing, Internet of Services and Advanced 
Software Engineering 
Internet of Things and enterprise environments 
Advanced ICT for Risk Assessment and Patient Safety 
Virtual Factories and enterprises 
e-Science environment
Application experiments for robotics and simulation 
Digital Libraries : European Digital Library - services 
Future Internet Research Experimentation (FIRE) 
Microsystems and Smart Miniaturised Systems 
Open Innovation for Internet-enabled services and next 
generation access (NGA) in smart cities 
Flexible, organic and large area electronics 
Core and disruptive photonic technologies 
SME initiative on analytics 
Language-based interaction 
ICT for water resources management 
Personal Health Systems 
Digital factories: Manufacturing design and product 
lifecycle management 
Data infrastructures for e-Science 
Smart Factories: ICT for agile and environmentally 
friendly manufacturing 
Photonics 
Collective awareness platforms for Sustainability and 

51 
34 

59 
42 

83 
137 
16 
16 

32 
8 

64 

8 
103 

47 
62 
77 
37 
7 
7 
7 
84 
7 

69 
87 
20 
46 
19 
63 
25 

29 
34 

62 
11 

6 
4 

7 
5 

10 
17 
2 
2 

4 
1 

8 

1 
13 

6 
8 
10 
5 
1 
1 
1 
12 
1 

10 
13 
3 
7 
3 
10 
4 

5 
6 

11 
2 

8.5 
8.5 

8.4 
8.4 

8.3 
8.1 
8.0 
8.0 

8.0 
8.0 

8.0 

8.0 
7.9 

7.8 
7.8 
7.7 
7.4 
7.0 
7.0 
7.0 
7.0 
7.0 

6.9 
6.7 
6.7 
6.6 
6.3 
6.3 
6.3 

5.8 
5.7 

5.6 
5.5 

Final Study Report- D4 - SMART 2015/0078  123 

Social Innovation 
Digital Libraries and Digital Preservation 
Exa-scale computing platforms, software and 
applications 
ICT for environmental services and climate change 
adaptation 
ICT for the fully electrical vehicle 
Scalable data analytics 
High-Tech Research Intensive SMEs in FET research 
Design studies for e-Infrastructures 
First implementation phase of the European High 
Performance Computing (HPC) service PRACE 
ICT for Environmental Management and Energy 
Efficiency 
Preparatory phase for "Computer and Data Treatment" 
research infrastructures in the 2006 ESFRI Roadmap 
Scientific Digital Repositories 
ICT systems for energy efficiency 
Digital Enterprise 
Europeana and creativity 
ICT for efficient water resources management 
Use Case scenarios and early trials 
ICT for energy efficiency 
ICT for Intelligent Vehicles and Mobility Services 
ICT for learning 
More efficient and affordable solutions for digital 
preservation developed and validated against public 
sector needs through joint Pre-Commercial 
Procurement (PCP) 
Very advanced nanoelectronic components: design, 
engineering, technology and manufacturability 
SME initiative on Digital Content and Languages 
ICT for Cooperative Systems 
FET-Open Xtrack 
Supplements to strengthen cooperation in ICT R&D in 
an enlarged European Union 
Coordination actions, conferences and studies 
supporting policy development, including international 
cooperation, for e-Infrastructures 
elearning for mathematics and science 
Empowering patients and supporting widespread 
deployment of telemedicine services 
General Accompanying Measures 
ICT for energy-efficient buildings and spaces of public 

55 
11 

33 

22 
11 
21 
10 
5 

50 

5 

10 
49 
9 
18 
9 
45 
31 
43 
4 
4 

16 

30 
22 
7 
40 

16 

3 
3 

3 
27 

10 
2 

6 

4 
2 
4 
2 
1 

10 

1 

2 
10 
2 
4 
2 
10 
7 
10 
1 
1 

4 

8 
6 
2 
12 

5 

1 
1 

1 
9 

5.5 
5.5 

5.5 

5.5 
5.5 
5.3 
5.0 
5.0 

5.0 

5.0 

5.0 
4.9 
4.5 
4.5 
4.5 
4.5 
4.4 
4.3 
4.0 
4.0 

4.0 

3.8 
3.7 
3.5 
3.3 

3.2 

3.0 
3.0 

3.0 
3.0 

Final Study Report- D4 - SMART 2015/0078  124 

use 
Identifying new research topics, Assessing emerging 
global SIdentifying new research topics, Assessing 
emerging global trends in ICT for future FET Proactive 
initiatives 
Microelectronics/Microsystems 
Personalised Health, active ageing and independent 
living 
Second implementation phase of the European High 
Performance Computing (HPC) service PRACE 
ICT solutions for governance and policy modelling 
ICT for energy-positive neighbourhoods 
Open data 
Aggregating content in Europeana 
Digital Libraries : European Digital Library aggregating 
digital content in Europeana 
Digital Preservation 
e-Infrastructures 
e-Science Grid infrastructures 
EU-Japan research and development Cooperation 
FET Proactive: Neuro-Bio-Inspired Systems (NBIS) 
ICT for Energy Efficiency in Public Buildings 
Smart CNECTed Electro-Mobility 
Scientific Data Infrastructure 
International Cooperation 
Cloud of public services 
FET Proactive: Atomic and Molecular Scale Devices and 
Systems 
Future Internet 
GEANT 
ICT for energy efficiency, environment and smart 
mobility : ICT for energy efficiency in social housing 
ICT for health, ageing and inclusion : e-Accessibility 
thematic network 
Improving certification of eHealth products 
Studies, conferences and coordination actions 
supporting policy development, including international 
cooperation, for e-Infrastructures 
Supplements to support International Cooperation 
between ongoing projects 

6 

3 
6 

3 

19 
8 
9 
2 
2 

4 
2 
2 
2 
2 
2 
2 
3 
5 
1 
1 

1 
1 
1 

1 

1 
1 

1 

2 

1 
2 

1 

7 
3 
4 
1 
1 

2 
1 
1 
1 
1 
1 
1 
2 
4 
1 
1 

1 
1 
1 

1 

1 
1 

1 

3.0 

3.0 
3.0 

3.0 

2.7 
2.7 
2.3 
2.0 
2.0 

2.0 
2.0 
2.0 
2.0 
2.0 
2.0 
2.0 
1.5 
1.3 
1.0 
1.0 

1.0 
1.0 
1.0 

1.0 

1.0 
1.0 

1.0 

 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  125 

Annex 2: Distribution of projects across European regions, NUTS 3, top 5 countries 
in terms of productivity. 

Germany 
NUTS3 

DE212 
DE300 
DE122 
DE111 
DE21H 
DEA21 
DED21 
DEA22 
DEA23 
DE133 
DE501 
DE600 
DE125 
DE716 
DE929 
DEB32 
DE712 
DEA52 
DEA11 
DEC01 
DE131 
DEB35 
DE911 
DEA13 
DEG03 
DE115 
DEA26 
DE271 
DED11 
DE144 
DE142 
DE943 
DE263 
DEA51 
DE915 
DE411 

Nr of 
proj. 
848 
357 
185 
170 
137 
134 
122 
117 
108 
102 
95 
93 
90 
87 
79 
78 
70 
53 
49 
48 
43 
43 
43 
42 
39 
37 
37 
34 
32 
31 
30 
28 
27 
27 
26 
26 

Italy 
NUTS3 

ITE43 
ITC45 
ITC11 
ITD20 
ITD55 
ITE17 
ITC33 
ITE14 
ITD52 
ITD44 
ITD36 
ITE12 
ITE32 
ITF33 
ITD54 
ITD35 
ITD31 
ITC48 
ITG17 
ITF35 
ITG27 
ITD56 
ITE19 
ITD53 
ITC41 
ITD34 
ITF42 
ITF45 
ITG12 
ITE21 
ITD10 
ITE15 
ITE33 
ITC46 
ITC16 
ITC47 

Nr of 
proj. 
803 
543 
279 
191 
167 
167 
165 
91 
52 
44 
43 
42 
36 
36 
35 
25 
24 
24 
20 
19 
19 
18 
17 
16 
16 
15 
14 
13 
12 
12 
12 
12 
11 
11 
10 
10 

UK 
NUTS3 

UKI11 
UKH12 
UKM25 
UKI23 
UKJ14 
UKD31 
UKJ23 
UKI12 
UKJ11 
UKK11 
UKJ32 
UKE32 
UKG31 
UKM34 
UKK14 
UKJ12 
UKG33 
UKF14 
UKE21 
UKN01 
UKE42 
UKC22 
UKD43 
UKL22 
UKH33 
UKJ13 
UKD72 
UKI22 
UKH23 
UKF22 
UKI21 
UKJ42 
UKJ21 
UKK43 
(blank) 
UKF21 

Spain 
NUTS3 

ES300 
ES511 
ES523 
ES213 
ES212 
ES618 
ES130 
ES243 
ES114 
ES617 
ES620 
ES220 
ES111 
ES418 
ES120 
ES514 
ES211 
ES521 
ES425 
ES230 
ES512 
ES532 
ES522 
ES431 
ES614 
ES705 
ES513 
ES616 
ES709 
ES611 
ES424 
ES415 
ES412 
ES413 
ES422 
ES432 

Nr of 
proj. 
570 
149 
141 
137 
122 
115 
110 
95 
94 
91 
87 
86 
84 
74 
63 
60 
58 
44 
40 
39 
38 
33 
30 
30 
30 
23 
23 
23 
22 
22 
22 
20 
19 
18 
17 
17 

Nr of 
proj. 
935 
700 
163 
160 
138 
71 
66 
42 
41 
39 
29 
28 
26 
25 
24 
23 
22 
22 
17 
15 
14 
14 
13 
13 
9 
6 
5 
5 
5 
5 
3 
3 
3 
2 
2 
2 

France 
NUTS3  Nr of 
proj. 
1205 
322 
256 
170 
163 
137 
93 
74 
67 
58 
51 
51 
43 
39 
39 
33 
26 
26 
17 
15 
15 
14 
11 
11 
11 
9 
8 
8 
7 
6 
6 
5 
4 
4 
4 
4 

FR101 
FR105 
FR103 
FR823 
FR714 
FR104 
FR623 
FR716 
FR523 
FR824 
FR612 
FR107 
FR102 
FR106 
FR301 
FR813 
FR421 
FR431 
FR411 
FR724 
FR715 
FR511 
FR718 
FR251 
FR521 
FR261 
FR711 
FR615 
FR108 
FR825 
FR522 
FR514 
FR422 
FR717 
FR212 
FR222 

Final Study Report- D4 - SMART 2015/0078  126 

DE138 
DEA47 
DE222 
DEG0F 
Total 

24 
24 
24 
767 
4406 

ITF11 
ITD58 
ITF61 
Other 
Total 

9 
9 
8 
133 
3183 

UKM21 
UKM22 
UKJ33 
Other 
Total 

15 
14 
13 
320 
3008 

ES416 
ES241 
ES612 
ES613 
Total 

1 
1 
1 
1 
2694 

FR244 
FR246 
FR231 
Other 
Total 

4 
4 
4 
48 
3072 

 
 
 
Final Study Report- D4 - SMART 2015/0078  127 

Annex 3:  Summary statistics of the publication productivity model. 

5
1

4
1

3
1

2
1

1
1

0
1

9

8

7

6

5

2
0

.

0

6
0

.

0

0

1
0

.

0
-

1
0

.

0
-

6
0

.

0
-

3
0

.

0
-

2
0

.

0
-

1
0

.

0
-

1
0

.

0
-

4
0

.

0
-

3
0

.

0
-

2
0

.

0
-

1
0

.

0
-

2
0

.

0

9
4

.

0
-

3
6

.

0
-

4

.

0
-

8
2

.

0
-

1
1

.

0
-

1
0

.

0
-

9
1

.

0
-

1
0

.

0
-

2
0

.

0
-

1
0

.

0
-

1
0

.

0
-

0

4
0

.

0

9
0

.

0

8
1

.

0
-

4
1

.

0

3
0

.

0
-

2

.

0

0

7
0

.

0

7
0

.

0
-

5
0

.

0
-

2
6

.

0

6
1

.

0
-

2
1

.

0
-

1
5

.

0

5
0

.

0
-

9
4

.

0
-

3
1

.

0
-

4
1

.

0
-

2

.

0
-

7
0

.

0

4
1

.

0

9
1

.

0
-

7
1

.

0

1

.

0
-

2
2

.

0

9
0

.

0

9
0

.

0

1

.

0
-

4
6

.

0

5
6

.

0

4
4

.

0

1
0

.

0

6
2

.

0

9
0

.

0

6
4

.

0
-

6
0

.

0

4
0

.

0
-

9
0

.

0
-

2
0

.

0
-

5
3

.

8
1

1
5

.

1
1

6
9

.

4

9
7

.

4
1

3
9

.

0

1

1

1

1

1

1

1

1

4

3
5

.

0

4
2

.

0

4
4

.

0

7
1

.

0

4
0

.

0
-

5
2

.

0

1
0

.

0
-

2
4

.

0
-

3
1

.

0

2
0

.

0
-

1
0

.

0
-

1

.

0
-

9
6

.

0

6
8

.

9

0

0

0

0

0

0

0

0

0

3

6
0

.

0

4
1

.

0

4
0

.

0

3
1

.

0

3
0

.

0

2
0

.

0
-

8
0

.

0

6
0

.

0
-

8
0

.

0
-

3
0

.

0

4
0

.

0
-

1
0

.

0
-

5
1

.

0
-

x
a
m

4
1
7
3

7
3
8
1

2

.

2

2

.

4

1

0

0

6
5

.

2

i

n
m

d
s

n
a
e
m

e

l

b
a

i
r
a
V

r
N

7
5

.

6
0
1

9
1

.

9
1

s
n
o

i
t
a
c
i
l

b
u
P

3
4

.

1
6

1
0

.

8
1

n
o

i
t
a
c
i
l

b
u
P
/
n
o

i
t
a
t
i

C

7
3

.

0

2

.

0

4
6

.

0

4
4

.

0

9
4

.

0

1

.

0

7
0

.

0

2
3

.

0

7
1

.

0

2
2

.

0

4
1

.

0

1

.

0

4
0

.

0

5
4

.

0

2

0

4
0

.

0

4
0

.

0
-

5
0

.

0
-

3
1

.

0

4
6

.

3

5
9

.

4
1

6
3

.

2

7

.

2
1

7

.

0

0

8
8

.

0

3
0

.

0

5
0

.

0

2
0

.

0

1
0

.

0

0

2
7

.

0

1

0

2
0

.

0

8
0

.

0

6
0

.

0

8
0

.

0

1
0

.

0

9
0

.

0

1
0

.

0
-

0

2
0

.

0

0

0

2
0

.

0
-

1
0

.

0
-

1
0

.

0
-

1
0

.

0
-

1

.

0
-

1
0

.

0

3
0

.

0
-

6
2

.

0

2
0

.

0
-

1
0

.

0
-

1
0

.

0
-

n
o

i
t
a
r
u
d
t
c
e

j

o
r
P

g
n

i

d
n
u
F

l

a
t
o
T

s
t
n
e
t
a
P

s
r
e
n
t
r
a
p
r
e
p
g
n

i

d
n
u
F

l

a
n
o

i
t
a
n
f
o
y
t
i
s
r
e
v
i
D

s
t
n
t
a
p

i
c
i
t
r
a
p
f
o
r
N

e
s
a
b
e
c
n
e

i
c
s

N
P
B
_
r
t
n
o
c

A
S
C
_
P
C
_
r
t
n
o
c

1
1

P
C
_
r
t
n
o
c

0
1

A
S
C
_
r
t
n
o
c

2
1

E
O
N
_
r
t
n
o
c

3
1

B
P
_
r
t
n
o
c

4
1

N
T
_
r
t
n
o
c

5
1

)
1
=
8
0
0
2
>
(
y
m
m
u
D

6
1

x
i
r
t
a
m
n
o

i
t
a

l

e
r
r
o
C

e

l

b
a

i
r
a
V

r
N

n
o

i
t
a
c
i
l

b
u
P
/
n
o

i
t
a
t
i

C

s
n
o

i
t
a
c
i
l

b
u
P

n
o

i
t
a
r
u
d
t
c
e

j

o
r
P

g
n

i

d
n
u
F

l

a
t
o
T

s
t
n
t
a
p

i
c
i
t
r
a
p
f
o
r
N

s
t
n
e
t
a
P

l

a
n
o

i
t
a
n
f
o
y
t
i
s
r
e
v
i
D

e
s
a
b
e
c
n
e

i
c
s

N
P
B
_
r
t
n
o
c

A
S
C
_
P
C
_
r
t
n
o
c

1
1

P
C
_
r
t
n
o
c

0
1

A
S
C
_
r
t
n
o
c

2
1

E
O
N
_
r
t
n
o
c

3
1

B
P
_
r
t
n
o
c

4
1

N
T
_
r
t
n
o
c

5
1

)
1
=
8
0
0
2
>
(
y
m
m
u
D

6
1

1

2

3

4

5

6

7

8

9

1

2

3

4

5

6

7

8

9

0

1
0

.

0
-

s
r
e
n
t
r
a
p
r
e
p
g
n

i

d
n
u
F

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  128 

Annex 4: Summary statistics of the patent productivity model. 

2
1

1
1

0
1

9

8

7

6

5

1
0

.

0

1
0

.

0
-

8
0

.

0
-

1
0

.

0
-

2
0

.

0
-

1
0

.

0
-

9
4

.

0
-

5
3

.

0
-

8
7

.

0
-

2
0

.

0

3
2

.

0
-

5
1

.

0

2
0

.

0
-

9
1

.

0

7
0

.

0
-

3
1

.

0

7
2

.

0

7
0

.

0

6
2

.

0
-

7
2

.

0
-

2
1

.

0

5
0

.

0

6

.

0

7
2

.

0
-

1
2

.

0

2
0

.

0
-

1
2

.

0

1
0

.

0
-

5
7

.

0

9
6

.

0

2
5

.

0

1
0

.

0
-

2

.

0

9
1

.

0
-

3
0

.

0
-

7
0

.

0

x
a
m

8

5

.

0
2

8
0

.

5

8
0

.

4

i

n
m

1

0

0

2
2

.

3

5
5

.

7
1

9
5

.

3
1

6
0

.

4

6
8

.

3
1

9

.

0

1
6

.

1

4

.

1
1

3

.

0

1

1

1

1

1

4

3
1

.

0

3
0

.

0

6
1

.

0

9
0

.

0

2
0

.

0

2
1

.

0
-

4
0

.

0

4
0

.

0

3
1

.

0
-

0

0

0

0

0

3

7
0

.

0

6
0

.

0

9
1

.

0

1
1

.

0
-

2
2

.

0

3
1

.

0
-

2
0

.

0

1

.

0

1

.

0

1
2

.

0
-

d
s

2
4

.

1

5
2

.

3

7
4

.

1

6
1

.

0

4
6

.

0

2
4

.

0

2
4

.

0

1
1

.

0

1
2

.

0

1
1

.

0

7
0

.

0

7
1

.

0

5

.

0

n
a
e
m

8
9

.

1

4
4

.

1

9
1

.

2

7
6

.

3

7
1

.

5
1

4

.

2

7

.

0

6
9

.

0

1
0

.

0

1
0

.

0

3
0

.

0

6
5

.

0

n
o

i
t
a
t
i
c
d
r
a
w

r
o
F

n
o

i
t
a
r
u
d
t
c
e

j

o
r
P

g
n

i

d
n
u
F

l

a
t
o
T

s
n
o

i
t
a
c
i
l

b
u
P

s
t
n
e
t
a
P

s
t
n
t
a
p

i
c
i
t
r
a
p
f
o
r
N

e

l

b
a

i
r
a
V

r
N

l

a
n
o

i
t
a
n
f
o
y
t
i
s
r
e
v
i
D

e
s
a
b
e
c
n
e

i
c
s

P
C
_
r
t
n
o
c

A
S
C
_
P
C
_
r
t
n
o
c

0
1

A
S
C
_
r
t
n
o
c

1
1

E
O
N
_
r
t
n
o
c

2
1

)
1
=
8
0
0
2
>
(
y
m
m
u
D

3
1

x
i
r
t
a
m
n
o

i
t
a

l

e
r
r
o
C

1

2

3

4

5

6

7

8

9

7
8

.

2
1

s
r
e
n
t
r
a
p
r
e
p
g
n

i

d
n
u
F

8
1

.

0

4
0

.

0
-

8
0

.

0

9
0

.

0

2
0

.

0

2
0

.

0

6
0

.

0

3
0

.

0
-

3
0

.

0
-

4
0

.

0
-

9
0

.

0
-

8
1

.

0

2
2

.

0

9
0

.

0
-

5
1

.

0

3
1

.

0

9
0

.

0

8
0

.

0

4
0

.

0
-

7
0

.

0
-

5
0

.

0
-

2
1

.

0

1
1

.

0
-

n
o

i
t
a
t
i
c
d
r
a
w

r
o
F

n
o

i
t
a
r
u
d
t
c
e

j

o
r
P

g
n

i

d
n
u
F

l

a
t
o
T

s
n
o

i
t
a
c
i
l

b
u
P

s
t
n
e
t
a
P

s
r
e
n
t
r
a
p
r
e
p
g
n

i

d
n
u
F

l

a
n
o

i
t
a
n
f
o
y
t
i
s
r
e
v
i
D

s
t
n
t
a
p

i
c
i
t
r
a
p
f
o
r
N

e
s
a
b
e
c
n
e

i
c
s

P
C
_
r
t
n
o
c

1

2

3

4

5

6

7

8

9

)
1
=
8
0
0
2
>
(
y
m
m
u
D

3
1

A
S
C
_
P
C
_
r
t
n
o
c

0
1

A
S
C
_
r
t
n
o
c

1
1

E
O
N
_
r
t
n
o
c

2
1

2

1

e

l

b
a

i
r
a
V

r
N

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Final Study Report- D4 - SMART 2015/0078  129 

European Commission 

Analysis of patents, publications and new organisations from FP7-ICT and CIP-
ICT projects 

Luxembourg, Publications Office of the European Union 

2019 - 130 pages 

ISBN 978-92-76-00595-7 

doi: 10.2759/644641

K
K
-
0
4
-
1
9
-
2
2
6
-
E
N
-
N

doi: 10.2759/644641    

ISBN 978-92-76-00595-7