Ex-post evaluation of science in society in FP7 : executive summary.
Date: 2016-06-14
Institutions: Directorate-General for Research and Innovation
Subjects: EU research policy, Framework Programme for Research and Development, added value, civil society, innovation, report, research policy

Ex-post Evaluation 
of Science in Society 
in FP7 
Executive Summary

 
 
 
  
 
 
 
 
 
 
 
 
EUROPEAN COMMISSION 

Directorate-General for Research and Innovation 
Directorate B — Open Innovation and Open Science 
Unit B7 — Science with and for Society 

Contact: Philippe Galiay 

E-mail:  Philippe.Galiay@ec.europa.eu 

RTD-PUBLICATIONS@ec.europa.eu 

European Commission 
B-1049 Brussels 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
EUROPEAN COMMISSION 

Ex-post Evaluation of Science 
in Society in FP7 

Executive Summary 

Prepared by Charu Wilkinson, Jan Franke, James Stroyan 

Submitted by ICF Consulting Services, in association with Delft University, 
Facts of Life and Technopolis 

2016 

Directorate-General for Research and Innovation 
FP7 — Science in Society 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
EUROPE DIRECT is a service to help you find answers  
to your questions about the European Union 

Freephone number (*): 
00 800 6 7 8 9 10 11 

(*) The information given is free, as are most calls (though some operators, phone boxes or hotels may charge you) 

LEGAL NOTICE 

This  document  has  been  prepared  for  the  European  Commission  however  it  reflects  the  views  only  of  the 
authors,  and  the  Commission  cannot  be  held  responsible  for  any  use  which  may  be  made  of  the  information 
contained therein. 

More information on the European Union is available on the internet (http://europa.eu). 

Luxembourg: Publications Office of the European Union, 2016. 

PDF  

ISBN 978-92-79-58544-9 

doi: 10.2777/896520 

KI-02-16-493-EN-N 

© European Union, 2016. 
Reproduction is authorised provided the source is acknowledged. 

Cover images: © Lonely, # 46246900, 2011. © ag visuell #16440826, 2011. © Sean Gladwell #6018533, 
2011. © LwRedStorm, #3348265. 2011. © kras99, #43746830, 2012. Source: Fotolia.com

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
EXECUTIVE SUMMARY 

The FP7 Science in Society programme 

The Science in Society (SiS) programme formed part of the FP7 'Capacities' programme and had a 
budget  allocation  of  €312  million  for  the  period  2007-  2013.  The  overall  aim  of  the  FP7  SiS 
programme  was  to  improve  linkages  between  science  and  society.  Specifically,  the  programme 
sought to: 

  Develop ways to enhance the numbers of young people from all backgrounds entering science 

education and careers;  

 

Identify and address barriers to gender equality in research; 

  Develop mechanisms to improve understanding of the role of science in society; 

 

 

Increase the role of the wider public and non-research actors in science policy making and 
agenda setting;  

Promote the development of an ethically sound and responsible European Research Area 
(ERA); 

  Make the results of publicly funded research more accessible. 

During the second half of the programme, the focus of FP7 SiS shifted to developing and promoting 
the  concept  of  Responsible  Research  and  Innovation  (RRI).  The  concept  of  RRI  is  based  on  the 
desire to steer research and innovation towards socially desirable and acceptable ends1. As such, it 
encapsulates three core ideas2: 

1.  A purposeful focus of research and innovation activities on the values, needs and expectations 

of society; 

2.  A comprehensive and meaningful involvement of all relevant societal stakeholders throughout 

the research and innovation process, with the aim of enhancing public good; 

3.  The development of flexible governance arrangements in order to better respond and adapt to 

changing circumstances and new knowledge. 

The  RRI  concept  was  operationalised  by  the  European  Commission  in  the  form  of  six  ‘keys’  or 
‘dimensions’ which coincide with the different thematic areas of intervention within FP7 SiS: 

(i)  Science education – making science education and careers attractive to young people and 

scientific knowledge accessible to all; 

(ii)  Gender – fostering gender balance in research teams and decision making, and integrating 

the gender dimension in research and innovation content;  

(iii) 

Public engagement – establishing iterative, inclusive and participatory multi-actor dialogues 
between  researchers,  policy  makers,  industry  and  civil  society  organisations,  NGOs,  and 
citizens;  

(iv) 

Ethics  –  helping  ensure  that  research  is  based  on  shared  values  and  to  highest  ethical 
standards; 

(v)  Open  access  –  providing  free  online  access  to  the  results  of  publicly-funded  research 

(publications and data); 

(vi)  Governance – developing ‘harmonious’ models of RRI that integrate the above five themes. 

Although  the  FP7  SiS  programme  ended  in  2013,  Horizon  2020  (H2020)  retains  a  strong 
commitment to SiS/RRI issues. Specifically, H2020 continues to support SiS/RRI actions under the 
‘Science with and for Society’ (SwafS) programme via: 

  Dedicated projects in the five thematic areas of Public engagement, Open access, Gender, 

Ethics, and Science education.  

1 Euroscientist (2014) Tackling grand challenges with socially acceptable solutions. Available at: 

http://www.euroscientist.com/special-issue-rri-overview-print-edition/.  

2 European Commission (2012) Responsible Research and Innovation: Europe’s ability to respond to 

societal challenges. Available at: 
https://ec.europa.eu/research/swafs/pdf/pub_public_engagement/responsible-research-and-
innovation-leaflet_en.pdf. 

3 

 
 
                                                 
 

Integrated actions that for example promote institutional change, to foster the uptake of the 
RRI approaches by stakeholders and institutions (the “Governance” theme). 

Furthermore, RRI has also been incorporated as a ‘cross-cutting issue’ within H2020. 

Objectives and scope of the evaluation 

This ex-post evaluation is both summative and formative in nature. The summative element of the 
evaluation takes stock of and critically examines the following aspects of the programme: 

The rationale for intervention; 
Internal and external coherence of the programme; 

 
 
  Aspects related to implementation, such as the nature and size of projects funded, types of 

participants involved, thematic and geographic coverage of funded activities; 

  Achievements and impacts; 
 

EU added value. 

The formative element of the evaluation seeks to draw lessons which could be applied to on-going 
and future EU activity aimed at supporting RRI issues. 

Data sources and methods 

The  evaluation  was  designed  to  respond  to  a  specific  set  of  evaluation  issues  and  questions,  as 
articulated in the Terms of Reference. The overall design of the evaluation was based on a mixed-
methods approach comprising: 

  Desk research to establish the intervention logic for the programme, to derive first answers to 
evaluation  questions  (on  the  basis  of  existing  evidence)  and  to  identify  gaps  in  evidence 
requiring primary data collection. 

  A full analysis of FP7 proposals, participations in  proposals,  and  projects and participations in 
projects (based on data extracted from CORDA database) to examine patterns of demand and 
participation in FP7 SiS as compared to FP7 overall. 

  An  analysis  of  final  reporting  questionnaires  completed  by  FP7  SiS  project  coordinators  to 
understand  the  outputs  produced  by  FP7  SiS  projects  and  the  number  and  type  of 
dissemination activities undertaken by funded projects (based on data extracted from SESAM).  

  An analysis of final reporting questionnaires completed by FP7 Cooperation project coordinators 
(based on data extracted from SESAM), as well as an online survey of FP7 Cooperation project 
coordinators to determine the extent to which SiS aspects were mainstreamed in other parts of 
FP7. 

 

Interviews  with  over  100  stakeholders  to  address  programme-level  questions,  particularly 
those  relating  to  its  relevance,  EU  added  value,  alignment  with  national  activities,  and  main 
achievements and impacts. 

  An online survey of FP7 SiS project participants to collect data on impacts and achievements of 

the individual actions supported under SiS. 

  Network analysis to map and analyse the linkages between SiS actors, i.e. the nature and scale 
of  partnerships  between  programme  actors  and  their  centrality  within  global  knowledge 
networks. 

  Bibliometric  analysis  to  determine  co-authorship,  citation  impact  and  content  analyses  of 
scientific literature and to establish the extent to which the outputs of supported projects and 
researchers had an impact on scientific community within and beyond the SiS domain. 

 

Forty-four case studies (based on desk research and interviews) to explore the impacts of SiS 
projects and the embeddedness of SiS aspects within other FP7 projects. 

4 

 
 
 
 
Limitations 

As  indicated  above,  a  variety  of  data  sources  were used  to  build  a  comprehensive  evidence  base 
for  the  evaluation.  Complementary  research  methods  were  used  as  a  means  to  enhance  the 
reliability  and  validity  of  the  data  collected  and  to  provide  the  basis  for  triangulation  of  results. 
However, as with any evaluation, there were limitations to the methodologies applied: 

  Many stakeholders interviewed did not have a detailed overview and understanding of the FP7 
SiS programme and were consequently only able to provide general feedback on specific issues 
(such as implementation) or specific themes (such as gender); 

 

 

The response rate to online surveys was relatively low (17 to 19%), presumably due to survey 
fatigue among participants; 

It was not possible to draw strong and definitive  evaluative judgements on certain aspects of 
the  programme  (such  as  achievement  and  impacts  or  EU  added  value)  for  the  following 
reasons: 

─ 

─  Over  half  of  the  funded  projects  (94  out  of  184)  were  still  being  implemented  at  the 
time of the evaluation. The results and impacts of these projects could therefore not be 
captured at this stage. 
It is methodologically quite difficult to capture the impacts of a programme such as SiS, 
which  is  quite  fragmented  in  nature,  supports  a  diverse  set  of  activities  and  seeks  to 
achieve  ‘soft’/intangible  outcomes  such  as  behavioural,  institutional  or  policy  change. 
This  problem  was  further  compounded  by  the  lack  of  quantitative  or  even  qualitative 
information  on  the  results  and  impacts  of  funded  projects.  The  study  team  tried  to 
collect  this  information  through  interviews  with  project  participants.  However, 
participants  were  often  unable  to  articulate  or  provide  evidence  of  the  results  and 
impacts of their activities.  

─  Other  gaps  in  evidence  (e.g.  data  on  national  funding  for  SiS  themes)  could  only 
partially be filled through data collection activities that could be undertaken within the 
time and budget available for the evaluation. 

Main findings of the evaluation 

The main findings of the evaluation are set out below. These should, however, be interpreted with 
the above caveats and limitations in mind.  

Rationale for the FP7 SiS programme 

There  was  a  clear  political  and  practical  rationale  for  the  FP7  SiS  programme.  SiS  reflected  the 
growing  political  acknowledgement  of  the  need  to  ensure  societal  relevance  and  acceptability  of 
research and innovation. At a practical level, it was framed by a continuing desire to improve the 
consideration and integration of gender issues, public engagement, science education, ethics, open 
access  and  governance  within  research  and  innovation  processes  and  systems  (Figure  ES1.1). 
There was also  a strong EU  added value  rationale for the programme, as highlighted by the high 
demand  for  FP7  SiS  funding  and  the  absence  of  strong  national  debates  and  dedicated  funding 
programmes for SiS issues in several Member States.  

Each of the six themes addressed pertinent and important issues, as illustrated in  Figure ES1.1. It 
could,  however,  be  argued  that  the  science  education  and  ethics  themes  were  too  narrowly 
focused: 

 

The intervention logic for the science education theme (which accounted for almost of third of 
the programme budget) was built around the premise that young people were disinterested in 
science  because  of  the  way  it  was  being  taught  in  schools  and  that  Inquiry  Based  Learning 
(IBL)  techniques  could  help  address  this  as  advised  in  the  Rocard  Report.  Consequently,  the 
programme  focused  heavily  on  developing  and  promoting  IBL  approaches,  even  though  the 
effectiveness of these approaches has not yet been proven. 

 

The ethics theme can be criticised for  not  addressing  important but politically sensitive issues 
such as the debates on GMO, energy, genetic engineering, stem cell research or germ lines.  

5 

 
 
Figure ES1.1 Issues addressed by FP7 SiS 

Internal coherence 

The overall intervention logic for  the programme  was reasonably  clear. The thematic intervention 
logics were also coherent, i.e. there was a logical link between programme activities and the issues 
being  addressed  and  the  budget  allocation  to  the  programme  as  well  individual  projects  was 
sufficient.  The  programme  evolved  over  time  in  response  to  new  developments  and  lessons 
learned, though two major programming developments, (i) shift towards the RRI concept and (ii) 
the introduction of MMLs in 2010, were not well understood by stakeholders and initially resulted in 
confusion. 

There were also some issues with the ways in which the programme was structured and objectives 
were formulated: 

 

The programme was both ambitious in scope and imprecise in laying out its expected results. 
Programme  objectives  indicated  the  ‘direction’  of  travel,  but  provided  no  indication  of  the 
‘distance to be travelled’. This lack of precision in objective setting to some extent undermined 
overall programme coherence and effectiveness.  

  Action and activity lines (as set out in programming documents) did not directly correspond to 

the six RRI themes, although the links could be implicitly deduced.  

 

The design, implementation and monitoring of the programme was heavily focused on activities 
and outputs, rather than results and impacts.  

External coherence with ERA policy objectives 

The FP7 SiS programme was fully aligned with two out of five ERA priorities:  

 

 

The  objectives  and  activities  of  the  gender  theme  within  FP7  SiS  programme  were  coherent 
with the ERA policy objective of “gender equality and gender mainstreaming in research”.  

The  open  access  theme  –  by  piloting  open  access  activities  and  promoting  dialogue  on  the 
subject – was highly relevant for the ERA policy objective of “optimal access to and circulation 
and transfer of scientific knowledge”. 

6 

 
 
 
Take-up of funding  

FP7  SiS  had  an  overall  financial  envelope  of  €312  million3  for  the  period  2007-2013.  This  budget 
was split as follows: 

  €265 million was allocated to funding projects selected through calls for proposals; 

 

The  remaining  amount  of  €47  million  was  allocated  to  grants  to  named  beneficiaries,  expert 
group activities and public procurement activities. 

The  FP7  SiS  programme  attracted  841  proposals,  of  which  184  projects  (22%  of  the  proposals) 
were  eventually  selected  for  funding.  This  suggests  that  there  was  high  demand  for  FP7  SiS 
funding.  

Scale of projects funded 

The  programme  succeeded  in  its  ambition  to  develop  larger  projects  both  in  terms  of  budget 
allocation and the number of partners involved: 

 

 

 

 

The  average  financial  size  of  the  projects  supported  under  FP7  SiS  was  €2.52  million.  This  is 
almost five times larger than the average size of projects supported under FP6 SaS (€547,000 
per project). 

The average size of SiS projects increased from €1 million (during 2007-2009), to €2-3 million 
during 2010 to 2013; those launched in the final year (2014) were even larger (€3.3 million). 

The average number of partners in the projects supported under FP7 SiS was ten. This is 50% 
greater  than  the  average  consortium  size  in  the  projects  supported  under  FP6  SaS  (at  6.5 
participants per project). 

The  SiS  programme  continued  to  support  larger  consortia  over  time,  with  projects  launched 
from  2007-2010  averaging  8.5  partners  per  project,  and  those  launched  from  2011-2014 
averaging 11.5 partners per project. 

Nature of partnerships 

Compared  to  the  overall  FP7  profile,  the  SiS  programme  had  a  relatively  high  participation  by 
Higher or Secondary Education (46% in SiS versus 37% in FP7 overall), ‘Other’ (16% in SiS versus 
just  3%  in  FP7  as  a  whole)  and  Public  Bodies  (8%  in  SiS  versus  5%  in  FP7  as  a  whole). 
Correspondingly, the SiS programme had a relatively low participation by Research Institutes and 
Industry. The exceptionally high level of involvement by the ‘other’ category – predominantly Civil 
Society Organisations (CSOs) – is a major achievement in efforts to diversify project contexts and 
partnerships  beyond  academic  circles.  Moreover,  the  share  of  participations  held  by  the  ‘other’ 
category  (CSOs  and  others)  increased  over  time,  from  12%  in  2008  to  16%  by  the  end  of  the 
programme. 

Mobilisation and Mutual Learning (MML) Actions 

MMLs were a new programming instrument introduced in 2010 to achieve scale and facilitate long-
term  SiS  partnerships  that  mobilised  key  actors  across  the  system  in  question  with  a  view  to 
finding  more  effective  solutions  through  mutual  learning  and  cooperation.  They  sought  to  bring 
researchers  together  with  a  much  wider  community  than  they  would  ordinarily  associate  with, 
including members of CSOs, policy makers, the public and various other media and business actors 
connected to the particular societal challenge being addressed. 

A  total  of  18  MMLs  were  supported  under  SiS,  making  up  approximately  10%  of  the  SiS  project 
portfolio as a whole. In comparison with non-MML SiS projects, the 18 MMLs: 

  Were  much  larger  in  terms  of  average  project  costs,  with  MMLs  averaging  €4.48  million  per 

project compared to €1.68 million for non-MML projects4. 

  Were  three  times  larger  in  terms  of  the  EU’s  financial  contribution,  with  MMLs  receiving  on 

average €3.86 million in EU funding each compared to €1.32 million each for non-MMLs. 

  Were twice as big in terms of consortium size. MMLs involved much larger consortia, with MMLs 
averaging  19.5  participants  per  project  compared  to  8.9  for  non-MML  projects.  The  smallest 

3 Total funding allocated through annual work programmes. 
4 While MMLs were on average larger, they were not in fact the largest SiS projects. Only three of the ten 

largest SiS projects were MMLs. 

7 

 
 
                                                 
number of partners in a MML project was 12 and the largest was 39. For non-MML projects, the 
range was 1 to 35. 

 

Involved  a  larger  proportion  of  organisations  with  very  limited  or  no  experience  of  the 
Framework Programmes. 

Moreover, non-researchers occupied a higher share of the project costs within MMLs and therefore 
had  a  deeper  or  more extensive  involvement  in  the  work  than  they  did  within  non-MML  projects. 
This  was  particularly  the  case  for  public  bodies  and  private  commercial  companies.  MMLs  can 
therefore be considered a successful tool for improving the involvement of non-researchers in the 
SiS programme. 

Thematic coverage of the programme 

The  relative  weight  of  themes  in  terms  of  EU  funding  and  overall  number  of  projects  was  very 
uneven, with Public engagement (53 projects) and Science education (47 projects) each receiving 
around  €86  million  and  together  representing  more  than  half  of  the  €289  million  of  SiS  funding 
allocated to  projects  through  calls  for  proposals  and  grants  to  named  beneficiaries5.  Of  particular 
note  is  the  rather  limited  funding  dedicated  to  the  Ethics  (€26  million)  and  Open  access  (€12 
million) themes.  

The rationale behind the relative weight of individual themes within the programme is not evident. 
In  the  absence  of  a  specific  ex-ante  evaluation  of  the  SiS  programme,  it  is  not  possible  to  tell 
whether the above thematic spread was intentional and whether it was appropriate considering the 
urgency/scale of the problems being addressed as well as the level of pre-existing activity. 

Figure ES1.2 Thematic coverage of FP7 SiS 

Geographic coverage of the programme 

Significant  progress  was  made  in  increasing  participation  from  ‘new’  member  states  in  FP7  SiS, 
over and above their participation within FP7 as a whole. Collectively, the new member states (EU-
13) had a much higher rate of participation in SiS than in FP7 as a whole in relative terms (15.5% 
of SiS participations compared to 8.0% of FP7 participations). 

Similarly,  third  countries  accounted  for  13.2%  of  SiS  participations,  a  slightly  higher  share  than 
achieved  within  FP7  as  a  whole  (12.0%).  The  FP6  SaS  programme  involved  project  participants 
from a total of 57 different countries and FP7 SiS involved participants from 71 countries, so it is 
clear  that  the  geographical  scope  of  the  programme  increased.  The  number  of  participating 
countries  that  are  not  EU  member  states increased  by  almost  50%,  from  29  in  FP6  SaS  to  43  in 
FP7  SiS.  While  14  more  countries  took  part  in  SiS  than  in  SaS,  22  ‘new’  countries  in  fact 
participated in SiS (but not SaS) and eight countries that had participated in SaS did not go on to 
participate  in  SiS.  This  suggests  that  the  programme  was  successful  in  quantitative  terms  in 
increasing the international focus of the programme.  

5 According to data extracted from CORDA, €289 million was allocated to projects through 1) calls for 

proposals = €272 million comprising EU funding of €265 million and third country contributions of €7 
million and 2) grants to named beneficiaries = €17 million. 

8 

 
 
 
                                                 
FP7 SiS was thus, highly successful in broadening its geographical coverage and participation over 
time.  

Dissemination 

The  vast  majority  of  the  SiS  dissemination  activities  were  directed  towards  the  scientific 
community. A relatively low share of dissemination activity was targeted to policy makers, industry 
and the media, particularly in the case of Open access and Ethics. Poor dissemination of outputs to 
policy  makers  and  industry  inevitably  limited  the  overall  impacts  of  a  programme  such  as  SiS, 
which seeks to have an impact through behavioural, institutional and policy change.  

There  was  a  low  level  of  awareness  of  the  content  of  the  SiS  programme  and  its  results  among 
these groups and an evident ‘gap’ in terms of efforts to consolidate the programme’s outputs and 
results into a digestible and usable form and to disseminate this information to relevant audiences. 
Although this work is partly underway as part of the FP7 SiS meta-analysis and individual FP7 SiS 
projects, such as RRI tools, Scientix 2 and Engagement 2020, these actions do not go far enough 
in  terms  of  identifying  and  promoting  best  practice  –  they  just  provide  a  large  and  growing 
collection of tools and methodologies that prospective users have to wade through. 

Achievements and impacts 

The  general  nature  of  the  stated  objectives  of  the  programme  and  the  absence  of  pre-defined 
‘indicators  of  success’  meant  that  evaluating  the  overall  progress  and  impact  of  the  programme 
was difficult. However, some qualitative statements and evidence of achievements and impacts are 
provided  below  on  the  basis  of  evidence  collated  from  a  variety  of  sources  including  interviews, 
case studies and project documentation. 

The main achievements and impacts of SiS can be summarised as follows: 

  Mainstreaming of SiS/RRI aspects within H2020. Specifically: 

─  RRI and gender are incorporated as cross-cutting issues within H20206; 
─  The rules of participation in H20207 include an explicit requirement for ethics reviews 

of proposals submitted under the research programme;  

─  Article  40  of  Regulation  1290/2013  mandates  the  inclusion  of  CSO  representatives 
among  independent  experts  in  charge  of  evaluation  of  proposals  and  programmes, 
monitoring of projects and establishment of research policies. 

─  The legal base for H2020 also requires open access to publications and data resulting 

from H2020 research (Article 18 of Regulation 1291/2013); 

─  The  Commission  published  guidance  on  gender  equality  in  Horizon  20208,  providing 
practical  guidance  on  how  to  integrate  gender  equality  issues  at  each  stage  of  the 
research  cycle,  from  programming  through  to  implementation,  monitoring  and 
programme evaluation.  

  Structuring  and  informing  discussion  and  debate  in  relation  to  RRI/SiS  concepts  (including 
problems  and  needs)  at  an  international  as  well  as  at  EU  level,  and  helping  to  develop  the 
underpinning evidence base. 

 

Taking the debate forward through a wider range of (non-FP7) collaborations and networks and 
through  promotion  of  a  common  understanding  of  RRI/SiS  concepts  among  the  scientific 
community. 

  Various  successes  in  getting  RRI  concepts  and  issues  featured  in  national  science  strategies, 
statements  from  funding  bodies  and  foundations,  in  research  agendas,  and  in  research  and 
innovation funding and selection criteria, can also be attributed to the programme. 

  Development  of  new  tools  and  working  methods  that  enhance  the  engagement  of  the  public, 
policy  makers  and  businesses  in  scientific  debate  and  improve  the  provision  of  scientific 
information to relevant audiences. 

6 Article 14 of Regulation 1291/2013 establishing Horizon 2020 - the Framework Programme for 

Research and Innovation (2014-2020). 

7 Article 14 of Regulation 1290/2013 laying down the rules for participation and dissemination in "Horizon 

2020 - the Framework Programme for Research and Innovation (2014-2020). 

8 European Commission (2014) Gender Equality in Horizon 2020, version 1, 26 February 2014. 

9 

 
 
                                                 
  Definition  of  relevant  indicators  and  collection  of  data  on  the  role  of  women  in  science  (She 

Figures), and hence to a coordinated EU-wide monitoring system. 

  Significant advances in the understanding of gender imbalance in science, its impacts, and the 

corrective measures needed.  

 

Improving  knowledge  and  understanding  in  EU  Member  States  of  the  importance  of  the 
relationships  between  science  and  society,  the  need  for  all  parts  of  society  to  become  fully 
engaged in science and technological progress and development, and the positive benefits this 
would bring in terms of scientific excellence, innovation and economic growth. 

  Developing  and  sharing  good  practice,  in  particular  developing  and  promulgating  tools, 
methods and techniques to promote and support  good SiS  and  RRI principles and behaviours 
within all groups of stakeholders. 

  Building  and  strengthening  the  community  of  actors  engaged  in  SiS/RRI  research  and  action 
through support and coordination for the (often small or fragmented) communities engaged in 
the  various  themes  within  the  SiS  area,  and  acting  as  a  focal  point  for  communities  and 
concepts to come together, interact and evolve. 

  Acting as a central point of knowledge and guidance to the Commission and to national actors 

across the ERA. 

Notwithstanding  the  above  achievements,  the  overall  impacts  of  the  programme  have  no  doubt 
been limited by: 

 

 

The fragmented nature of activity, particularly during the first half of the programming period. 
The  programme  suffered  from  a  lack  of  clear  focus  and  intent  during  its  first  few  years  and 
consequently  a  number  of  small,  disconnected  projects  were  funded.  For  example,  within  the 
science  education  theme,  24  projects  were  funded  on  the  topic  of  IBL  techniques.  These 
projects carried out similar activities (such as testing or piloting IBL approaches, or conducting 
research  on  the  prerequisites  for  implementing  IBL  in  different  levels  of  schooling),  but  in 
different  contexts.  They  differed  with  respect  to  the  EU  Member  States  and  third  countries 
involved,  and  the  teaching  subjects  covered.  A  more  strategic  and  joined-up  approach  that 
involved  funding  complementary  projects  and  exploiting  linkages/synergies  between  funded 
activities would have generated stronger impacts. 
Lack  of  focus  and  attention  on  ensuring  that  projects  have  an  impact  beyond  their  own 
boundaries. While lots of interesting ideas and pilots were pursued, it is not clear  – other than 
in isolated examples – if or how the project results are being embedded more widely, beyond 
the confines of the individual projects and the partners involved. 

  Related to the above, insufficient engagement with industry and policy makers, particularly in 
areas such as ethics and  open access. Some isolated good practice examples can be found of 
projects that were successful in bringing on board industry and policy makers (e.g.  INgenious 
and Scientix 2 funded under the science education theme). These projects were, however, an 
exception rather than the rule.  

Furthermore, although significant strides were made to mainstream SiS/RRI aspects within H2020, 
there remains a gap between de jure measures and de facto implementation. An internal analysis 
of H2020 work  programmes for the years 2014-2015 carried out by the Commission reveals that 
RRI dimensions have not yet been fully integrated within H2020  – only a limited number of topics 
in the work programmes explicitly mention RRI themes, activities related to open access continue 
to  be  predominantly  funded  through  SwafS  and  ICT  programmes,  and  work  programmes  contain 
partial references to RRI (e.g. “responsible research” or “responsible innovation”). 

Finally,  the  SiS  programme  does  not  perform  strongly  on  most  of  the  Commission’s  standard 
output metrics, which are reported in the Annual Monitoring Report. This reflects the programme’s 
emphasis on coordination and  support  activities, and  the relatively lower level of interest in more 
conventional  types  of  research  outputs,  whether  that  is  international  peer-reviewed  articles  or 
intellectual property.  This  is  understandable  considering  that  the  focus  of the  programme  was on 
structuring of ERA and changing the governance framework within the EU, rather than generating 
new knowledge. H2020, on the other hand, seeks to reinforce the knowledge base for RRI and as 
such, more funding would be allocated to research projects within the SwafS programme. 

10 

 
 
 
 
Impact on policy making 

The evaluation identified several projects that had a notable impact on decision making at various 
levels.  The  ETICA  project,  for  example,  was  seen  as  having  a  high  impact  on  the  European 
Commission’s  agenda  around  ethical  issues  in  information  and  communication  technologies.  The 
PACITA  project spread  technology assessment approaches across Europe, resulting in two further 
Member States taking up technology assessment in their parliamentary and government processes 
for  reviewing  R&D-related  legislation  and  policy.  Other  impacts  were  identified  at  the  local  level 
were,  for  instance,  the  EFORTT  project  that  provided  a  blueprint  ethical  framework  which  was 
taken up by a local council in the process of procuring telecare services for up to 7,000 patients.  

Moreover,  over  40%  of  FP7  SiS  participants  reported  that  their  project  had  a  large  or  very  large 
impact  on  scientific  advice  and  decision  making  at  a  European  (44%)  or  local  (45%)  level  – 
however, this pattern of expected impacts differs to some extent by theme. This suggests that in 
the  future,  greater  consideration  could  be  given  to  whether  certain  themes  are  more  effective  at 
influencing  the  SiS  agenda  at  a  European  level  (such  as  ethics  and  governance),  at  a  local  level 
(such  as  science  education)  or  are  effective  at  influencing  the  agenda  at  both  EU  and  local  level 
(gender, open access and public engagement).  

Mainstreaming of FP7 SiS aspects 

The  analysed  project  final  reporting  data  shows  that  the  extent  to  which  FP7  SiS  aspects  were 
addressed within the FP7-Cooperation programme varies to a large degree between the SiS theme 
under investigation and the individual FP7-Cooperation theme. Nevertheless, some mainstreaming 
is measurable across all  FP7-Cooperation themes. There is a high degree of  variability across the 
FP7-Cooperation themes, with Security (94%), Health (76%), and Food (73%) projects most likely 
to  have  had  SiS/RRI  obligations  placed  on  them,  and  Transport  (19%),  Energy  (40%)  and 
Nanosciences  (44%)  least  likely.  There  is  also  a  high  degree  of  variability  as  to  which  SiS/RRI 
issue,  FP7-Cooperation  projects  were  required  to  address.  Overall,  ethical  issues  were  most 
frequently  cited  by  FP7-Cooperation  project  coordinators,  followed  by  gender  balance  in  research 
teams and the need to ensure multi-actor engagement in projects. Results of an online survey of 
FP7-Cooperation  project  coordinators  indicate  that  efforts  in  the  areas  of  multi-actor  and  public 
engagement, and on governance aspects (aligning research with societal needs and applying good 
practice in research) are considered as having the largest positive benefit on their projects.  

EU added value 

At the national level, SiS research communities and activities are small and fragmented, and where 
funding  is  available  it  tends  to  be  limited,  ad  hoc  and  embedded  within  other  programmes.  As  a 
result,  systematic  and  concentrated  national  research  efforts  are  scarce,  and  while  the  FP7  SiS 
programme  had  a  relatively  limited  budget  given  its  scope,  it  is  still  seen  as  one  of  the  main 
vehicles for accelerating the promotion of SiS/RRI efforts at the national level. 

A review of national activities suggest that issues covered by SiS are sometimes taken up as small, 
complementary parts of collaborative research programmes, but seldom funded through dedicated 
programmes. 

Therefore, the availability of a dedicated pot of funding for addressing SiS/RRI issues was the most 
attractive  aspect  of  the  FP7  SiS  programme  from  a  national  perspective  because  of  the  lack  of 
comparable and sizeable alternatives. As a result, however, it suffered from low success rates, and 
had  difficulties  in  involving  less  well-established  groups.  National  representatives  would  have 
welcomed a larger and more ambitious SiS programme. From their perspective, a programme such 
as FP7 SiS has strong EU added value, in particular as regards: 

 

 

 

Its role as the only specific funding programme at EU and in many cases, national level, to fund 
research around pertinent themes at the intersection of science and society; 
Its  capacity  building  function  for  new  (EU-13)  Member  States  and  Associated  Countries  to 
enhance  the  performance  and  institutional  capacity  of  national  research  and  innovation 
landscapes; 
Its  role  as  an  agent  of  change  at  the  institutional  and  policy  level  regarding  themes  such  as 
ethics, gender equality and open access. 

Furthermore, case studies provide evidence of a scaling-up effect of FP7 SiS, which enabled issues 
funded as a side note in national research to be scaled up, in terms of thematic scope, the amount 
of funding available and the involvement of European and international project partners. 

11 

 
 
 
 
Moving forwards, the SiS/RRI agenda is regarded as highly important, both as a research topic and 
as a cross-cutting issue within research and innovation activities. National representatives are keen 
that the momentum and progress achieved so far is not lost. 

Recommendations for the future 

The  evaluation  confirms  the  relevance  and  importance  of  an  EU-level  programme  addressing 
SiS/RRI issues. However, it also flags a number of areas where improvements can be made. These 
are as follows: 

Strategic recommendations 

Strengthen linkages between the programme and high-level policy priorities 

SiS/RRI  issues  should  be  an  integral  part  of  the  Commission’s  high-level  policy  on  research  and 
innovation.  The  Commission  should  explicitly  define  the  linkages  between  SiS/SwafS  programme 
and its high-level political priorities, most notably: “Open science", “Open innovation” and “Open to 
the world”.  

Clarity of focus and objectives 

Future SiS programmes should have well-defined goals, providing a clear sense of direction as well 
as  the  ‘distance  to  be  travelled’.  Admittedly,  it  can  be  challenging  to  set  SMART  objectives  for  a 
programme  such  as  SiS  which  seeks  to  have  an  impact  through  behavioural,  institutional  and 
policy change, it is nevertheless possible to define SMART(er) objectives, particularly with respect 
to the internal dimension of the FP7  SiS  programme, i.e. promoting SiS aspects in other parts of 
FP7.  It  is  understood  that  work  is  on-going  to  define  such  indicators.  The  MORRI  project  will 
develop  indicators  to  measure  aspects  of  RRI  at  an  aggregate  (national  and  EU)  as  well  as  at 
institutional level.  

An  outcomes  or  impacts-based  approach  to  objective  setting  would,  however,  also  require 
investment  in  monitoring  and  evaluation  tools  that  go  beyond  the  collection  of  outputs  data  to 
performance tracking after the project has ended to capture outcomes and impacts.  

Greater coherence and clarity of thematic objectives and how projects contribute to these 

The  objectives  of  some  of  the  themes  (Governance/RRI,  Public  engagement,  Open  access)  were 
difficult  to  identify.  In  the  future  the  Commission  should  seek  to  define  its  objectives  for  each 
theme  more  clearly,  and  ensure  that  there  is  coherence  between  the  stated  objectives  and  the 
actions  funded  under  each  theme.  Any  synergies  and  complementarities  between  projects  should 
be  made  explicit,  in  order  to  ensure  that  projects  do  not  compete  with  each  other  or  duplicate 
activity. 

Stronger focus on ‘on-the-ground’ impacts as opposed to new knowledge  

While  the  projects  funded  under  FP7  SiS  were  successful  in  generating  new  knowledge  and 
disseminating  their  results  among  participants  and  known  stakeholder  groups,  they  could  have 
been  more  ambitious  in  terms  of  generating  wider  impacts,  particularly  at  a  policy  level.  The 
Commission  should  ensure  that  projects  are  designed  to  generate  such  wider  impacts  from  the 
outset  and  have  sufficient  time  and  resource  to  do  so  effectively.  Greater  involvement  of  policy 
makers  at  programme  level  should  help  ensure  the  policy  relevance  of  the  knowledge  and  tools 
developed in the future. 

Work programmes and individual calls need to include more specific guidance and requirements on 
how to measure impacts at the project level, and mandate the inclusion of a SMART impact section 
in  project  proposals.  The  quality  of  these  impact  sections,  in  particular  the  degree  to  which  they 
are SMARTly defined, should form part of the proposal evaluation process. 

Maximise the added value and impact of FP7 investments 

There is a strong need (and also a big challenge) to package up the results of SiS projects in a way 
that makes them meaningful to potential ‘users’ such as policy makers, practitioners and industry. 
To maximise the  impact  of  SiS-type activities,  it  is necessary to  identify,  extract and disseminate 
information on (a) the key things that have been learned from the funded projects and (b) the key 
changes/practice improvements that could be applied by the various ‘user groups’. 

Increase the use of theme-based advisory groups 

Greater  use  of  expert  advisory  groups  for  individual  SiS/RRI  themes,  not  just  to  inform 
deliberations  at  the  start  of  the  programme,  but  also  to  assist  DG  RTD  with  strategic  advice  on 
future priorities, call preparation, progress and exploitation of results. Group membership (in terms 
of  expert  backgrounds)  should  be  as  broad  as  possible,  comprising  representatives  of  main  user 
groups  of  SiS  outputs  (industry,  public  authorities,  civil  society,  and  the  scientific  and  academic 

12 

 
 
community). This would ensure that the programme is more demand led (“pull model”) rather than 
the more traditional approach (“push model”). 

Greater involvement by policy makers at national and EU levels 

Future work on Governance/RRI would benefit from greater involvement of policy makers (science 
ministries)  and  other  bodies  (research  councils,  scientific  foundations),  with  the  power  to 
implement changes in governance arrangements at both national and EU levels. If the programme 
is to deliver impacts in terms of improved governance of the European Research Area in line with 
the  RRI  concepts  and  principles  then  there  needs  to  be  much  greater  engagement  with  such 
entities.  The  objective  of  this  deeper  engagement  with  policy  makers  should  be  to  help  set 
priorities and ensure that  the programme  is investing  in relevant actions that have the scale and 
reach to deliver results of utility to policy makers. 

Stronger involvement of industry 

Given  the  important  role  of  industry  in  carrying  out  research  and  exploiting  its  results,  closer 
involvement of industry in all stages of the programme is important to ensure that the outputs of 
the  programme  are  practically  relevant  and  useful,  and  are  ultimately  adopted  to  improve  the 
functioning of ERA. 

Operational recommendations 

Additionally,  a  number  of  practical  suggestions  were  made  by  programme  participants,  which 
would improve the overall relevance and effectiveness of the programme: 

More  open  calls  –  participants  suggested  that  calls  for  proposals  should  be  less  prescriptive  and 
give more room for creativity. More open calls would give more scope for project consortia to set 
appropriate  timescales  and  budgets  for  the  project,  and  would,  according  to  some  participants, 
make it easier for new entrants to become involved. 

Easier access for new entrants – European Commission to provide more help and make it easier for 
new  types  of  organisation  to  participate,  particularly  CSOs  and  SMEs.  Simpler  application  and 
reporting rules, improved financing terms and conditions, and smaller projects would all help make 
it easier for civil society and SMEs to participate. 

Improved funding levels – some Higher Education and CSO participants argued that funding levels 
should be increased to cover the full/a higher share of participation costs, in order to facilitate CSO 
access to the programmes. 

Lighter  administration/reporting  –  some  Higher  Education  and  CSO  participants  made  the  usual 
request  for  lighter  administration  and  reporting  across  the  board,  suggesting  that  journal 
publications should be prioritised over  management  reporting. CSO participants asked for simpler 
rules  for  themselves.  It  is  understood  that  a  number  of  administrative  simplifications  have  been 
implemented under H2020. The impact of these simplifications should be evaluated as part of the 
H2020 interim evaluations. 

More flexibility – participants noted that they would have liked greater scope to make adjustments 
to project workplans, financial plans and timetabling mid-course. More flexibility here would allow 
projects  to  adjust  in  the  light  of  stakeholder  input  (gathered  through  early  parts  of  the  projects) 
and to respond appropriately to the situation as it unfolds. 

Follow-up funding for dissemination/take-up – participants from the public engagement and science 
education themes (of all types) indicated that there is a need for more dedicated follow-up funding 
for dissemination and wider take-up of project results. Participants argued that three years and the 
budgets  available  have  not  been  sufficient  to  make  maximum  use  of  the  results  and  tools 
developed  through  the  projects.  When  projects  have  been  successful  then  follow-on  funding  and 
political  support  should  be  given  to  ensure  maximum  take-up  and  impact  from  the  original 
investment. 

Longer  timescales  –  many  of  the  comments  indicated,  inter  alia,  that  it  takes  a  long  time  to 
mobilise  actors,  implement  activities,  generate  results,  and  then  encourage  other  actors  to  take 
notice  of  and  take-up  the  outputs.  The  three  year  timeframe  for  most  projects  was  considered 
insufficient, often meaning that the critical dissemination/take-up phase was limited. 

More  synergies  –  there  should  be  more  synergies  between  projects  to  enable  improved 
coordination and sharing of results/best practices. 

13 

 
 
 
 
14 

 
 
 
 
 
How to obtain EU publications 

Free publications: 

•  one copy: 

        via EU Bookshop (http://bookshop.europa.eu); 

•  more than one copy or posters/maps: 

        from the European Union’s representations (http://ec.europa.eu/represent_en.htm);  
        from the delegations in non-EU countries (http://eeas.europa.eu/delegations/index_en.htm);  
        by contacting the Europe Direct service (http://europa.eu/europedirect/index_en.htm) or 
        calling 00 800 6 7 8 9 10 11 (freephone number from anywhere in the EU) (*). 

        (*) The information given is free, as are most calls (though some operators, phone boxes or hotels may charge you). 

Priced publications: 

•  via EU Bookshop (http://bookshop.europa.eu).  

15 

 
 
 
         
 
 
K
I
-
0
2
-
1
6
-
4
9
3
-
E
N
-
N

K

I

-

0

2

-

1

6

-

4

9

2

-

E

N

-

N

The  FP7  Science  in  Society  (SiS)  programme  was  part  of  FP7  Capacities  and  had  a 
budget  of  €312  million  allocated  for  the  period  2007-2013.  It  attracted  a  total  of  841 
research, 
proposals,  of  which  184  were 
CORDA/SESAM/bibliometric/network analysis, stakeholder interviews, online surveys and 
project  case  studies  this  ex-post  evaluation  assesses  the  rationale,  design  and 
implementation  of  the  programme  and  evaluates  its  achievements,  impacts  and  EU 
added value. 

funded.  Drawing  on  desk-based 

Studies and reports 

ISBN 978-92-79-58544-9